My game uses an isometric camera and renders my Spine sprites as 2D billboards in a 3D world. Generally everything is working fine, but I've recently I've updated the shaders I am using to ensure the sprites are properly testing and writing to the depth buffer to ensure they sort properly when moving throughout the 3D world. This works well and my individual units sort properly with the world and each other, but it looks like there may be Z-fighting with themselves. I've attached a gif showing the problem.

I suspect the issue is simply that at the end of SkeletonRenderer's Draw call when it sends the vertex data off to the batcher, the Z position is always 0 for every vertex. This of course makes sense, but seems like it may cause problems once the controller shader is utilizing the depth buffer for testing and writing.

I tried playing around with the Draw Order but didn't see much success. I'm guessing because regardless of draw order once we start executing shader code it is using the Z buffer anyway.

I tried playing around with the ZFunc inside my shader, but I couldn't get exactly what I wanted. Setting ZFunc to "Always" fixes the Z-fighting issue entirely, but then of course my units will then always render on top of all my 3D geometry which is not what I want.

What would the proper solution be for handling this?

Some additional information:

This seems to only be happening when I'm doing some custom depth calculations in my shaders. I need to do this in order to prevent my sprites from clipping into 3D geometry when they're close to it since they're billboards.

The way I'm doing this is relatively simple - I pass down two World matricies - your standard "World" matrix I use for the bulk of calculations and a second "DepthWorld" matrix which is exactly the same as the billboarded "World" matrix except its Y axis always points up. I only use this matrix to calculate what the depth value would be if the quad was standing upright in the scene.

Here is the shader code for reference (with most of the unrelated lighting code removed):

matrix World;
matrix DepthWorld;
matrix View;
matrix Projection;

float3 AmbientLightColor;

struct VSInput
{
   float4 position      : POSITION0;
   float4 color      : COLOR0;
   float2 texCoord      : TEXCOORD0;
   

   //Passed down but unused
   //float4 color2      : COLOR1;
};

struct VSOutput
{
   float4 position      : SV_Position;
   float4 color      : COLOR0;
   float2 texCoord      : TEXCOORD0;

   float depthValue : TEXCOORD1;
};

sampler TextureSampler : register( s0 );

VSOutput VSQuad(VSInput input)
{
   VSOutput output;

   float4 worldPosition = mul(input.position, World);
   float4 viewPosition = mul(worldPosition, View);
   output.position = mul(viewPosition, Projection);

   float4 depthWorldPosition = mul(input.position, DepthWorld);
   float4 depthViewPosition = mul(depthWorldPosition, View);
   output.depthValue = mul(depthViewPosition, Projection).z;

   output.texCoord = input.texCoord;
   output.color = input.color;

   return output;
}

struct PSOutput
{
   float4 color : SV_Target;
   float depth : SV_Depth;
};

PSOutput PSQuad(VSOutput input)
{
   float4 baseColor = tex2D( TextureSampler, input.texCoord );
   baseColor *= input.color;
   clip(baseColor.a < 0.8f ? -1 : 1);

   // Final light value starts off with the ambient defined in the scene
   float3 totalLight = AmbientLightColor;

   PSOutput output;
   output.color = baseColor * float4(totalLight, 1.0f);
   output.depth = input.depthValue;
   return output;
}

PSOutput PSQuadAlpha(VSOutput input)
{
   float4 baseColor = tex2D( TextureSampler, input.texCoord );
   baseColor *= input.color;
        clip(baseColor.a >= 0.8f ? -1 : 1);

   // Final light value starts off with the ambient defined in the scene
   float3 totalLight = AmbientLightColor;

   PSOutput output;
   output.color = baseColor * float4(totalLight, 1.0f);
   output.depth = input.depthValue;
   return output;
}

technique CharacterLit
{
   pass P0
   {
      AlphaBlendEnable = false;
      ZEnable = true;
      ZWriteEnable = true;

  VertexShader = compile vs_4_0 VSQuad();
  PixelShader = compile ps_4_0 PSQuad();
   }
   pass P1
   {
      AlphaBlendEnable = true;
      ZEnable = true;
      ZWriteEnable = false;

  PixelShader = compile ps_4_0 PSQuadAlpha();
   }
}

If I use the default SV_DEPTH value (which would be the Z value of the position vector) I'm not seeing this issue. However change it to use the depthValue causes this. I'm not sure why this would be the case considering it should really only be making the depth buffer think the top of the quad is slightly closer than it really is.

Still hoping to get some insight on this, but I've been messing around with it a little bit.

By adding a small Z offset to the vert positions in the SkeletonRenderer Draw call the problem goes away

VertexPositionColorTextureColor[] itemVertices = item.vertices;
            for (int ii = 0, v = 0, nn = verticesCount << 1; v < nn; ii++, v += 2) {
               itemVertices[ii].Color = color;
               itemVertices[ii].Color2 = darkColor;
               itemVertices[ii].Position.X = vertices[v];
               itemVertices[ii].Position.Y = vertices[v + 1];
               temVertices[ii].Position.Z = i * 0.01f;
               itemVertices[ii].TextureCoordinate.X = uvs[v];
               itemVertices[ii].TextureCoordinate.Y = uvs[v + 1];
               if (VertexEffect != null) VertexEffect.Transform(ref itemVertices[ii]);
            }

So it does seem like the issue is basically one of Z-Fighting. However I am still very confused as to why this isn't a problem normally?

Thanks for the explanation Harald, now I'm seeing where the difference was coming from. I didn't even stop to consider the value that before my changes for the custom depth, the values were of course all the same since the billboarded quad was aligned to the camera. My change introduces different depth values for each vertex since I'm using the non-billboarded World matrix.

As a follow-up question, do you think my altered code I have at the bottom of the original post is the best place to go about this? As far as I know there isn't any preexisting way to add some z offset for each quad is there? Adding a small Z value in that loop is definitely simple and looks like it solves the issue, but I figured I would ask if there was any other way to handle this. If at all possible I prefer not to edit the Spine runtime code since I'll have to deal with merging the code anytime I want to update the version of our Spine runtime.

Thanks for pulling that functionality over! I'll sync up and give it a test over the weekend.