Hey mitch, thanks for the reply! I hadn't even thought of using the cutouts as a starting point
I guess I'd been assuming that there must be more specialized stuff going on with the spine shaders. Doesn't using alpha testing rather than alpha blending mean the spine animation will have rough, pixelated edges? The cutout does indeed receive shadows though, I'm going to take a look at this now and see what I can do from here. Thanks for your suggestion, Mitch. I'm really looking forward to those amazing tools you've been working on!
edit wait, your unity tools just got merged... time to play!
Alright, I looked into this a bit today. Getting dynamic shadow receiving to work on a transparent shader like the shaders used in spine is not trivial in Unity. If anyone is reading this and has gotten shadow casting to work in combination with nice alpha blending, please post to let me know I'm overcomplicating this, I would love to know it's simpler than I'm making it.
Let me preface this by saying that I'm using spine animations in full 3D space in my game, with a downward angled camera that's meant to look kind of like Xenogears or Grandia
games with 2d characters in 3d worlds. So draw order is a bit more complex here than in some 2D cases.
Anyway, here's what I've got:
In Unity, nothing in the render queue after 2500 can have a dynamic shadow cast onto it. I'm assuming that's because dynamic shadows are drawn just after 2500 in the render queue.
Getting into shader queue settings for a moment, the AlphaTest queue value is equal to 2450, and the Transparent queue value is equal to 3000.
Only things drawn in the Transparent queue (starting at 3000) are sorted strictly back to front. As far as I can tell, this is a built in behavior of Unity.
The problem is, the only way to get perfectly clean alpha blending is to draw things back to front with z writing off.*
So, to get perfect alpha blending we need to sort spine characters back to front, which is only done in the Transparent section of the render queue, but to get shadows casting onto our spine characters they need to be drawn at 2500 or earlier, before the Transparent part of the render queue.
The best solution I've found is to draw Spine characters at or before 2500 and to do the back-to-front sorting manually. Fortunately, in the 2D additions to Unity they introduced sorting layers, which actually seem to give the exact type of control needed
the ability to sort draw order however you want between groups of meshes at any point in the render queue. My plan is to write a sorting layer manager which will go through every frame and assign a sorting order on each spine mesh so that all spine animations are being drawn back to front. I haven't actually built the manager, but a quick test of the sorting order thing does seem to work. The spine shader itself can be a dirt simple surface shader
it seems like pretty much any shader will work with this method, once the draw order is being correctly sorted back to front AND and render queue value is low enough that shadows can be cast onto the object. The cheapest solution is just to use a shader with zwriting completely turned off, though this might mean you need to sort everything transparent with the sorting layers (stuff like particle effects). The best solution may end up being to write z values for all of the opaque pixels in the spine animation, but to turn z writing off for the edges, so most things can be left in the transparent queue
I'm guessing in most cases the artifacts would be subtle for things like quick particle effects.
After this, I have one more shader problem to deal with, which is that shadow casting and projector casting don't work on spine attachments which have been mirrored. This is the case whenever a backface is being used to draw the attachment. Obviously one workaround would be to not use any negative scaling, but that would get pretty annoying to not be able to flip all my east animations westward, not to mention all the texture duplication needed, so hopefully I can solve this at some point. I know other people have run into this while experimenting with normal mapped spine shaders, so hopefully someone has already solved this. Maybe I'll create another thread about it!
Anyway, when I actually set this up I'll post shader code and sorting layer manager code here, in case anyone else runs into the same problem.
*There's also a two pass method used on alpha blended grass, in which solid pixels are drawn with zwriting and then soft edges are drawn in a second pass without zwriting, but this method still introduces jagged edged artifacts when things are drawn out of order with lighting.