Jyll

Platform: Nintendo Switch
Unity: 2019.3.10f1
Spine Runtime: 3.8-2020-04-14

Summary:

Some players are experiencing crashes that feel almost random, but they all seem to revolve around Spine use cases.

  1. A Spine Skeleton may have completely white textures. When this is witnessed, the game may continue running for a short while but it is guaranteed to crash soon. In other cases, it crashes immediately.
  2. A Spine Skeleton may appear with "messed up" or "glitched" textures (e.g. the character's hair appears on his shoes etc). If a precise screenshot is needed I can e-mail it to you. When this is witnessed, the game crashes shortly after.

Some details:
  1. It is not possible to reproduce in a consistent manner. It "randomly" happens after a few, or sometimes several hours of gameplay - or, in some cases, never at all.
  2. Upgrading Unity or the Spine runtime must be avoided if at all possible.
  3. We use Unity UI, so it's SkeletonGraphic instances. No, we have not confirmed whether it happens with regular Skeleton renderers, but it's not really an option at this point.
  4. Unfortunately, this isn't something we personally managed to reproduce with either dev kits or retail hardware. What we know is from user reports, but they have all been consistent in the symptoms listed above.
  5. This is not confirmed and may be entirely wrong, but it may be ultimately crashing in UnityGfxDeviceWorker. This suspicion is just based on similar crash reports from other platforms, but it might be unrelated.
  6. The files are "correct". That is to say, we have successfully loaded every single skeleton that the game has and also played all of their animations, so it's not a file corruption issue.
  7. Almost nothing uses the skin system. Only one entity has its skin changed dynamically in the game, and yes that entity has had this issue as well.
  8. We always disable "Use Clipping" for performance reasons.
  9. These issues are not observed on neither Standalone nor Playstation 4. However for PS4 specifically, it uses Unity 2019.4.2f1. Standalone does use the same 2019.3.10f1 though, and our playerbase on it is large and has not encountered this issue on Desktop.

While it feels random, it always happens with Spine entities - perhaps a character, or an animated background, etc. Below I will describe the general workflow for, say, our characters.

When building the game:
  1. One folder per character.
  2. The folder contains the Spine export of said character (the atlas, the json, the generated skeleton, material, etc).
  3. The folder itself is an AssetBundle. Therefore, one AssetBundle per character.
  4. Generated bundles are placed in StreamingAssets.

When running the game:
  1. When a character is needed, we read the bytes from the bundle and then load it:
    AssetBundle.LoadFromMemory(byte[])
    1. Note: does not make a difference to use LoadFromFile() instead.
    2. Note: not tested with async loading, but it's not advisable to go async at this point.
  2. Once the bundle is loaded, we load the skeleton:
    var skeletonData = bundle.LoadAsset<SkeletonDataAsset>(skeletonDataName);
  3. And then, once we have the skeleton, we load it into a SkeletonGraphic that's already in the scene:
    SkeletonGraphic skeleton = GetComponent<SkeletonGraphic>();
    skeleton.skeletonDataAsset = skeletonData;
    skeleton.Initialize(true);
    skeleton.AnimationState.SetAnimation(0, "idle", true);
  4. Finally, we unload the bundle but do not unload the game objects (otherwise we'd lose the textures):
    bundle.Unload(false);
    1. Note: originally we didn't even unload it. In other words, this issue occurs regardless of unloading the bundle or persisting it, so we don't believe than Unload() is related.
  5. When the character is no longer needed (e.g. it dies), the object is destroyed. In the future if the same skeleton is needed, the steps above are repeated (the bundle is reloaded all over again).
    1. Note: we also tried by caching the bundles, skipping step 1.

Insights:
  1. The game is not running out of memory, or even close to its limit.
  2. Some skeletons use POT textures, a others NPOT. So far there is no indication that one is more responsible than the other.
  3. Most skeletons use a texture that is at most 2048x2048. Very few ones, like large backgrounds, may be up to 4096x4096.
  4. This is not confirmed and may be entirely wrong, but it may be ultimately crashing in UnityGfxDeviceWorker. This suspicion is just based on similar crash reports from other platforms, but it might be unrelated since they did not present the above symptoms (white textures etc).
  5. Large skeletons might be more susceptible to the crash. I say this simply because it often happens when entering a scene with an animated background (which is the largest kind of skeleton we have).
    1. Note: perhaps it's not because of a large skeleton, but several skeletons within the same frame.

I believe that a project upload is not necessary because you will likely not crash even after several hours, and the above snippets already outline what we're doing in general.

Thank you for the help.

---

Additional info, which might be relevant perhaps:

  1. Multi-threaded rendering is turned on.
  2. Graphics Jobs is turned off.
Jyll
  • Posts: 1

Mario

We don't have access to a Switch dev kit, so it will be even harder to reproduce the issue on our end. We'll have to go by your diagnostics. A couple of questions and comments.

  • I assume you can't get definite stack traces for the crashes from users?
  • Your code/sequence of invoking the API doesn't look suspicious.
  • POT or NPOT textures shouldn't make a difference on the Switch as far as I know.
  • You may not run out of heap memory, but given the symptoms, it sounds more like there's a bug in the GPU side resource management (textures, buffers), i.e. tiny leaks that eventually culminate in a crash after hours of playtime. While this could be a bug in spine-unity, I think it's more likely a bug in Unity's Switch graphic stack. I assign the likelyhood based on the fact that you do not see this issue on other platforms (which, to be fair, also have more resources to be used by the game). It's really hard to pinpoint just based on the symptoms. FWIW, we have other customers deploying to Switch via Unity which have not reported such issues. Do Unity release changelogs for version updates for a specific platform? Maybe we could see if a newer Unity version changelog for Switch mentions any bug fixes in the rendering system that might be worth trying?
  • Related to the last point, could you disable multi-threaded rendering? I guess since this only happens for end users, deployment of "debug" builds to Switch for end users is out of the question?
User avatar
Mario

Mario
  • Posts: 2447

Harald

One more thing to mention that caused undefined behaviour on some devices in the past:
Please double check that every SkeletonGraphic component uses Spine/SkeletonGraphic * shaders exclusively, and no non-UI shader like e.g. Spine/Skeleton Fill or any URP/Spine/* shader.

If this is not the case, then you could try setting up a stress test of a loop using your Spine methods that allocate and free resources or otherwise modify Textures, like repacking skins or attachments to atlas textures. If you do not use such methods, perhaps you find similar Spine calls that when repeated in a loop cause the issue to occur.
User avatar
Harald

Harri
  • Posts: 2272


Return to Bugs