davidsvson

I'm going to need to do lip sync in my current project, but am completley new to the subject.

I found this: https://github.com/DanielSWolf/rhubarb-lip-sync

Any better sugestions? Where whould I start to get this going with spine and Unity?
davidsvson
Posts: 51

Nate

I've not see Rhubarb. It looks nice, that is probably how I would design such a tool. There is also Papagayo and Annosoft, though I have no experience with those either. Someone wrote a Papagayo to Spine exporter you might find useful. Otherwise, I would load the Rhubarb data and programmatically generate an animation which has an AttachmentTimeline that hides/shows the appropriate attachments at the right times. I'd probably do this at runtime rather than convert Rhubarb output to Spine JSON.
User avatar
Nate

Nate
Posts: 7433

davidsvson

Papagayo looks interesting, Annosoft looks :think: ... o wow that is alot of $$ :whew:

But I really like the simpleness of rhubarb. Thanks for the tips. I'll try to figure out how to generate an animation programmatically then.
davidsvson
Posts: 51

DanielWolf

Hi, I'm the author of Rhubarb. Converting Rhubarb's lip animation to Spine sounds interesting. Let me know if there's any way I can help!
User avatar
DanielWolf
Posts: 7

davidsvson

Both papagayo and rhubarb can output at textfile with the times (papagayo: frames) when to attach the different attachments (different mouthshapes).

Example rhubarb (time in seconds first then witch attachment):
0.00 X
0.05 D
0.27 C
0.31 B
0.43 X
0.47 X
Example Papagayo (witch frame first (30f/s) then witch attachment):
1 etc
4 O
7 etc
16 E
19 MBP
22 etc
How do I generate an animation programmatically and show/hide the different attachments?
davidsvson
Posts: 51

DanielWolf

I believe the most elegant solution would be to add Spine support right into Rhubarb. I'm prepared to do that, but I'd like some feedback first. Here are my thoughts:

Typically, people have an existing character in Spine and want to add dialog animations to it. So rather than generating a new Spine file from scratch, the user should specify an existing Spine JSON file. This file should contain a rigged skeleton with a slot for the mouth. This slot contains multiple image attachments for the various mouth shapes.

The user would now call Rhubarb with a WAVE file and the existing Spine JSON file. Rhubarb would then generate a new Spine JSON file that's a copy of the original. In addition, it contains a new animation that's named after the WAVE file, with a prefix like "say_". So If you call Rhubarb with a recording named "hi_there.wav", you'd get a new animation called "say_hi_there".

I think this would be pretty straightforward to implement. But I'm new to Spine, so I'm not sure if it's the best solution from a user perspective. Please comment:

Is the behavior I describe above what you're looking for? Is there a better approach?
User avatar
DanielWolf
Posts: 7

Nate

Welcome, DanielWolf! :)

You're right that generating an entire skeleton would make it difficult to get the animation into an existing skeleton. It could be possible using Import Project, which allows importing an animation from another project file. Rhubarb would write a skeleton JSON file, then you'd need to Import Data to get it into a project file, then you could Import Project to merge in an animation. Note this requires both skeletons to have the same objects that were keyed (slots, attachments, events, etc).

Your proposal would work fine and is simpler than the above juggling. Users would export their project to JSON, add animations to it with Rhubarb, then they can use the data as is or import it back into a Spine project for further editing. I don't think there is a better approach.
User avatar
Nate

Nate
Posts: 7433

DanielWolf

Thanks for your detailed explanation. That's exactly what I was assuming, so I'll implement it that way.

I'm not sure how soon I will find the time to implement the exporter. As soon as there are any news, I'll update this topic.
User avatar
DanielWolf
Posts: 7

davidsvson

So, I try to generate a animation programmatically from the papagayo output as it can be seen above. But no success, I get no error messages but no attachments seems to get set on slot "Mouth". Any suggestions? Here is my code:
public TextAsset speechTxt;
private int speechTxtFrameRate = 24;

void Talk() {
var lines = speechTxt.text.Split("\n"[0]);
AttachmentTimeline timeLine = new AttachmentTimeline (lines.Length-1);
timeLine.slotIndex = skeletonAnimation.skeleton.data.FindSlotIndex("Mouth");

for (int frameIndex = 1; frameIndex < lines.Length-1; frameIndex++) {
var frameData = lines[frameIndex].Split(" "[0]);
timeLine.SetFrame (frameIndex - 1, float.Parse (frameData [0]) / speechTxtFrameRate, "Mouths/"+frameData[1]);
}

ExposedList<Timeline> timelines = new ExposedList<Timeline>();
timelines.Add(timeLine);
Spine.Animation anim = new Spine.Animation("talk", timelines, Speech.Length);
skeletonAnimation.skeleton.data.animations.Add(anim);

skeletonAnimation.state.SetAnimation (1, "talk", false);
}
davidsvson
Posts: 51

Nate

It looks OK. You'll have to debug it further. Does the AttachmentTimeline try to set the attachment when it should? If so, when it sets the attachment is it able to find the attachment with the Mouths/xxx name?
User avatar
Nate

Nate
Posts: 7433

davidsvson

Thank you for looking!
I have been tearing my hair, but I just figured out that it was my line endings in the text file that was my problem :bang: ! when I converted them everything worked :rofl: !
davidsvson
Posts: 51

jamqdlaty

DanielWolf wrote:Thanks for your detailed explanation. That's exactly what I was assuming, so I'll implement it that way.

I'm not sure how soon I will find the time to implement the exporter. As soon as there are any news, I'll update this topic.
Hi. do you still plan to implement the exporter?
jamqdlaty
Posts: 15

DanielWolf

do you still plan to implement the exporter?
Absolutely. I've been postponing this feature for some time, mainly because I heard that Esoteric Software was planning to add audio support to Spine. I felt that it made sense to wait for that, since it makes things easier for me.

Now that Spine 3.7 beta is out, I'll soon start implementing Spine support in Rhubarb Lip Sync.
User avatar
DanielWolf
Posts: 7

Erikari

DanielWolf wrote:I heard that Esoteric Software was planning to add audio support to Spine.
And now we have it!:D
Thanks for doing this, I myself would love to use it in some projects!
User avatar
Erikari

Erikari
Posts: 551

DanielWolf

Erikari wrote:I myself would love to use it in some projects!
That's great to hear! :) And it definitely makes more sense now that Spine has audio support.

By the way: Integrating third-party tools such as Rhubarb with Spine would be much easier if Spine had a plugin interface. I recently integrated Rhubarb with Adobe After Effects, and the UX is much better than what I'll be able to provide for Spine. Is there a chance to see a plugin/scripting system for Spine any time soon?
User avatar
DanielWolf
Posts: 7

Raeleus

Rhubarb looks great! I would love to see integration of this tool in one way or another.
Raeleus
Posts: 27

AlaNintendo

following!
User avatar
AlaNintendo
Posts: 62

DanielWolf

I've been busy with other stuff, but now I'm ready to add Spine support to Rhubarb Lip Sync! :)

Now that Spine has audio support, I'm thinking of the following rough workflow:

1. The user creates a Spine project with a skeleton. They give the skeleton a slot for the mouth. This slot contains multiple image attachments for the various mouth shapes, named using a fixed schema.
2. The user has one or more audio file to lip-sync. For each audio file, they create an event.
3. This step is optional: Rhubarb Lip Sync can create good mouth animation from audio alone. However, the results are even better if Rhubarb is told the dialog text for each sound file. If the user wants to, they may set the event's default string value to the dialog text.
4. The user exports the skeleton in JSON format.
5. The user starts "Rhubarb Lip Sync for Spine", which will be a cross-platform GUI application. There, they open the exported JSON file.
6. The user now has the following controls:
  • a dropdown to select the slot that represents the mouth
  • a list of all events with associated audio file. Each event has a checkbox that can be checked.
6. At the click of a button, automatic mouth animation is performed for all checked events:
  • For each checked event, a new animation is created that's named after the event, with an added prefix like "say_". So if the user created the event 'hi_there' based on the file 'hi_there.wav', there will now be the animation 'say_hi_there'.
  • Each generated animation contains the matching audio event at frame 0, plus the actual mouth animation.
  • The original JSON file is overwritten to contain the new animations.
7. The user imports the JSON file back into Spine to get the new animations.

I don't expect any technical problems, but I'm concerned about the user experience. My experience with Spine is very limited, so I'd like to hear from some power users:

Does this workflow make sense to you?
Is there aspect that may be inconvenient or unnecessarily complicated?


Any feedback is welcome!
User avatar
DanielWolf
Posts: 7

Nate

Sounds like a good plan!
User avatar
Nate

Nate
Posts: 7433


Return to Unity