Auto Lip syncing with Rhubarb

Demos and Tutorials
#1

I just wanted to demonstrate an example of auto lip syncing. I’m going to attach a wick project to demonstrate this very, very rough script that reads off from a rhubarb data output. I will also give a quick tutorial on how to use rhubarb.

Download rhubarb here: https://github.com/DanielSWolf/rhubarb-lip-sync/releases

use the command like so: “rhubarb -o output.txt audiofile.wav”

This will spit out output.txt which has the data you can plug into the script as demonstrated in the project file. I’ve included an image set with the project file for all the mouth shapes as guides.

I’m sure there are more talented programmers that can optimize this script and make it work better.

Merry Christmas!

rhubarb.wick (1.8 MB)

2 Likes
Automatic Lip Syncing
#2

@Luxapodular would be really cool to have some kind of automatic tool for this which uses rhubarb on the backend like you use ffmpeg on the backend.

You could make it a built-in asset like a button. Except instead of the button states you could use the phoneme states for mouth shapes and you could run it on the backend for performance. That’d be really cool.

As far as this script goes, I may come back to this at some point, clean it up, and make it into a class or something to be more usable. Also, I didn’t know that I had access to the current frame number so I didn’t use that. Lots of things need to be cleaned up. But, hey, at least I’ve given the community a general direction on how to get auto lip syncing up and going.

#3

thankyou wbc
the project is really interesting and i’m sure it’s the answer (or at least a solid base to build on) for many of the lip sync questions found in this forum
Personally I dont think that including rhubarb in the editor would be a good move.
Importing the audio file is already possible and I dont see much added value of having rhubarb included in the face of the added weight and possible issues for the editor
It would be great if we could add non basic features like this one as plugin or addons.

#4

Hey this is a great suggestion. I definitely think rhubarb could make a cool built-in asset that people could drop into their projects. Really interesting stuff!

1 Like
#5

I don’t know. I don’t think it would be that bad. Here’s what I had in mind. You could have a built-in asset for the rhubarb phoneme states. you link it to an audio asset. You click “generate” and it will call rhubarb on the backend (like it does ffmpeg on the backend for video export) and generate the output. Then the built-in asset would have a script similar to the one I have in the demo. Alternatively, instead of a script you could just have wick “bake” the necessary key frames based on the rhubarb output.

It wouldn’t cause any overhead anymore than ffmpeg does. And I wouldn’t call auto lipsyncing tools bloat. So I just think it’s something that should definitely get serious consideration.