Lip-Sync?
Blitz3D Forums/Blitz3D Programming/Lip-Sync?
| ||
| Anyone done any work on Lip-Sync? And please don't say 'BlitzBass', coz at 995$ (or whatever) for a commercial license what's the point? |
| ||
| Hi John, No not yet, But sooner or later I've got to go there. I had a few thinks about this before. I think it would be too complext to 'deform geometry' (for me at least;) So my approach would be to create different textures for a characters mouth- and replace/animate (multiple frames texture) them according to the speech. If I can't replace his mouth texture - I'll stick and match a sprite in front of his mouth to get the same effect. Then basically making (for example) 8 different 'mouth poses' for the A, E, I, O, U, etc. (also, a smile, grin, and other basic expressions. To 'sync' then with the exactual voice (=wav file) I'd create a long string like this for example: "E...L.L..O.... E..A.A..G...ER..." wich could translate in something like 'hello stranger' ;) Where each letter corresponds to one of the x textures (or A, E, I, O, U, 'phonyms'). And each character in that string would ie. be 1/4 of a second. That way to 'only' have to translate each wav file to one of those text strings. This should be super-easy to code (and doesn't cost you anything else except time ;) Actuall speach-to-text or speach-to-text would be too complex I guess and take too much processor time. Also remember you don't have to translate each letter to get proper lip-sync - quite the contrary. The 'string' technique works really well, I've used something like this for many years doing commercials and such in 3D. Ofcourse I didn't do it using textures for mouth positions but with actual 3D blend shapes. Textures will work for most basic human characters I figure, although for a 'dragon's mouth' an animated texture would look rather silly I guess ;) But, as I said before, I've never actually tried this in Blitz :) Most importantly I still think that it's better NOT to have any lip-sync, than BAD lip-sync... Hope this helps, Danny. |
| ||
| . |
| ||
| You can also model your characters head seperately from the body, and make an animation for the mouth for each different mouth pose... I think this would look better than texture, and it would be just as easy to code for as with a texture. |
| ||
| This doesn't help now, but if you look in "Animated mesh with multiple weights" Mark should have support for blended vertex weights in the next update, at which point using bones for facial animation becomes an attractive option. |
| ||
| I believe there are tutorials online on how to model mouth animations for lip syncing for the different vowels/sounds etc. you would just have to do those animations for the model. |
| ||
| Somewhere in these forums there is a demo of lip syncing. I don't know whatever happaned to it - I'll have a look on my trusty disk drive for you! IPete2. edit. I found it - No Frontiers "Talk Test". It's not source code and requires the bass thing. |
| ||
| I think if you analyze the pitch and ranges of the sound, it should match generally with a base set of facial animaitons for each sound in the file. |
| ||
| As of last week, Cletus lip-syncs. |
| ||
| instead of making an animation sequence for each word or phrase you want the character to say, make sequences for each vowel/sound they will say, then play the anims in the correct sequence for the words. i'll try and find the tuturial i was looking at. |
| ||
| Hi Folks, John, depending on the type of game you could forget lip-synching and go for Kung-Fu movie style dubbing with genuine dodgy accents. That way the lips and speech don't have to match up. Good for a laugh, anyway. Later, Jes |
| ||
| You might like to try a search on 'visemes' (the visual counterpart of phonemes) before tackling this... most vowels will have the same viseme (try saying 'a..e..i' and look at how much the mouth changes - very little until you try 'o' and 'u') many sounds can use the same viseme - e.g. 'p' and 'b' both force the speaker to purse the lips... You might also do some research on phonetically equivalent 'strings' - search for 'Soundex Coding' so that you can play the same series of lip motions for the string 'feenicks' and 'phoenix' |
| ||
| Woah! Lots of replies! Thanks to you all, especially Jes : ) The problem is, Jes, that my chars look like that already... I merely animate the jaw up and down until the wave file isn't playing. Then I got an actor to do the voice who used lots of meaningful pauses, which brought to my attention the fact that the method should be one of 'tracking' the amplitude of the wave file as it plays, and opening or closing the mouth accordingly. So... BlitzBass can do this this, but is too costly. Anyone know of another dll that can play/track the wave file (in real time)? - That appears to be the method that Deus Ex uses, and very effective too. |
| ||
| Stumbled across this post by accident (looking for soundex code). If i were coding your project, for each of your wav files, I would create a guide string and use that to control the animation. For example "Hello. (meaningful pause) Greatings from Parth!" might have a duration of 8 seconds. My guide string, each character representing 0.5 sec would be "1110001111111111". A whole bunch of preprocessing (write a program!), store all the guide strings - job done. |
| ||
| Thats how a lot of games around half-life 1 era did it - each conversation sound file was accompanied by a file that described the animations to use - pretty much just a set time interval string with letters for the different animations. Dunno if BlitzAL has any similiar features I've never looked at everything it can do tho I'm guessing it doesn't. |
   |