🎥🎤😍 Kling AI LipSync: Easy to use, very cool but not perfect

My favorite AI video generator Kling AI has recently introduced a LipSync function, which I will briefly introduce to you here today:

- LipSync works with models 1.0 and 1.5 and is completely uncomplicated:
- You upload a picture of a person/face (image-to-video)
- The animation should not contain too much movement (prompt “static shot of”)
- I also usually add “a [man, woman, witch, ...] is speaking” to the prompt
- This will generate a video as normal (I recommend Pro mode)
- Below the generated video you will find a button “Match Mouth Type”
- This prepares the video for the LipSync or checks whether the LipSync works
- If LipSync is possible, a pop-up window opens
- There you upload the file with the spoken text (either self-recorded or AI-generated, as I did with ElevenLabs)
- You may have to shorten the audio file to the length of the video
- Start LipSync, this costs 5 credits extra and never took me longer than 5 minutes

How well does LipSync work?

- In my own videos, I have to admit that there are a few strange distortions in the lip movements. The teeth don't fit either. But that's not really a fair comparison, as Kling AI wasn't trained with me. ❌
- In the AI-generated templates, there are also sometimes very short sequences in which the spoken text and lip movements are not completely synchronized. ❌
- But in most cases, the lip sync works really well. ✅
- I am particularly impressed by the passages in which the face turns in a different direction or even away. Even there, the lips continue to animate synchronously. ✅
- 2 people in one video doesn't work yet. You can only have one person speak, Kling AI decides for itself which person speaks. Even regardless of whether it is a male or female voice. ❌
- It doesn't work at all with animals. ❌
- It sometimes works with comic/anime characters. ✅

✅ From my point of view, Kling AI LipSync is therefore already a very useful tool, and I'm already looking forward to shooting, er, generating my first AI movie with spoken texts. ✅

свернуть