Notifications
Notification preferences
notifications_off

No unread notifications right now

You're up to date
View All

You're up to date

File information

Last updated

Original upload

Created by

rmemr

Uploaded by

rmemr

Virus scan

Safe to use

Tags for this mod

Tag this mod
Current section

About this mod

radish lipsync for REDkit projects is an alternative lip animation generator which offers improved animation quality and easier voiceover management compared to the vanilla REDkit version.

Permissions and credits
Changelogs
radish lipsync 4 REDkit is an updated version of the lipsync animation generator feature from the radish community modding tools. The package is streamlined and adapted for usage in REDkit projects and supports two main usecases:

  • generation of lip movement animations synced to prerecorded audio
  • generation of lip movement animations based on input text only if audio is not available

Multilingual preschool education in 13th century (click on image for video)


Text to Lip Animation, also 13th century (click on image for video)


Note on quality: given the limited resources (free time, zero underlings) this is most probably about as good as I can get it without investing significantly more time.


Main Features

  • generate lip animations from audio
  • generate lip animations from text
  • support for multiple languages (de, en, fr, pl, ru) (*)
  • support for different animation profiles for different actors (**)
  • generated animations can be used in REDkit scene editor and/or loaded with the REDkit blender plugin
  • (optionally) automatically pack as w3speech file
(*) English is included in the download package and Polish, French, German and Russian language support is available as optional drop-in add-on package.
(**) Lipsync profiles are based on the poseweights from REDkit patch3 blender plugin and adjust animations for different head meshes.

Compared to REDkit lipsync solution

  • generally higher quality lip animations (*)
  • easier voiceover management (no blender required, no manual audio downsampling required)
  • bulk generation of animations
  • heuristic quality assessment: detection and visual cues for common problems (unusually long/short phoneme blocks, timing gaps in words, etc.)
  • easy to interactively adjust specific timings in GUI
  • easy to use multi language support for supported languages
  • easy assigning of a specific animation generation profile to actors in GUI
(*) It's not perfect and there are cases where timings can be screwed up. However it is very easy to adjust the timings in the GUI. My general advice is to check quality for voiceover lines used in close-up shots of scenes and if necessary tweak the timings for these lines.


Usage

The package contains a HOWTO text file with more details and more options but the general gist is:

  • unpack tools package (radish.lipsync.4.REDkit zip)
  • unpack project template package, read HOWTO and adjust settings
  • export REDkit strings csv into project root
  • (optional) put recorded audio files in language dependent subdirectory
  • (optional) link textlines with audio files (see example)
  • run generation batch file and open scene(s) in REDkit

There is also a little more background information and details in the wiki documentation for radish modding tools. However some of the info is outdated and tailored for radish modding tools projects. But it might be still interesting.


Thanks:
@Ard Carraigh: for french configuration tweaking
@lindrole, @jontel: for project template testing


Credits:
Text for example scene "The Boy Who Cried 'Wolf'" by Aesop (source)
Translation into different languages via DeepL
Voiceover audio generated with ElevenLabs


Copyrights
The phoneme-extractor tool uses the CMU Pocketsphinx library (https://github.com/cmusphinx/pocketsphinx),
the eSpeak Library (http://espeak.sourceforge.net).


Sourcecode
Sourcecode for the phoneme extractor is hosted at codeberg.org


Checksums
w3speech-phoneme-extractor.exe v0.7.2
size: 3813888 bytes
md5: 1cc25db5f77394e77f1379bfb41d9434
sha256: 26dba13823295e12c2434eb7c4a599ca3666740e614e86a6852426c669d3775e

espeak_lib.dll v0.7.2
size: 457728 bytes
md5: cb922b5f03d9ea1123738b93233e4cde
sha256: aba903d79cb0144ff57afbd608aae783d3a1aacf1aa3b3d8aec54250330ee03e

w3speech-lipsync-creator.exe v0.7.8 (git:9f642fa5 build:2025-01-05 17:00)
size: 932864 bytes
md5: cf0d9d17a605c1ccba0cea8b64867c87
sha256: 7303b4e2ab0800904c33a22efc072595a45bbaf4385517d6b632be7bcfdea361

w3speech.exe v0.7.8 (git:9f642fa5 build:2025-01-05 17:00)
size: 3509760 bytes
md5: 81d290c0a1324fc307b1d26f4f79e14f
sha256: 588017f183a8d4ff361c7ca52ade6a9dc3666287d736197c692ffc18f685ee88

w3speech-converter.exe v0.1.0 (git:5ba5356 build:2024-10-10 20:10)
size: 4387840 bytes
md5: 08dff9d799dcdf064f5af82ecd823b89
sha256: da1dce283d17949260ffae8f6d76e3587434e1557db357c1c9143bb77133d71c