We have released v0.31.0-alpha. It supports Anniversary Edition and includes a brand new optional speech recognition engine: Voice2Json to replace the old Microsoft speech recognition engine.
Note the engine is always optional. If you remove the Engine=Voice2Json line from the configuration file, the Microsoft Speech Recognition Engine will be used. Then you can use the alpha version like the beta version, and you don't need to install Docker for Windows.
The new engine will solve many problems: 1. Microsoft's speech recognition engine is bound to the system locale, making it difficult to make the engine work properly. 2. Lower recognition rate. You have to speak clearly like an announcer to be likely to be recognized. 3. The probability of false triggering is high. After a while, any noise the microphone picks up will trigger a random command.
The new engine does not have the aforementioned drawbacks. For English speakers, it has far better recognition accuracy than Microsoft and is less prone to false triggering.
However, the new engine is not perfect, and some non-English languages may not work at all (Known issues: Chinese and Russian works poorly). We look forward to your feedback.
Also, in order to run the new engine, you must have Docker Desktop installed. You will also need to modify the configuration file for the new engine to take effect (the default engine is still Microsoft).
dragonborn_speaks_naturally.dll contains code that hooks into SkyrimSE/VR.exe in order to take over dialogue and console commands. This hooking method is similar to a virus. Meanwhile, DragonbornSpeaksNaturally.exe has code to execute cmd commands in Windows in order to connect to the Voice2Json speech engine in docker. This makes some antivirus software think they are suspicious. In addition, DragonbornSpeaksNaturally.exe uses your microphone, which further increases the suspicion of antivirus software on it.
Therefore, every new version we upload will be quarantined and need to wait for the review of a NexusMods staff. This is why most of our releases are flagged for manually verified.
So are these files really doing the malicious things that antivirus software fears? I think the answer is no. dragonborn_speaks_naturally.dll just does what you want it to do: hooks into Skyrim, takes over Skyrim's dialogue and console commands, and simulates keypresses as you dictate. DragonbornSpeaksNaturally.exe also does what you ask it to do: record from your microphone, pass it to one of the speech recognition engines (Microsoft or Voice2Json), recognize the command you want to run and send it to dragonborn_speaks_naturally.dll. Then, dragonborn_speaks_naturally.dll will execute these commands (console, dialog or simulated keystrokes). During the entire process, your microphone recording never leaves your computer. And all the behavior of the mod depends on the configuration files you write and what you say through the microphone. No third party can remotely control its behavior.
The mod will not connect to the Internet while it is running (If you choose the Voice2Json engine, some model files will be downloaded from GitHub on the first launch, and once the download is complete, no internet connection is required; the Microsoft engine also does not use an internet connection). If you see an Internet connection in the VirusTotal behavior report, it's not the mod, but the VirusTotal's scanner are downloading the zip file from their own servers. The same page will also record the behavior of the scanner to decompress the DSN zip with 7z, which proves that not all the behaviors recorded are from the scanning target.
If you still have doubts about the safety of the mod, you can review its code yourself or ask experienced people. All of our releases are compiled from the following repositories, using Visual Studio Community 2022, to the x64 Release target: https://github.com/YihaoPeng/DragonbornSpeaksNaturally
Did I spend all night making a fat .ini file for users? Yes. https://pastebin.com/qBwrQd1L Includes: All spells. Left, right, dual wield and cast commands. Equip all shouts / shout command. Quick "Shoot" command for archers plus "Aim" and "Fire". (Hold and release.) Menu / Map / Back commands that work. Funnier thieves command. "Mine" (Seagull RP)
Coming soon... All apocalypse spells.
P.S Auto Input Switch should really be listed as a requirement. https://www.nexusmods.com/skyrimspecialedition/mods/54309
That would be great to have an official installation and setup guide to show what to do to install it properly and make it work.
It's very obscure and somehow when Docker start up, it turns in circle, it's incredibly long. I'm on 64Gb of DDR5 7200mhz Ram and i9 13900K and 2To SSD to say the least, so it's not my computer limitations.
Hello, thank you for this amazing and immersive mod - I was wondering if any help could be possible here - The game seems to crash if I say "hello / excuse me" to an npc and I am not entirely sure why. I run it on VR. It works as normal when in dialogue but as soon as I try to greet someone it crashes.
I've got this mod working decently well on the equipping and using spells side of things, but I'm having an issue with the dialogue recognition. It seems that the program starts listening around 3 seconds after the NPC finishes their line, as it will catch me saying the line the second or third time I say it or if I wait a few seconds after the NPC finishes. The log confirms this - it skips the first few words, but gets the rest of the sentence perfectly. I've tried each of the subset matching settings with no luck. Is there any workaround for this? Example below.
DragonbornSpeaksNaturally.exe Information: 0 : Recognition log: 'like those nords were giving you trouble.' (Confidence: 0) DateTime=2024-06-02T07:09:52.5778121Z DragonbornSpeaksNaturally.exe Information: 0 : Recognition log: 'Looked like those Nords were giving you trouble.' (Confidence: 1) DateTime=2024-06-02T07:09:57.2763250Z DragonbornSpeaksNaturally.exe Information: 0 : Recognized phrase 'Looked like those Nords were giving you trouble.' (Confidence: 1) DateTime=2024-06-02T07:09:57.2763250Z
It works fine for me, the only problem that I've got is that I cn'a equip spells in both hands, only one. This is a example of my spells so far: Curacion menor=player.Equipspell 00012FCC right Llamas a mi=player.Equipspell 00012FCD right Relampago=player.Equipspell 0002DD29 left Curar aliado=player.Equipspell 0004D3F2 right
Even when I change left or right with both... It doen's work. Any tips? thank you!
Thank you for one of the most immersive mods. The first time I found this mod was a few years ago, and I was so excited to find (from talking with the other user here) that the Voice2JSON can recognize my speech very well.
That time, I remember I was amazed that it could distinguish between firebolt and fireball (Custom Command to cast) - 95% of the time.
Please feel free to put this information on the front page, something like mod conflict, for others to know.
/////
I'd like to contribute that after testing, I found {Alternate Conversation Camera} to conflict (partially) with this mod. {Alternate Conversation Camera} https://www.nexusmods.com/skyrimspecialedition/mods/21220
Issue: When using both mods, Can't use {DSN}'s Dialogue Function. Speaking won't select the dialogue choice. Like, it's not hooked to the choices. Goodbye Phrases also don't exit the conversation.
Solution: 1. Disable {Alternate Conversation Camera} for immersively speaking or 2. If you don't mind moving up/down to select the dialogue choice the old way, just leave them be. - Probably set dialogue function to 0 for {DSN} as well. or 3. Disable {DSN} if you don't use the talk - But I guess this is not a solution any of us would want here.
I'm not well-versed in modding/coding, so I won't know how to let both mods work together perfectly.
-----
Other things still work fine:
-Custom Console Commands - Very good -Shout with {SkyVoice Reloaded VR for VoiceMacro or DSN or VoiceAttack or WSR} - very good https://www.nexusmods.com/skyrimspecialedition/mods/17840 I like that with DSN, I don't need VoiceMacro now. (Tested with Fus Ro Dah - 1.5-1.8 sec after the last word spoken - Nice) ("Fus" alone doesn't shout for me, I had to say "Fus Ro") (Maybe I might add "Ha" in front of any 1 word shout later)
-Favourites - Doesn't work well for me, I can only equip Iron Shield and Long Bow but can't equip Iron Sword/Iron War Axe/Flames spell/Healing spell. I had to add "knownEquipmentTypes= ... " line into the ini file. From the log file: Equip Sword (I read as saude, like sauce but with d), the DSN hear as equip zol (Confidence 0). Equip Axe, the DSN hear as equip equip. XD (Didn't check with the spell) Might be different with other items later.
Should be fine in Skyrim VR as the {ACC} doesn't work in VR anyway.
/////
Version tested on 2024-5-18:
Windows 10
Skyrim SE - 1.5.97 - on SSD (Both game and mods on SSD) {Dragonborn Speaks Naturally} - 0.31.1 {xSHADOWMANx's Dll Loader} - 1.0.0.4 (Not like it's updated recently) {Alternate Conversation Camera} - 2.4.4 (the version for SE)
DSN - Voice2Json (Docker Desktop)
Locale and recognizer: Locale=en-us_kaldi-zamia My accent - Asian accent English
en-us_kaldi-rhasspy and en-us_kaldi-zamia Works well for me, at least for now.
Hey just want to say I love this mod, I'm wondering though, is there a version or way to use this with the dialogue interface option only? I loved being able to talk to NPCs and companions directly but I was very confused at first when my character started throwing gang signs (from me talking into my mic randomly and it switching weapons or active spells). It took me a while to realise it was from this and my mic 😅. I'm planning on doing a playthrough when I get ahold of the anniversary edition in the near future and I would love to give this another try. Just hoping I don't have to stay still and quiet while exploring or battling. Thanks!
1131 comments
We have released v0.31.0-alpha. It supports Anniversary Edition and includes a brand new optional speech recognition engine: Voice2Json to replace the old Microsoft speech recognition engine.
The new engine will solve many problems:
1. Microsoft's speech recognition engine is bound to the system locale, making it difficult to make the engine work properly.
2. Lower recognition rate. You have to speak clearly like an announcer to be likely to be recognized.
3. The probability of false triggering is high. After a while, any noise the microphone picks up will trigger a random command.
The new engine does not have the aforementioned drawbacks. For English speakers, it has far better recognition accuracy than Microsoft and is less prone to false triggering.
However, the new engine is not perfect, and some non-English languages may not work at all (Known issues: Chinese and Russian works poorly). We look forward to your feedback.
Also, in order to run the new engine, you must have Docker Desktop installed. You will also need to modify the configuration file for the new engine to take effect (the default engine is still Microsoft).
The specific installation steps are here: https://github.com/YihaoPeng/DragonbornSpeaksNaturally/releases
---
Manually Verified / Quarantined Files
dragonborn_speaks_naturally.dll contains code that hooks into SkyrimSE/VR.exe in order to take over dialogue and console commands. This hooking method is similar to a virus. Meanwhile, DragonbornSpeaksNaturally.exe has code to execute cmd commands in Windows in order to connect to the Voice2Json speech engine in docker. This makes some antivirus software think they are suspicious. In addition, DragonbornSpeaksNaturally.exe uses your microphone, which further increases the suspicion of antivirus software on it.
Therefore, every new version we upload will be quarantined and need to wait for the review of a NexusMods staff. This is why most of our releases are flagged for manually verified.
So are these files really doing the malicious things that antivirus software fears? I think the answer is no. dragonborn_speaks_naturally.dll just does what you want it to do: hooks into Skyrim, takes over Skyrim's dialogue and console commands, and simulates keypresses as you dictate. DragonbornSpeaksNaturally.exe also does what you ask it to do: record from your microphone, pass it to one of the speech recognition engines (Microsoft or Voice2Json), recognize the command you want to run and send it to dragonborn_speaks_naturally.dll. Then, dragonborn_speaks_naturally.dll will execute these commands (console, dialog or simulated keystrokes). During the entire process, your microphone recording never leaves your computer. And all the behavior of the mod depends on the configuration files you write and what you say through the microphone. No third party can remotely control its behavior.
The mod will not connect to the Internet while it is running (If you choose the Voice2Json engine, some model files will be downloaded from GitHub on the first launch, and once the download is complete, no internet connection is required; the Microsoft engine also does not use an internet connection). If you see an Internet connection in the VirusTotal behavior report, it's not the mod, but the VirusTotal's scanner are downloading the zip file from their own servers. The same page will also record the behavior of the scanner to decompress the DSN zip with 7z, which proves that not all the behaviors recorded are from the scanning target.
If you still have doubts about the safety of the mod, you can review its code yourself or ask experienced people. All of our releases are compiled from the following repositories, using Visual Studio Community 2022, to the x64 Release target: https://github.com/YihaoPeng/DragonbornSpeaksNaturally
https://pastebin.com/qBwrQd1L
Includes:
All spells. Left, right, dual wield and cast commands.
Equip all shouts / shout command.
Quick "Shoot" command for archers plus "Aim" and "Fire". (Hold and release.)
Menu / Map / Back commands that work.
Funnier thieves command. "Mine" (Seagull RP)
Coming soon... All apocalypse spells.
P.S Auto Input Switch should really be listed as a requirement.
https://www.nexusmods.com/skyrimspecialedition/mods/54309
P.P.S New engine rocks.
That would be great to have an official installation and setup guide to show what to do to install it properly and make it work.
It's very obscure and somehow when Docker start up, it turns in circle, it's incredibly long. I'm on 64Gb of DDR5 7200mhz Ram and i9 13900K and 2To SSD to say the least, so it's not my computer limitations.
Thanks.
DragonbornSpeaksNaturally.exe Information: 0 : Recognition log: 'like those nords were giving you trouble.' (Confidence: 0)
DateTime=2024-06-02T07:09:52.5778121Z
DragonbornSpeaksNaturally.exe Information: 0 : Recognition log: 'Looked like those Nords were giving you trouble.' (Confidence: 1)
DateTime=2024-06-02T07:09:57.2763250Z
DragonbornSpeaksNaturally.exe Information: 0 : Recognized phrase 'Looked like those Nords were giving you trouble.' (Confidence: 1)
DateTime=2024-06-02T07:09:57.2763250Z
Curacion menor=player.Equipspell 00012FCC right
Llamas a mi=player.Equipspell 00012FCD right
Relampago=player.Equipspell 0002DD29 left
Curar aliado=player.Equipspell 0004D3F2 right
Even when I change left or right with both... It doen's work. Any tips? thank you!
Thank you for one of the most immersive mods. The first time I found this mod was a few years ago, and I was so excited to find (from talking with the other user here) that the Voice2JSON can recognize my speech very well.
That time, I remember I was amazed that it could distinguish between firebolt and fireball (Custom Command to cast) - 95% of the time.
Please feel free to put this information on the front page, something like mod conflict, for others to know.
/////
I'd like to contribute that after testing, I found {Alternate Conversation Camera} to conflict (partially) with this mod.
{Alternate Conversation Camera}
https://www.nexusmods.com/skyrimspecialedition/mods/21220
Issue: When using both mods, Can't use {DSN}'s Dialogue Function. Speaking won't select the dialogue choice. Like, it's not hooked to the choices.
Goodbye Phrases also don't exit the conversation.
Solution:
1. Disable {Alternate Conversation Camera} for immersively speaking or
2. If you don't mind moving up/down to select the dialogue choice the old way, just leave them be. - Probably set dialogue function to 0 for {DSN} as well. or
3. Disable {DSN} if you don't use the talk - But I guess this is not a solution any of us would want here.
I'm not well-versed in modding/coding, so I won't know how to let both mods work together perfectly.
-----
Other things still work fine:
-Custom Console Commands - Very good
-Shout with {SkyVoice Reloaded VR for VoiceMacro or DSN or VoiceAttack or WSR} - very good
https://www.nexusmods.com/skyrimspecialedition/mods/17840
I like that with DSN, I don't need VoiceMacro now.
(Tested with Fus Ro Dah - 1.5-1.8 sec after the last word spoken - Nice)
("Fus" alone doesn't shout for me, I had to say "Fus Ro")
(Maybe I might add "Ha" in front of any 1 word shout later)
-Favourites - Doesn't work well for me, I can only equip Iron Shield and Long Bow but can't equip Iron Sword/Iron War Axe/Flames spell/Healing spell.
I had to add "knownEquipmentTypes= ... " line into the ini file.
From the log file:
Equip Sword (I read as saude, like sauce but with d), the DSN hear as equip zol (Confidence 0).
Equip Axe, the DSN hear as equip equip.
XD (Didn't check with the spell)
Might be different with other items later.
Should be fine in Skyrim VR as the {ACC} doesn't work in VR anyway.
/////
Version tested on 2024-5-18:
Windows 10
Skyrim SE - 1.5.97 - on SSD (Both game and mods on SSD)
{Dragonborn Speaks Naturally} - 0.31.1
{xSHADOWMANx's Dll Loader} - 1.0.0.4 (Not like it's updated recently)
{Alternate Conversation Camera} - 2.4.4 (the version for SE)
DSN - Voice2Json (Docker Desktop)
Locale and recognizer:
Locale=en-us_kaldi-zamia
My accent - Asian accent English
en-us_kaldi-rhasspy
and
en-us_kaldi-zamia
Works well for me, at least for now.
I'm planning on doing a playthrough when I get ahold of the anniversary edition in the near future and I would love to give this another try. Just hoping I don't have to stay still and quiet while exploring or battling.
Thanks!