We have released v0.31.0-alpha. It supports Anniversary Edition and includes a brand new optional speech recognition engine: Voice2Json to replace the old Microsoft speech recognition engine.
Note the engine is always optional. If you remove the Engine=Voice2Json line from the configuration file, the Microsoft Speech Recognition Engine will be used. Then you can use the alpha version like the beta version, and you don't need to install Docker for Windows.
The new engine will solve many problems: 1. Microsoft's speech recognition engine is bound to the system locale, making it difficult to make the engine work properly. 2. Lower recognition rate. You have to speak clearly like an announcer to be likely to be recognized. 3. The probability of false triggering is high. After a while, any noise the microphone picks up will trigger a random command.
The new engine does not have the aforementioned drawbacks. For English speakers, it has far better recognition accuracy than Microsoft and is less prone to false triggering.
However, the new engine is not perfect, and some non-English languages may not work at all (Known issues: Chinese and Russian works poorly). We look forward to your feedback.
Also, in order to run the new engine, you must have Docker Desktop installed. You will also need to modify the configuration file for the new engine to take effect (the default engine is still Microsoft).
dragonborn_speaks_naturally.dll contains code that hooks into SkyrimSE/VR.exe in order to take over dialogue and console commands. This hooking method is similar to a virus. Meanwhile, DragonbornSpeaksNaturally.exe has code to execute cmd commands in Windows in order to connect to the Voice2Json speech engine in docker. This makes some antivirus software think they are suspicious. In addition, DragonbornSpeaksNaturally.exe uses your microphone, which further increases the suspicion of antivirus software on it.
Therefore, every new version we upload will be quarantined and need to wait for the review of a NexusMods staff. This is why most of our releases are flagged for manually verified.
So are these files really doing the malicious things that antivirus software fears? I think the answer is no. dragonborn_speaks_naturally.dll just does what you want it to do: hooks into Skyrim, takes over Skyrim's dialogue and console commands, and simulates keypresses as you dictate. DragonbornSpeaksNaturally.exe also does what you ask it to do: record from your microphone, pass it to one of the speech recognition engines (Microsoft or Voice2Json), recognize the command you want to run and send it to dragonborn_speaks_naturally.dll. Then, dragonborn_speaks_naturally.dll will execute these commands (console, dialog or simulated keystrokes). During the entire process, your microphone recording never leaves your computer. And all the behavior of the mod depends on the configuration files you write and what you say through the microphone. No third party can remotely control its behavior.
The mod will not connect to the Internet while it is running (If you choose the Voice2Json engine, some model files will be downloaded from GitHub on the first launch, and once the download is complete, no internet connection is required; the Microsoft engine also does not use an internet connection). If you see an Internet connection in the VirusTotal behavior report, it's not the mod, but the VirusTotal's scanner are downloading the zip file from their own servers. The same page will also record the behavior of the scanner to decompress the DSN zip with 7z, which proves that not all the behaviors recorded are from the scanning target.
If you still have doubts about the safety of the mod, you can review its code yourself or ask experienced people. All of our releases are compiled from the following repositories, using Visual Studio Community 2022, to the x64 Release target: https://github.com/YihaoPeng/DragonbornSpeaksNaturally
It works fine for me, the only problem that I've got is that I cn'a equip spells in both hands, only one. This is a example of my spells so far: Curacion menor=player.Equipspell 00012FCC right Llamas a mi=player.Equipspell 00012FCD right Relampago=player.Equipspell 0002DD29 left Curar aliado=player.Equipspell 0004D3F2 right
Even when I change left or right with both... It doen's work. Any tips? thank you!
Thank you for one of the most immersive mods. The first time I found this mod was a few years ago, and I was so excited to find (from talking with the other user here) that the Voice2JSON can recognize my speech very well.
That time, I remember I was amazed that it could distinguish between firebolt and fireball (Custom Command to cast) - 95% of the time.
Please feel free to put this information on the front page, something like mod conflict, for others to know.
/////
I'd like to contribute that after testing, I found {Alternate Conversation Camera} to conflict (partially) with this mod. {Alternate Conversation Camera} https://www.nexusmods.com/skyrimspecialedition/mods/21220
Issue: When using both mods, Can't use {DSN}'s Dialogue Function. Speaking won't select the dialogue choice. Like, it's not hooked to the choices. Goodbye Phrases also don't exit the conversation.
Solution: 1. Disable {Alternate Conversation Camera} for immersively speaking or 2. If you don't mind moving up/down to select the dialogue choice the old way, just leave them be. - Probably set dialogue function to 0 for {DSN} as well. or 3. Disable {DSN} if you don't use the talk - But I guess this is not a solution any of us would want here.
I'm not well-versed in modding/coding, so I won't know how to let both mods work together perfectly.
-----
Other things still work fine:
-Custom Console Commands - Very good -Shout with {SkyVoice Reloaded VR for VoiceMacro or DSN or VoiceAttack or WSR} - very good https://www.nexusmods.com/skyrimspecialedition/mods/17840 I like that with DSN, I don't need VoiceMacro now. (Tested with Fus Ro Dah - 1.5-1.8 sec after the last word spoken - Nice) ("Fus" alone doesn't shout for me, I had to say "Fus Ro") (Maybe I might add "Ha" in front of any 1 word shout later)
-Favourites - Doesn't work well for me, I can only equip Iron Shield and Long Bow but can't equip Iron Sword/Iron War Axe/Flames spell/Healing spell. I had to add "knownEquipmentTypes= ... " line into the ini file. From the log file: Equip Sword (I read as saude, like sauce but with d), the DSN hear as equip zol (Confidence 0). Equip Axe, the DSN hear as equip equip. XD (Didn't check with the spell) Might be different with other items later.
Should be fine in Skyrim VR as the {ACC} doesn't work in VR anyway.
/////
Version tested on 2024-5-18:
Windows 10
Skyrim SE - 1.5.97 - on SSD (Both game and mods on SSD) {Dragonborn Speaks Naturally} - 0.31.1 {xSHADOWMANx's Dll Loader} - 1.0.0.4 (Not like it's updated recently) {Alternate Conversation Camera} - 2.4.4 (the version for SE)
DSN - Voice2Json (Docker Desktop)
Locale and recognizer: Locale=en-us_kaldi-zamia My accent - Asian accent English
en-us_kaldi-rhasspy and en-us_kaldi-zamia Works well for me, at least for now.
Hey just want to say I love this mod, I'm wondering though, is there a version or way to use this with the dialogue interface option only? I loved being able to talk to NPCs and companions directly but I was very confused at first when my character started throwing gang signs (from me talking into my mic randomly and it switching weapons or active spells). It took me a while to realise it was from this and my mic 😅. I'm planning on doing a playthrough when I get ahold of the anniversary edition in the near future and I would love to give this another try. Just hoping I don't have to stay still and quiet while exploring or battling. Thanks!
Having an issue where the mod works sometimes, but not always. I dont know if it works only on conversations with few dialogue choices, but I have that suspicion. I feel like it works when there are only 3-4 choices, but never works when there are 10+.
I have a downgraded SSE and this mod keeps crashing my game, it always happens when I am in dialogue. No crashes when the mod is disabled. Load order crash log
If you have installed a DSN for AE, you will need to remove it from the mod manager and re-add it. Then you need to select the correct game version (Skyrim SE 1.5.97) in the DSN's installation dialog.
AE's dll does not work with SE 1.5.97 and will crash the game. Different selections in the installation dialog will cause different dlls to be released.
Note: We have released a version for AE 1.6.1170, which can be used if you need. (As before, because the virus scan issue is pending administrator review.)
When you launch Skyrim for the first time after installing **Docker for Windows**, DSN will automatically install **voice2json** with the **docker** command for you. When the installation is complete, you will see a `dsn_voice2json` container in the Docker Desktop UI.
This isn't the mod authors fault really its the new 1130 update on steam. Since it removed steam integration for the game things like achievement enablers and a lot of skse plugins just don't work with the update. I have heard that this issue will be fixed in the next update from steam but we shall see.
1125 comments
We have released v0.31.0-alpha. It supports Anniversary Edition and includes a brand new optional speech recognition engine: Voice2Json to replace the old Microsoft speech recognition engine.
The new engine will solve many problems:
1. Microsoft's speech recognition engine is bound to the system locale, making it difficult to make the engine work properly.
2. Lower recognition rate. You have to speak clearly like an announcer to be likely to be recognized.
3. The probability of false triggering is high. After a while, any noise the microphone picks up will trigger a random command.
The new engine does not have the aforementioned drawbacks. For English speakers, it has far better recognition accuracy than Microsoft and is less prone to false triggering.
However, the new engine is not perfect, and some non-English languages may not work at all (Known issues: Chinese and Russian works poorly). We look forward to your feedback.
Also, in order to run the new engine, you must have Docker Desktop installed. You will also need to modify the configuration file for the new engine to take effect (the default engine is still Microsoft).
The specific installation steps are here: https://github.com/YihaoPeng/DragonbornSpeaksNaturally/releases
---
Manually Verified / Quarantined Files
dragonborn_speaks_naturally.dll contains code that hooks into SkyrimSE/VR.exe in order to take over dialogue and console commands. This hooking method is similar to a virus. Meanwhile, DragonbornSpeaksNaturally.exe has code to execute cmd commands in Windows in order to connect to the Voice2Json speech engine in docker. This makes some antivirus software think they are suspicious. In addition, DragonbornSpeaksNaturally.exe uses your microphone, which further increases the suspicion of antivirus software on it.
Therefore, every new version we upload will be quarantined and need to wait for the review of a NexusMods staff. This is why most of our releases are flagged for manually verified.
So are these files really doing the malicious things that antivirus software fears? I think the answer is no. dragonborn_speaks_naturally.dll just does what you want it to do: hooks into Skyrim, takes over Skyrim's dialogue and console commands, and simulates keypresses as you dictate. DragonbornSpeaksNaturally.exe also does what you ask it to do: record from your microphone, pass it to one of the speech recognition engines (Microsoft or Voice2Json), recognize the command you want to run and send it to dragonborn_speaks_naturally.dll. Then, dragonborn_speaks_naturally.dll will execute these commands (console, dialog or simulated keystrokes). During the entire process, your microphone recording never leaves your computer. And all the behavior of the mod depends on the configuration files you write and what you say through the microphone. No third party can remotely control its behavior.
The mod will not connect to the Internet while it is running (If you choose the Voice2Json engine, some model files will be downloaded from GitHub on the first launch, and once the download is complete, no internet connection is required; the Microsoft engine also does not use an internet connection). If you see an Internet connection in the VirusTotal behavior report, it's not the mod, but the VirusTotal's scanner are downloading the zip file from their own servers. The same page will also record the behavior of the scanner to decompress the DSN zip with 7z, which proves that not all the behaviors recorded are from the scanning target.
If you still have doubts about the safety of the mod, you can review its code yourself or ask experienced people. All of our releases are compiled from the following repositories, using Visual Studio Community 2022, to the x64 Release target: https://github.com/YihaoPeng/DragonbornSpeaksNaturally
Curacion menor=player.Equipspell 00012FCC right
Llamas a mi=player.Equipspell 00012FCD right
Relampago=player.Equipspell 0002DD29 left
Curar aliado=player.Equipspell 0004D3F2 right
Even when I change left or right with both... It doen's work. Any tips? thank you!
Thank you for one of the most immersive mods. The first time I found this mod was a few years ago, and I was so excited to find (from talking with the other user here) that the Voice2JSON can recognize my speech very well.
That time, I remember I was amazed that it could distinguish between firebolt and fireball (Custom Command to cast) - 95% of the time.
Please feel free to put this information on the front page, something like mod conflict, for others to know.
/////
I'd like to contribute that after testing, I found {Alternate Conversation Camera} to conflict (partially) with this mod.
{Alternate Conversation Camera}
https://www.nexusmods.com/skyrimspecialedition/mods/21220
Issue: When using both mods, Can't use {DSN}'s Dialogue Function. Speaking won't select the dialogue choice. Like, it's not hooked to the choices.
Goodbye Phrases also don't exit the conversation.
Solution:
1. Disable {Alternate Conversation Camera} for immersively speaking or
2. If you don't mind moving up/down to select the dialogue choice the old way, just leave them be. - Probably set dialogue function to 0 for {DSN} as well. or
3. Disable {DSN} if you don't use the talk - But I guess this is not a solution any of us would want here.
I'm not well-versed in modding/coding, so I won't know how to let both mods work together perfectly.
-----
Other things still work fine:
-Custom Console Commands - Very good
-Shout with {SkyVoice Reloaded VR for VoiceMacro or DSN or VoiceAttack or WSR} - very good
https://www.nexusmods.com/skyrimspecialedition/mods/17840
I like that with DSN, I don't need VoiceMacro now.
(Tested with Fus Ro Dah - 1.5-1.8 sec after the last word spoken - Nice)
("Fus" alone doesn't shout for me, I had to say "Fus Ro")
(Maybe I might add "Ha" in front of any 1 word shout later)
-Favourites - Doesn't work well for me, I can only equip Iron Shield and Long Bow but can't equip Iron Sword/Iron War Axe/Flames spell/Healing spell.
I had to add "knownEquipmentTypes= ... " line into the ini file.
From the log file:
Equip Sword (I read as saude, like sauce but with d), the DSN hear as equip zol (Confidence 0).
Equip Axe, the DSN hear as equip equip.
XD (Didn't check with the spell)
Might be different with other items later.
Should be fine in Skyrim VR as the {ACC} doesn't work in VR anyway.
/////
Version tested on 2024-5-18:
Windows 10
Skyrim SE - 1.5.97 - on SSD (Both game and mods on SSD)
{Dragonborn Speaks Naturally} - 0.31.1
{xSHADOWMANx's Dll Loader} - 1.0.0.4 (Not like it's updated recently)
{Alternate Conversation Camera} - 2.4.4 (the version for SE)
DSN - Voice2Json (Docker Desktop)
Locale and recognizer:
Locale=en-us_kaldi-zamia
My accent - Asian accent English
en-us_kaldi-rhasspy
and
en-us_kaldi-zamia
Works well for me, at least for now.
I'm planning on doing a playthrough when I get ahold of the anniversary edition in the near future and I would love to give this another try. Just hoping I don't have to stay still and quiet while exploring or battling.
Thanks!
Any ideas on how to fix this?
When it works, it's awesome.
crash log
AE's dll does not work with SE 1.5.97 and will crash the game. Different selections in the installation dialog will cause different dlls to be released.
Note: We have released a version for AE 1.6.1170, which can be used if you need. (As before, because the virus scan issue is pending administrator review.)
damn stromcloaks , skyrim was safe before they make mess, mods was stable and good,
well , I just starting new playthrough, don't wanna break my game yet with this update, let's see ...