We have released v0.31.0-alpha. It supports Anniversary Edition and includes a brand new optional speech recognition engine: Voice2Json to replace the old Microsoft speech recognition engine.
Note the engine is always optional. If you remove the Engine=Voice2Json line from the configuration file, the Microsoft Speech Recognition Engine will be used. Then you can use the alpha version like the beta version, and you don't need to install Docker for Windows.
The new engine will solve many problems: 1. Microsoft's speech recognition engine is bound to the system locale, making it difficult to make the engine work properly. 2. Lower recognition rate. You have to speak clearly like an announcer to be likely to be recognized. 3. The probability of false triggering is high. After a while, any noise the microphone picks up will trigger a random command.
The new engine does not have the aforementioned drawbacks. For English speakers, it has far better recognition accuracy than Microsoft and is less prone to false triggering.
However, the new engine is not perfect, and some non-English languages may not work at all (Known issues: Chinese and Russian works poorly). We look forward to your feedback.
Also, in order to run the new engine, you must have Docker Desktop installed. You will also need to modify the configuration file for the new engine to take effect (the default engine is still Microsoft).
dragonborn_speaks_naturally.dll contains code that hooks into SkyrimSE/VR.exe in order to take over dialogue and console commands. This hooking method is similar to a virus. Meanwhile, DragonbornSpeaksNaturally.exe has code to execute cmd commands in Windows in order to connect to the Voice2Json speech engine in docker. This makes some antivirus software think they are suspicious. In addition, DragonbornSpeaksNaturally.exe uses your microphone, which further increases the suspicion of antivirus software on it.
Therefore, every new version we upload will be quarantined and need to wait for the review of a NexusMods staff. This is why most of our releases are flagged for manually verified.
So are these files really doing the malicious things that antivirus software fears? I think the answer is no. dragonborn_speaks_naturally.dll just does what you want it to do: hooks into Skyrim, takes over Skyrim's dialogue and console commands, and simulates keypresses as you dictate. DragonbornSpeaksNaturally.exe also does what you ask it to do: record from your microphone, pass it to one of the speech recognition engines (Microsoft or Voice2Json), recognize the command you want to run and send it to dragonborn_speaks_naturally.dll. Then, dragonborn_speaks_naturally.dll will execute these commands (console, dialog or simulated keystrokes). During the entire process, your microphone recording never leaves your computer. And all the behavior of the mod depends on the configuration files you write and what you say through the microphone. No third party can remotely control its behavior.
The mod will not connect to the Internet while it is running (If you choose the Voice2Json engine, some model files will be downloaded from GitHub on the first launch, and once the download is complete, no internet connection is required; the Microsoft engine also does not use an internet connection). If you see an Internet connection in the VirusTotal behavior report, it's not the mod, but the VirusTotal's scanner are downloading the zip file from their own servers. The same page will also record the behavior of the scanner to decompress the DSN zip with 7z, which proves that not all the behaviors recorded are from the scanning target.
If you still have doubts about the safety of the mod, you can review its code yourself or ask experienced people. All of our releases are compiled from the following repositories, using Visual Studio Community 2022, to the x64 Release target: https://github.com/YihaoPeng/DragonbornSpeaksNaturally
Thank you for this wonderful mod which has brought joy to my family and allowed me to enjoy a game which was otherwise 90% menu management in order to access spells etcetera. Now I can switch spells on the family without being limited to just a few things.
I have a few issues.
1. For WSL, it would be wonderful if there was a way to enable and disable speech recognition with a button press instead of having to voice the command. This is because after disabling speech recognition, ongoing speech will ALWAYS eventually re-enable voice commands, even if what I'm saying doesn't match activation pattern at all! This matters when I'm throwing in ambient chatter for my kids to enjoy, and is the biggest reason I was hoping Voice2JSON would work.
For reference, my disable phrase is "Please don't listen to me;Yes darling what is it", while the re-enable phrase is "Magic Elephant Says Activate Voice Features" and it STILL randomly re-enables while I'm reading a book out. lol
2. I have been trying to move to Voice2JSON to avoid the looming deletion of WSL, but could not work out what the issue is. I might try installing Voice2JSON without Docker to see if that's the problem, but the actual word recognition in the log seems worse than WSL. Voice2JSON does not seem to implement commands, as posted on Github. I can get it to activate and de-activate, but it won't equip things like WSL does. I tried dropping commandMinConfidence=0.7 from my preferred 0.9.
3. Is there any reason why the command "use shield" would often equip the spell "healing"?
I'm SO confused. i uncommented some of the commands in the .ini and made some of my own following the layout, and my engine loops the commands and now the dialogue side of the mod isn't even working.(after changing things, i can't even use the mod to pick responses in chats.) i've almost triple checked to make sure the only things i changed were the lines that i either added commands to, or removed the semicolon to make the example commands i wanted work.
DSN tries to find DragonbornSpeaksNaturally.ini in multiple locations, maybe it loads another DragonbornSpeaksNaturally.ini in a different location. It is recommended that you check Documents\DragonbornSpeaksNaturally\DragonbornSpeaksNaturally.log to confirm which configuration file is loaded.
So if I want to use this is there a way to get it to select the options based on keywords rather than having to say the entire line? Because I think it would be awesome to not have to read the lines like a script and just say what you want as long as you have the keyword.
Set Engine=Microsoft allow you to do this, then find SubsetMatchingMode=None and change it to one of the followingSubsetMatchingMode=OrderedSubsetContentRequired SubsetMatchingMode=SubsequenceContentRequired SubsetMatchingMode=OrderedSubset SubsetMatchingMode=Subsequence
filepath not found: I:\Steam\steamapps\common\SkyrimVR\Data\Plugins\Sumwunn\DragonbornSpeaksNaturally.ini filepath not found: I:\Steam\steamapps\common\SkyrimVR\DragonbornSpeaksNaturally.ini filepath not found: C:\Users\USER\Documents\DragonbornSpeaksNaturally\DragonbornSpeaksNaturally.ini filepath not found: C:\Users\USER\Documents\DragonbornSpeaksNaturally\DragonbornSpeaksNaturally.ini.ini filepath not found: I:\Steam\steamapps\common\SkyrimVR\DragonbornSpeaksNaturally.ini.ini Failed to watch for config files: System.Exception: Configuration file does not exist.
I've installed it as part of Mad God Overhaul, so the "DragonbornSpeaksNaturally.ini" file is in my wabbajack mods folder. How can I direct it to this directory?
Just copy DragonbornSpeaksNaturally.ini to the same folder as the log and everything should be fine.
If anyone doesn't know where the directory is, it's in "File Explorer > Documents > DragonbornSpeaksNaturally".
If someone still can't find the folder, there is another place to put it. Right-click your game in your Steam library, select Manage > Browse Local Files, and place DragonbornSpeaksNaturally.ini in the folder that opens.
The "DragonbornSpeaksNaturally.ini" file is in my wabbajack mods folder. How can I direct it to this directory?
Copying the file directly to Documents folder is always the recommended way. But if you don't want to copy files, you can redirect files using a feature called symbolic links. Right-click the Start button, select "Administrator Terminal" or "Administrator Command Prompt", then enter the following command (replace the path with your own) and press Enter:
Replace "C:\Users\USER\Documents\DragonbornSpeaksNaturally\DragonbornSpeaksNaturally.ini" to one path of your "filepath not found" error; Replace "X:\xxx\xxx\DragonbornSpeaksNaturally.ini" to the realpath of the DragonbornSpeaksNaturally.ini file (the path in your wabbajack mods folder).
You can do the same with modded followers but the ID's will depend on your load order so just check in console when you recruit them and add them with the same format. Hope this helps some people. For generic start conversation with any NPC then the talk to close ones mod is better as discussed in other posts. But this method does not require looking at the follower.
Worked well for me on Windows 11 for Skyrim VR on my Oculus Quest 2, and it worked right out of the box. I thought it might not work since I saw someone say somewhere that Windows cancelled Windows Speech Recognition (I think on the Dragonborn Unlimited posts section), but Speech Recognition worked for this mod.
As far as finding Windows Speech Recognition on Windows 11, it didn't show up in the control panel, but when I pressed windows key+ctrl+s, it popped up; so if you want to calibrate and configure, or just verify that you have it, I guess that's where you go.
Having some serious issues with false positives in VR. When trying to talk to Mantella, it goes haywire and triggers all my console commands. Is there a way to in game toggle the speech on and off? I only really need it in specific circumstances to trigger my commands, and can be off most other times.
1151 comments
We have released v0.31.0-alpha. It supports Anniversary Edition and includes a brand new optional speech recognition engine: Voice2Json to replace the old Microsoft speech recognition engine.
The new engine will solve many problems:
1. Microsoft's speech recognition engine is bound to the system locale, making it difficult to make the engine work properly.
2. Lower recognition rate. You have to speak clearly like an announcer to be likely to be recognized.
3. The probability of false triggering is high. After a while, any noise the microphone picks up will trigger a random command.
The new engine does not have the aforementioned drawbacks. For English speakers, it has far better recognition accuracy than Microsoft and is less prone to false triggering.
However, the new engine is not perfect, and some non-English languages may not work at all (Known issues: Chinese and Russian works poorly). We look forward to your feedback.
Also, in order to run the new engine, you must have Docker Desktop installed. You will also need to modify the configuration file for the new engine to take effect (the default engine is still Microsoft).
The specific installation steps are here: https://github.com/YihaoPeng/DragonbornSpeaksNaturally/releases
---
Manually Verified / Quarantined Files
dragonborn_speaks_naturally.dll contains code that hooks into SkyrimSE/VR.exe in order to take over dialogue and console commands. This hooking method is similar to a virus. Meanwhile, DragonbornSpeaksNaturally.exe has code to execute cmd commands in Windows in order to connect to the Voice2Json speech engine in docker. This makes some antivirus software think they are suspicious. In addition, DragonbornSpeaksNaturally.exe uses your microphone, which further increases the suspicion of antivirus software on it.
Therefore, every new version we upload will be quarantined and need to wait for the review of a NexusMods staff. This is why most of our releases are flagged for manually verified.
So are these files really doing the malicious things that antivirus software fears? I think the answer is no. dragonborn_speaks_naturally.dll just does what you want it to do: hooks into Skyrim, takes over Skyrim's dialogue and console commands, and simulates keypresses as you dictate. DragonbornSpeaksNaturally.exe also does what you ask it to do: record from your microphone, pass it to one of the speech recognition engines (Microsoft or Voice2Json), recognize the command you want to run and send it to dragonborn_speaks_naturally.dll. Then, dragonborn_speaks_naturally.dll will execute these commands (console, dialog or simulated keystrokes). During the entire process, your microphone recording never leaves your computer. And all the behavior of the mod depends on the configuration files you write and what you say through the microphone. No third party can remotely control its behavior.
The mod will not connect to the Internet while it is running (If you choose the Voice2Json engine, some model files will be downloaded from GitHub on the first launch, and once the download is complete, no internet connection is required; the Microsoft engine also does not use an internet connection). If you see an Internet connection in the VirusTotal behavior report, it's not the mod, but the VirusTotal's scanner are downloading the zip file from their own servers. The same page will also record the behavior of the scanner to decompress the DSN zip with 7z, which proves that not all the behaviors recorded are from the scanning target.
If you still have doubts about the safety of the mod, you can review its code yourself or ask experienced people. All of our releases are compiled from the following repositories, using Visual Studio Community 2022, to the x64 Release target: https://github.com/YihaoPeng/DragonbornSpeaksNaturally
I have a few issues.
1. For WSL, it would be wonderful if there was a way to enable and disable speech recognition with a button press instead of having to voice the command. This is because after disabling speech recognition, ongoing speech will ALWAYS eventually re-enable voice commands, even if what I'm saying doesn't match activation pattern at all! This matters when I'm throwing in ambient chatter for my kids to enjoy, and is the biggest reason I was hoping Voice2JSON would work.
For reference, my disable phrase is "Please don't listen to me;Yes darling what is it", while the re-enable phrase is "Magic Elephant Says Activate Voice Features" and it STILL randomly re-enables while I'm reading a book out. lol
2. I have been trying to move to Voice2JSON to avoid the looming deletion of WSL, but could not work out what the issue is. I might try installing Voice2JSON without Docker to see if that's the problem, but the actual word recognition in the log seems worse than WSL. Voice2JSON does not seem to implement commands, as posted on Github. I can get it to activate and de-activate, but it won't equip things like WSL does. I tried dropping commandMinConfidence=0.7 from my preferred 0.9.
3. Is there any reason why the command "use shield" would often equip the spell "healing"?
First hello won't register, 2nd hello will.
First "what have you got for sale" won't register, 2nd will. This also applies to goodbye phrases
Any way to fix this annoying problem?
hello=press e; sleep 10; release e
dsn_plugin/ConsoleCommandRunner.cpp#L75
So if it does work, it's probably because the extra invalid commands add processing delays.
So
hello=press e 100may works too. This will keep e pressed for 100 milliseconds instead of the default 50 milliseconds.Engine=Microsoft
allow you to do this, then findSubsetMatchingMode=None
and change it to one of the followingSubsetMatchingMode=OrderedSubsetContentRequired
SubsetMatchingMode=SubsequenceContentRequired
SubsetMatchingMode=OrderedSubset
SubsetMatchingMode=Subsequence
This link shows the difference of the four: https://learn.microsoft.com/en-us/dotnet/api/system.speech.recognition.subsetmatchingmode?redirectedfrom=MSDN&view=netframework-4.8#remarks
Engine=Voice2Json doesn't support it.
My log gets spammed with:
filepath not found: I:\Steam\steamapps\common\SkyrimVR\Data\Plugins\Sumwunn\DragonbornSpeaksNaturally.ini
filepath not found: I:\Steam\steamapps\common\SkyrimVR\DragonbornSpeaksNaturally.ini
filepath not found: C:\Users\USER\Documents\DragonbornSpeaksNaturally\DragonbornSpeaksNaturally.ini
filepath not found: C:\Users\USER\Documents\DragonbornSpeaksNaturally\DragonbornSpeaksNaturally.ini.ini
filepath not found: I:\Steam\steamapps\common\SkyrimVR\DragonbornSpeaksNaturally.ini.ini
Failed to watch for config files: System.Exception: Configuration file does not exist.
I've installed it as part of Mad God Overhaul, so the "DragonbornSpeaksNaturally.ini" file is in my wabbajack mods folder. How can I direct it to this directory?
If anyone doesn't know where the directory is, it's in "File Explorer > Documents > DragonbornSpeaksNaturally".
If someone still can't find the folder, there is another place to put it. Right-click your game in your Steam library, select Manage > Browse Local Files, and place DragonbornSpeaksNaturally.ini in the folder that opens.
---------------------------------------------------
Copying the file directly to Documents folder is always the recommended way. But if you don't want to copy files, you can redirect files using a feature called symbolic links. Right-click the Start button, select "Administrator Terminal" or "Administrator Command Prompt", then enter the following command (replace the path with your own) and press Enter:
cmd /c mklink "C:\Users\USER\Documents\DragonbornSpeaksNaturally\DragonbornSpeaksNaturally.ini" "X:\xxx\xxx\DragonbornSpeaksNaturally.ini"
Replace "C:\Users\USER\Documents\DragonbornSpeaksNaturally\DragonbornSpeaksNaturally.ini" to one path of your "filepath not found" error; Replace "X:\xxx\xxx\DragonbornSpeaksNaturally.ini" to the realpath of the DragonbornSpeaksNaturally.ini file (the path in your wabbajack mods folder).
;Talk to NPC with Hey
;;;Base Game
Hey Lydia=player.StartConversation 000A2C94
Hey Iona=player.StartConversation 000A2C93
Hey Jordis=player.StartConversation 000A2C95
Hey Argis=player.StartConversation 000A2C92
Hey Aela=player.StartConversation 0001A697
Hey Farkas=player.StartConversation 0001A693
Hey Vilkas=player.StartConversation 0001A695
Hey J'zargo=player.StartConversation 0001C1A3
Hey Onmund=player.StartConversation 0001C1A2
Hey Brelyna=player.StartConversation 0001C1A4
Hey Cicero=player.StartConversation 0009BCB0
Hey Jenassa=player.StartConversation 000E1BA9
Hey Marcurio=player.StartConversation 000B9986
Hey Vorstag=player.StartConversation 000B9984
Hey Erik=player.StartConversation 000656E2
Hey Mjoll=player.StartConversation 00019DF7
Hey Ahtar=player.StartConversation 000198B0
Hey Illia=player.StartConversation 0004B22E
Hey Faendal=player.StartConversation 0001348C
Hey Benor=player.StartConversation 0001AA68
Hey Derkeethus=player.StartConversation 0001C197
Hey Ghorbash=player.StartConversation 0001CA7D
Hey Uthgerd=player.StartConversation 00091918
;;;Dawnguard DLC Followers
Hey Serana=player.StartConversation 02002B74
Hey Isran=player.StartConversation 02003B8E
Hey Celann=player.StartConversation 02015C15
Hey Durak=player.StartConversation 02015C16
Hey Ingjard=player.StartConversation 02015C17
Hey Agmaer=player.StartConversation 02015C18
Hey Beleval=player.StartConversation 02015C19
;;;Dragonborn DLC Followers
Hey Frea=player.StartConversation 04017A0D
Hey Teldryn=player.StartConversation 04038565
Hey Ralis=player.StartConversation 04018109
Hey Talvas=player.StartConversation 040177DB
Hey Neloth=player.StartConversation 040177DB
Hey Riekling=player.StartConversation 0401FB99
;;;Hearthfire DLC Followers
Hey Rayya=player.StartConversation 03005216
Hey Valdimar=player.StartConversation 0300521D
Hey Gregor=player.StartConversation 0300521F
You can do the same with modded followers but the ID's will depend on your load order so just check in console when you recruit them and add them with the same format. Hope this helps some people. For generic start conversation with any NPC then the talk to close ones mod is better as discussed in other posts. But this method does not require looking at the follower.
As far as finding Windows Speech Recognition on Windows 11, it didn't show up in the control panel, but when I pressed windows key+ctrl+s, it popped up; so if you want to calibrate and configure, or just verify that you have it, I guess that's where you go.
A spell or MCM option would be a godsend here.