Give Feedback
0 of 0
About this mod
Generates a spell research patch for any spell mod with a few clicks of a few buttons.
- Requirements
- Permissions and credits
- Changelogs
If you are like me you probably really like the mod Spell Research, however you would also be aware that the mod has one glaringly obvious problem, it doesn't work with new spell mods. Sure eventually someone creates a patch, but this can take a painfully long time, the reason is its pretty tedious to create a Spell Research patch, even with the Importer Tool. This patcher aims to rectify that.
This tool can generate a spell research patch for any spell mod. The necessary scripts and esp file are generated automatically. Archetypes are selected by parsing the esp as well as comparing semantic similarity of the spell descriptions to each archetype using an NLP (Natural Language Processing) model.
The patcher comes in a 2 step process, the first step is to generate the archetypes by parsing the esp and generating an output file (called out.json). This is done using the sr_autopatch.exe tool. After generation you can either leave the file as it is or edit it to your liking. The second step is to use the output file to generate the corresponding script and esp file for the spell research patch, this is done using zEdit.
Experimental Patches
As a proof of concept, I generated (untested) patches for the following mods, find them in the optional section.
- Colorful Magic by 184Gesu SE
- All Makers' Lexicon - Solstheim Magic
- Reach Magic
Detailed Usage:
Step 1: Archetype Generation
Step 2: Patch Generation
Nitty Gritty
This section covers the selection process for the spell archetypes, most of these are unambiguous in the esp, some of them are however up to interpretation. You don't need to read this if you are not interested.
Configuration
WARNING: I did some limited testing to make sure this thing works. For now, consider this to be in experimental stages, I would look over the generated esp and psc files carefully before committing them to your playthrough.
On the other hand, its pretty unlikely to break anything, and you can always re-import spells.
I highly recommend you use this in conjunction with my other mod:
Spell Research - Altered Tomes zEdit Patcher
The idea is that we make patches for all the amazing spell mods that currently do not have a patch. If you use this to make a patch you like, please make a mod of it, or pm me and ill add it as an optional file, or give it to the original mod author.
Post any feedback in the comments.
This tool can generate a spell research patch for any spell mod. The necessary scripts and esp file are generated automatically. Archetypes are selected by parsing the esp as well as comparing semantic similarity of the spell descriptions to each archetype using an NLP (Natural Language Processing) model.
The patcher comes in a 2 step process, the first step is to generate the archetypes by parsing the esp and generating an output file (called out.json). This is done using the sr_autopatch.exe tool. After generation you can either leave the file as it is or edit it to your liking. The second step is to use the output file to generate the corresponding script and esp file for the spell research patch, this is done using zEdit.
Experimental Patches
As a proof of concept, I generated (untested) patches for the following mods, find them in the optional section.
- Colorful Magic by 184Gesu SE
- All Makers' Lexicon - Solstheim Magic
- Reach Magic
Detailed Usage:
Step 1: Archetype Generation
Spoiler:
Requirements - Bethesda Toolkit
1. Obtain Bethesda Toolkit and use it to generate an esx file for the mod you are patching.
eg. bethkit.exe convert "Triumvirate - Mage Archetypes.esp" "Triumvirate - Mage Archetypes.esx"
Bethesda Toolkit is a command line tool, this mod covers the steps on how to use BethKit somewhat, just ignore the xEdit stuff, when you have the esx file you'r done. https://www.nexusmods.com/skyrimspecialedition/mods/34036/
2. run the SR autopatcher. Also a command line tool.
The patcher takes in 4 arguments,
-f or --file - this is the path to the esx
-m or --model (optional) - this is the path to the model, this can be omitted, but then techniques/elements will not be populated, and you will need to fill them in manually
-M or --masters (optional) - this allows you to include master files when parsing SPEL and MGEF records (not books). The masters must also be in esx format and must be in the same location as the target esx. Usually this option isn't needed, since typically mod authors will add their own spells and effects as part of their mod. It is only required for mods that have certain spells that rely only on vanilla effects, colorful magic has a few of these, which if you try to run without this argument will give an error, typically you would not use this argument.
If the mod has an esp and an esm of the same name, e.g._1ex0_ True Fantasy Skyrim.esp, _1ex0_ True Fantasy Skyrim.esm, add an _m to the end of the esx of the master. eg _1ex0_ True Fantasy Skyrim_m.esx
-V or --verbose - verbose mode, prints out a bunch of extra info when working, also saves the name and description tokens into out.json
examples:
sr_autopatch.exe -f "Triumvirate - Mage Archetypes.esx" -m "GoogleNews-vectors-negative300.bin"
sr_autopatch.exe -f "Colorful_Magic_SE.esx" -m "GoogleNews-vectors-negative300.bin" -M
sr_autopatch.exe -f "Colorful_Magic_SE.esx" -m "GoogleNews-vectors-negative300.bin" -M -V
The google news model can be downloaded from https://code.google.com/archive/p/word2vec/, it is probably the most complete model with 100 billion words,
however it is quite large (1.6 Gb download), there are other models here https://github.com/RaRe-Technologies/gensim-data. Some might be a lot smaller
but they might not have the words in the archetypes/descriptions.
After running the patcher, read the output, it will tell you if it succeeded, if it mention any records that failed, you will need to manually enter them in the output (out.json) file.
3. After running srpatch, an out.json file will be generated in the same folder, you should look over it carefully, especially concerning targets and techniques/elements. You can make any changes you want to the output file, as long as your keywords are consistent.
Show
Requirements - Bethesda Toolkit
1. Obtain Bethesda Toolkit and use it to generate an esx file for the mod you are patching.
eg. bethkit.exe convert "Triumvirate - Mage Archetypes.esp" "Triumvirate - Mage Archetypes.esx"
Bethesda Toolkit is a command line tool, this mod covers the steps on how to use BethKit somewhat, just ignore the xEdit stuff, when you have the esx file you'r done. https://www.nexusmods.com/skyrimspecialedition/mods/34036/
2. run the SR autopatcher. Also a command line tool.
The patcher takes in 4 arguments,
-f or --file - this is the path to the esx
-m or --model (optional) - this is the path to the model, this can be omitted, but then techniques/elements will not be populated, and you will need to fill them in manually
-M or --masters (optional) - this allows you to include master files when parsing SPEL and MGEF records (not books). The masters must also be in esx format and must be in the same location as the target esx. Usually this option isn't needed, since typically mod authors will add their own spells and effects as part of their mod. It is only required for mods that have certain spells that rely only on vanilla effects, colorful magic has a few of these, which if you try to run without this argument will give an error, typically you would not use this argument.
If the mod has an esp and an esm of the same name, e.g._1ex0_ True Fantasy Skyrim.esp, _1ex0_ True Fantasy Skyrim.esm, add an _m to the end of the esx of the master. eg _1ex0_ True Fantasy Skyrim_m.esx
-V or --verbose - verbose mode, prints out a bunch of extra info when working, also saves the name and description tokens into out.json
examples:
sr_autopatch.exe -f "Triumvirate - Mage Archetypes.esx" -m "GoogleNews-vectors-negative300.bin"
sr_autopatch.exe -f "Colorful_Magic_SE.esx" -m "GoogleNews-vectors-negative300.bin" -M
sr_autopatch.exe -f "Colorful_Magic_SE.esx" -m "GoogleNews-vectors-negative300.bin" -M -V
The google news model can be downloaded from https://code.google.com/archive/p/word2vec/, it is probably the most complete model with 100 billion words,
however it is quite large (1.6 Gb download), there are other models here https://github.com/RaRe-Technologies/gensim-data. Some might be a lot smaller
but they might not have the words in the archetypes/descriptions.
After running the patcher, read the output, it will tell you if it succeeded, if it mention any records that failed, you will need to manually enter them in the output (out.json) file.
3. After running srpatch, an out.json file will be generated in the same folder, you should look over it carefully, especially concerning targets and techniques/elements. You can make any changes you want to the output file, as long as your keywords are consistent.
Step 2: Patch Generation
Spoiler:
1. Download the zEdit module and copy it into zEdit/modules
2. Copy the out.json into the zEdit patcher folder (zEdit/modules/spellResearchPatchGenerator).
3. Run zEdit with the esp you are patching enabled, open the patchers modal (right click -> manage patchers), select Spell Research Patch Generator, in the box enter the desired script name, e.g. _SR_ImportTriumvirate.
4. Select Build Patches, Add Plugin, rename the newly created patch to whatever you want (e.g. SR_Triumvirate_Patch.esp), drag Spell Research Patch Generator under the new patch and tick its box
5. Select 'Build' next to the patch.
6. A psc file will be generated in the same folder, look over it carefully, search for 'undefined' (if its there its an error)
7. Compile the psc with PapyrusCompiler or the Creation Kit and put it into scripts/<name>.pex
8. The generated esp should be safe to eslify, you can check by running xEdit with the -PseudoESL argument
9. Done, in game reimport the spells from the spell research MCM menu
Show
1. Download the zEdit module and copy it into zEdit/modules
2. Copy the out.json into the zEdit patcher folder (zEdit/modules/spellResearchPatchGenerator).
3. Run zEdit with the esp you are patching enabled, open the patchers modal (right click -> manage patchers), select Spell Research Patch Generator, in the box enter the desired script name, e.g. _SR_ImportTriumvirate.
4. Select Build Patches, Add Plugin, rename the newly created patch to whatever you want (e.g. SR_Triumvirate_Patch.esp), drag Spell Research Patch Generator under the new patch and tick its box
5. Select 'Build' next to the patch.
6. A psc file will be generated in the same folder, look over it carefully, search for 'undefined' (if its there its an error)
7. Compile the psc with PapyrusCompiler or the Creation Kit and put it into scripts/<name>.pex
8. The generated esp should be safe to eslify, you can check by running xEdit with the -PseudoESL argument
9. Done, in game reimport the spells from the spell research MCM menu
Nitty Gritty
This section covers the selection process for the spell archetypes, most of these are unambiguous in the esp, some of them are however up to interpretation. You don't need to read this if you are not interested.
Spoiler:
The following archetypes are selected from the esp they are pretty solid, not really left up to interpretation.
skill - MGEF\Magic Effect Data\DATA\Magic Skill (i.e. destruction, alteration...etc)
cast type - SPEL\SPIT\Cast type (i.e. Fire and Forget, Concentration... etc)
level - MGEF\Magic Effect Data\DATA\Minimum Skill level (0 Novice, 25 Apprentice, 50 Adept...etc)
In case a spell has more than 1 magic effect (MGEF), the effect with the description is chosen. The target's dont line up completely with the archetypes in spellresearch, the esp has a 'touch' option for targets that spellresearch doesn't have, the esp also has no 'AOE' target.
targets - MGEF\Magic Effect Data\Delivery
- SPEL\SPIT\Flags (for AOE)
Techniques and Elements:
Techniques and elements are chosen by tokenising the name and description of the spell, where tokens are nouns and verbs. Then the semantic similarity of each archetype with a listen of tokens is compared using an pre-trained NLP word2vec model.
e.g.
"Creatures and people up to level <mag> fight for you for <dur> seconds, and obey orders when not in combat."
may have tokens:
[creatures, people, level, fight, seconds, obey, orders, combat]
example of semantic similarity between 2 words:
model.similarity('curing', 'healing') -> 0.40927798 (according to Google News)
for comparison here are the most similar words to 'curing' with that model.
model.most_similar('curing')
[('cure', 0.6621213555335999), ('cured', 0.6294988393783569), ('Curing', 0.6107527017593384),
('cures', 0.588561475276947), ('uncurable', 0.5072536468505859), ('diagnosing', 0.5018425583839417),
('eradicating', 0.49097612500190735), ('curing_diseases', 0.4870814085006714), ('curer', 0.48673000931739807),
('alleviating', 0.48111990094184875)]
For each list of name/description tokens, the archetype(s) with the highest similarity is selected. The amount of archetypes selected is dependant on the level of the spell and the configuration file.
e.g.
Novice - 1 technique/element archetype
Master - 5 technique/element archetypes
The name and description of the spell is taken from the esp:
- MGEF\DNAM (description)
- BOOK\FULL (name)
Show
The following archetypes are selected from the esp they are pretty solid, not really left up to interpretation.
skill - MGEF\Magic Effect Data\DATA\Magic Skill (i.e. destruction, alteration...etc)
cast type - SPEL\SPIT\Cast type (i.e. Fire and Forget, Concentration... etc)
level - MGEF\Magic Effect Data\DATA\Minimum Skill level (0 Novice, 25 Apprentice, 50 Adept...etc)
In case a spell has more than 1 magic effect (MGEF), the effect with the description is chosen. The target's dont line up completely with the archetypes in spellresearch, the esp has a 'touch' option for targets that spellresearch doesn't have, the esp also has no 'AOE' target.
targets - MGEF\Magic Effect Data\Delivery
- SPEL\SPIT\Flags (for AOE)
Techniques and Elements:
Techniques and elements are chosen by tokenising the name and description of the spell, where tokens are nouns and verbs. Then the semantic similarity of each archetype with a listen of tokens is compared using an pre-trained NLP word2vec model.
e.g.
"Creatures and people up to level <mag> fight for you for <dur> seconds, and obey orders when not in combat."
may have tokens:
[creatures, people, level, fight, seconds, obey, orders, combat]
example of semantic similarity between 2 words:
model.similarity('curing', 'healing') -> 0.40927798 (according to Google News)
for comparison here are the most similar words to 'curing' with that model.
model.most_similar('curing')
[('cure', 0.6621213555335999), ('cured', 0.6294988393783569), ('Curing', 0.6107527017593384),
('cures', 0.588561475276947), ('uncurable', 0.5072536468505859), ('diagnosing', 0.5018425583839417),
('eradicating', 0.49097612500190735), ('curing_diseases', 0.4870814085006714), ('curer', 0.48673000931739807),
('alleviating', 0.48111990094184875)]
For each list of name/description tokens, the archetype(s) with the highest similarity is selected. The amount of archetypes selected is dependant on the level of the spell and the configuration file.
e.g.
Novice - 1 technique/element archetype
Master - 5 technique/element archetypes
The name and description of the spell is taken from the esp:
- MGEF\DNAM (description)
- BOOK\FULL (name)
Configuration
Spoiler:
There are a few things that can be configured, everything that can be configured is in the config.json file.
maxtechniqueelement: the number of techniques + elements for a level of spell, by defauly, Novice spells have 1 technique or element, master has 5.
stopwords: words in the description of spells that should be ignored, add things here if the mod has too much of a vertain word you don't like
exclusions: archetypes to be excluded, these archetypes will not be used at all.
synonyms: alternative names for archetypes, e.g. if 'Ice' is a synonym of 'Frost', it can be used in lieu of the Frost archetype. Each archetype can have multiple synonyms, the one with the highest similarity with the tokens will be used. This section has significant impact on accuracy.
Show
There are a few things that can be configured, everything that can be configured is in the config.json file.
maxtechniqueelement: the number of techniques + elements for a level of spell, by defauly, Novice spells have 1 technique or element, master has 5.
stopwords: words in the description of spells that should be ignored, add things here if the mod has too much of a vertain word you don't like
exclusions: archetypes to be excluded, these archetypes will not be used at all.
synonyms: alternative names for archetypes, e.g. if 'Ice' is a synonym of 'Frost', it can be used in lieu of the Frost archetype. Each archetype can have multiple synonyms, the one with the highest similarity with the tokens will be used. This section has significant impact on accuracy.
WARNING: I did some limited testing to make sure this thing works. For now, consider this to be in experimental stages, I would look over the generated esp and psc files carefully before committing them to your playthrough.
On the other hand, its pretty unlikely to break anything, and you can always re-import spells.
I highly recommend you use this in conjunction with my other mod:
Spell Research - Altered Tomes zEdit Patcher
The idea is that we make patches for all the amazing spell mods that currently do not have a patch. If you use this to make a patch you like, please make a mod of it, or pm me and ill add it as an optional file, or give it to the original mod author.
Post any feedback in the comments.