Q:Why aren't there installation instructions? Why do we have to change settings to make it work? Why doesn't it just work immediately? A:
Spoiler:
Show
This script was written as an informative learning tool for authors first and a conversion tool for users second. When it comes right down to it, the ideal would be authors taking the information provided and making the changes on their end. However, we did still want to allow the average user to take advantage of the benefits it can offer, but they should at the very least be familiar with xEdit already, know how to perform at least minor troubleshooting by themselves, and be prepared to read documentation. The script has clear, concise comments for pretty much everything.
Q:Why does this exist? Why is it important? A:
Spoiler:
Show
That's a complicated question. The simplest explanation is: there is something called a Reference Handle Cap. It's very high, above a million entries, every active reference is counted towards this, and if you reach it, you risk CTDs. There are two kinds of references: temporary and persistent. Temporary should only be active when you're nearby, persistent are active all the time. A non-ESM plugin (without the .esm file extension, and/or not using the ESM plugin flag), which is most mods, is treated by the engine as though all references are active all the time, even the temporary ones. With large mods this can add up extremely quickly. Ever run several large "new world" or "new dungeon" mods in the same install and kept crashing randomly but couldn't figure out why? The Reference Handle Cap may have been the reason. If as a community we'd been building our mods as Bethesda originally intended, it's unlikely we'd ever have this problem. This script, and its predecessors, act as a stopgap to at least bring us a bit closer to their original intentions.
Q:Mods have been made like this for like 15 years, why should we trust your claim that it's wrong? A:
Spoiler:
Show
It's less our claim and more Bethesda's development methods, how the CK normally operates, and evidence uncovered over the years. Something that was never really explained to the general community is that Bethesda uses a "version control" method of development (think similar to git and Github), and all of their content is released as a "master". The way "masters" and "plugins" are treated by the game, it may be more accurate to refer to them as "main" and "patch". The latter has a number of behavior changes that seem to be intended to force new content to work as long as it's been built even remotely functional, but those will fail as soon as the plugin is changed to a master. All of the evidence we've gathered over the years tells us that your bog-standard .esp file was never truly intended to be the delivery method for new content, and should probably have only been used for overriding and changing already existing content at most.
Q:What is this "Persistent Location" thing that's in the script because I totally actually read through the code to see what it's doing? A:
Spoiler:
Show
From what we've been able to determine, this record element is the middle ground between no persistence and global persistence from the record flag. When a reference has a "Persistent Location" filled, rather than only being active when you're within range of the cell it was placed in, instead the reference is active whenever you're in any cell associated with that Location. That means that, for instance, instead of being within eyesight of someone within Whiterun, you can be anywhere within Whiterun, and that reference will still be processed by the game. Much more performance friendly than using the "global" persistence flag on the record.
Q:Ok, so why doesn't your script add "Persistent Location"s to each reference instead of using the globally persistent flag? A:
Spoiler:
Show
Because it can't read minds. We explored several methods of targeting a potential location through a reference's associated records, but even in Bethesda's own core masters that can be inconsistent. We settled on just informing the user and, if enabled, resorting to the global flag. The ideal solution would be mod authors taking the information provided and making the changes to their mods themselves, using Locations and "Persist Location" where appropriate, instead of just flipping the global flag and forgetting about it.
Q:Can this also remove the Persistent flag? A:
Spoiler:
Show
Not at this time, but that's a possible addition in the future. The same filters used to check if a record should be flagged could probably relatively easily be repurposed from the current "skip record" behavior to indicate removing an unnecessary flag.
You might be wondering: "Hey, they were talking about the script not being done yet, but that was over a month ago, where are the updates?!" The answer is: I jump around modding projects quite a bit. Recently I've been working more on a classic DOOM project. No, this is far from abandoned. I actually have a Skyrim install that I want to play myself, probably over the Christmas period due to visiting relatives for a month, and I refuse to play without my install being properly configured. That includes what this script is intended to cover.
Most of the processing functionality should actually be done on my local version of the script, but I intend on retooling it to catch some additional details that I'm like 95% sure were being missed before. Keep in mind that this is work to catch as many necessary cases as possible. While I do want to avoid over-flagging, the fact still stands that flagging extra references is still better than having them break during plugin conversion, so I'm more concerned about finding everything at the moment.
More details to come, as well as a flowchart showing how the script processes.
I would say use this one. The reasoning behind the additional filters is solid, and it should provide more accurate results. Additionally it also flags some other references as persistent which were not covered by my script.
I also re-esmified my existing plugins using it and everything is working as expected.
@cavy8 This should be safe to use in the same circumstances as you'd use the ESMifyer, it's just more selective about what it flags. That way you don't have things like random rabbit AI being processed by Riften while you're in Solitude. Just don't forget to turn the dry run "devMode" off!
-----
@Blackread I'm incredibly honored, and relieved, to see you say that!
I didn't want to be stepping on any toes when I first started this project, but what got me thinking was "if ESM flagging a plugin is supposed to reduce the number of persistent reference handles, this approach could probably be improved to lower them even further".
To everyone else: Even if this script does end up taking the place of the ESMifyer, the groundwork that KoKauH97 and Blackread did to set the stage shouldn't be forgotten. This script wouldn't exist at all if it weren't for their efforts!
got this error while trying to apply the script to Clockwork mod [00:00] Exception in unit line -1: Error in unit 'Persistentify_Those_Plugins' on line 168 : Undeclared Identifier 'GetLoadOrderFileID' [00:00] Error during Applying script "00_Persistentify_Those_Plugins": Error in unit 'Persistentify_Those_Plugins' on line 168 : Undeclared Identifier 'GetLoadOrderFileID'
Is there any way to see (in game) how close one is to the reference cap in different places, meaning how many references are currently being handled? I assume this is something that cannot be checked "statically" on the load order itself, because it also depends on which temp references are actually loaded (i.e. which cells are loaded in that moment) - I hope my understanding is correct (?)
Engine fixes will throw a warning at the main menu once you surpass 800k. And after loading a save if you surpass 1 million. If you do not see these warning popups then you are not above their respective thresholds.
You can actually check statically on the load order by running the "Count Loaded Refs In Load Order" script that is included with xEdit. It'll count all Persistent flagged references, and all Non-persistent flagged references in non-ESM flagged plugins. Take whatever number this spits out and add a couple thousand to count for loaded cells during play.
How common is it that the script doesn't find anything to change? I ran it on EasierRider's Dungeon Pack, Caves of Morrowind, Ljungvunde, and Solstheim Dungeonpack, and it found nothing to change in any of them. I guess it's fine to just flag those plugins as esm as they are?
That's not too uncommon. If it found nothing you can probably add the flag and then load it up in the CK. And see if it mentions anything that needs to be persistent but isn't. As a safety check. We're pretty thorough ATM but it never hurts to double check.
Hi, i am geting these errors? * Plugin: [16] Midwood Isle.esp [01:01] Exception in unit line -1: Error in unit 'Persistentify_Those_Plugins' on line 168 : Undeclared Identifier 'GetLoadOrderFileID' [01:01] Error during Applying script "00_Persistentify_Those_Plugins": Error in unit 'Persistentify_Those_Plugins' on line 168 : Undeclared Identifier 'GetLoadOrderFileID'
GetLoadOrderFileID is new in xEdit 4.1.5c and replaced GetLoadOrder. It uses the same syntax, but is properly coded/setup to work with/around esl flagged files.
Ok newer Update, i am using now 4.1.5 not 4.1.5c But i cant seem to find the Download for the 4.1.5c Version? The only thing i can find is the 4.1.5c for Fallout4 on Github?
sorry but i'm super dumb so i'll ask a simple question i know about the 1 million reference handle tag cap where if u approach it or pass it ur game will crash forever which limits us to play with heavy mods for a good chunk of time true what i don't understand is what your mod does. can this change the cap? or do something to mitigate us from approaching it?
ESP (standard) plugins: The game considers all references to be persistent, making them run at all times.
ESM (master) plugins: The game treats temporary-flagged references as actually temporary, causing them to only function when you're nearby (or certain other cases)
However, many authors aren't aware of how references should be used because of the way ESPs are treated by the engine, it having been this way for a very long time. If you flag a plugin as a master without properly managing how references are handled, things will often break, from strangeness in-game all the way to crashes.
This script runs logic based on common uses of references to determine what references should be flagged as persistent in order to function correctly, flags those references accordingly, then can also flag the plugin as ESM. This means that temporary references will only function when you're nearby, as they should, which reduces the number of active references, making it less likely you'll reach that handle cap.
You don't have to code anything, all you have to do is run the script on plugins in your load order which have a higher count of references. To identify them, run this other script first. Usual "culprits" are new worldspace mods, new dungeons mods, and new followers mods such as 3DNPC for instance.
- To enable plugin edits with this Beta version, find **dryRun** under **USER SETTINGS** and change it to **false** . - To also enable ESM flagging of the plugin, find **ESMify** under **USER SETTINGS** and change it to **true** .
When you select a script to run in SSEEdit, the code shows up in the window. Scroll down in that window to find the user settings. You can edit what you want in there, and when you run the script, it'll ask you if you want to save your changes.
Thanks for the update. I use this mod in FO4 as well as SSE and never had an issue. Can I assume it will work the same way for Starfield, or is it too early yet to say?
I would defacto assume no xEdit scripts will work across games. Differences in the record definitions make that unreliable. If something happens to work consider it a fluke.
Starfield idk they changed so much the circumstances for something to need to be persistent could be wildly different.
This might or might not sound dumb but, although i never done it, is it safe to use this on automatically generated esp like bashed, smash, dyndolod and synthesis patches?.
Do not run this script on skyrim.esm there's no point as you can't save it.
Running on the other vanilla masters is also not really worth it although adding the persistent flag to references in those masters certainly won't break anything.
Steps to reproduce: 1) Have just USSEP, Curse of Hound Amulet and Alternate Start in your modlist 2) Choose the Arrived by Boat -> Solitude start 3) Go to Solitude from the docks and you should see Caraleth straight ahead - she walks towards you to initiate the mod's quest when in range 4) Run ESMify script with forceEsmify = true on the mod's ESP and reproduce steps 1 to 3, Caraleth will neither initiate dialogue nor let you talk to her on Activating her
Any insight on why the different behavior after ESMifying would help identify potential other mods that might have the same issue on ESMification
That sound's like a broken mod. Load it up in the CK and see if it has any "this XXX should be persistent but it is not" errors specifically, or any other notable errors.
Any mod that is properly built can be ESM flagged without issue. Mods where flagging causes issues by definition fundamentally means something is wrong with the mod itself. whether it be something as simple as a reference not being persistent that should be (that we aren't aware of to flag it with our script) or Location Records aren't setup correctly, or a host of other issues.
Thank you for the response, I loaded up the ESMed mod in the CK and just got these in the console. QUESTS: Could not find previous info (02004DE7) for TopicInfo (02004DE8) in Topic "" (02004DAA). It is time MASTERFILE: Unique ActorBase 'CursedHoundBasic' (0402D336) has more than one reference in the world. Data access when unloaded may not work correctly. MASTERFILE: Unique ActorBase 'CursedHoundRunt' (0406B293) has more than one reference in the world. Data access when unloaded may not work correctly. MASTERFILE: Ref 'GuardSolitudeSonsTrainingBow' (0007E144) uses location but is not in the unloaded ref data.
so either 2 things to solve that "unique ActorBase" errors
1. remove the Unique flag from those NPC base records or 2. The "second" Reference placing that npc is actually supposed to be an override of the existing one and for some reason it isn't. If that is the case, edit the mod that places that new reference and change it to an override
Typically number 2 is evident if the original reference is disabled. If you can't tell that its 2, do #1
that last one means you need to look at that reference in xEdit, go to the CELL record it is contained in, look at its XLCN - Location tag. go to the LCTN record that is listed, and copy that as override into the mod. then re-save the plugin. At least thats generally the solution. That specific solitude guard one is a vanilla issue and is resolved by ussep as it contains the SolitudeLocation already. so that one can be ignored with respect to the mod your editing.
Thanks for that, I changed both Hound actors to non-unique because I've seen them as generic enemies in one of the videos I've seen of the mod (also saw multiple enabled references scattered throughout the dungeon in xEdit) but the issue with not being able to actually initiate the quest still remains.
So an update on this, the CK now gives a different error on loading after removing the Unique flag from those actors: QUESTS: Could not find previous info (02004DE7) for TopicInfo (02004DE8) in Topic "" (02004DAA). AI: RELATIONSHIP: NPC 'CursedHoundBasic' (0402D336) is not unique. Relationship '' (0406DB35) may inadvertently apply to multiple actors. AI: RELATIONSHIP: NPC 'CursedHoundRunt' (0406B293) is not unique. Relationship '' (0406DB36) may inadvertently apply to multiple actors. MASTERFILE: Ref 'GuardSolitudeSonsTrainingBow' (0007E144) uses location but is not in the unloaded ref data.
Those 2 npcs that you removed the Unique flag from, now need to have their relationships deleted as relationships can only be applied to unique npcs. It lists the form id of the Relationship later in each message
So I did those changes and still no effect on the initialization of the quest dialogue. Reverting the ESM flag from the header allows the dialogue to happen.
Sorry for the trouble, but it seems I will need to revert the ESMification changes for my modlist and forgo the use of this script. Eagerly tracking any future developments it has and I will give this a try again after maybe a future update.
Thanks again for taking the time and guiding me through the CK errors.
Then the only conclusion I have left to draw is something is seriously broken about the quest/scripting used by that mod. For that reason I must recommend not using it.
Just an update on this. It seems using the optional script from here to ESMify makes changes to some additional CELL subrecords that fixes this issue (if you used the Persistentify script)
That's actually specifically what we want to avoid with Persistentify. That script just blanket flags all unique npcs as persistent, which is wasteful. Our script here only wants to flag things that actually need to be flagged and skip any others that don't.
Give Feedback
200 comments
2022-11-19
You might be wondering: "Hey, they were talking about the script not being done yet, but that was over a month ago, where are the updates?!"
The answer is: I jump around modding projects quite a bit. Recently I've been working more on a classic DOOM project. No, this is far from abandoned. I actually have a Skyrim install that I want to play myself, probably over the Christmas period due to visiting relatives for a month, and I refuse to play without my install being properly configured. That includes what this script is intended to cover.
Most of the processing functionality should actually be done on my local version of the script, but I intend on retooling it to catch some additional details that I'm like 95% sure were being missed before. Keep in mind that this is work to catch as many necessary cases as possible. While I do want to avoid over-flagging, the fact still stands that flagging extra references is still better than having them break during plugin conversion, so I'm more concerned about finding everything at the moment.
More details to come, as well as a flowchart showing how the script processes.
I also re-esmified my existing plugins using it and everything is working as expected.
This should be safe to use in the same circumstances as you'd use the ESMifyer, it's just more selective about what it flags. That way you don't have things like random rabbit AI being processed by Riften while you're in Solitude. Just don't forget to turn the dry run "devMode" off!
-----
@Blackread
I'm incredibly honored, and relieved, to see you say that!
I didn't want to be stepping on any toes when I first started this project, but what got me thinking was "if ESM flagging a plugin is supposed to reduce the number of persistent reference handles, this approach could probably be improved to lower them even further".
To everyone else: Even if this script does end up taking the place of the ESMifyer, the groundwork that KoKauH97 and Blackread did to set the stage shouldn't be forgotten. This script wouldn't exist at all if it weren't for their efforts!
[00:00] Exception in unit line -1: Error in unit 'Persistentify_Those_Plugins' on line 168 : Undeclared Identifier 'GetLoadOrderFileID'
[00:00] Error during Applying script "00_Persistentify_Those_Plugins": Error in unit 'Persistentify_Those_Plugins' on line 168 : Undeclared Identifier 'GetLoadOrderFileID'
You can actually check statically on the load order by running the "Count Loaded Refs In Load Order" script that is included with xEdit. It'll count all Persistent flagged references, and all Non-persistent flagged references in non-ESM flagged plugins. Take whatever number this spits out and add a couple thousand to count for loaded cells during play.
* Plugin: [16] Midwood Isle.esp
[01:01] Exception in unit line -1: Error in unit 'Persistentify_Those_Plugins' on line 168 : Undeclared Identifier 'GetLoadOrderFileID'
[01:01] Error during Applying script "00_Persistentify_Those_Plugins": Error in unit 'Persistentify_Those_Plugins' on line 168 : Undeclared Identifier 'GetLoadOrderFileID'
Update: ok it still doesn't work, did i understand something wrong?
But i cant seem to find the Download for the 4.1.5c Version?
The only thing i can find is the 4.1.5c for Fallout4 on Github?
Also the 4.1.5c on GitHub is for all games. Every xEdit version posted anywhere is by definition every single game. It's all one exe.
i know about the 1 million reference handle tag cap where if u approach it or pass it ur game will crash forever which limits us to play with heavy mods for a good chunk of time true
what i don't understand is what your mod does.
can this change the cap? or do something to mitigate us from approaching it?
- ESP (standard) plugins: The game considers all references to be persistent, making them run at all times.
- ESM (master) plugins: The game treats temporary-flagged references as actually temporary, causing them to only function when you're nearby (or certain other cases)
However, many authors aren't aware of how references should be used because of the way ESPs are treated by the engine, it having been this way for a very long time. If you flag a plugin as a master without properly managing how references are handled, things will often break, from strangeness in-game all the way to crashes.This script runs logic based on common uses of references to determine what references should be flagged as persistent in order to function correctly, flags those references accordingly, then can also flag the plugin as ESM. This means that temporary references will only function when you're nearby, as they should, which reduces the number of active references, making it less likely you'll reach that handle cap.
- To also enable ESM flagging of the plugin, find **ESMify** under **USER SETTINGS** and change it to **true** .
Where exactly is this USER SETTINGS?
Starfield idk they changed so much the circumstances for something to need to be persistent could be wildly different.
Skyrim.esm (19 references) (NPC)
Dragonborn.esm (3 reference) (NPC)
Colorful Magick.esp (11 reference) (NPC)
Animated Armoury.esp (3 reference) (WPN)
Running on the other vanilla masters is also not really worth it although adding the persistent flag to references in those masters certainly won't break anything.
Mod : Curse of the Hound Amulet
Steps to reproduce:
1) Have just USSEP, Curse of Hound Amulet and Alternate Start in your modlist
2) Choose the Arrived by Boat -> Solitude start
3) Go to Solitude from the docks and you should see Caraleth straight ahead - she walks towards you to initiate the mod's quest when in range
4) Run ESMify script with forceEsmify = true on the mod's ESP and reproduce steps 1 to 3, Caraleth will neither initiate dialogue nor let you talk to her on Activating her
Any insight on why the different behavior after ESMifying would help identify potential other mods that might have the same issue on ESMification
Any mod that is properly built can be ESM flagged without issue. Mods where flagging causes issues by definition fundamentally means something is wrong with the mod itself. whether it be something as simple as a reference not being persistent that should be (that we aren't aware of to flag it with our script) or Location Records aren't setup correctly, or a host of other issues.
QUESTS: Could not find previous info (02004DE7) for TopicInfo (02004DE8) in Topic "" (02004DAA).It is time
MASTERFILE: Unique ActorBase 'CursedHoundBasic' (0402D336) has more than one reference in the world. Data access when unloaded may not work correctly.
MASTERFILE: Unique ActorBase 'CursedHoundRunt' (0406B293) has more than one reference in the world. Data access when unloaded may not work correctly.
MASTERFILE: Ref 'GuardSolitudeSonsTrainingBow' (0007E144) uses location but is not in the unloaded ref data.
1. remove the Unique flag from those NPC base records
or
2. The "second" Reference placing that npc is actually supposed to be an override of the existing one and for some reason it isn't. If that is the case, edit the mod that places that new reference and change it to an override
Typically number 2 is evident if the original reference is disabled. If you can't tell that its 2, do #1
that last one means you need to look at that reference in xEdit, go to the CELL record it is contained in, look at its XLCN - Location tag. go to the LCTN record that is listed, and copy that as override into the mod. then re-save the plugin. At least thats generally the solution. That specific solitude guard one is a vanilla issue and is resolved by ussep as it contains the SolitudeLocation already. so that one can be ignored with respect to the mod your editing.
QUESTS: Could not find previous info (02004DE7) for TopicInfo (02004DE8) in Topic "" (02004DAA).
AI: RELATIONSHIP: NPC 'CursedHoundBasic' (0402D336) is not unique. Relationship '' (0406DB35) may inadvertently apply to multiple actors.
AI: RELATIONSHIP: NPC 'CursedHoundRunt' (0406B293) is not unique. Relationship '' (0406DB36) may inadvertently apply to multiple actors.
MASTERFILE: Ref 'GuardSolitudeSonsTrainingBow' (0007E144) uses location but is not in the unloaded ref data.
specifically: 0406DB35 and 0406DB36
Sorry for the trouble, but it seems I will need to revert the ESMification changes for my modlist and forgo the use of this script. Eagerly tracking any future developments it has and I will give this a try again after maybe a future update.
Thanks again for taking the time and guiding me through the CK errors.