0 of 0
About this mod
This utility is a bash shell script to browse archived files in BA2 and loose files as well, keeping track of their load order to determine conflicts, conflict winners and their source repositories.
- Requirements
- Permissions and credits
- Changelogs
Introduction, motivation and caveats
If I understand, loading mechanism of conflicting files is based on the following rule: the last is the winner. If two mods provide the same file the conflict is solved by a mod manager like Vortex by means of a load order rule.
Just to make it clear, a definition is needed: I call 'conflicting files' two or more files with the same name and path relative to 'Data' directory.
The task of resolving the conflict is easily accomplished if both files are not archived in a .BA2/BSA container: the last 'loose file' takes the precedence and will be loaded. Vortex allows the user to modify the loading order to finely tune which file will be the winner.
Even in the case of a set of more archived files and a single 'loose file' we can quickly solve the conflict: 'loose file' takes the precedence and will be loaded instead of any file of the same name stored in an archived container, whatsoever it would be fetched from. 'Loose file' will always be the winner.
Problems arise when two or more files are hidden inside an archive: in this case the 'loadorder.txt' file keeps track of the order in which plugins and their connected archives are loaded, but it is not immediately apparent which will be the conflict winner. I mean: the rule still applies but 'behind the curtain' in some sense.
This script tries to make clear what will be the result of the loading rule, listing all conflicting files, stating which will be the winner and, in case of a loose file, from which mods it comes from.
The script relies on the Nexus mod number 17061, BSA Browser (portable version) by AlexxEG but you need not to install his great mod: the needed executable file, a CLI version of his archive browser, is already embedded in the archive, with his permission, of course. I would like to thanks AlexxEG again.
All this work relies on my understanding of loading mechanisms, so if I'm wrong please tell me (and throw away this mod).
Prerequisites
A working modern bash shell with some built-ins like getopts.
Along with the shell a working implementation of 'find', 'sed', 'awk', 'grep', 'unix2dos'/'dos2unix', 'sort', 'diff', 'stat' is needed but I assume that, if you use bash you probably have all the standard *nix utilities (I mean, GNU coreutils, diffutils, fileutils and so on). My setup is built around a pretty standard MinGW-Msys implementation.
This utility relies on the 'hardlink' deployment mechanism that Vortex uses to build up 'Data' directory but I think that any other mod manager using hardlinks would fit. By the way this is the reason why there will be one and only one file with a given name (technically: with the same inode) in the deployed 'Data' directory. Of course the 'source' of the file may vary instead.
You can run the script whether Vortex is running or not (but after a full deploy of all the mods) but you probably can not integrate this script into it: it needs to be launched inside a shell but Vortex does not invoke the proper shell ("cmd" and not "bash", I mean).
Please google for "Link Shell Extension" by Schinagl for an excellent utility to clearly mark hardlinks in Explorer.
Setup and run
Create a directory of your choice and unpack the archive. You will find a shell script, 'Conflict_Detector.sh' along with two directories, 'snpashots' in which results will be collected and stored, and 'BSA_Browser' with a single executable inside, 'bsap.exe'. I will call 'working directory' the directory in which the script is stored.
Due to a feature of 'sed' please check that you have 'Full Control' permissions on the working directory and, in case, set them accordingly. See comments inside the script, line 222 of version 1.1 (by the way: the script is extensively commented but I realise that the word 'extensively' is pretty subjective. If you do not understand something it's my fault, not yours: I made my code obfuscated and unclear, and this is a cardinal sin...).
The script needs a little editing: the line
DATADIR="/e/Home/SteamLibrary/steamapps/common/Fallout 4/Data"
line 5 of the script, needs to be edited (by your text editor of choice) to reflect your setup. Please note quotation marks and the reference to the mounted drive (in my case E:\ which is rendered in "/e" by the Msys shell).
Now you can open a bash shell, change your dir to the working directory and launch the script. Please don't you forget that THIS IS A COMMAND LINE INTERFACE script. A 'double click' on the file from a GUI (I mean: from within an "Explorer" window) is not the intended way to get the script working. So please launch the script from within a bash shell CLI.
With no options the script runs (almost) quietly and saves the results in the 'snapshots' directory. Results are collected in three text files, eg:
'Snapshot_2019_08_11_20_42_51_All_Conflicts.txt'
'Snapshot_2019_08_11_20_42_51_Non_Textures_Conflicts.txt'
'Snapshot_2019_08_11_20_42_51_Textures_Conflicts.txt'
A single run takes its time. On my setup it takes about five minutes to process more than 1000 conflicts so please be patient, I have kept runs as silent as possible. If you prefer you can run a verbose ("-v" flag) version but your screen will be cluttered.
============= Conflict Detection ===========
Result files are identified by a timestamp (YYYY_MM_DD_HH_mm_SS) coded in the name and are organized in a sequence of 'Conflicts' like these:
------------------
General - Conflict number 11
5 DLCCoast - Main.ba2 Materials/DLC03/Landscape/Trees/RedPineQuarter01.BGSM
- 159 FO4 Seasons - Autumn_GoldenGrasses - AllDLC - Main.ba2 materials/DLC03/Landscape/Trees/RedPineQuarter01.BGSM
Conflict winner: FO4 Seasons - Autumn_GoldenGrasses - AllDLC - Main.ba2 materials/DLC03/Landscape/Trees/RedPineQuarter01.BGSM
------------------
How to decript this 'Conflict'?
First line shows the Conflict number, in this case in the 'General list' is the eleventh conflict.
Both conflicting files, namely
Materials/DLC03/Landscape/Trees/RedPineQuarter01.BGSM
are stored in two archives, 'DLCCoast - Main.ba2' (fifth in load order) and 'FO4 Seasons - Autumn_GoldenGrasses - AllDLC - Main.ba2', 159th, so the conflict winner is recorded on the last line of the 'conflict' section.
Another 'Conflict' section involves archived and loose files:
------------------
General - Conflict number 2
112 DOOMMerged - Main.ba2 Materials/Weapons/DOOM/ThatGun/ThatGun.BGSM
- 999 Loose Files Materials/weapons/DOOM/ThatGun/ThatGun.BGSM
Conflict winner: Loose Files Materials/weapons/DOOM/ThatGun/ThatGun.BGSM
Source:
\Home\Vortex\Fallout 4\mods\50 Ways to Die at Dr. Nick's - v.0.99c-25564-0-99c\materials\Weapons\DOOM\ThatGun\ThatGun.BGSM
Target:
\Home\SteamLibrary\steamapps\common\Fallout 4\Data\Materials\weapons\DOOM\ThatGun\ThatGun.BGSM
------------------
In this case the 'Loose file' is the conflict winner but another information is needed: from which 'mod' the winner comes from? This information is obtained using a windows file utility, 'fsutil', see comments on line 147 and followings.
An interesting conflict could be the following:
------------------
General - Conflict number 38
3 DLCworkshop01 - Main.ba2 Meshes/AnimTextData/AnimationOffsets/PersistantSubgraphInfoAndOffsetData.txt
- 5 DLCCoast - Main.ba2 Meshes/AnimTextData/AnimationOffsets/PersistantSubgraphInfoAndOffsetData.txt
- 7 DLCworkshop02 - Main.ba2 Meshes/AnimTextData/AnimationOffsets/PersistantSubgraphInfoAndOffsetData.txt
- 9 DLCworkshop03 - Main.ba2 Meshes/AnimTextData/AnimationOffsets/PersistantSubgraphInfoAndOffsetData.txt
- 11 DLCNukaWorld - Main.ba2 Meshes/AnimTextData/AnimationOffsets/PersistantSubgraphInfoAndOffsetData.txt
- 102 mcgFemaleWalk - Main.ba2 Meshes/AnimTextData/AnimationOffsets/PersistantSubgraphInfoAndOffsetData.txt
- 135 RemoteExplosives - Main.ba2 meshes/AnimTextData/AnimationOffsets/PersistantSubgraphInfoAndOffsetData.txt
Conflict winner: RemoteExplosives - Main.ba2 meshes/AnimTextData/AnimationOffsets/PersistantSubgraphInfoAndOffsetData.txt
------------------
Here we have seven conflicting files, each one superseding its previous instance, till the last one, the conflict winner coming from the archive associated to RemoteExplosives (mod id 22444 by fadingsignal). Please note that this utility gives no advice about the conflict itself. How to handle the last conflict? Is that a fatal one? Or it's just Mostly Harmless? Who knows? Not me, for sure... I would consider this utility a forensic tool in the spirit of HijackThis, generating a file to be submitted to a good willing guru.
There are other two files; all the conflicts are split into two categories, 'Textures' and 'Non_Textures': the very same structure of the 'All_conflicts' is maintained.
A 'verbose' run (invoking script with a "-v" flag) keeps track of each and every file generated during the script execution. Mostly for debug and informations.
============= Mapping =======================
From version 1.1 some functionalities are added. Invoking the script with the "-M" the script generates another report stored in
'Snapshot_2019_08_11_20_42_51_Map_Target_to_Source.txt'
listing all the files with a hardlink equal to 2, that is to say all the files coming from a mod and deployed by the mod manager. In other words, this report gives you a "map" of all the files, listing where the files come from, from which mod and/or from the original installation or not managed by the mod manager or the like... A typical entry would be:
-----------------------------------------------------------
Target: d3d11.dll
Mod source: /Home/Games/Vortex/Fallout 4/mods/enbseries_fallout4_v0385/d3d11.dll
-----------------------------------------------------------
(all target names are relative to your game directory installation).
How to interpret this entry? It's quite straightforward, I think. My 'd3d11.dll' comes from a mod staged in /Home/Games/Vortex/Fallout 4/mods/enbseries_fallout4_v0385
which contains the same file d3d11.dll.
Another example would be:
-----------------------------------------------------------
Target: Data/3DNPC_FO4_DLC.esp
Mod source: /Home/Games/Vortex/Fallout 4/mods/DLC Commentary for TFTC v2.5-8704-2-5/3DNPC_FO4_DLC.esp
-----------------------------------------------------------
File 'Data/3DNPC_FO4_DLC.esp' comes from mod id 8704, "Atomic radio and Tales from the Commonwealth".
Some sanity checks are performed: if a file happens to have three or more link counts, something is going wrong and a report will be issued. It's left to the user to take appropriate measures. As said, this is a forensic tool.
A full "-M" invocation takes quite the time, so a 'light' mapping version is invoked with the "-m" option, which generates maps only and no conflict reports at all. "-M" and "-m" options are mutually exclusive and a check is performed. The "-v" options can be used as well and the only effect on the mapping process is to keep all the intermediate working files: this is intended as a tool to scrutiny the guts of your installation.
Files tend to be quite lengthy: a full map of more than 10000 files (not this big number, I think) has more than 30000 lines. Files are ordered in a tidy way, I think, but it's not that easy to get a picture of what is going on in its entirety.
Concluding remarks
I was trying to debug a game setup of mine and I weren't be able to detect what files were conflicting. So I felt the need to write a tool to fullfil this in my opinion essential request.
Suggestions, corrections (grammar as well as code, I'm not a native speaker), comments are welcome. More: are kindly requested.
Regards.
If I understand, loading mechanism of conflicting files is based on the following rule: the last is the winner. If two mods provide the same file the conflict is solved by a mod manager like Vortex by means of a load order rule.
Just to make it clear, a definition is needed: I call 'conflicting files' two or more files with the same name and path relative to 'Data' directory.
The task of resolving the conflict is easily accomplished if both files are not archived in a .BA2/BSA container: the last 'loose file' takes the precedence and will be loaded. Vortex allows the user to modify the loading order to finely tune which file will be the winner.
Even in the case of a set of more archived files and a single 'loose file' we can quickly solve the conflict: 'loose file' takes the precedence and will be loaded instead of any file of the same name stored in an archived container, whatsoever it would be fetched from. 'Loose file' will always be the winner.
Problems arise when two or more files are hidden inside an archive: in this case the 'loadorder.txt' file keeps track of the order in which plugins and their connected archives are loaded, but it is not immediately apparent which will be the conflict winner. I mean: the rule still applies but 'behind the curtain' in some sense.
This script tries to make clear what will be the result of the loading rule, listing all conflicting files, stating which will be the winner and, in case of a loose file, from which mods it comes from.
The script relies on the Nexus mod number 17061, BSA Browser (portable version) by AlexxEG but you need not to install his great mod: the needed executable file, a CLI version of his archive browser, is already embedded in the archive, with his permission, of course. I would like to thanks AlexxEG again.
All this work relies on my understanding of loading mechanisms, so if I'm wrong please tell me (and throw away this mod).
Prerequisites
A working modern bash shell with some built-ins like getopts.
Along with the shell a working implementation of 'find', 'sed', 'awk', 'grep', 'unix2dos'/'dos2unix', 'sort', 'diff', 'stat' is needed but I assume that, if you use bash you probably have all the standard *nix utilities (I mean, GNU coreutils, diffutils, fileutils and so on). My setup is built around a pretty standard MinGW-Msys implementation.
This utility relies on the 'hardlink' deployment mechanism that Vortex uses to build up 'Data' directory but I think that any other mod manager using hardlinks would fit. By the way this is the reason why there will be one and only one file with a given name (technically: with the same inode) in the deployed 'Data' directory. Of course the 'source' of the file may vary instead.
You can run the script whether Vortex is running or not (but after a full deploy of all the mods) but you probably can not integrate this script into it: it needs to be launched inside a shell but Vortex does not invoke the proper shell ("cmd" and not "bash", I mean).
Please google for "Link Shell Extension" by Schinagl for an excellent utility to clearly mark hardlinks in Explorer.
Setup and run
Create a directory of your choice and unpack the archive. You will find a shell script, 'Conflict_Detector.sh' along with two directories, 'snpashots' in which results will be collected and stored, and 'BSA_Browser' with a single executable inside, 'bsap.exe'. I will call 'working directory' the directory in which the script is stored.
Due to a feature of 'sed' please check that you have 'Full Control' permissions on the working directory and, in case, set them accordingly. See comments inside the script, line 222 of version 1.1 (by the way: the script is extensively commented but I realise that the word 'extensively' is pretty subjective. If you do not understand something it's my fault, not yours: I made my code obfuscated and unclear, and this is a cardinal sin...).
The script needs a little editing: the line
DATADIR="/e/Home/SteamLibrary/steamapps/common/Fallout 4/Data"
line 5 of the script, needs to be edited (by your text editor of choice) to reflect your setup. Please note quotation marks and the reference to the mounted drive (in my case E:\ which is rendered in "/e" by the Msys shell).
Now you can open a bash shell, change your dir to the working directory and launch the script. Please don't you forget that THIS IS A COMMAND LINE INTERFACE script. A 'double click' on the file from a GUI (I mean: from within an "Explorer" window) is not the intended way to get the script working. So please launch the script from within a bash shell CLI.
With no options the script runs (almost) quietly and saves the results in the 'snapshots' directory. Results are collected in three text files, eg:
'Snapshot_2019_08_11_20_42_51_All_Conflicts.txt'
'Snapshot_2019_08_11_20_42_51_Non_Textures_Conflicts.txt'
'Snapshot_2019_08_11_20_42_51_Textures_Conflicts.txt'
A single run takes its time. On my setup it takes about five minutes to process more than 1000 conflicts so please be patient, I have kept runs as silent as possible. If you prefer you can run a verbose ("-v" flag) version but your screen will be cluttered.
============= Conflict Detection ===========
Result files are identified by a timestamp (YYYY_MM_DD_HH_mm_SS) coded in the name and are organized in a sequence of 'Conflicts' like these:
------------------
General - Conflict number 11
5 DLCCoast - Main.ba2 Materials/DLC03/Landscape/Trees/RedPineQuarter01.BGSM
- 159 FO4 Seasons - Autumn_GoldenGrasses - AllDLC - Main.ba2 materials/DLC03/Landscape/Trees/RedPineQuarter01.BGSM
Conflict winner: FO4 Seasons - Autumn_GoldenGrasses - AllDLC - Main.ba2 materials/DLC03/Landscape/Trees/RedPineQuarter01.BGSM
------------------
How to decript this 'Conflict'?
First line shows the Conflict number, in this case in the 'General list' is the eleventh conflict.
Both conflicting files, namely
Materials/DLC03/Landscape/Trees/RedPineQuarter01.BGSM
are stored in two archives, 'DLCCoast - Main.ba2' (fifth in load order) and 'FO4 Seasons - Autumn_GoldenGrasses - AllDLC - Main.ba2', 159th, so the conflict winner is recorded on the last line of the 'conflict' section.
Another 'Conflict' section involves archived and loose files:
------------------
General - Conflict number 2
112 DOOMMerged - Main.ba2 Materials/Weapons/DOOM/ThatGun/ThatGun.BGSM
- 999 Loose Files Materials/weapons/DOOM/ThatGun/ThatGun.BGSM
Conflict winner: Loose Files Materials/weapons/DOOM/ThatGun/ThatGun.BGSM
Source:
\Home\Vortex\Fallout 4\mods\50 Ways to Die at Dr. Nick's - v.0.99c-25564-0-99c\materials\Weapons\DOOM\ThatGun\ThatGun.BGSM
Target:
\Home\SteamLibrary\steamapps\common\Fallout 4\Data\Materials\weapons\DOOM\ThatGun\ThatGun.BGSM
------------------
In this case the 'Loose file' is the conflict winner but another information is needed: from which 'mod' the winner comes from? This information is obtained using a windows file utility, 'fsutil', see comments on line 147 and followings.
An interesting conflict could be the following:
------------------
General - Conflict number 38
3 DLCworkshop01 - Main.ba2 Meshes/AnimTextData/AnimationOffsets/PersistantSubgraphInfoAndOffsetData.txt
- 5 DLCCoast - Main.ba2 Meshes/AnimTextData/AnimationOffsets/PersistantSubgraphInfoAndOffsetData.txt
- 7 DLCworkshop02 - Main.ba2 Meshes/AnimTextData/AnimationOffsets/PersistantSubgraphInfoAndOffsetData.txt
- 9 DLCworkshop03 - Main.ba2 Meshes/AnimTextData/AnimationOffsets/PersistantSubgraphInfoAndOffsetData.txt
- 11 DLCNukaWorld - Main.ba2 Meshes/AnimTextData/AnimationOffsets/PersistantSubgraphInfoAndOffsetData.txt
- 102 mcgFemaleWalk - Main.ba2 Meshes/AnimTextData/AnimationOffsets/PersistantSubgraphInfoAndOffsetData.txt
- 135 RemoteExplosives - Main.ba2 meshes/AnimTextData/AnimationOffsets/PersistantSubgraphInfoAndOffsetData.txt
Conflict winner: RemoteExplosives - Main.ba2 meshes/AnimTextData/AnimationOffsets/PersistantSubgraphInfoAndOffsetData.txt
------------------
Here we have seven conflicting files, each one superseding its previous instance, till the last one, the conflict winner coming from the archive associated to RemoteExplosives (mod id 22444 by fadingsignal). Please note that this utility gives no advice about the conflict itself. How to handle the last conflict? Is that a fatal one? Or it's just Mostly Harmless? Who knows? Not me, for sure... I would consider this utility a forensic tool in the spirit of HijackThis, generating a file to be submitted to a good willing guru.
There are other two files; all the conflicts are split into two categories, 'Textures' and 'Non_Textures': the very same structure of the 'All_conflicts' is maintained.
A 'verbose' run (invoking script with a "-v" flag) keeps track of each and every file generated during the script execution. Mostly for debug and informations.
============= Mapping =======================
From version 1.1 some functionalities are added. Invoking the script with the "-M" the script generates another report stored in
'Snapshot_2019_08_11_20_42_51_Map_Target_to_Source.txt'
listing all the files with a hardlink equal to 2, that is to say all the files coming from a mod and deployed by the mod manager. In other words, this report gives you a "map" of all the files, listing where the files come from, from which mod and/or from the original installation or not managed by the mod manager or the like... A typical entry would be:
-----------------------------------------------------------
Target: d3d11.dll
Mod source: /Home/Games/Vortex/Fallout 4/mods/enbseries_fallout4_v0385/d3d11.dll
-----------------------------------------------------------
(all target names are relative to your game directory installation).
How to interpret this entry? It's quite straightforward, I think. My 'd3d11.dll' comes from a mod staged in /Home/Games/Vortex/Fallout 4/mods/enbseries_fallout4_v0385
which contains the same file d3d11.dll.
Another example would be:
-----------------------------------------------------------
Target: Data/3DNPC_FO4_DLC.esp
Mod source: /Home/Games/Vortex/Fallout 4/mods/DLC Commentary for TFTC v2.5-8704-2-5/3DNPC_FO4_DLC.esp
-----------------------------------------------------------
File 'Data/3DNPC_FO4_DLC.esp' comes from mod id 8704, "Atomic radio and Tales from the Commonwealth".
Some sanity checks are performed: if a file happens to have three or more link counts, something is going wrong and a report will be issued. It's left to the user to take appropriate measures. As said, this is a forensic tool.
A full "-M" invocation takes quite the time, so a 'light' mapping version is invoked with the "-m" option, which generates maps only and no conflict reports at all. "-M" and "-m" options are mutually exclusive and a check is performed. The "-v" options can be used as well and the only effect on the mapping process is to keep all the intermediate working files: this is intended as a tool to scrutiny the guts of your installation.
Files tend to be quite lengthy: a full map of more than 10000 files (not this big number, I think) has more than 30000 lines. Files are ordered in a tidy way, I think, but it's not that easy to get a picture of what is going on in its entirety.
Concluding remarks
I was trying to debug a game setup of mine and I weren't be able to detect what files were conflicting. So I felt the need to write a tool to fullfil this in my opinion essential request.
Suggestions, corrections (grammar as well as code, I'm not a native speaker), comments are welcome. More: are kindly requested.
Regards.