----------------------------------------------------------------

ESP Scalpel - Nov 2013 Aaron Logue http://www.cryogenius.com/

ESP Scalpel is a fast and powerful tool for performing surgery on
ESP file elements. It does this by loading an ESP file into memory
as a linked list of groups. Command line switches can be used to select
a specific group for expansion into subchains of records and subrecords.
Records and subrecords that have been expanded can then be operated
on using generic primitives such as extract and write to file, delete,
insert, replace using previously extracted records, overwrite with
string or binary literals, and so on. Changed block sizes are
recalculated, compressed sections are recompressed, and new files
can be written to (or old ones overwritten if one is confident or
foolhardy or both). ESP Scalpel has very little knowledge about
flags, fields or the values of specific records. It manipulates
blocks of data at the record or subrecord level while handling
decompression, recompression, and resizing of records and subrecords.

ESP Scalpel and the portions of its source code written by me
(see below for a clear description of the portions not written by me)
is provided freely for use by others along with my sincere thanks for
all of the time and effort that others have put into mods,
documentation, tutorials, and tools that I have enjoyed so much.

ESP Scalpel uses code from zlib 1.2.8 by Mark Adler and Jean-loup Gailly
and available from http://zlib.net/ to handle ESP file compression.
Source files from zlib128.zip used, without alteration and included in
the scalpel_source directory purely for convenience by those wishing to
recompile scalpel.exe are: crc32.h deflate.h gzguts.h inffast.h inffixed.h
inflate.h inftrees.h trees.h zconf.h zlib.h zutil.h adler32.c crc32.c
deflate.c infback.c inffast.c inflate.c inftrees.c trees.c zutil.c
See the license in zlib.h and http://zlib.net/ for further information.

ESP Scalpel comes without warranty or guarantee of any kind.
Under no circumstances will the author be liable for any problems
that may arise from its use. It is a sharp instrument, so if you are
using the -w switch to write, keep your stuff backed up and know what
you've told it to do before you hit Enter.

----------------------------------------------------------------

INSTALLATION

Unzip the contents of the data directory - scalpel.exe, scalpel.dat,
and if you'd like to check out the demonstration, ethel.bat -
to your skyrim\data directory. Unzip the text files wherever you
would like to read them from.

ESP Scalpel is a command-line tool that you run repeatedly from a
C:\...> command prompt, so you may also want to make a shortcut
to a command prompt for convenience. To do that, right-click on
your desktop, select New, then Shortcut, enter or browse to
C:\windows\system32\cmd.exe, then re-edit the properties of your
cmd.exe icon and change "Start in:" to
C:\games\steam\steamapps\common\skyrim\data or whatever the correct
path on your machine is. Click on the cmd.exe shortcut that you
created, and you should get a command window with a
C:\yaddayaddayadda\skyrim\data> prompt.

ESP Scalpel does not need to be in skyrim\data. If you'd like to play
around with it separate from your Skyrim install, what you can do instead
is copy scalpel.exe, scalpel.dat, ethel.bat, and the text files into
the directory of your choice and copy skyrim.esm from your skyrim\data
directory to there as well. If using Windows Explorer, be sure to use
ctrl-drag or copy and paste so that you copy it rather than move it.

scalpel.exe does not care what its own name is, so you could copy it
to s.exe if.. lol, that wasn't intentional, I swear.. you were using
it and wanted to type less.

UNINSTALLATION

ESP Scalpel doesn't create registry entries or write to My Documents
or anything like that, so all you need to do is delete whatever
you unzipped. That was most likely scalpel.exe and scalpel.dat at
the very least, but probably scalpel_readme.txt, ethel.txt, ethel.bat,
and the scalpel_source directory as well. The whole shebang is only
a couple hundred K, so you could also just stop using it and not
worry about deleting it, but if you created the decomp.dat file -
the decompressed version of skyrim.esm to do searches with -
deleting that will free about 350M.

----------------------------------------------------------------

GETTING STARTED

If you are brand-new to command line things but interested,
read on, because one of my goals is to make this work as an
introduction to command line utilities. If you're brand-new to
the ESP file format as well - as I was when I started writing
this thing - you've got some work ahead of you, but I muddled
through it so you can too. If you're an old hand at this stuff,
run ESP Scalpel for a Usage: message and have at it. ESP Scalpel
is read-only without the -w switch, so as long as
-w is not used, it's safe to play with.

Follow the installation instructions above and get to the point
where you can open up a command window to a C:\...\skyrim\data>
prompt. Commands are entered simply by typing them and pressing
the Enter key. The first command to know is "exit". That'll close
the window. Getting in the habit of exiting that way instead of
hitting the little red X will make sure that you never accidently
abort a task that hadn't completed yet.

The term "folder" can get off our lawns now, because they were
"directories" before that. The next command to know is "dir",
which will give you a list of files in your current directory.
Like many command line commands, you can make dir do lots of
other stuff by adding parameters and switches. Type "dir *.exe"
and hit enter. The * is a wildcard, so that command will show
you all of the files that end in ".exe" in the skyrim data directory.
You should see scalpel.exe there.

The third thing to know is that the up arrow will recall your last
command. This is slightly broken in cmd.exe and sometimes it's the
down arrow instead, but even so, it's a feature that endows command
line users with the ability to get stuff done fast. Scalpel's command
line parameters and switches have been designed with the up arrow in mind.

The next things to know are "help", "help ", and that
adding "/?" to the end of many commands will cause them to display
a list of additional switches and parameters that they support.
Type "dir /?" or "help dir" and it will show you that "dir /od"
will sort a directory of files by their timestamps. "Help" will
give you a list of other commands. Ones that I use a lot include
findstr, cd, copy, and rename. I also use erase a lot, but save
that one until you feel confident about it, because it erases things.

"dir scalpel.*" should show you scalpel.exe and scalpel.dat.
"dir skyrim.esm" should show you that skyrim.esm is there, and
that it's roughly 250MB in size.

Type "scalpel" and press Enter to run it without any parameters.
It will print a usage message. Like many command line things,
scalpel has a simple syntax. The less-than/greater-than brackets
around parameters mean that they are required. Square brackets
mean that they are optional. Scalpel's switches are prefixed
with a dash, which is slightly different from the native MS-DOS
commands which use a forward slash to prefix switches.
(eg: "dir /?" versus "scalpel -?") The dash prefixing of switches
is the Linux, Mac OS, and multiplatform command line way of doing
things. The forward slash prefixing is the Microsoft way.

Scalpel requires an input file. It does not care about the extension
(the part after the ".") but only that the file contains some groups,
records, or subrecords for it to manipulate. Type "scalpel skyrim.esm"
and press Enter. 119 lines of data should be displayed, prefixed with
A0000 through A0118. You can select a block to look at in greater
detail with the -b switch. Press the up arrow key to recall
"scalpel skyrim.esm" and then add "-b A0011" or "-ba11" to select
the EYES group. It's not picky about the leading zeros.
"scalpel skyrim.esm -ba11" and Enter will display the contents of
the A0011 block. Now let's select the blue eyes. Again, up arrow to
recall the last command and add "b7" to select the particular EYES
record with "scalpel skyrim.esm -ba11b7". Finally, select the ICON
subrecord for the selected EYES record with "scalpel skyrim.esm -ba11b7c2"
and press Enter. You should see:
C0002: ICON 0x00000022 / 34 bytes
00000000: 4163746f 72735c43 68617261 63746572 |Actors\Character|
00000010: 5c457965 735c4579 65426c75 652e6464 |\Eyes\EyeBlue.dd|
00000020: 7300 |s. |

Now, try changing the selected subrecord by adding "-lhonk.dds" to get
"scalpel skyrim.esm -ba11b7c2 -lhonk.dds". Don't worry: no changes to
disk files will be made without the -w switch. You should see:
C0002: ICON 0x00000008 / 8 bytes
00000000: 686f6e6b 2e646473 |honk.dds |

Without the -w switch, ESP Scalpel displays the change that it has
made to the subrecord in memory and then throws the change away
without writing anything. That gives you a chance to see if that's
really what you want to write to a file or not. In this case, you
would probably wind up redoing it to add a "Actors\Chara.." path,
and then again with doubled backslashes to get the backslashes
inserted (see the section on LITERALS below for what that's about)
and again to add the 0x00 NULL terminator to the string. Let's stop
there though, because you don't want to be changing skyrim.esm anyway.

What you just accomplished, however, was that you learned how to
specify an input file, how to use the -b switch to select a group,
record, or subrecord, and then once you have what you want selected,
you can add a switch to perform an operation on that group, record,
or subrecord. Finally, if you don't like what you see, you can just
abandon it with no harm done. You saw that a typical way of working
with ESP Scalpel is to hit the up arrow, add to the parameters or
switches that you've already got, hit Enter to see the next result,
and repeat until you like what you see.

Try exploring some ESP files that are in your data directory,
or if you'd like, check out...

----------------------------------------------------------------

The ETHEL.BAT Demo

If you unzipped ethel.bat into your data directory, you can run it
with the command "ethel". A bunch of stuff should scroll by for a
few seconds as scalpel is run multiple times to extract bits and
pieces of skyrim.esm and use them to construct ethel.esp. After it
completes, you can list its contents with "scalpel ethel.esp"

A0000: TES4 0x00000059 / 89 bytes
A0001: GRUP NPC_ 0x0000020d / 525 bytes
A0002: GRUP CELL 0x000001ab / 427 bytes

I have included a file, ethel.txt, that documents my thought
processes as I went about coming up with the commands that I
used to construct ethel.esp.

You can experiment with ethel.esp and try writing to it with
-w ethel.esp. If it gets damaged beyond repair, you can just
regenerate it by running ethel.bat again. An experiment might
be to see if you can replace Ethel's iron dagger with something else.

scalpel decomp.dat -tz -m OrcishDaggerShock could be used to begin
the hunt for the ID of a more powerful replacement. An NPC's
inventory begins with the COCT subrecord, which is a count of how
many CNTO subrecords follow it. The CNTO subrecords are an item ID
and a count, so if you find an item ID to use, you should be able
to replace the ID in an existing CNTO record, or add a new CNTO
record with the ID you want to add to the NPC's inventory.

The -o switch works well to overwrite 4-byte ID values, and the
-y switch works to insert new subrecords. For example, gold
has an ID of 0000000f, so this should add 50 gold to Ethel's inventory:
scalpel ethel.esp -ba1b0c8 -y CNTO0x08000f00000032000000 -w ethel.esp
Annnnd that didn't work... until I remembered COCT and bumped the count:
scalpel ethel.esp -ba1b0c6 -o 0x03 -w ethel.esp
Annnnd it still didn't work because Ethel's CNTO d1461000 "Citizen Loot"
item was replacing the 50 gold with a small random amount of gold.
Swapping the order of the CNTOs didn't fix it, so whatever fills
in the loot and clobbers existing inventory numbers must do that
after all of the inventory slots are filled. I had to delete the
loot item and put the COCT record count back before Ethel had 50 gold:
scalpel ethel.esp -ba1b0c8 -d -w ethel.esp
scalpel ethel.esp -ba1b0c6 -o 0x02 -w ethel.esp

For me, the above flailing about is typical of Skyrim modding regardless
of the tools being used. I tend to spend the majority of my time trying
to understand why something that seemed reasonable to expect to work did
not work, then after searching in vain for documentation, experimenting
until an adequate workaround is found. ESP Scalpel was such an experiment
as I struggled to try to get the CK to add one door triangle to an exterior
cell without needlessly modifying the navmeshes of the surrounding cells.

If you are hacking on ethel.esp, I recommend avoiding loading it
up in-game and making a Save file with a character that you care
about and then continuing to play long-term on that Save thread.
The reason is that, while ESP Scalpel will certainly allow one to
scramble an ESP file catastrophically by deleting the wrong GRUP
or loading an entire CELL into the middle of some poor NPC's eyeball,
it will also allow one to do something like forget to change an ID
in a copied record and corrupt an ESP in a subtle way that will appear
to work for awhile but fail later for reasons that are not obvious.

----------------------------------------------------------------

SELECTING GROUPS, RECORDS, and SUBRECORDS

In general, the top level of an ESP file consists of a collection of
groups. ESP Scalpel lists them as blocks of data labelled A0000,
A0001, etc. Within any particular group may be one or more records,
which ESP Scalpel will list as B0000, B0001, etc, but only if it
has been asked to descend into that group with the -b switch.
The selection of a group might look like "-b A0042" or "-ba42".
The -b switch isn't picky about the space or leading zeros.
Selecting a record is done by adding it after the group.
The selection of a record might look like "-b A0042B0555" or
"-ba42b555". Selecting a record will cause any subrecords that
it contains to be listed as C0000, C0001, etc, and selecting one
of those subrecords is done as above, by adding it after the record.
Selecting a subrecord might look like "-ba42b555c0". A selected
record must be preceded by a selected group, and a selected
subrecord must be preceded by a selected record.

ESP Scalpel will work with headerless ESP file fragments, labelling
whatever structures it finds at the top level as A0000, A0001, etc.
This is useful because it also supports extraction of fragments to
standalone files and import of fragments from files.

By default, the selected block is displayed as a hexdump, with a
list of any subblocks beneath. The hexdump display can be modified
with the -t and -q switches, and the block list display can be modified
with the -f and -z switches. The -p switch will cause parent blocks
that are normally hidden when drilling down to be included. In addition,
the -p switch can be used to cause parent records to be included in
lists, hexdumps, or fragment extracts when they would normally not be
as a result of having selected a subblock. For an example, compare the
outputs of "scalpel ethel.esp -pt -ba2b2c1" with and without the -p switch.

ESP Scalpel relies on the external file scalpel.dat to tell it which
codes are groups, records, or subrecords. It matters because
groups and records are followed by 4-byte lengths while subrecords
are followed by 2-byte lengths, along with a few other differences.
scalpel.dat is just a text file, editable with notepad or whatever.
Hopefully there will be no need ever to change it, but I did it
as an external file in case I missed some codes or got some wrong.

Some records are compressed, and are loaded into memory but left
compressed unless selected with the -b switch. Compressed or
decompressed records are shown as such. By default, records that
were decompressed as a result of being selected will be recompressed
upon being written. This can be overridden with the -n switch.

----------------------------------------------------------------

LITERALS

Four of scalpel's switches take a as a parameter.
A literal is a string of characters without any intervening spaces
that can be used to specify an arbitrary sequence of bytes.
"0x" will cause the parser to switch from character string to
hexadecimal mode, and "\" (or encountering a non-hexadecimal
character) will cause the parser to switch from hexadecimal mode
back to character string mode. Thus, "The Bannered Mare" with
two embedded spaces and a NULL terminator could be specified as
"The0x20\Bannered0x20Mare0x00" and a TES4 header's MAST subrecord
might be inserted with "-ba0b1 -y MAST0x0B00Skyrim.esm0x00"
The "\" character also works to escape the "0x" sequence and
the "\" itself, so if you need "0x00" entered as a 4-character
string as opposed to a single byte with a value of zero, you can
use "\0x00". If you need a single backslash, which commonly occur
in DDS texture filename paths, use "\\".

A single byte is 8 bits, and each hexadecimal character is 4 bits,
so a byte requires 2 hexadecimal characters. When you're in 0x
hexadecimal mode, remember to always enter hex digits in pairs.
When mixing hex and strings, remember that "abcdef" are valid
hex digits, so you'll need to switch out of hex mode with \
if the string begins with one of those letters. 0x20 is a space,
so "Bannered" was a good example of this. Without the \ to exit
hex mode and interpret the B literally, the "Ba" would have been
interpreted as a single byte in hex. Scalpel's hex interpreter
is not case-sensitive.

The Intel or AMD processor in your PC stores numbers as "big endian"
(as opposed to Motorola processors, which are "little endian". What
that means is that things that are multiple-byte values, like
Form IDs, will appear in reverse order, with their least significant
byte first and their most significant byte last. If you are searching
for Ysolda's ID 00013BAB, for example, you will need to do this:

scalpel skyrim.esm -ba42 -tz -m 0xAB3B0100

because that is how the ID is stored in Ysolda's NPC_ record:

scalpel skyrim.esm -ba42b4789 -tz
B4789: NPC_ 0x00000449 / 1097 bytes (decompressed)
00000000: 00000000 ab3b0100 1b695500 28000c00 |.....;...iU.(...|
00000010: 45444944 07005973 6f6c6461 00564d41 |EDID..Ysolda.VMA|
...

Here are some examples:
"Ethel" : "Ethel0x00" (don't forget to null-terminate your strings!)
"actor\swamp zombie\foo.dds" : "actor\\swamp0x20\zombie\\foo.dds0x00"
"0x0x0\DethBlyd/0x0x0" : "\0x\0x0\\DethBlyd/\0x\0x\00x00"

----------------------------------------------------------------

SEARCHING

Searching large files like skyrim.esm is complicated by the fact that
ESP and ESM files can contain multiple individually-compressed records.
While the -m switch is used to search for a string or binary
value as described above, it does not by default decompress records to
search them. This is often not a problem for Form IDs of cells or NPCs,
which tend to be found in uncompressed record headers, but additional
steps must be taken to search for strings.

For ESP files, which tend to be much smaller than skyrim.esm, an easy
solution is to use the -e switch to decompress all of the records in
the file before searching them: scalpel ethel.esp -ezpm Ethel
The -z switch inhibits display of records that did not match, and the
-p switch forces parent records to be considered to have matched if
a subrecord matched. That allows you to see which A0000,B0000 etc
records to specify with the -b switch on a subsequent command to select
the record that matched.

Another approach is to use -e with the -n and -w switches to expand
everything and write it to a decompressed version of the file, which
is then used for fast searching even of large files like skyrim.esm.
This is described in more detail in ethel.txt, in which the command

scalpel skyrim.esm -eqzn -w decomp.dat

is used to create a decompressed version of skyrim.esm.

Here are some examples of searching for the IDs and records of
NPCs, places, and items in decomp.dat and ethel.esp for you to try:

scalpel ethel.esp -etzp -m Ethel (Shows you that -ba1b0 will display
Ethel's NPC_ record)
scalpel ethel.esp -ba1b0 (Display's Ethel's NPC_ record, along with
a list of subrecords. If you know what COCT
and CNTO records are, then you see that
-ba1b0c7 and c8 will show you the item IDs
of what's in Ethels' inventory.)
scalpel ethel.esp -etzpm Banner (Shows you that -ba2b2 will display
the Bannered Mare's CELL record and
give you its ID, 0001605E)

When working with IDs, remember that the PC is a "big endian" machine,
meaning that you'll constantly be having to reverse bytes in your head.
Don't worry.. it's not that bad, especially with Skyrim IDs which have
certain characteristics that you'll start to recognize. If you see
5e600100 where the ID is supposed to be in a hexdump, read it as 0001605E.
If you want to search for that ID, use 0x5e600100 as the -m literal value
to search for since that's the order it is stored in memory and on disk.

Beware that searches are case sensitive, so "ysolda" has no matches
in decomp.dat, while "Ysolda" has several.

scalpel decomp.dat -m Ysolda (list all of the groups containing Ysolda.
Shows you that -ba42 is the NPC_ group)
scalpel decomp.dat -zm Ysolda (don't display the groups she's not in)
scalpel decomp.dat -ba42 -tzm Ysolda (add -t to print only the first 8 lines
of the 4-megabyte A42 NPC_ group.
Shows you that -ba42b4789 is Ysolda)
scalpel decomp.dat -zm SleepingGiant (Same basic search, only for a place)
scalpel decomp.dat -ba58 -tzm SleepingGiant (finds B29621)
scalpel decomp.dat -ba58b29621 (selects the Sleeping Giant Inn cell)
scalpel decomp.dat -ba58 -tzm DraugrMelee (finds 20 candidates for my wife's
swamp zombies project)

----------------------------------------------------------------

DELETING

Selected groups, records, or subrecords can be deleted with the -d switch.
Without the -w switch, the -d switch displays a "Would delete ..."
message with the selected block and any subblocks removed from the
block list display. The sizes of structures that change as a result
of a delete (for example, the GRUP NPC_ becoming smaller because
it contained multiple NPCs and one was deleted) are updated.

Deleting a group or record will not automatically cause the TES4 HEDR
group+record count to be updated. A -u update will need to be done,
but it can be done at any point; given on the same command line along
with -d, or done later as a standalone operation.

ESP Scalpel reassigns its A0000, B0000, etc labels to groups, records,
and subrecords every time it is run. That means that if, for example,
you wish to delete two consecutive records, the -b block selection
may be same each time. For example, suppose you wished to delete
two records that happened to have ESP Scalpel block labels B0007 and
B0008 and you deleted B0007 first. The record that was B0008 would
become B0007, so the second delete would need to delete B0007 rather
than B0008. If it deleted B0007 and then B0008, it would actually
be deleting whatever was B0009 at the time that B0007 was deleted.
This isn't an issue if the higher numbered blocks are deleted first.
You can see this being done both ways in ethel.bat.

----------------------------------------------------------------

EXTRACTING FRAGMENTS

Selected groups, records, or subrecords can be extracted as an ESP fragment
with the -x switch. An ESP file fragment created with the -w switch can be
edited with ESP Scalpel even though it is not a complete ESP file. In fact,
ESP Scalpel is intended for use with incomplete or work-in-progress ESP files.

Selected records that were originally compressed will be recompressed
before writing, but this can be overridden with the -n switch.

The -p switch can be added to cause the parents of the selected block
to be included in the output file. This is generally useful only
to grab the parent GRUP header of a record when you are going to
insert that record in an ESP that you're working on that does not
yet have a GRUP header to contain that type of record. Recompression
of records is inhibited by the use of -p (compressed records with single
selected subrecords are pointless), but records can be compressed
later with -c if desired.

----------------------------------------------------------------

IMPORTING FRAGMENTS

The -i and -r switches accept filenames of ESP fragments to import.
-i will insert the file's contents after the block selected with -b,
and -r will replace the selected block with the file's contents.
Structures that change size as a result of

The -s switch is a standalone operation to swap the selected block
with the block following it, but it can also be used in conjunction
with -i to cause -i to effectively insert before the selected block
rather than after it.

----------------------------------------------------------------

EDITING WITH LITERALS

The -y and -l switches are analogous to -i and -r, with the primary
difference being that they accept literals as command line parameters
as the data to be inserted after the selected block or used to replace
the selected block with, respectively. While it's technically possible
to enter an entire GRUP or record with -y or -l, these switches are
intended for entering subrecord data. The -s swap switch works with
-y to cause it to insert before a selected record in the same way that
it works with -i. Sizes of structures that change as a result of an
insert or replace will be updated.

The literal accompanying a -y or -l switch should start with the
4-character code, and must be followed by the length bytes, which are
dummies and can be any value. They will be replaced with the actual
length of the data that follows.

Note that the -l switch can thus be used to change a selected subrecord
code to a different code. The code is specified with -l, even though
it is replacing an existing code.

The -o data overwrite switch expects only the group, record, or subrecord
data as the literal, and not a code or length as with the -y and -l switches.
It overwrites starting at the beginning of a block's data up to the end
of the block. It cannot change the size of a structure. A common use of
-o is to change something's Form ID, which typically occurs 4 bytes into
a record rather than at the beginning of it. The four bytes to be skipped
must be provided and are not dummies - they will overwrite whatever is
there on the way to what you want to change.

----------------------------------------------------------------

WRITING

Without the -w switch, ESP Scalpel is read-only, performing all of
its operations on linked lists in memory and displaying the results,
but terminating without performing a write operation or changing
anything on disk.

The -w switch takes a filename. It can be a new file or it can
be the name of a file to overwrite - including the input file.

When using ESP Scalpel interactively, a common way to use it is
to see what it's going to do without the -w switch before adding it.
After a time, you might get comfortable enough with it to hit enter on
scalpel myfile.esp -ba4b7c3 -y FULL0x0000New0x20Name0x00 -w myfile.esp
without bothering to see what it looks like without the -w switch first,
but I'm not there yet and I wrote the thing. :)

----------------------------------------------------------------

May your hard drive always spin up and all your mods be successful.