0 of 0
About this mod
Convert x86/x64 hkx <-> XML,
Debug bytes,
Show hexdump,
Show deps tree,
hkx bytes diff
- Permissions and credits
- Changelogs
Release: https://github.com/SARDONYX-sard/serde-hkx/releases
Havok Behavior (De)Serialization CLI written in Rust(for ver. hk_2010.2.0-r1)
CLI created for hkx debugging purposes.
This is one of my attempts.
- Is it possible to serialize/deserialize Havok behavior quickly without ownership errors with Rust static dispatch?
On the binary data, it outputs as if there is a vtable similar to the official SDK, but we thought it might be faster by not using a vtable in the Havok pseudo-C++ class expressed in Rust.
- I need a way to debug hkx. (I wanted to enable debugging by writing to log the location of the binary data and the information contained in the binary data.) If there is a bug in hkx, I wanted to clarify what is logically wrong with it. (Because nothing can be done in such an ambiguous state as “it errors for some reason.”)
<Features>
- Convert x86/x64 hkx <-> XML
- Debug bytes
- Show hexdump
- Show deps tree
- hkx bytes diff
<Bug report>
Reporting via GitHub issue is the easiest way to be noticed, but POSTS are also fine.
However, to fix a bug, you will need at least the following (since the only way to find a bug is to debug it manually)
- Nexus Mod name & ver. & URL
- hkxc ver.
- What kind of conversion did you do? (e.g. xml -> win32)
- Name of the hkx file where the bug occurred
- error message
<Not testing is the source of bugs>
- Verify(since v0.7.0)
When editing hkx files like `defaultmale.hkx`, it is recommended to first test if you can revert to the original hkx with the verify command. This command does not use the official SDKs, so there is no guarantee that you will be able to restore the original hkx. (Animation.bsa has already been tested, but it is unknown if the mod's hkx can be reproduced)
Example
./hkxc verify ./defaultmake.hkx
./hkxc verify ./input_dir/
hkx -> XML -> hkx => There is no guarantee of reproduction at all. (even the hkxcmd command using the official SDK)
I found out while coding this that converting to XML and then to hkx causes some data loss and differences. hkxcmd has the same behavior. I don't yet know how the differences affect the results, but this can be seen by looking at the diff.
`hkxc diff . /skeleton_x86_hkxc.hkx . /skeleton_x86.hkx --color`.
Conversion from hkx to hkx does not cause any loss.
Also, the extra_fmt cli released on GitHub can output json, yaml, and in this case can restore.
<Tips>
- Duplicate XML(<hkparam>) is ignored.
- Missing XML items(<hkparam>) are automatically initialized with their initial values.
- For IO-bound processing, conversion can be a few seconds faster using tokio asynchronous, but as far as we tested with Animation.bsa, the speed is almost the same as rayon mode, a library for CPU-bound processing.
<Known issues>
A diff on the test file confirms that the following is not reproduced.
- QsTransform's scale field (Vector4.w that cannot be obtained in XML, but exists in the binary data in the game. 0 is entered in the same way when converted with hkxcmd)
- Some of the data in f32 cannot be reproduced. -> The problem was f16. Fix complete.(v0.1.2)
- The reproduction of binary data in global_fixups does not match.((There is no resolution in sight for the design problem.) -> (Fixed v0.2.0)
- The hkxcmd XML index(e.g. `#0001`) rules are nearly figured out base index probably involves a `__types__` section. But I don't know how to reproduce the order of serialization from the independent classes.
The current indexing method of serde-hkx adheres to the serialization order specification of hkx files. hkx serialization follows a tree structure starting from `hkRootLevelContainer`, ensuring consistency, which is why this rule is used for indexing. (hkxconv uses a similar method, but it starts from index 50, whereas my command starts from 1.)
By the way, you can see the serialization order in the `./hkxc.exe tree ./defaultmale.hkx`. This is a dependency tree of pointers, and the order of the tree is the serialization order.
<Frequently Asked Questions>
Q. Can I convert to fallout 4 hkx?
A. Probably not. The binary data depends on the offset and field information of the C++ classes defined in hk2010. it is not compatible because the classes and definitions that exist change depending on the SDK version.(My code is designed so that if a C++ class definition is defined in a json file, the class code for that version can be generated automatically.)
Q. You already have the command, don't you?
A. The existing commands cannot parse the binary data information and debug the results. We wanted to find out which byte position is the data in a certain field.
<Ref>
<Related Projects(GUI)>
Havok Behavior (De)Serialization CLI written in Rust(for ver. hk_2010.2.0-r1)
CLI created for hkx debugging purposes.
This is one of my attempts.
- Is it possible to serialize/deserialize Havok behavior quickly without ownership errors with Rust static dispatch?
On the binary data, it outputs as if there is a vtable similar to the official SDK, but we thought it might be faster by not using a vtable in the Havok pseudo-C++ class expressed in Rust.
- I need a way to debug hkx. (I wanted to enable debugging by writing to log the location of the binary data and the information contained in the binary data.) If there is a bug in hkx, I wanted to clarify what is logically wrong with it. (Because nothing can be done in such an ambiguous state as “it errors for some reason.”)
<Features>
- Convert x86/x64 hkx <-> XML
- Debug bytes
- Show hexdump
- Show deps tree
- hkx bytes diff
<Bug report>
Reporting via GitHub issue is the easiest way to be noticed, but POSTS are also fine.
However, to fix a bug, you will need at least the following (since the only way to find a bug is to debug it manually)
- Nexus Mod name & ver. & URL
- hkxc ver.
- What kind of conversion did you do? (e.g. xml -> win32)
- Name of the hkx file where the bug occurred
- error message
<Not testing is the source of bugs>
- Verify(since v0.7.0)
When editing hkx files like `defaultmale.hkx`, it is recommended to first test if you can revert to the original hkx with the verify command. This command does not use the official SDKs, so there is no guarantee that you will be able to restore the original hkx. (Animation.bsa has already been tested, but it is unknown if the mod's hkx can be reproduced)
Example
./hkxc verify ./defaultmake.hkx
./hkxc verify ./input_dir/
hkx -> XML -> hkx => There is no guarantee of reproduction at all. (even the hkxcmd command using the official SDK)
I found out while coding this that converting to XML and then to hkx causes some data loss and differences. hkxcmd has the same behavior. I don't yet know how the differences affect the results, but this can be seen by looking at the diff.
`hkxc diff . /skeleton_x86_hkxc.hkx . /skeleton_x86.hkx --color`.
Conversion from hkx to hkx does not cause any loss.
Also, the extra_fmt cli released on GitHub can output json, yaml, and in this case can restore.
<Tips>
- Duplicate XML(<hkparam>) is ignored.
- Missing XML items(<hkparam>) are automatically initialized with their initial values.
- For IO-bound processing, conversion can be a few seconds faster using tokio asynchronous, but as far as we tested with Animation.bsa, the speed is almost the same as rayon mode, a library for CPU-bound processing.
<Known issues>
A diff on the test file confirms that the following is not reproduced.
- QsTransform's scale field (Vector4.w that cannot be obtained in XML, but exists in the binary data in the game. 0 is entered in the same way when converted with hkxcmd)
-
- The hkxcmd XML index(e.g. `#0001`) rules are nearly figured out base index probably involves a `__types__` section. But I don't know how to reproduce the order of serialization from the independent classes.
The current indexing method of serde-hkx adheres to the serialization order specification of hkx files. hkx serialization follows a tree structure starting from `hkRootLevelContainer`, ensuring consistency, which is why this rule is used for indexing. (hkxconv uses a similar method, but it starts from index 50, whereas my command starts from 1.)
By the way, you can see the serialization order in the `./hkxc.exe tree ./defaultmale.hkx`. This is a dependency tree of pointers, and the order of the tree is the serialization order.
<Frequently Asked Questions>
Q. Can I convert to fallout 4 hkx?
A. Probably not. The binary data depends on the offset and field information of the C++ classes defined in hk2010. it is not compatible because the classes and definitions that exist change depending on the SDK version.(My code is designed so that if a C++ class definition is defined in a json file, the class code for that version can be generated automatically.)
Q. You already have the command, don't you?
A. The existing commands cannot parse the binary data information and debug the results. We wanted to find out which byte position is the data in a certain field.
<Ref>
- Source Code(serde-hkx/cli)
- hkx docs(hex dump journey)
- hkx2library (64bit C++ definitions from `comments`)
- hkxcmd (32bit C++ definitions by `Report` command)
<Related Projects(GUI)>
- Hkx Conversion Tool
- Composite HKX Conversion GUI
- D merge(diff & merge nemesis altnative patcher)