Structure and Class Documentation

[Seeky]

I've put together some documentation of various structures and classes in the game that I'm familiar with, check it out here. This isn't meant to be a 1 to 1 decompilation or anything, just a guide to what various fields are and some functions related to the classes. Please be sure to credit me in any codes you make with this information

If you have anything to add, it'd be appreciated!

[em_eight]

I've put together some documentation of various structures and classes in the game that I'm familiar with, check it out here. This isn't meant to be a 1 to 1 decompilation or anything, just a guide to what various fields are and some functions related to the classes. Please be sure to credit me in any codes you make with this information

If you have anything to add, it'd be appreciated!

This project is extremely interesting and I made an account here just for this. I was wondering why kaitai struct is not used https://kaitai.io/. It seems to fit exactly for this purpose, makes documentation very clear and you could get compilable code across many languages. Am I missing something?

[Seeky]

I didn't know about katai at the time I was working on this, and it was mainly just meant to be a reference to get the information out there, rather than something that could be directly used in code; almost everything was written in asm at the time, and I didn't really want to spend time making sure the syntax of everything was valid

Edit: looking at katai again, it also seems like it's not ideal for this purpose at all. Compiling it to C++ creates a huge amount of overhead compared to a simple struct/class definition, and since it's mainly meant for file formats, you have to add a load of useless steps to each struct definition just to tell it it's one block of information

[em_eight]

I didn't know about katai at the time I was working on this, and it was mainly just meant to be a reference to get the information out there, rather than something that could be directly used in code; almost everything was written in asm at the time, and I didn't really want to spend time making sure the syntax of everything was valid

Edit: looking at katai again, it also seems like it's not ideal for this purpose at all. Compiling it to C++ creates a huge amount of overhead compared to a simple struct/class definition, and since it's mainly meant for file formats, you have to add a load of useless steps to each struct definition just to tell it it's one block of information

The only amount of overhead is the generation of the parser, which shouldn't be comparable to the parsing itself, given the size of the data. Still, the parser generation is a one-off thing and would not at any case run real-time, but the parser itself would be as fast as a human would write it. I don't see it ever being human-noticeable slow anyway.

Can you maybe clarify a little bit what you mean by "mainly used for file formats" and "useless steps"? Kaitai has been used for executables, core dumps, filesystems and many things https://formats.kaitai.io/. I don't see any limitations for MEM1 dumps. I don't see any useless steps to definitions, you just need to be completely precise so parsers can be generated. That's the point of the tool. You can write the kaitai file as a draft if you want and not make sure it compiles into a proper parser and this would be equivalent to the current project. The only difference is that it would be scalable to an at least partially actual working one, if eventually someone finds the time.

In the end, this can be a tradeoff of documentation time and utility. A basic kaitai file could be used to create very powerful tools, such as a custom real time dashboard of all variables of interest in MKW memory, wiimm like tools (for static files, not RAM memory) and so on. I think it would be timelessly useful tool that can be scaled into arbitrarily interesting stuff.

[Seeky]

Take this struct for example:

Code:

typedef struct {
  uint8_t unknkown_0x0;
  uint8_t unknkown_0x1;
  uint8_t padding[2]; // Ignored when copying to array
  uint32_t objectId; // see http://wiki.tockdom.com/wiki/List_of_Identifiers#Items
  uint32_t numberOfItems;
  uint32_t unknown_0xc;
  uint8_t unknown_0x10;
  uint8_t padding2[3]; // Ignored when copying to array
  ItemUseType useType;
  void * useFunction; // Item is draggable if this is null
} ItemBehaviour; // Total size 0x1c

In katai, it becomes:

Code:

meta:
  id: itembehaviour
  endian: be
seq:
  - id: body
    type: body
types:
  body:
    seq:
    - id: unknown_0x0
      type: u1
    - id: unknown_0x1
      type: u1
    - id: padding_0x2
      size: 2
    - id: objectid
      type: u4
    - id: numberofitems
      type: u4
    - id: unknown_0xc
      type: u4
    - id: unknown_0x10
      type: u1
    - id: padding_0x11
      size: 3
    - id: usetype
      type: u4
    - id: usefunction
      type: u4

The useless steps being:

Code:

meta:
  id: itembehaviour
  endian: be
seq:
  - id: body
    type: body
types:

vs the lines

Code:

typedef struct {
} ItemBehaviour; // Total size 0x1c

and each field's declaration is less concise

As for the overhead, it generates a 80 line header and a c++ file that needs to be compiled (as well as depending on the katai library to build) for a struct that could've been read in one pointer dereference with the definition above just put in a header

Katai could be useful for other languages, but considering most coding done for the game above asm is in C++ (or even pure C, which katai doesn't support), it's not ideal here imo

[em_eight]

Take this struct for example:

Code:

typedef struct {
  uint8_t unknkown_0x0;
  uint8_t unknkown_0x1;
  uint8_t padding[2]; // Ignored when copying to array
  uint32_t objectId; // see http://wiki.tockdom.com/wiki/List_of_Identifiers#Items
  uint32_t numberOfItems;
  uint32_t unknown_0xc;
  uint8_t unknown_0x10;
  uint8_t padding2[3]; // Ignored when copying to array
  ItemUseType useType;
  void * useFunction; // Item is draggable if this is null
} ItemBehaviour; // Total size 0x1c

In katai, it becomes:

Code:

meta:
  id: itembehaviour
  endian: be
seq:
  - id: body
    type: body
types:
  body:
    seq:
    - id: unknown_0x0
      type: u1
    - id: unknown_0x1
      type: u1
    - id: padding_0x2
      size: 2
    - id: objectid
      type: u4
    - id: numberofitems
      type: u4
    - id: unknown_0xc
      type: u4
    - id: unknown_0x10
      type: u1
    - id: padding_0x11
      size: 3
    - id: usetype
      type: u4
    - id: usefunction
      type: u4

The useless steps being:

Code:

meta:
  id: itembehaviour
  endian: be
seq:
  - id: body
    type: body
types:

vs the lines

Code:

typedef struct {
} ItemBehaviour; // Total size 0x1c

and each field's declaration is less concise

As for the overhead, it generates a 80 line header and a c++ file that needs to be compiled (as well as depending on the katai library to build) for a struct that could've been read in one pointer dereference with the definition above just put in a header

Katai could be useful for other languages, but considering most coding done for the game above asm is in C++ (or even pure C, which katai doesn't support), it's not ideal here imo

I can get the less concise part, but that's what happens when you require precision. You would have to write additional code to enforce strict alignment for structs in C too.

Trust me there is practically no computational overhead in compiling the C files generated. You would need to compile a custom parser anyway, how is that different? If you are talking about human overhead, all of this is easily automated in Makefile/CMake/shell script or w/e.

I am not suggesting this as a stylistic preference, it is absolutely harder to read and write. I am suggesting it for additional functionality. As mkw-structures grows, no progress will be made in creating an actual memory parser. And such parser is way more complicated than wimm's tools, as those work in non-dynamic memory and do not need to do pointer-chasing. WIll there be interest for anyone to develop+debug it from scratch?
I just think that work(documentation in pseudoC+custom parser)>>work(documentation in kaitai)

Anyway, I know your effort is still very helpful and as you do it for fun I am in no position to force you do anything, so I just wanted to drop a suggestion. I am interested in the automation of extraction of large amount of data from the game, so I might join your efforts at some point

[Seeky]

You would have to write additional code to enforce strict alignment for structs in C too.

I was referring to the type & name being on separate lines, any aligmnent needed is achievable with padding fields in the same way for both katai and C++

You would need to compile a custom parser anyway, how is that different?

I'm talking in the context of custom C++ code running inside the game, you don't need a custom parser to read a field of a struct (and would a katai parser even have a way to write back to these structs in RAM, or handle things like method calls?)

If I was writing a custom memory parser then, sure, katai might be a good choice, but that was never my intention, I wanted to document the game for code mods, either as a reference for assembly modding or as something that could be cleaned up into c/c++ headers for modding in those languages. mkw-structures is a pretty dead project at this point anyway, I've moved on from mkw modding (got sick to death of the wider community, but I do stick around in some places to help people) and the other main contributor, stebler, moved their focus to their ghidra repo + riidefi's decomp

[em_eight]

Ye, that's a fair thing to say, you had completely different goals.

Can't blame anyone though, decompilation and even documentation are enormous challenges, both in time and technical skill, it's quite remarkable that MKW has even reached that level of modding.