Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[zilpin]

Dwarf Fortress World.dat File Format for df34.07

dfWorldDat current version: 0

Link to dfWorldDat SourceForge Project

Primary Goal:
To provide synchronization between a World.dat or World.sav and it's raws, so that modders may quickly test changes without re-genning a new world.
(in other words, replace DF World Tinker)

Secondary Goal:
To provide an export mechanism giving world information so that third party GUIs may be made to render the world, view it in detail, search it's layers for ideal embark sites, generate movies of world history (oh yeah), etc, etc.

About:
I've been working on this since the early releases of df2010, extending as needed when needed.
Getting burned out deciphering the cryptic world format, and too busy to pursue time consuming trial-and-error discovery at the moment.
So I cleaned up the file, commented it, and opened up what I have to create a team project for it.
The SF project has one file, a template for the 010 Hex Editor by Sweetscape.
This will parse most of the header, RAW tags, geology data, regions and some subregions.  Then it will error out on a subregion.
I need help completing this, by people with a reasonable understanding of hex editing, data structures, grok 'little-endian', basic C knowledge.
And most importantly, people willing to do trial-and-error experimentation to discover the inner workings of the World.dat format.

Tasks:
Listed roughly in the order found in the world.dat file, difficulty varies.

Spoiler (click to show/hide)

   Task 0: Figure out the rest of the header.
   Task 1: Figure out Animal People
   Task 2: Region Meta-data.
   Task 3: Region Map.  This may be related to how it is displayed on world maps.
   Task 4: Map Element probably not be best name.  Will be renamed once it is decyphered.
   Task 5: Figure out what LayerType values are valid and what they mean.
   Task 6: Figure out what soil knows about itself.
   Task 7: Verify and clarify descriptions in Rock data.
   Task 8: Geology unknown word.
   Task 9: Subregion Unknowns preceding Name.
   Task 10: Subregion Unknowns following Name
   Task 11: Subregion Unknowns following Map Grid Coordinates.
   Task 12: Subregion Unknown 192 byte block.
   Task 13: Subregion of unknown repeating shorts.
   Task 14: Figure out the dfsubregionUnkStruct20 data structure.
   Task 15: Figure out unknown at end of subregions.

Getting Started:

• Turn off compressed saves in init.txt, and use vanilla RAWs
• Get 010 Hex Editor by Sweetscape
• Get the latest DwarfFortress3407Template.bt from the dfWorldDat SF project.
• Read the 010 Editor help files about scripts and templates.
• Gen a generic pocket world, open the world.dat file in 010, load the template, run it.
• It will fail somewhere in the subregions.  Take a look around, get a feel for it.
• Now is a good time to skim through the whole template file, to understand what it is doing.
• Dive in.

I've documented the template file fairly thoroughly.
Some things I learned with dfWorldTinker have not been incorporated yet, but not much.
If you want to contribute, come back to this page to see what tasks have been assigned, then pick one still open and PM me that you're working on it.
(All of them are open right now.)
If there is enough interest, I'll get the SourceForge page better organized.
If there is a very large amount of interest, I'll move over to GitHub.

[zilpin]

(placeholder)

[zilpin]

(placeholder)

[zilpin]

(placeholder)

[zilpin]

One last note, back before civ sprawl was implemented, I was able to read the Site data as well (caves, cities, towers).  Some of the complications with subregions may be due to cites spanning multiple embark regions, farms, and all that was added later being stored in the subregion data.
Worth thinking about, but until you've gotten into viewing the data in hex, that probably does not mean much to you.

[Alem]

I heartily approve of this concept. I have zero skills to help you, but I highly approve. World Tinker is handy, but feels a little on the temperamental side currently, and apparently found at least one of my fortresses delicious

[Grimlocke]

Not sure how I missed this, but following it now. This could save loads of time with testing mods, and save quite a few of those fortresses I end up abandoning due to some fault in my attempts at modding.

Sadly best I can do to help you is to cheers you on from the sidelines, as I am programming language illiterate.

[Thundercraft]

Posting to follow this thread.

[darius]

Pls add power goal: ability to synchronize two (and more) different braches of same world (reassigning id's of creatures and artifact items) so that it could be synced between many players thus forming an indirect multiplayer mode (e.g. i killed something so update server that he is dead, and tell legends about my character and so on)
Powergoal because hard to achieve ( almost everything must be found out) and because it would be awesome

[zilpin]

All goals at this point will be in reading and understanding the file format.
This is a discovery project, at the moment.  After enough is understood about the file format, an API will be designed.  Then a command line program using it. 
If sufficient people step up to help with the trial and error hex editing.

So far, no one has.

[edit]
Or, if the Almighty Toad smiles upon us, and tells us what the file format is.
Savegame reading/writing code, even without required header files, would have no risk on his part and save us a great deal of time.
Not likely, but it will be in my prayers to Armok to speak favorably with his favored son 

[Artanis00]

Or, if the Almighty Toad smiles upon us, and tells us what the file format is.
Savegame reading/writing code, even without required header files, would have no risk on his part and save us a great deal of time.
Not likely, but it will be in my prayers to Armok to speak favorably with his favored son 

I believe his no API stance extends (or would extend) to this kind of thing, though slightly mitigated by the fact that the save format isn't as flexible as internals (he has to consider backward compatibility and all).

[zilpin]

Several months, but with only 32 downloads of the template file, and no one volunteering, this is not going anywhere right now.

Anybody who can contribute, need you.

[Dienes]

Oh golly this looks neat. There was a mad plan discussed that could be very fun but would need to know what was going on in world.dat to happen. I did some looking but must have missed this so I had mostly given up in despair.

My hex editing is rusty but I should have some time next week to see if I can made some sense of this.

Prayers and catnip for Scamps will probably pay off before Armok.

[Andux]

I did some work on documenting DF's file formats earlier this year; never got very far, but maybe it can help fill in some gaps:

http://df.magmawiki.com/index.php/User:Andux/Format_research

[Ghills]

The hex editor program you recommend is only free for 30 days.  Does your program need that hex editor, or would another one work?