Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Markavian]

Hi I'm Markavian, (aka John Beech), and once upon a time in 2008 I made a website called the DF Map Archive (https://mkv25.net/dfma/) featuring a flash based viewer in collaboration with an encoder programmed by ShadowLord. I then continued to build out the site on my domain. You and the community may have used it from time to time...

Between then and now, I went from hobby coder, to aspiring software engineer, to where I am now; a software team lead at a big international broadcaster, married to my beautiful wifey (who is also an engineer), and adopted two cats Ezio, and Icarus. Basically life happened.

Meanwhile, the rest of the world moved on as well; Flash doesn't work on mobile, and is being blocked or deprecated by most browser vendors. This means that the map decoder / viewer doesn't work for most peple, and the the sizeable map archive 11800 maps, and 33400 points of interest are at risk to the future internet community.

With problems, come solutions... and I'd like to call out and thank the work of Jacob Blomquist, who has made a P5.js and HTML5 Canvas version of the map viewer, that should run from modern browsers. He contacted me recently, and reminded me of the on going need to support the development of the map archive.

So that brings me to this post...

Spoiler (click to show/hide)

Markavian is taken by a fey mood!

https://github.com/df-map-archive/dfma-map-file-archive/

DFMA Map File Archive

Motivation
To help share, and preserve Dwarf Fortress stories for aeons to come.

Purpose
The DFMA is place to upload maps, tell stories, tracks points of interest, comments, and community favourites - bringing to life fortresses through interactive snapshots submitted by the community.

Underlying Technology
The Dwarf Fortress Map Archive (DFMA) is a repository and online viewer for player created maps exported from Dwarf Fortress. The site revolves around an optimized file format (.FDF-MAP), which can produce files 1/2000th the size of the ouputted bitmaps (for a 28 layer map) and 1/20th the size of equivalent PNG files. This makes them small enough to upload and transfer across the web without chewing up bandwidth.

Map File Archive
The map file archive is an archive of dwarf fortress maps (.fdf-map files) and any associated descriptive data, such as author, submitter, fortress name, map year, map season, submission date, points of interest, and description.

There are gaps in this data, this is a stalled project that needs love, work, and attention.

Project Roadmap
Any/all future work is being tracked through the shared github project page for through the df-map-archive organisation on github.

See: https://github.com/orgs/df-map-archive/projects/1

Timeline
2017 November
- Initial idea to create HTML/JS version of Map Archive discussed with Casey Link
- Created df-map-archive organisation on github
- Created dfma-map-file-archive namespace for archiving maps

2018 April
- Download and preserve .fdf-map files from https://mkv25.net/dfma/ to local machine
- Archiving stalled due to file size restrictions on repositories - lost motivation

2019 February
- Archiving stalled due to file size restrictions on repositories
- Looking for solutions; may just pay the cost

2020 May
- Jacob Blomquist creates a new P5.js and HTML5 Canvas map viewer for fdf-maps, and releases it on an MIT license: http://blomquist.xyz/dfma/ / https://github.com/JacobRBlomquist/DFMA-Viewer-HTML5
- Created a new DF Map Archiive project space to track and prioritise work

New Plan
- Break repo down into multi year backups, e.g. pre-2008, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, and 2019
- Automate process of creating multi-repo backups using template script
- Communicate intentions on forum

Footnote
The map archive has slowed in its use, but is an excellent resource for early Dwarf Fortress gameplay, and has allowed many great stories to be shared over the years. I hope that by providing this archive of the website that we can build a new site that will continue to support these stories into the future.

Fey Demands
There's a lot on my shoulders here. I want to try and open up the archive and provide long term support to the community by backing stuff up in Github as a place where other hackers, coders, programmers, engineers, etc. can go use and make stuff. It's the community data after all, my website is just hanging it all together.

I'm not look for money, or adverts, or any programming expertise. I think I'm just in need of support at this stage. I've got the outline of a plan worked out, and I've got lots of data to move and push around, and hopefully I'll have lots of exciting news to share with people. Maybe its just the ideas and inspiration I need, and that'll hopefully make this all worth while.

Also while I'm here, and incase they read this, congrats to Tarn, Zach, (and the community) for keeping Dwarf Fortress going for all these years. Among all the games I've embraced DF has carved a very precious rune on my heart.

[Uthimienure]

While I've only played for a couple of years, it is very clear there is great value to your project!
A while back I did check out the DFMA and enjoyed browsing it.
Thank you for your labor of love and I sincerely hope you can maintain the motivation to carry this through.

[Blomquistador]

My name is Jacob Blomquist (Blomquistador).
I wanted to revive the DFMA mostly because I don't want these amazing builds and fortresses to be lost just because flash has lost support.
I also wanted to revive this project so that it would be ready for the eventual Steam release of DF.
I'm a longtime player of Dwarf Fortress, but this is the first contribution I've made to the community.

This is also a rare opportunity for us to add features and functionality to the map viewer. I would be happy to hear any contribution requests.

All in all I hope that this will revive interest in the DFMA and allow for more DF history to be shared amongst the community.

I also want to echo Markavian in thanking Tarn and Zach Adams for building such a profound and influential game that has touched the hearts of so many. I hope that the Steam release will go well and that we'll see more newcomers who can tackle the learning curve.

[Salmeuk]

Wow, I had my heart all set to be broken, expecting this thread to be the announcement that the archive would be finally shutting down . . but no!

I've uploaded maps to DFMA since it's creation. It's wonderful to search back and see all the strange and wonderful things I was building so long ago. And before the early 3d visualizers there was DFMA's ability to see a vertical y-slice of your upload, allowing all the strange megaproject sculptures to be visualized despite the impenetrable ASCII. Indispensable, really. 

So, from the bottom of my heart, thank you greatly for your donated time, effort, and hosting bandwidth!

I created a thread a few years back that you might enjoy reading, since many people shared their enjoyment of DFMA.

[Markavian]

Wow, I had my heart all set to be broken, expecting this thread to be the announcement that the archive would be finally shutting down . . but no!

I've uploaded maps to DFMA since it's creation. It's wonderful to search back and see all the strange and wonderful things I was building so long ago. And before the early 3d visualizers there was DFMA's ability to see a vertical y-slice of your upload, allowing all the strange megaproject sculptures to be visualized despite the impenetrable ASCII. Indispensable, really. 

So, from the bottom of my heart, thank you greatly for your donated time, effort, and hosting bandwidth!

I created a thread a few years back that you might enjoy reading, since many people shared their enjoyment of DFMA.

Thanks Salmeuk, I started reading through the first page, and that's really heart warming, thank you again fingers crossed I can get the site back open for business.
I'm getting stoked by the DF Steam release, feels like a big time for the community.

[Fleeting Frames]

I was expentant of the archive being abandoned (already so, in fact, with your latest uploads being decade ago and latest cmment being half a decade ago on DFMA) with flash support dropped. It makes me glad this is only not so, but that there's already a translating to HTML5 project that I was unaware of (gj & gz to Blomquistador).

Additionally, whenever you're ready, you might want to post the update onto reddit as well; while DFMA is linked on the sidebar I still see maps shared as collection of images on imgur, which may make sense for certain graphic set cases but is frequently done with ascii as well. Additional exposure would lead to additional users, I expect.

Though tbf I have to recheck "how did I use this again" every time I upload a map which might hurt usage numbers, so seeing if it can be simplified might be worth consideration -  for example mifki made 3d viewer which simplifies uploading to single copy-paste into dfhack console.

[Markavian]

I was expentant of the archive being abandoned (already so, in fact, with your latest uploads being decade ago and latest cmment being half a decade ago on DFMA) with flash support dropped. It makes me glad this is only not so, but that there's already a translating to HTML5 project that I was unaware of (gj & gz to Blomquistador).

Additionally, whenever you're ready, you might want to post the update onto reddit as well; while DFMA is linked on the sidebar I still see maps shared as collection of images on imgur, which may make sense for certain graphic set cases but is frequently done with ascii as well. Additional exposure would lead to additional users, I expect.

Though tbf I have to recheck "how did I use this again" every time I upload a map which might hurt usage numbers, so seeing if it can be simplified might be worth consideration -  for example mifki made 3d viewer which simplifies uploading to single copy-paste into dfhack console.

Wow, that 3D viewer is amazing. I think I always wanted to do something like that with the existing data on the map archive. I had experimented with an isometric viewer using a custom tileset, but I could never really iron out the bugs in the system. I think once I get over the hill of "fixing the map archive", it would definitely be worth looking at the "upload my fortress" journey - make it more drag and drop ... and lets be real, people's bandwidth is much better these days, if we tried, we could probably do something with the actual game/map data (like the exporter script) rather then just compressing bitmaps. I think the data already in the DFMA (.fdf-map files) would be compatible with the 3D viewer as well - we've got colour information and layered blocks, so rendering cubes should be easy.

Good shout on reddit. I think I came back here to the forums first to dip my toe back into the community.

My update for this week:
- Updated the roadmap - got lots of ideas on the project page, got some more to add now
- Recovered the SVN repo for the old DFMA, wasn't much in there, but I've got a snapshot of the code dfma website in github now
- Started linting the dfma website using modern techniques, its slow going, I've got another 60 files to go, its quite painful - setup a deployment pipeline to make site updates easier / more controlled
- Started this morning putting together a spike for Jacob's decoder/viewer in https://github.com/df-map-archive/dfma-html5-map-viewer

Jacob made a few changes as well:
- Added progress bar and async
- Changed scaling factors
- Fixed a bug with smooth zooming

[Blomquistador]

...... and lets be real, people's bandwidth is much better these days, if we tried, we could probably do something with the actual game/map data (like the exporter script) rather then just compressing bitmaps.

At the very least it would be nice to implement the .BMP compressor online so that people wouldn't need to download the compressor to upload .fdf-map files.
Creating a DFHack script that exports even more information (such as an entity list, water flow, etc) would be amazing and we could do so much with it.

[Markavian]

...... and lets be real, people's bandwidth is much better these days, if we tried, we could probably do something with the actual game/map data (like the exporter script) rather then just compressing bitmaps.

At the very least it would be nice to implement the .BMP compressor online so that people wouldn't need to download the compressor to upload .fdf-map files.
Creating a DFHack script that exports even more information (such as an entity list, water flow, etc) would be amazing and we could do so much with it.

Completely agree; something to add to the R&D list...

Weekly update, mostly for my own benefit:
- Jacob did an update to the viewer; now packaged a bit more friendlier
- I set up a build pipeline to deploy the new viewer code to the DFMA website : https://github.com/df-map-archive/dfma-html5-map-viewer
- I've got a "working" wrapper of Jacob's viewer code hosted on my website - it looks identical; but I'm now in a position to start feeding in DF map IDs
  - See: https://mkv25.net/dfma/xdfmadev/parcel/index.html

What's next:
- I'm still fighting with PHP Linting of the old website - I need to do this so I can start quality control around the map viewer changes - I want to avoid pushing broken code to the website when I have a limited amount of time to fix things
- Ready to start integrating DFMA Maps with Jacob's HTML viewer, using the above code/project
- Need to work out how Points of Interest will be supported with the new viewer - its not enough to just load a map (that should be easy), we also need a way to navigate using the existing PoI system

Project page:
- https://github.com/orgs/df-map-archive/projects/1

[Fleeting Frames]

Oh, that's some fast progress. Few obvious notes that jump out to me:

- Display-wise, it seems the previous "multilevel" view in underground areas is gone, though.

- Navigation, might want to set the maps to be entirely navigatable with both mouse and keyboard (I'd suggest key placement that enables one to move around with one hand in both cases).

- POIs, current flash viewer had weird thing where logged out it directs you to a new page while logged in it updates the flash viewer. Limitation arising from lack of cookies, I assume? (Mind the GDPR.)

[Starver]

I've been meaning to PTW this thread for a while now, still not found a good thing to justify it, though.

The 'best' qualifying comment I've yet come up with is

Spoiler: (edit) actually, you can skip this bit entirely... (click to show/hide)

that an old machine with Flash on, rarely connected to the internet because I don't need it to be, tends to never do anything when booted up connected, but the next time (cable unplugged) it suggests it needs a Flash update, as if suddenly discovering this fact by spooky-action-at-a-distance. To be charitable, it's maybe discovering a behind-in-version one time, but only provoking the start-up 'checker' to report this when next booted, yet sequential online boot-ups never provoke the alert.

Plugging in and getting the splash-notice to do its thing leads to it successfully test itself in-browser at the end, like it does, no complaints) . Yet a later (month or so?) boot-online, boot-offline goes and provokes another update, regretably I've never thought to note if it's truly a new version of install file invoked.

Not plugging into ethernet and ignoring it also means no more prompts to update (for a month, or so, but a calendar month seems like a reasonable "pester the user, regardless" period to be programmed in, albeit in the faulty way that doesn't seem to work with my operational manner) regardless of booting/online status in the immediate interim.

(As contrast, the AV on the device moans at me if I've not updated for a couple of months, when I've refrained from connecting at all for that long, but updates itself merrily (give or take the seasonal demand to "renew my free licence or pay for bells and whistles (and synchronising with mobile devices, and cloud storage, and other things I neither jeed nor could effectively us) pretty please!" for which I have to work not to update it to "paid version on free trial" mode) immediately upon touching the outside world, if necessary, and complains not one jot (beyond the immediate) if I curtail it and reboot without ethernet in. Which is more logical.)

As this is my designated DF computer (well behind the curve, in spec, but sufficient and convenient) it's what I use for occasional (not uploaded for a long time) .fdf-map creation/viewing, but with an HTML5-capable browser on there, I'm definitely interested to see where this is all going, how much I may need/might be able to change this mainly offline 'working' practice of mine under the new scheme.

Which is why I'm PingTW. With absolutely no need for any direct response to me on the above rambling screed that is mostly only talking about things you have no control over, and I'm obviously just venting for the sake of it. 


I really should consider using the DFMA more, though, to make it obvious I support it, and you. Maybe when I've decided I've got a good 0.47.x embark going, with my Super-Turtlable design philosophy nicely established, or something. *shrug*

[lethosor]

Wow, I had my heart all set to be broken, expecting this thread to be the announcement that the archive would be finally shutting down . . but no!

Same here. DFMA was one of the first DF sites I signed up on. Glad to see it's still alive!

Regarding the map export side of things: I've done some work with DFHack, as well as cross-platform DF-related tools in Go, and could help out there if needed. The latter approach would almost certainly be a command-line tool, and I guess a DFHack plugin/script would be too, although people might be more accustomed to that. (Is the .fdf-map tool command-line-based? I've never actually used it, as I don't run Windows.)

I would be willing to take a look at frontend stuff too, although it looks like you might have that covered pretty well. (And despite my free time having increased in the past couple months, the number of side projects I've taken on has also increased, so I can't make too many commitments, haha.)

[Fleeting Frames]

I don't run windows (to export maps; still have copy if I need it) either; I run the .fdf-map export tool in linux with mono --debug iirc. Though it tends to complain about nr of different tiles; which is fair enough given TWBT multilevel & transparency mean there's far less repeating tiles.

PS: Also, "I've done some work with dfhack" is amusing to hear (, given that you seem to be one of the most active dfhack core devs here).

[Warmist]

Wow, I had my heart all set to be broken, expecting this thread to be the announcement that the archive would be finally shutting down . . but no!

Same here. DFMA was one of the first DF sites I signed up on. Glad to see it's still alive!

Regarding the map export side of things: I've done some work with DFHack, as well as cross-platform DF-related tools in Go, and could help out there if needed. The latter approach would almost certainly be a command-line tool, and I guess a DFHack plugin/script would be too, although people might be more accustomed to that. (Is the .fdf-map tool command-line-based? I've never actually used it, as I don't run Windows.)

I would be willing to take a look at frontend stuff too, although it looks like you might have that covered pretty well. (And despite my free time having increased in the past couple months, the number of side projects I've taken on has also increased, so I can't make too many commitments, haha.)

Ditto.

Composed a few mails and then deleted them to Markavian with title: let's rewrite CMV viewer in js. Mostly because i'm not a webdev and my js would be no-libs-scary mess. However "Warmist is ecstatic upon hearing this news".

Edit: i wonder how possible would it be to integrate something into forums? Direct iframe with movie viewer would be awesome!

[Markavian]

Oh, that's some fast progress. Few obvious notes that jump out to me:
- Display-wise, it seems the previous "multilevel" view in underground areas is gone, though.
- Navigation, might want to set the maps to be entirely navigatable with both mouse and keyboard (I'd suggest key placement that enables one to move around with one hand in both cases).
- POIs, current flash viewer had weird thing where logged out it directs you to a new page while logged in it updates the flash viewer. Limitation arising from lack of cookies, I assume? (Mind the GDPR.)

Display-wise: I'd forgotten about the multi-level view; I'll let Jacob know and we can get support for that back in. Loading maps from custom URLs and zoomTo are what I've asked him to look at next. I could help him directly with viewer development, but I figure while development is fresh in his mind, I can focus on tidying up the site, and sort out the direct integration with PHP + Database.

Navigation: I've let him know about the onscreen mouse controls; I can always remap the keyboard in DFMA, and eventually make this a user-profile setting if people want.

POIs: Yes I hated that bug, I never got to the bottom of what caused it, some sort of browser change - the links should dynamically update the map, but they "fall back" to a full page refresh.

@Starver - that whole comment is fascinating. I can totally imagine it.

Wow, I had my heart all set to be broken, expecting this thread to be the announcement that the archive would be finally shutting down . . but no!

Same here. DFMA was one of the first DF sites I signed up on. Glad to see it's still alive!

Me too. Keeping the server alive is the easy bit; plenty of money, hardly any time - I'm working hard, sacrificing computer games at the moment.

Ditto.

Composed a few mails and then deleted them to Markavian with title: let's rewrite CMV viewer in js. Mostly because i'm not a webdev and my js would be no-libs-scary mess. However "Warmist is ecstatic upon hearing this news".

Edit: i wonder how possible would it be to integrate something into forums? Direct iframe with movie viewer would be awesome!

Oh please send the emails I'll add cards to the project page I think technology has moved on sufficiently to do a purely HTML/JS/Canvas solution - Jacob's map viewer is proof of that. My .NET CMV editor is... somewhere, that has a decoder in it. Found it: https://github.com/Markavian/CMV-Editor/blob/master/CMV/CMV.cs