Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[alfie275]

I've had an idea regarding the multithreading system:
Basically the possibility to allow two instances of DF to interact, possible via the file system. It would allow forts to dispatch and accept caravans to and from other forts as well as migrants.
If this were implemented you could simply have two sites adjacent to each other rather than one big site, effectively splitting it across two threads.
There'd obviously be issues, but it'd also be a good step towards any kind of multiplayer interactions that may occur in the future.

[Telgin]

This has been suggested a few times before I believe, and there are a fair number of problems with it.  Synchronization is a big one: what happens when you pause one instance?  Pause the other?  What if they get out of sync in other ways?  Which one is the master?  Does only one maintain world state?

I'm pretty sure Toady has shot down any chances of multiplayer as well.  Largely but surely not exclusively for the reasons I just mentioned.

[alfie275]

They'd be stored in a folder until the recieving fort unpauses, then you'd get them all at once or with a random delay.

[Jeoshua]

This method is already what dfhack, stonesense, dfusion, dwarf therapist, and many other tools do.  So long as the one process states openly that it is doing something (like pausing), the other can respond accordingly.

I see this as a fairly good idea, but it isn't exactly a "simpler" alternative to multithreading.  It's multithreading, using two executable instances.  Same thing as far as difficulty is concerned, albeit maybe a bit more portable to other operating systems.

[thisisjimmy]

Multithreading isn't actually that hard.  I don't think there's any need for a simpler alternative.  Toady just needs to identify the performance bottlenecks and see what would benefit from multithreading.

[Telgin]

Multithreading isn't actually that hard.

Depends entirely on what you're talking about.  Some things are exceedingly easily parallelized, like a GPU pipeline.  Some things.... not so much.

There are two big problems with parallelizing Dwarf Fortress:

#1: The game wasn't designed with it in mind from the get-go.  Adding it back in, after this much code has been written, would be exceptionally difficult even if the game was easily parallelized.

#2: The game probably isn't really all that easily parellelized.  Much of the game update step probably could be split between several threads, but it's still a nontrivial thing.  There are going to be a lot of race conditions that have to be considered.  Things like having two dwarves move onto the same tile, for example.  Right now it happens serially so there's no chance of two dwarves trying to say they're the only creature on the tile.

Of course, there's more than that.  Things like items being marked for tasks, jobs in general, tile occupancy... there's a lot of stuff you have to consider.


Overall it's certainly doable, but it's unfortunately going to be a lot of work.  Profiling and identifying the bottlenecks will just be the first step on a long road.

[thisisjimmy]

#2: The game probably isn't really all that easily parellelized.  Much of the game update step probably could be split between several threads, but it's still a nontrivial thing.  There are going to be a lot of race conditions that have to be considered.  Things like having two dwarves move onto the same tile, for example.  Right now it happens serially so there's no chance of two dwarves trying to say they're the only creature on the tile.

But you wouldn't just try to parallelize everything.  That would be ridiculous and counterproductive.  You only benefit from speeding up the bottlenecks.

So, for example, you might parallelize pathfinding.  It could potentially be as simple as (borrowing the spawn and sync keywords from Cilk):
for each creature that needs pathfinding
    spawn pathfinding(creature)
sync

Granted, it might be somewhat more annoying to set up since DF wasn't designed with parallelization in mind, but it's still shouldn't be that hard.  The main barrier is probably that Toady doesn't have that much experience with multithreading.

[Jeoshua]

jimmy, it's not as easy as you claim.

So what does the main instance do while it's waiting for the pathfinding to complete?  Just... sit there?  Go on to something else?  What if one of the pathfinder threads fails?  Can't find a path?  Paths it's Dwarf into something else that's currently occuring, like a liquid (magma) flow?  There are many possiblilities that would need to be taken into account.

Adding multithreading would be a complete rewrite of the engine from the ground up, which is something Toady is (understandably) not anxious to do.

[thisisjimmy]

So what does the main instance do while it's waiting for the pathfinding to complete?  Just... sit there?  Go on to something else?

What main instance?  That's not how multithreading works.  In the pseudocode I posted, each pathfinding task is added the thread pool and executed in parallel.  All cores get used.  None are sitting idle.

What if one of the pathfinder threads fails?  Can't find a path?  Paths it's Dwarf into something else that's currently occuring, like a liquid (magma) flow?  There are many possiblilities that would need to be taken into account.

DF preprocesses the map to mark unconnected regions.  Pathfinding isn't run if there's no path.  There's no issue here.

[ccasper]

Hi I've been playing DF for a couple of years now, but i have only now registered to the forums to add my opinion to this thread. It was the most recent thread i could find discussing multithreading.
In my opinion DF is a great game, and it will only get better with future updates but the game has one major flaw and that is performance. I had to abandon many fortresses just because they got too big and the FPS drop became unberable and im not alone. One possible solution is getting a better pc with a monster CPU but the problem with singlethreaded applications these days is that singlethreaded performace improvment has really slowed down. If you read this article (http://preshing.com/20120208/a-look-back-at-single-threaded-cpu-performance) you'll see that the main focus for CPU manufactures is multithreaded performance. So we can't relly that a new CPU will greatly increase the FPS of DF. Another solution is opimization but it goes only so far and can be harder to do than implementing multithreading. Also i do not doubt that Toady hasnt allready done some serious optimizations. The complexity of the simulations involved in DF makes me really impressed it runs as good as it does .
So my oppinon is that regardless how difficult mulltithreading may be to implement to DF it may be inevitable and the sooner it is done the easier it will be to do it.

[MrWiggles]

Multithreading isn't actually that hard.  I don't think there's any need for a simpler alternative.  Toady just needs to identify the performance bottlenecks and see what would benefit from multithreading.

In a DF Talk, ToadyOne spoke on how Multithreading wouldnt really benefit DF. Considering the recent SCIENC THREADS in DF General sub forum, it seems that major bottle next for DF is Ram speed and cache space.

[slothen]

I've had an idea regarding the multithreading system:
Basically the possibility to allow two instances of DF to interact, possible via the file system. It would allow forts to dispatch and accept caravans to and from other forts as well as migrants.
If this were implemented you could simply have two sites adjacent to each other rather than one big site, effectively splitting it across two threads.
There'd obviously be issues, but it'd also be a good step towards any kind of multiplayer interactions that may occur in the future.

How is this an alternative?
How is this simple?

[Jeoshua]

My point exactly.  It's not really that different at all.  In fact somewhat more complex of an option due to the two instances being separate executables.

[thisisjimmy]

Multithreading isn't actually that hard.  I don't think there's any need for a simpler alternative.  Toady just needs to identify the performance bottlenecks and see what would benefit from multithreading.

In a DF Talk, ToadyOne spoke on how Multithreading wouldnt really benefit DF. Considering the recent SCIENC THREADS in DF General sub forum, it seems that major bottle next for DF is Ram speed and cache space.

You're talking about this from DF Talk #9:

It is my understanding that the SDL stuff is now multicored, with the graphics and the graphics display and so on. Now obviously people, when they're asking about it, that's one important point but people are curious like 'why can't I have seventeen dwarves pathing at once' or 'why can't you take that stupid ass weather simulation and make that go happen on some core, preferably not at all'. It's complicated, and from what I gather it would be a really difficult long project and it's probably not going to happen. That's what I gather. Now there are some things, like these micro multithreading of a smaller process just to get through one loop faster, it can break it up and then come back without it being a case of running path finding at the same time as you're running a fluid simulation, and that stuff, the more I know about that the better, probably, and maybe that's feasible. As for splitting up the path finding and stuff: some people have discussed about how that might work, but as for whether anything is actually going to come of that, I wouldn't bet on it anyway.

The quote shows that Toady doesn't or didn't know that much about multithreading.  From my experience, it's likely easier than he thought.

Whether multithreading parts of DF will actually improve performance is another concern.  The only way to really know is to measure.  However, if cache space is important, then multithreading may help simply because it increases the amount of cache space available.

[MrWiggles]

Hrm, misremembered.