Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[NW_Kohaku]

I've been trying to put my head to the concept of how to abstract the worldgen process without having an actual loss in the actual detail of the legends and adventure modes. 

This thread includes some science on the problem areas, and we have identified two major areas of problem to be werecritters and the trading that takes place in the new towns.

Toady appears to be currently working on trying to fix the problem of the slow worldgen, so this may be a chance to help and make a useful suggestion to improve game performance.

I want to try to get people brainstorming and critiquing ways to perform abstraction without a loss in actual meaningful data, including the ability for unusual events to occur in worldgen or for things like a dwarven dresser to appear in a human town because of a trade that took place 180 years before the adventurer was born.


First off, I want to suggest that werecritter attacks should have a random chance to trigger the deployment of a wave of monster-hunters from the victimized site.  This means promoting a nameless nobody to historical figure status if need be to deploy them.

The problem with werecritters is that they multiply exponentially, are capable of rampaging 3 times in a year, and eventually wind up with dozens of lairs around any given town and hundreds of attacks on a town per year. 

Setting up a counter-balancing force that only becomes active when the werecritters themselves are numerous is the most natural way of counterbalancing an overwhelming force, until it reaches a nice stable terminal velocity.


Secondly, we need a sanity check on the amount of random totem crap a single individual can own and wear.  The problem is especially apparent on werewolves, but even random historical soldiers must look like some sort of voodoo shaman they have so many whatever-man bone totems hanging off their neck, fingers, wrists, etc. 

Keeping only the 10 or so most valuable or important totems keeps things sane.


Thirdly, werecritter rampages (and rampages in general) need to be much less detailed.  We don't really need 10,000,000 lines in Legends Mode recording the same two lines over and over of a werecritter attacking some random civilian who manages to run away without getting harmed at all.  We can make do with much less rampages in general that result in many more actual maimings or kills, since those are the only rampages that matter, and it seems as though something like half of these rampages wind up doing nothing, and most of the rest only kill a chicken.


Fourthly, the same needs to be done for buildings in worldgen.  There is a bug involving "abandoned shops" getting overrun by tables and driving residents out because there is no longer room for them.  This is because shops apparently actually have their inventory tracked during all of worldgen, and when tables are built to display goods, they are never removed, even when shops are abandoned or the previous owners die.  This winds up filling the shop with tables until it becomes completely unusable, and whole buildings are abandoned as table graveyards.

While having a cleanup would be nice, there's an even simpler solution to this - do not even stock a town's shops until the end of worldgen.  There is no reason we need to be keeping track of the number of display tables in legends mode, and they only need to be genned when the world is ready for the player to start exploring.

This brings me to the next point...


Finally, we do not need to track overall totals of every type of material.

Seriously, consider how many different types of creature or individual stones we have.  Now consider all the different kinds of leather created from that, and then the different kinds of leather armor and different prepared brain tallies for each creature, and the different numbers of chests made of each type of stone... 

Now consider those tallies get traded around in worldgen, and you start to see why trading is such a bloater of your worldgen. 

What the game could use instead is a general pool of what resource types are available to a given region, and what sorts of craftsmen are working there to generate a probability pool for each type of item, its quality, and its material. 

Instead of trading individual items, what happens is that the trading pools get linked, and a small amount of everything that the trading partners have access to can be transferred between one another.  This means that if 10% of goods are available for trade, and the probability pool of one town includes materials or finished goods that are not available in another town that can trade with the first town, the second town may have a limited access to the probability pool of the first town.

This limited access to the probability pool can then be further traded, although with much more limited access unless something like a caravan is set up for the specific request of some good that is rare and useful enough to make such a thing viable.

The game will have to record when craftsmen die or when towns lose access to a certain type of material or lose connection to a trade partner to state that certain types of items (like the probability of getting a masterwork silver craft from a dwarven civ) so that there can be a small probability of finding something like a master craftsman that created the best silver crafts in history who lived 300 years ago, and whose crafts are spread within the region of the trade routes that existed in that particular time.

Because all that is being traded is a probability of a given set of items, it should be faster to handle the fungible nature of what is actually being traded, and the actual content of the warehouses of any given town or the actual furniture of a town don't need to be actually generated until the first time you actually walk into a town. (You just need to have a controlled, saved seed for the RNG to generate the positions of the stuff.)

[Kento]

If this is a brainstorm, let me submit some hurried ideas.

Based on the idea of generating items at the end of worldgen, and the trade pool: Instead of regions, how about each site (fortress/city/whatever) generates a generic profile based on the materials it has access to, and the civilization species, and perhaps a couple random tokens to boost or halt production of certain goods to give the location character. Population would be tracked for each site, and average population would be the basis for basic amount of production.

For example, if a site should produce .08 furniture for every citizen every year of its existence, the profile might generate the information (simplified for an example, of course)
furniture: .05 wood, .03 stone, .03 tables, .03 chairs, .02 weapon racks
So that 5/8ths of the furniture is wood, 3/8ths is stone, etc. 3/8ths is tables, etc.
But perhaps to give places more individual character, maybe a city prizes oak furniture, so a profile might include the information
furniture: .03 oak, .02 wood, .03 stone

Thresholds may also be necessary, a seven dwarf fortress does not produce 10% of the metal items of a 70 dwarf fortress.

The downside of this, if used as the sole way of tracking goods, is that it would make it so that complex industries that might need trade between two locations would be difficult to account for, something would be needed for addressing why a city might want goods from another city, and survival based on trade of weapons and consumables would probably need to be tracked separately as well.

But if that's not being done, why not have an abstract "weapon/armor supply" tracker for each site, as well as food and drink until the end of worldgen? Do we already have that?

An idea for masterwork items: Instead of tracking craftspeople, just generate artifacts, then use the artifacts to generate masterwork items. The person who generated the artifact probably did more in their lives than just make the one item, or the artifact may just be the best work of a rich period of the local artisan culture. These masterwork items can just be added as bonuses to the base supply from the main abstract inventory tracker.

[Kento]

The biggest challenge, I think, is finding a way for worldgen to be dynamic, rather than just having static growth. Cities should change hands based on their ability to defend themselves and the strength of aggressive neighbors, and that should be a product of their economy. Cities should face famine, etc. So tracking each city's food and drink (I think this can be a single metric), weapons and armor (multiple metrics for different weapon types and materials seem necessary, though "leather and bone armor" (those with access to leather almost always have access to bone)  "copper armor" "iron armor" etc is probably fine, "silver blunt" "copper blunt" "silver edge" "copper edge" "silver mixed blunt/edge" "copper blunt/edge," "silver range" "iron range," etc. That level of complexity may be desirable but I don't think we need more than that.

[Kogut]

Abstracting worldgen trade may not be the best idea as AFAIK in version 1.0 history is supposed to run not only during worldgen - so we may end with *adamantine toy boat* exported half world away. But townsfolk ignoring werefoo is ridiculous.

[NW_Kohaku]

Abstracting worldgen trade may not be the best idea as AFAIK in version 1.0 history is supposed to run not only during worldgen - so we may end with *adamantine toy boat* exported half world away.

Why is this a problem?

Having some random item, potentially from a player-built fortress, which has travelled halfway across the world because of trade routes would actually be kind of cool.

The only problem would be that you would have to have a special set of specific instances of anything that a player actually interacted with, which, instead of being a completely random mix-and-match materials and item type and improvement on-the-spot generation, would instead have to pull from a list of created items from your fort or handled by your character.

The general gist of it is that you can operate by the rules of Schrodinger's Gun, and simply keep track of anything that hasn't already been revealed to the player in a very abstract manner where everything is fungible, but then treat everything that has been specifically manipulated by the player (or artifacts or the like that have to be unique, and whose properties are important for shaping worldgen) as being distinct and individually-tracked... provided that they are within some reasonable sanity-checked limit.

I think part of the problem is that DF generally lacks sanity checks or counterbalancing forces in general.  Things are allowed to get to ridiculous proportions, like 80 pages of vampire trophies, because broad-picture extreme case scenarios are rarely struck with some offsetting force until it becomes a clear problem.

Granted, this is my own personal outlook on the world speaking, but I think something that DF keeps coming back to is that Toady is constantly so focused upon the fine grain of the bark on the trees that he keeps forgetting to keep track of the forest.

[NW_Kohaku]

Based on the idea of generating items at the end of worldgen, and the trade pool: Instead of regions, how about each site (fortress/city/whatever) generates a generic profile based on the materials it has access to, and the civilization species, and perhaps a couple random tokens to boost or halt production of certain goods to give the location character. Population would be tracked for each site, and average population would be the basis for basic amount of production.

For example, if a site should produce .08 furniture for every citizen every year of its existence, the profile might generate the information (simplified for an example, of course)
furniture: .05 wood, .03 stone, .03 tables, .03 chairs, .02 weapon racks
So that 5/8ths of the furniture is wood, 3/8ths is stone, etc. 3/8ths is tables, etc.
But perhaps to give places more individual character, maybe a city prizes oak furniture, so a profile might include the information
furniture: .03 oak, .02 wood, .03 stone

This is exactly what I mean: A probability pool of each type of item.  It tracks the probabilities of any given items and their probabilities of being of any given material. 

When trade occurs, a small probability of items from the other probability pool "seeps" into your own probability pool based upon the amount of trade.  Items would have to be eventually lost as new items were created, and items like food should be constantly consumed, forcing constant replacement, and meaning that food items from 100 years ago probably shouldn't still be around.

Thresholds may also be necessary, a seven dwarf fortress does not produce 10% of the metal items of a 70 dwarf fortress.

The downside of this, if used as the sole way of tracking goods, is that it would make it so that complex industries that might need trade between two locations would be difficult to account for, something would be needed for addressing why a city might want goods from another city, and survival based on trade of weapons and consumables would probably need to be tracked separately as well.

Industry is usually performed on the site of where the raw materials are extracted in this game. 

It should be possible for specific agreements to be made that dramatically increase the rate of trade of specific items along a trade corridor, which may be also reflected in an increased rate of training in the specific skills (such as more weaponsmiths being trained in the one dwarven fort that can make steel exporting steel items down to the humans), and that can still be handled while keeping the game in the abstract.

But if that's not being done, why not have an abstract "weapon/armor supply" tracker for each site, as well as food and drink until the end of worldgen? Do we already have that?

No, we have concrete numbers of weapons and food.  We should still track the total tally of food in worldgen as a concrete number for the purposes of starvation, of course, and generally how many people can be armed and of what quality weapons for the purposes of worldgen battles, but this can handle the specific details of who had what specific item only when it becomes specifically important (such as a historical figure's equipment being an important detail, as opposed to what specifically was in the quatermaster's tallies at the armory at the time).

An idea for masterwork items: Instead of tracking craftspeople, just generate artifacts, then use the artifacts to generate masterwork items. The person who generated the artifact probably did more in their lives than just make the one item, or the artifact may just be the best work of a rich period of the local artisan culture. These masterwork items can just be added as bonuses to the base supply from the main abstract inventory tracker.

Worldgen artifacts are not the same as fortress mode artifacts. 

Worldgen artifacts are slabs that turn you into a necromancer.  Fortress artifacts are the result of strange moods.

Besides, worksmen can hit legendary without a mood. 

[NW_Kohaku]

Still working on world gen. Made it a little faster, nothing spectacular (200 medium years in 10 minutes). There's more I can do there, in bits and pieces, but I'll probably go back to regular new bugs now for a bit, and then we can do another release.

I have to wonder if he read some of the bug reports or science threads...  If he doesn't realize that werecritters are a major problem, it's... well, it's a major problem.

[King Mir]

I actually think you do wanna track what animal the leather comes from. But you could have it track a generic "armor" item, instead of items that go on each body part. When a caravan enters a fortress, or an adventurer enters a town, you could resolve the armor to a pair of boots or a breastplate.

Similarly furniture could be abstracted into this generic object type. But keep track of the material, so towns will look different or the same depending on where their furniture comes from.

I'm not sure which and how many items can be abstracted this way. If there aren't enough, the optimization isn't worth while.

[NW_Kohaku]

Any optimization is worthwhile.  Any time you can do the same thing you were doing before, but better, and with no loss of functionality, you have achieved something worthwhile.

What you could do is simply track animal kills that are butchered, and add that to the probability pool.  Then, when the game is looking for a leather to make a clothing out of, it simply takes it randomly from the pool at large.  The same with meat or bones or other organs.

Since most of those things will decay, except maybe the bone carver stuff, you probably will wind up only needing a very vague "animal parts" pool that gets filled and depleted rapidly and repeatedly over the course of worldgen, and which only needs solid numbers at the end of worldgen, which is exactly what abstracting the early years is all about.

[Neonivek]

I think one good long term fix for Werecreatures is that later on. they turn into a group a "Werecreature Pack".

[Kogut]

Any optimization is worthwhile.  Any time you can do the same thing you were doing before, but better, and with no loss of functionality, you have achieved something worthwhile.

No, sometimes time needed to do this is so large that it is pointless (multithreading may be in this area).

[Sowelu]

Yeah, a day job in the software industry knocked the will to optimize right outta me.  You say "Here's how slow it is, is the value acceptable?"  You make the call, and if the answer is "We can deal with it", you accept the harsh reality that programmer time is the most valuable commodity in the world.

Also, the pool sounds...weird.  I mean, it makes sense for this case, but I don't know if it's generic and extensible, and it seems like this kind of change really needs to be in order to work.

[NW_Kohaku]

Also, the pool sounds...weird.  I mean, it makes sense for this case, but I don't know if it's generic and extensible, and it seems like this kind of change really needs to be in order to work.

I'm not sure I understand what your qualms are.  Could you be more specific?

[King Mir]

Any optimization is worthwhile.  Any time you can do the same thing you were doing before, but better, and with no loss of functionality, you have achieved something worthwhile.

What you could do is simply track animal kills that are butchered, and add that to the probability pool.  Then, when the game is looking for a leather to make a clothing out of, it simply takes it randomly from the pool at large.  The same with meat or bones or other organs.

Since most of those things will decay, except maybe the bone carver stuff, you probably will wind up only needing a very vague "animal parts" pool that gets filled and depleted rapidly and repeatedly over the course of worldgen, and which only needs solid numbers at the end of worldgen, which is exactly what abstracting the early years is all about.

There is a loss of functionality. You're grouping what would be separately demanded products into one. That means the price of chairs relative to tables becomes fixed, which should not necessarily be the case.

Again keeping track of animals kills doesn't work if the leather is traded away from the city of origin. If a city in the pole is trading leather with a temperate city, then the temperate city should have penguin leather armor, even though there are no local penguins.  On the other hand a nearby temperate city that produces it's own leather armor will not have penguin leather. Keeping track of the stock of killed animals would not give that result.

In short, I do think that materials should be precisely tracked.

[NW_Kohaku]

There is a loss of functionality. You're grouping what would be separately demanded products into one. That means the price of chairs relative to tables becomes fixed, which should not necessarily be the case.

Again keeping track of animals kills doesn't work if the leather is traded away from the city of origin. If a city in the pole is trading leather with a temperate city, then the temperate city should have penguin leather armor, even though there are no local penguins.  On the other hand a nearby temperate city that produces it's own leather armor will not have penguin leather. Keeping track of the stock of killed animals would not give that result.

In short, I do think that materials should be precisely tracked.

You are assuming that there is no way to abstract concepts.

If I tell you that a car is travelling at 60 mph to a destination 180 miles away at a constant rate with no stops, do you have to model this by pushing a little car 6 tiles along a track per unit time until you hit a point 18 tiles away to figure out this problem?

That's the way that the game handles most items and transactions now.  It follows completely unnecessary steps that can be replicated through a more efficient mathematical model with no loss of resolution in the actual data the model produces.

It is simple calculus, of the sort that is very easily handled by computers, to simply look at the rate of consumption and rate of production, compare the rates, and see whether or not trade must and can take place to compensate for shortfalls, and adjust prices accordingly.  That merely brings the level of math up to the level that us average fleshy human brains unoptimized for math process math problems. 

You don't have to constantly re-tally every single orthoclase table differently from the microcline tables differently than the slate tables, and then adjust prices, and then handle trade from there.  You can simply match excess production to market capacity, and set trade routes (and fluctuate price) accordingly.  All this can be handled on the abstract.

You only need to worry about whether the leather was penguin leather or dog leather, which makes absolutely no difference in-game, and is not treated by the game in any way other than being just as completely fungible as the way I have been treating it in this hypothetical model, when you actually have the player walk into the town itself and look at what items are sitting on a leatherworker's counter.

Already, the actual physical positions of objects in the game are not recorded unless a player changes them - the insides of towers have their walls rebuilt at the time that the player walks up to them, not just the first time, but every time, because it can just rebuild the actual floorplan from a re-usable seed.  You only need to track the changes from the initial positions, and only if it is a location that will actually be tracked (like a lair, as opposed to a random spot out in the wilderness).

This merely tries to extend that same principle to the trade mechanics going on in worldgen, which the player will never, ever be able to see or really care about except as far as the actual end result effects are, which this abstraction still produces.