Alright, so...
I was recently working on a personal mod to improve coat colors and patterns in dogs and cats, and found myself with some questions about how genetics are handled in DF. After much searching on the forum and elsewhere, I wasn't able to turn up much information. There has been some experimentation looking at attributes and body size, but very little about how hair or skin color is handled. Possibly because it's completely useless outside of flavor, but I think it's fun.

Anyway, to remedy this I decided to embark on some experimentation myself.
(I'm putting this here because animal breeding pertains to Fort mode, but it can be moved if people think it is better suited to DF General or Modding.)

So, here is what Toady has said about color genetics:

  Then all of the colours, like eye colours, hair colour, I think that uses a dominant/recessive thing now where you pass on two copies and then it picks probably the colour with the lowest index; maybe there's an alphabetic bias right now on which genes are dominant, or it might be the first you listed, it could be the first one you list in the raws that's the dominant gene.

…It kind of sounds like not even Toady's entirely sure on the details, but that gives us a good place to start from. With that in mind...

GOALS:

1. See if genetic inheritance of tissue layer/body part coloration actually happens.
This seems like a given, but you can never assume everything is working as stated in Dwarf Fortress. It's not impossible that the features on animal offspring are generated randomly or taken from an existing population somehow.

2. Verify whether or not creatures actually carry two different copies of a gene.
In which case the dominant copy would be expressed, and the recessive one would be hidden. Before I looked into it, I was under the impression that DF probably used a very simple system where it just picked either the color of the mother or father at random to pass on. According to that quote, it might be more complicated. Let's find out!

3. Check whether color dominance is based on the order of options listed in the creature's raw file, or the order in the color raw file (which would make it alphabetical).
Just because the quote was a little ambiguous. For most vanilla creatures, it comes out about the same because the options are listed alphabetically in the creature raws. But it's useful information for anyone who wants to modify things!

EXPERIMENTAL SETUP

For greater control and consistency, I created a custom creature to do my testing with:
It's a modified mongoose that can be brought on embark and always gives birth to 10 babies at a time. Its body parts and tissues are all default, I only changed the color options and descriptive names. Name starts with z for easy access on huge lists.

It has three features that come in different colors:

Fur that comes in white, ivory, or cream.
Stripes that come in red, sky blue, purple, or black.
Eyes that are usually black, but have a rare 1:10 chance of being golden yellow.

Assuming the non-alphabetical model is correct, the order of dominance should be:

Fur: WHITE > IVORY > CREAM
Stripes: RED > SKY BLUE > PURPLE > BLACK
Eyes: GOLDEN YELLOW > BLACK

I used Arena mode for my testing. If you put a mating pair of creatures on the same team in Arena mode, they will eventually breed and produce babies. The only catch is that all the babies are born on separate teams from the parents and each other, so un-pausing results in an instant bloodbath. No second generation testing, sadly.

Testing was done by setting the game to pause when it announced an animal giving birth. I opened up five instances of the game in Arena mode, and in each one placed one male and one female Zangoose in the enclosed central ring. Then I went and had lunch. When I came back, the games had paused and I was able to open up the unit lists and record the color-features of all of the offspring. One of the five pairs decided not to play ball on the romance game (c'est la vie), but the other four worked out perfectly.

RESULTS

Posting these as code tables to keep the formatting. I hope they're readable enough.

Pair #1
M/F | Fur Stripes Eyes
--------------------------------------------------------------
Father M ivory sky-blue black
Mother F ivory red black
--------------------------------------------------------------
Pups:

#1 F ivory sky-blue black
#2 F ivory sky-blue black
#3 F cream sky-blue black
#4 M cream red black
#5 F ivory red black
#6 F ivory red black
#7 F ivory sky-blue black
#8 M ivory sky-blue black
#9 M ivory sky-blue black
#10 M cream sky-blue black

Awesome results right out of the gate!
First of all, we can tell right away these results aren't random. They are definitely inheriting colors from their parents.
After that, the fur color is the main thing to look at here. The next thing we can rule out is the game simply picking a color from mom or dad, because we have two ivory parents producing three cream-colored babies. Based on that, we can assume each parent actually carries two separate copies of each gene and picks one at random to pass to the offspring, so we are indeed looking at some form of Mendelian inheritance here.
Thirdly, in order for the cream coloration to show up in the offspring but not the parents, it would need to be recessive to ivory and therefore hidden in the parents. Which in turn means we can throw out the alphabetical dominance theory because that would require cream (early alphabet) to be dominant to ivory (later alphabet). Yay!

So that's most of my questions answered already, but let's keep going...
In order to see cream from two ivory parents, they would have to both be carrying one copy of the ivory gene and one copy of the cream gene (IC IC). If you plug that into a classic Punnet square, you'd see that it would predict 75% ivory offspring and 25% cream offspring (25% II, 50% IC and 25% CC). Which is… Pretty much exactly what we got. It works!

We can also deduce that the parents have all black-eye genes, which isn't surprising based on how rare golden yellow is.

Pair #2
M/F | Fur Stripes Eyes
--------------------------------------------------------------
Father M white purple black
Mother F white red black
--------------------------------------------------------------
Pups:

#1 F white black black
#2 F white black black
#3 F ivory red black
#4 F white black black
#5 F white red black
#6 F white red black
#7 F white purple black
#8 M ivory black black
#9 M white red black
#10 M white red black

So, we've pretty much determined the model, but this is still interesting.
Parents with red stripes and purple stripes produced a mix of red-, purple-, and black-striped babies.  Black stripes are the most recessive possible stripes, which means both parents must be carrying black stripe genes, meaning dad is carrying purple and black (PB) and mom is carrying red and black (RB). That means the Punnet square prediction would be 50% red (RB), 25% black (BB) and 25% purple (PB). It ended up being 50%, 40%, and 10% respectively, but given the sample size that's still pretty close.

Pair #3
M/F | Fur Stripes Eyes
--------------------------------------------------------------
Father M white sky-blue golden-yellow
Mother F white purple black
--------------------------------------------------------------
Pups:

#1 F white sky-blue golden-yellow
#2 M white sky-blue golden-yellow
#3 F white sky-blue black
#4 M white sky-blue black
#5 F ivory purple black
#6 M white purple golden-yellow
#7 M white purple black
#8 F white sky-blue black
#9 M white purple black
#10 M white sky-blue black

Yay! Golden-yellow eyes!
Not a lot to say here, really. Dad is carrying one golden-eye and one black eye gene, and mom has two black eyes (ouch). That gives us a predicted ratio of 50% golden and 50% black, which... it's close enough. In terms of the fur, the one ivory girl shows that one of the parents is carrying a hidden ivory gene, and the other has either a hidden ivory or cream. If they had produced a litter containing only the nine white ones, it would've been easy to mistakenly assume they had only white genes. This is why big litters are good for testing!
Pair #4
M/F | Fur Stripes Eyes
--------------------------------------------------------------
Father M white red golden-yellow
Mother F ivory sky-blue golden-yellow
--------------------------------------------------------------
Pups:

#1 M white red golden-yellow
#2 M ivory red golden-yellow
#3 M ivory sky-blue black
#4 F white sky-blue golden-yellow
#5 M white red golden-yellow
#6 F white sky-blue golden-yellow
#7 M ivory red golden-yellow
#8 F white sky-blue black
#9 M ivory red golden-yellow
#10 F white red golden-yellow

Last one. I was really surprised to get two golden-eyes in one pair. Doesn't really tell us anything new, though if you had any lingering concerns that baby colors might be chosen randomly based on their rarity in the raws, this 80% golden-eyed litter should put that to rest.

CONCLUSIONS

I might have gotten a little carried away, but after all that I can state pretty definitively that:

YES, DF models a simple form of genetics.

For any given individual creature, for each body part or tissue that can have a range of colors, DF stores two (potentially different) color genes. When you check its description, what you see is the color for the more dominant of the two. Dominance is determined by the order the potential colors are listed in that creature's raw, with the first being most dominant in the last being most recessive. When two creatures breed, it randomly selects one of the two color genes from each parent and passes them to the offspring. This appears to be done separately for each color-defined body part in a given child (that is, hair color is determined entirely separately from skin color, etc.)


It would be awesome if people could do some more testing, especially second-generation testing after figuring out the genes of the parents. I'm not currently equipped to play fortress mode, so that's a bigger project I can't really tackle. I also don't use any third-party programs, but someone who does might be able to look into things more deeply. Anyone is welcome to use my creature or methods and expand on it.