Here is a brief overview of what I've learned about ecosystem generation while making this mod.

The game selects a fauna abundance during planet generation, and based on this abundance, it selects a role table (or two, for water) for each of the present habitat types (air, cave, ground, underwater). This seems to be at least partly controlled by the CREATUREROLEDISCRIPTION table, but I have not thoroughly tested the variables in that table. After selecting role tables, the game populates the planet with creatures defined by the roles in these tables, up to the limit of 18 species. A role might look like this (from my version of GROUNDTABLEBUSY, which is the "standard" table for "Abundant" fauna):

<Property value="GcCreatureRoleDescription.xml"><!--ROLE: Large Roaming Herd-->
      <Property name="CreatureRole" value="GcCreatureRoles.xml">
        <Property name="CreatureRole" value="Prey" />
      </Property>
      <Property name="CreatureType" value="GcCreatureTypes.xml">
        <Property name="CreatureType" value="None" />
      </Property>
      <Property name="ForceID" value="" />
      <Property name="MinGroupScale" value="1.5" />
      <Property name="MaxGroupScale" value="2.5" />
      <Property name="MinGroupSize" value="50" />
      <Property name="MaxGroupSize" value="60" />
      <Property name="GroupsPerSquareKm" value="25" />
      <Property name="FractionActiveInDay" value="1" />
      <Property name="FractionActiveInNight" value="0.1" />
      <Property name="ProbabilityOfBeingEnabled" value="1" />
      <Property name="IncreasedSpawnDistance" value="3" />
      <Property name="Filter" value="" />
</Property>

Here are what the entries do (to the best of my knowledge):

• <Property name="CreatureRole" value="Prey" />: the value can be "Prey", "Predator", or "Passive" for ground creatures. Predators attack prey and have a certain probability of becoming hostile to the player. Predators do not attack passive creatures. Values "FishPrey" and "FishPredator" are also possible, with analogous impacts. Birds require the "Bird" role, and Butterflies require the "Butterfly" role.
• <Property name="CreatureType" value="None" />: the value indicates whether the creature is a special type. Possible values are "Jellyfish", "Crab", "FlyingSnake", "FlyingLizard".
• <Property name="ForceID" value="" />: the value can be empty (select randomly) or an ID from the CREATUREDATATABLE, which forces the creature to be of a certain archetype (e.g., TREX, COW).
• GroupScale gives bounds on the size of individuals in a group. This value corresponds roughly to meters in height/length. Note that the in-game scanner is rarely accurate and seems to be capped at ~10m. The big creatures still get made. If an individual is selected to be a size that falls outside the allowed range for its genus (MinScale and MaxScale for each genus are set in CREATUREDATATABLE), its size will be set to the corresponding genus bound.
• GroupSize is the number of individuals that spawn in a group, NOT the size of its members. This number can be multiplied by the HerdGroupSizeMultiplier variable in the GCCREATUREGLOBALS file if the creature genus selected has the "Herd" movement flag set to true (I do not think the multiplier is applied if the role is that of a predator). However, the resulting number spawned does not always accurately reflect the range specified, and seems to be hard-capped somewhere around 100.
• GroupsPerSquareKm controls how far apart groups spawn. I do not know if this value accurately reflects the in-game distance scale (I doubt that it does).
• I'm not 100% sure, but I believe that FractionActive controls the fraction of individuals that spawn at day/night, not the fraction of groups.
• ProbabilityOfBeingEnabled controls the chance that the role is used, given that the role table has been selected. The 1 in the example does not mean 100% of planets have a large roaming herd species; instead, 100% of planets that use the "Busy" ground table have one. This is a planet-wide on/off switch, not a per group probability, so if I changed it to 0.2, there would be a 20% chance that this type of species exists on a "Busy" ground planet, and an 80% chance that the species is not present anywhere on the planet.
• IncreasedSpawnDistance multiplies the standard spawn distance (in vanilla it is used for large creatures only).