Dwarf Fortress Mod Structure - a Suggestion
Go back to the main page

Below is a mirror of the initial post of this thread over at the official Dwarf Fortress Suggestions forum board, posted November 26, 12021. It regards the mod structure of Dwarf Fortress, and how it should change for the then-upcoming Steam/Itch.io release - what we would come to know as DF v50. This mirror exists because having written it, why should it not put it on my personal website? The v50.04 structure still lacks some functions the suggestion names, so it is still relevant, and maybe it can get a smidge more attention this way. But also, it is something that I'm proud of. Judge for yourself whether that's warranted, but if you find it interesting, I recommend you read the rest of the thread.

// Voliol, 12022-12-30, HE.

Suggestions for how to rework the mod structure (mod manager etc.)

The below suggestion (and description of the current state of modding/mod installing) is in part based on the discussion held between primarily Mr_Crabman and I on this thread on the modding board. In case you decide to read it, the thread uses some different terminology, though this suggestion should stand on its own. When it comes to the mod manager mentioned, I have written a mockup which can hopefully help visualize the idea better, though it does not work exactly as written in this suggestion, or use identical syntax.

So the mod structure will soon be getting reworked. Let's talk about that.

There are three kinds of goals to achieve with this new reworked structure:

First off, mods should become more available. Players should be able to use mods without having knowledge of how they work, and be less alienated by the way mods are installed.
Second, everything that is possible using the current methods of creating, finding, and installing mods should also be possible in the new system. This might seem trivial, but it turns out to have some conflicts with making it more available, and moving away from folders and manual mod-merging.
Third, various goals related to improving the raw syntax. These are more of stretch goals, as none are needed as a core part of making and using mods, but many of them are still very useful, and/or have been on the to-do list for a long time, so doing some of them along with other mod structure changes might not be a bad idea.

There is also the non-functional requirement of having support for Steam Workshop, as that has been promised. At the same time, the system should work for those who play Classic, or Premium via an itch.io purchase. For that reason, it should not be too dependent on Steam Workshop.
Finally, there are goals related to the graphics' need to move out of the save files. I will not talk about them much in this initial post, but I encourage the discussion in the thread. This is obviously an important part of the rewrite, as it is half of what prompted it (the other being Steam Workshop)(as far as I know). The reason I won't is that I don't have a good grasp on graphics, so I simply don't have the suggestions.

In this post, I will first map out the way mods are currently installed, and then move on to suggesting how they could/should be installed (by using an in-game mod manager). This way we will see what is needed, and what can currently done (goals of the first and second kind) before arriving at the suggestion. Third, there is the technical aspect of how to merge mods, and finally the syntax improvement suggestions, as they tie into that technical aspect (and are also less important) they can't come before it.

How mods are currently installed:

The player first finds a mod online, likely on the Modding board here on the forum, or directly from DFFD (though these are my own guesses from where I know there are mods). Sometimes there are multiple versions of a mod available, the most common being modular (raw files only) vs. pre-installed (comes with a full install of Dwarf Fortress, and the mod already installed in it).
Mods may have installers of their own, like Masterwork, but these mods are by far in minority (and may indeed only be Masterwork), so I will not be going over them.
Then they download it.
The downloaded mod is usually found in a compressed folder of some kind, but the structure here may vary, even if we include the pre-installed. Some mods contain only one set of raw objects, some contain multiple subfolders with versions or parts of the mod to be chosen by the player. For this reason, most mods have an installation instruction somewhere, and these are not all identical.
In all cases but with pre-installed mods (and these are already installed, so they won't be mentioned from now), the player then drags some files into the appropriate folder (but which one this is may vary!) of their DF install, with the goal of getting the mod's raw files into the /raw/objects folder (and sometimes some other files into other folders, like graphics and speech files). The mod's raw files either overwrite existing files, or don't.
All raw files contain so-called raw objects describing the objects in the game, each with a supposedly unique ID (like CREATURE:DWARF for the dwarf creature), and a definition built up by "tokens" (like [DESCRIPTION:A sturdy creature fond of alcohol and industry.]). When mods overwrite existing files, this is usually to make changes to existing objects, by providing a version of the file where one or several objects have been manually edited by the modder. The modder may also add new objects to an existing file (adding a sphinx to creature_standard.txt), but this is considered bad practice for reasons that will become apparent. New objects are instead better placed in their own files, these are the files that do not overwrite.
When the mod has been installed, any new worlds created will use the mod. It is also possible to mod the raws of existing saves, though this comes with some limitations as many objects (such as creatures and entities) will only appear in-game if they were in the raws at the time of world generation.

Installing multiple mods at once is complicated, due to two kinds of conflicts.
The first is raw duplication, when two objects (possibly in different files) have the same object ID it causes some kind of offset error, with grievous (if funny) results. Raw duplication is usually easy to avoid by using prefixes/suffixes in object IDs; CREATURE:MOD1_SPHINX is a different id from CREATURE:SPHINX_MOD2, but this is a convention new modders often do not know about. Raw duplication may also arise even with unique names, for example when the same objects are used as a base in multiple mods by the same creator, and two or more are installed together.

The other kind of conflict is a file conflict, when two mods use files of the same name they overwrite another, and thus the content in all but last installed of those files is left out. Perhaps the most common example is two mods adding reactions to ENTITY:MOUNTAIN in entity_default.txt, as that is needed for dwarven civilizations (and thus players) to use the reactions in-game. As a solution for that case, some mods provide lists of reactions to manually add to the entities when installing the mod. While functional, this is very hefty.

Is is not possible to uninstall a mod that changes Vanilla files any other way than replacing your modded /raw folder with a clean one, nor is it possible to switch between mod loadouts quickly; the fastest way to do so is to open a separate installation of DF with the other set of mods already installed.

Suggestion:

What I suggest, is to have an in-game mod manager. The mod manager would depend on a folder of "all mods". In this folder, the player would place mods as separate sub-folders (or compressed .zip files, if that works better with Steam Workshop) in some unified format, unlike the current downloads. For those using Steam Workshop, the "subscribe" button would place a mod in the folder, and "unsubscribing" would remove it, while those not using Steam would have to find mods elsewhere and place them manually there. This is the extent that the player would have to dig in the files (not at all with Steam Workshop), with the mod manager handling the use cases of installing and uninstalling mods.

I suggest reusing many of the same GUI elements from the the trade screen for the mod manager screen, both to instill understanding through familiarity, and to save time developing (if it does, this is an assumption on my part).

When the player enters the mod manager screen for the first time, after having placed a few mods in the "all mods" folder, all mods are listed on the left side of the screen.
By selecting them they may see info on each of the mods, such as mod name, creator, version (of the mod), and what version of Dwarf Fortress it is meant for. It should also show a description (written by the creator), and dependencies with other mods, including what mods (the creator knows) the mod is incompatible with.
The player may also move mods to the right side of the screen, where they are listed as mods to be installed, and conversely mods may be moved from the right list to the left to be "uninstalled". When the player exits the mod manager this configuration is saved, so the to-be-installed mods are the same as last time when they come back.

There is also a "mod" which is on the right "to-be-installed" side by default: the vanilla raws. These may or may not be kept in the "all mods"folder - if they are kept separately there is a smaller risk of players accidentally erasing them, and it becomes easier to have a "vanilla toggle", turning off all mods without losing your configuration for the future. Regardless I argue they should be in the mod manager screen, to show vanilla raws are not hard-coded, and also that they should be possible to uninstall, for the sake of total conversion mods.
I'm not sure when the installing is best done, and where the mod manager is best placed. Perhaps on the main screen, or as a step when creating a new world, as new worlds are the prime targets.
In either case, at the time of "installing" mods (which could be as soon as the mod manager screen is exited), all the mods in the right list have their folders read, and their combined raws are what is used by a save. Here comes the difficult technical part, how do you merge mods? How do you deal with the aforementioned conflicts? The easier way is probably to do it with "patch-style edits" (rather than to use git-like merges of full files) where some parts of a mod relate to a base file (or base raw object) and have instructions on how to add/remove/change(convert) certain tokens, or remove the whole object.

There is one more integral part for the mod manager to cover the use cases we currently have (before getting to patches): mod options and/or mod bundles. The point of both of them is to allow variants of a mod and mods subdivided into optional parts, without separate downloads.
With mod options, the idea is to have options for each mod (presumably set using similar methods to the "main" options, as those will have some in-game solution as well). These can toggle parts of the mod on/off, and set values within the raws. From a technical standpoint I imagine they would work similar to the arguments used in creature variations (this explanation/suggestion might be expanded as well), but for the whole mod.

A mod bundle is something like a bin in the trade screen, it contains several "mods" and you can choose to only include some of them to be installed. In the "all mods" folder, a mod bundle is a folder containing multiple mod folders. Mod bundles cover the same use cases as mod options (the bundle containing all variations of the mod as separate mods, the same thing with optional parts), though they also have a technical function related to the merging of mods, if one of the "extra" goals below is met. Mod bundles could also be used for mod packs, though I think the collections on Steam Workshop work pretty well to fulfill that niche.

Patch-style edits:

The point of a patch-style edit (or a patch, or an edit) is to describe only the changes made to a base object, if tokens are added/removed/changed(converted) or if the whole object is removed.

When applying a patch, the base object must first be read by whatever installs the mods, so some sort of order is implied. Patches should also be able to negate one another. For example: we have Mod A, Mod B and Mod C.

Mod A includes the base version of the object (let's call it TOAD)
Mod B includes a patch to remove some tokens in TOAD, but also other patches/additions
Mod C is meant to revert the changes Mod B made to TOAD, by including a patch which re-adds the removed tokens

Here, it is important that the base Mod A is first, and the patches in B and C must be applied in order, otherwise C will add the tokens only for B to remove them along with the original tokens. Solving this order for every set of mods is difficult, I think this is a case where the task is best offloaded onto the player. The mods in the right list are ordered, and the player is able to change the order, guided by instructions in the mod descriptions, and the dependencies (as these imply which mods are before).
Whether this order should be top-down or bottom-up, with the base mods on the top or bottom of the list, is arbitrary. There is intuition and examples in other games for both.

But before that, here's aside on how to solve duplication errors, have "later" objects (in the installation order) overwrite "earlier" ones. This can be a useful tool, redefining an object from scratch when the corresponding patch would be too substantial, but mostly it solves duplication errors. Even when misused, the worst case scenario is that objects that should not be overwritten are, and parts of mods are left out like with the current file conflicts.

For the patches to work, I suggest EDIT "objects", to exist alongside the normal objects in a mod's raws.
EDIT objects do not describe a new object, instead they describe a patch to apply on an already existing object.
EDIT objects should have the powers that the current creature variations (though they shouldn't replace them) have, they should be able to add, remove, and convert tokens. This can be done with tokens like ADD_TAG, REMOVE_TAG, or CONVERT_TAG (the raws themselves call tokens "tag"), but it should be noted that ADD_TAG can as well be left out. If it is, then EDIT objects adding tokens will look mostly like the objects they edit, making it easy to copy-and-paste between them during the modding process. Not having this is currently an issue with the creature variations and CV_ADD_TAG.
An EDIT object might look something like this:

[EDIT:ENTITY:MOUNTAIN] [PERMITTED_REACTION:MITHRIL_MAKING] [PERMITTED_REACTION:MITHRIL_MAKING2]

It adds two PERMITTED_REACTION tokens to the end of ENTITY:MOUNTAIN, presumably allowing the dwarves to create mithril (the reactions and materials for this still have to be defined elsewhere).

In addition, EDIT objects should be able to remove objects altogether. This is probably best done by adding a token like REMOVE_OBJECT, which slates an object for removal, but doesn't remove it there and then in the installation process. This way mods can easily add back objects removed by earlier mods.
For example: Mod A removes an object (CREATURE:TOAD) using the following EDIT object:

[EDIT:CREATURE:TOAD] [REMOVE_OBJECT]

which adds the REMOVE_OBJECT token to the end of CREATURE:TOAD. Then all Mod B has to do to add it back is using:

[EDIT:CREATURE:TOAD] [REMOVE_TAG:REMOVE_OBJECT]

The alternative would be for B to include a full definition of CREATURE:TOAD, to add it back. This latter solution has the weakness of B having to keep up with the original definition of CREATURE:TOAD, should it be changed. Any edits of CREATURE:TOAD before A (say by Mod C) would also be ignored by this new definition.

However, creature variations (and COPY_TAGS_FROM) complicate EDIT objects. I will assume a familiarity with the former here, otherwise I refer to their Wiki article. Say you want to be kind to slug men, and allow them to use APPLY_CREATURE_VARIATION:ANIMAL_PERSON instead of APPLY_CREATURE_VARIATION:ANIMAL_PERSON_LEGLESS. This sounds reasonable, how would we do it with an EDIT object?

[EDIT:CREATURE:SLUG_MAN] [CONVERT_TAG] [CT_MASTER:APPLY_CREATURE_VARIATION] [CT_TARGET:ANIMAL_PERSON_LEGLESS] [CT_REPLACEMENT:ANIMAL_PERSON]

Now, we are also upset that slug men get the SAVAGE token from CREATURE_VARIATION:ANIMAL_PERSON; we want our legged slug men to appear in all biomes. We only want that for slug men though, not other animal people. How would that look?

[EDIT:CREATURE:SLUG_MAN] [REMOVE_TAG:SAVAGE]

Finally, we want to re-add the PREFSTRING which was removed by CREATURE_VARIATION:ANIMAL_PERSON.

[EDIT:CREATURE:SLUG_MAN] [PREFSTRING:slimy trails]

Of course, these three can be written together, like

[EDIT:CREATURE:SLUG_MAN] [CONVERT_TAG] [CT_MASTER:APPLY_CREATURE_VARIATION] [CT_TARGET:ANIMAL_PERSON_LEGLESS] [CT_REPLACEMENT:ANIMAL_PERSON] [REMOVE_TAG:SAVAGE] [PREFSTRING:slimy trails]

These are reasonable, and can be achieved using manual edits, but here they won't work together; the first example can only be done if EDIT objects are applied before APPLY_CREATURE_VARIATION (because creature variations are irreversible), while the latter two require that they are applied after it (SAVAGE must be removed after it is added, and the creature variation removes any instances of PREFSTRING when applied). I said this can be achieved using manual edits, but how?
The former is very simple, replace [APPLY_CREATURE_VARIATION:ANIMAL_PERSON_LEGLESS] with [APPLY_CREATURE_VARIATION:ANIMAL_PERSON].
To do the latter you use creature variations, you add the following to the slug man raws after (below) [APPLY_CREATURE_VARIATION:ANIMAL_PERSON]:

[CV_REMOVE_TAG:SAVAGE] [APPLY_CURRENT_CREATURE_VARIATION] [PREFSTRING:slimy trails]

Note that we would also need to apply an EDIT before if it removed or added any APPLY_CREATURE_VARIATION. What this means is essentially that we need to add/remove/convert any instances of APPLY_CREATURE_VARIATION in a step before they are applied, and add/remove/convert normal tokens in a step after.

EDIT objects also need to be able to edit creature variations, e.g. in case you want to change all animal people, or giant animals, or [insert custom modded creature variation here]. This is of course also possible to do currently, manually.

With these use cases explained, I think it is possible to take multiple paths. The one followed in the thread leading up to writing this suggestion (and when writing the mockup) was treating the add/remove/convert tokens within EDIT objects like they were creature variation tokens, and applying them as such. I am unsure if that is the best path, it lead to certain complications and was conceptually confusing, so I won't go into details here. That said, it did lead to many insights about creature variations, that have been baked into the "goals/suggestions of the third kind" below. The best path probably depends on the structure of the code as well, something we can't know.
Toady, if you choose to go this far with the mod structure rework, I believe any solution will be a good one :)

Goals/suggestions of the third kind:

Generalize creature variations to be useable by all raw objects. They are general enough that it feels strange they only exist for creatures. As an asset, that kind of very general templates is really useful, though it has been underutilized in the modding community. A proposed name is "object templates", skipping "variation" as that presumably had to do with the animal people and giant animals being variations on the base animal, and with that origin being so removed from the use it makes little sense to keep it.

A wish from those making a DF markup language (Mr_Crabman being one of them) is that if many kinds of objects (entities, creatures, tools, etc.) use the same kind of template, they should be distinguished somehow to clarify the context for the parser.
Make creature variations/object templates nestable. This is an old goal from when creature variations were first revealed. It is also a good idea, and could be done "while you're at it", though that is probably a dangerous mindset.
Merge body detail plans with creature variations/object templates. Likewise, an old goal.

Condense the syntax of X_CONVERT_TAG tokens. Why have
[CV_CONVERT_TAG] [CVCT_MASTER:BODY] [CVCT_TARGET:QUADRUPED] [CVCT_REPLACEMENT:HUMANOID] [CV_CONVERT_TAG] [CVCT_MASTER:BODY] [CVCT_TARGET:QUADRUPED_FRONT_GRASP] [CVCT_REPLACEMENT:HUMANOID] [CV_CONVERT_TAG] [CVCT_MASTER:BODY] [CVCT_TARGET:QUADRUPED_NECK] [CVCT_REPLACEMENT:HUMANOID_NECK:3FINGERS] [CV_CONVERT_TAG] [CVCT_MASTER:BODY] [CVCT_TARGET:QUADRUPED_HOOF] [CVCT_REPLACEMENT:HUMANOID_HOOF:3FINGERS]
when it can be condensed into the following?
[CV_CONVERT_TAG:BODY] [CVCT_TARGET:QUADRUPED] [CVCT_REPLACEMENT:HUMANOID] [CVCT_TARGET:QUADRUPED_FRONT_GRASP] [CVCT_REPLACEMENT:HUMANOID] [CVCT_TARGET:QUADRUPED_NECK] [CVCT_REPLACEMENT:HUMANOID_NECK:3FINGERS] [CVCT_TARGET:QUADRUPED_HOOF] [CVCT_REPLACEMENT:HUMANOID_HOOF:3FINGERS]
This applies to whatever conversion tokens are used in the EDIT objects as well.
Split CVCT_TARGET (and corresponding EDIT token) into CVCT_TARGET and CVCT_TARGET_STRING. The way CVCT_TARGET currently works by looking for strings is quite uncumbersome, as seen with the body conversions which have to be in a certain order so [CVCT_TARGET:QUADRUPED] won't affect [BODY:QUADRUPED_HOOF]. The suggestion is to let CVCT_TARGET look for whole token arguments, as separated by colons, and give the string-searching functionality (which is still needed for e.g. giant bushmasters) to CVCT_TARGET_STRING.
Give all objects "classes" like the CREATURE_CLASS and SYNDROME_CLASS that already exist. Primarily for the mass-editing purposes described below, but also because these kinds of arbitrary tokens are good to build off of and use in the future. They give freedom, and modders love that.
Allow EDIT objects to select not only one, but multiple objects at once, by following some criteria. Suggested criteria are specific tokens, object classes (see above) and of course object IDs. This is both so you can shorten some expressions, instead of having
[EDIT:ENTITY:MOUNTAIN] [PERMITTED_REACTION:MITHRIL_MAKING] [PERMITTED_REACTION:MITHRIL_MAKING2] [EDIT:ENTITY:MOUNTAIN_DARK] [PERMITTED_REACTION:MITHRIL_MAKING] [PERMITTED_REACTION:MITHRIL_MAKING2] [EDIT:ENTITY:MOUNTAIN_DEEP] [PERMITTED_REACTION:MITHRIL_MAKING] [PERMITTED_REACTION:MITHRIL_MAKING2]
to add the mithril reactions to the vanilla dwarven entity and their two modded cousins (made up on the spot as examples) it could be shortened to
[EDIT:ENTITY:MOUNTAIN] [PLUS_SELECT:ENTITY:MOUNTAIN_DARK] [PLUS_SELECT:ENTITY:MOUNTAIN_DEEP] [PERMITTED_REACTION:MITHRIL_MAKING] [PERMITTED_REACTION:MITHRIL_MAKING2]
or even to
[EDIT:ENTITY:SEL_BY_CLASS:DWARVEN] [PERMITTED_REACTION:MITHRIL_MAKING] [PERMITTED_REACTION:MITHRIL_MAKING2]
if we imagine all these dwarven civ to have a shared "DWARVEN" class. This class-based selection may also select other "DWARVEN" civs, even ones the person creating the mithril mod had no knowledge of (for instance because they were made after it). Another example would be
[EDIT:ENTITY:ALL] [ALL_MAIN_POPS_CONTROLLABLE]
to make all entities available for adventurers to come from. This kind of mass-editing is thus a boon for the connectivity between mods, as long as the community can come up with some standard for classes. Though as I have already seen propositions for the limited creature classes, and other modding communities like the Minecraft one has been able, I have no doubts we would.

The examples given here are all inclusive (like a logic "or"), but it should also be able to be restrictive (like a logic "and"), e.g. only select the plants that are both giant mushrooms and found underground. Or a combination thereof.

This is also where the "technical function" of mod bundles lie. Take the mithril mod we've used as an example. It consists in part of the mithril raws themselves (which we haven't seen) and the reactions, and in part of the EDIT object giving the mithril reactions to all dwarven entities. We also assume it contains those dark dwarves and deep dwarves implied by the object names. Now imagine another mod, let's call it the orichalcum mod. Much like the mithril mod, the orichalcum mod adds a new material, and a few dwarven civs (forge dwarves and moss dwarves), and it too intends for the new metal to be usable for all dwarven civs, by using something like the SEL_BY_CLASS construction seen above.

How do you order these mods?
If you put the mithril mod before the orichalcum mod, then the dark and deep dwarves will get both mithril and orichalcum, but the forge and moss dwarves will only get the latter. If you reverse the situation and put the orichalcum mod the reverse will be true, with dark and deep dwarves only getting mithril but no orichalcum. The solution is to split the mods into two parts each, one for the definitions and one for the EDIT-based assignments. The assignment parts. As you don't want separate downloads/subscriptions for these parts, and want them grouped together in the left list for readability, mod bundles are perfect for this.

(As a final note (for now, and obviously I want people to reply here, I just didn't know where to put this), Steam Workshop mods may be auto-updated, so if that feature is used then presumably the mods in the "all mods" folder are periodically overwritten with newer versions. This could be an issue if mods aren't stored somehow in conjunction with each save file, as changing the raws of existing saves doesn't always work well.)