Engine System update: the .fsm format, a validating builder and binary loading

Course: GAMES 6320-001 Game Engineering II (Fall 2026)

1. What the system is, and where it stands

My engine system is a finite state machine library for game AI. The states of an enemy, the transitions between them and the conditions that trigger each transition live in a human-readable .fsm file, and the game only sets parameters (like the distance to the player) and reacts to the current state.

The proposal split the work into three steps. This is where Update #1 is:

What exists now:

• A new static library, Engine/StateMachine, with cStateMachineDefinition (namespace eae6320::AI). It loads a built file with Platform::LoadBinaryFile(), finds its arrays with pointer offsets, looks up names to handles and is reference counted like cMesh.
• A new tool, Tools/StateMachineBuilder, that reads .fsm files with Lua, checks them, turns every name into an index and writes the binary file.
• MyGame builds guard.fsm as part of its assets, loads it at startup and writes everything it loaded to the log.
• All four configurations (Debug/Release, x64/x86) build without warnings.

The public interface is platform-independent. Right now the library has no platform-specific code at all; the only planned .win.cpp file is the debug output for state changes, which belongs to Update #2.

2. The final .fsm format

This is MyGame_/Content/StateMachines/guard.fsm:

return
{
	initialState = "Patrol",

	-- The values that the game sets every update
	parameters =
	{
		{ name = "distanceToPlayer", type = "float", default = 1000.0 },
		{ name = "canSeePlayer", type = "bool", default = false },
		{ name = "health", type = "float", default = 100.0 },
		{ name = "wasHit", type = "trigger" },
	},

	-- Checked from every state, before the state's own transitions
	anyState =
	{
		{ to = "Flee", conditions = { { "health", "<", 20.0 } } },
	},

	-- Each state's transitions are checked in the order they are written,
	-- and the first one whose conditions are all true fires
	states =
	{
		{
			name = "Patrol",
			transitions =
			{
				{ to = "Chase", conditions = { { "canSeePlayer", "==", true } } },
				{ to = "Chase", conditions = { { "wasHit" } } },
			},
		},
		{
			name = "Chase",
			transitions =
			{
				{ to = "Attack", conditions = { { "distanceToPlayer", "<", 2.0 } } },
				-- Give up after 3 seconds without seeing the player
				{
					to = "Patrol",
					conditions = { { "canSeePlayer", "==", false } },
					minimumTimeInState = 3.0,
				},
			},
		},
		{
			name = "Attack",
			transitions =
			{
				{ to = "Chase", conditions = { { "distanceToPlayer", ">", 2.5 } } },
			},
		},
		{
			name = "Flee",
		},
	},
}

The rules:

• parameters is a list. Each parameter has a name, a type ("float", "bool" or "trigger") and an optional default (0 or false if it is left out; a trigger can’t have one).
• A condition is { parameter, comparison, value } with one of <, <=, >, >=, ==, ~= (Lua’s spelling of “not equal”). For a trigger it is just { "wasHit" }, which means “the trigger was set”. The same short form on a bool means == true.
• All conditions of one transition have to be true. A transition can also have a minimumTimeInState in seconds.
• anyState transitions are checked from every state, before the state’s own ones.

Changes from the proposal

States are a list with a name field instead of a table keyed by name. In the proposal I wrote states = { Patrol = { ... }, Chase = { ... } }. When I started on the builder I found two problems with that:

1. If two states have the same key, Lua silently keeps the last one. The builder never sees the first one, so it can’t report the duplicate, which was one of the errors I promised to catch.
2. Lua doesn’t keep the order of keys in a table. The state indices would depend on the hash order, so the same file could build into a different binary file, and “the first state is index 0” wouldn’t mean anything.

With a list both problems go away: every state is seen, and the index of a state is its position in the file. It costs one extra line per state, which I think is fine.

The binary header has a version number. If I change the layout later and someone runs the game with an old built file, the loader reports “build the assets again” instead of reading garbage.

3. The builder

Registering it

StateMachineBuilder is a normal builder like MeshBuilder. To use it in a game, three things are needed.

In Tools/AssetBuildLibrary/AssetBuildFunctions.lua, a new asset type (built files get the extension .fsmbin, the same way built meshes are .meshbin):

NewAssetTypeInfo( "stateMachines",
	{
		ConvertSourceRelativePathToBuiltRelativePath = function( i_sourceRelativePath )
			-- Built state machines are binary, and so they get a different extension
			-- than the human-readable source files
			local relativeDirectory, file = i_sourceRelativePath:match( "(.-)([^/\\]+)$" )
			local fileName = file:match( "([^%.]+)" )
			return relativeDirectory .. fileName .. ".fsmbin"
		end,
		GetBuilderRelativePath = function()
			return "StateMachineBuilder.exe"
		end,
	}
)

In the game’s Content/AssetsToBuild.lua, the list of files:

stateMachines =
{
	"StateMachines/guard.fsm",
},

And in Visual Studio, the game’s asset build project (BuildMyGameAssets) gets a build dependency on StateMachineBuilder (right-click the project, Build Dependencies > Project Dependencies…), so that the builder is always built before it runs. The game project itself references the StateMachine library.

Validation

The builder checks everything I listed in the proposal:

• a transition to a state that doesn’t exist,
• a condition on a parameter that doesn’t exist,
• an ordered comparison (<, <=, >, >=) on a bool, or any comparison on a trigger,
• a missing or unknown initialState,
• duplicate state names and duplicate parameter names,

plus the smaller things: wrong types, unknown comparison strings, a negative minimumTimeInState, a default value on a trigger.

It doesn’t stop at the first problem. Every error is reported, and the build fails at the end if there was any. Errors are written in the file(line) : error : message format, so they show up in the Visual Studio Error List and double-clicking one jumps to the line.

To test this I made a copy of guard.fsm with six mistakes in it (a misspelled initial state, a duplicated parameter, canSeePlayer < true, a misspelled trigger, a misspelled target state and a duplicated state) and added it to the asset list for a moment. This is the Error List after building:

Each of the six errors points at the right line. The first and last rows are the generic “the builder failed” messages from the asset build system.

Getting line numbers was the interesting part. Lua syntax errors already come with a line, and I reused the code from my mesh builder for those. But a missing state isn’t a Lua error: by the time the builder sees the table, Lua has forgotten which line each value came from. So the builder also keeps the source text and searches it for the line that contains the name the error is about (for example a line with both to and "Atack"). The search only moves forward through the file, in the same order the builder reads the tables, so a duplicated name is reported on the line of the second copy and not the first. It is a heuristic, and if it can’t find the line it reports the error without one instead of guessing.

4. The binary format

The built file is six parts, one after the other:

Some decisions behind this:

• Indices instead of names. A state refers to its transitions as a range (first index and count) in one shared transition array, and a transition refers to its conditions the same way. The “any state” transitions are simply the first ones in the transition array, so they need no extra list.
• Alignment. The arrays mix float, uint16_t and uint8_t. I gave every struct a size that is a multiple of 4 (with explicit padding fields) and put the header first, so every array starts on a 4-byte boundary and the floats can be read in place. The name block is last because it is only chars and can have any length. The structs live in one header, Engine/StateMachine/BinaryFormat.h, which both the builder and the library include, with static_asserts on their sizes.
• Bools are stored as floats (0 or 1) in defaults and conditions. That keeps every parameter and condition the same size, and the run time can compare everything as a number.
• Names are kept even though the run time works with indices, because they are needed to look up handles and to print useful debug messages.

Unlike the mesh files, the built file is identical on every platform: I checked that the Debug and Release builds for x64 and x86 produce byte-for-byte the same 269-byte file. There is nothing like the winding order that depends on the graphics API.

This is the built guard.fsmbin, with each part outlined:

A few things to find in it: the header starts with 01 00 (version 1) and 04 00 04 00 06 00 06 00 (4 parameters, 4 states, 6 transitions, 6 conditions). The first parameter’s default 00 00 7A 44 is 1000.0f. Transition 4 (Chase back to Patrol) starts with 00 00 40 40, which is 3.0f, its minimum time in state. And the end of the file is the names, readable on the right.

5. Loading it in the game

cStateMachineDefinition::Load() loads the whole file as one block of memory and keeps it. The arrays are found by adding the sizes of the parts before them, with no copying and no loop per element:

m_parameters = reinterpret_cast<const BinaryFormat::sParameter*>( currentOffset );
currentOffset += parameterArraySize;
m_states = reinterpret_cast<const BinaryFormat::sState*>( currentOffset );
currentOffset += stateArraySize;
m_transitions = reinterpret_cast<const BinaryFormat::sTransition*>( currentOffset );
currentOffset += transitionArraySize;
m_conditions = reinterpret_cast<const BinaryFormat::sCondition*>( currentOffset );
currentOffset += conditionArraySize;
m_names = reinterpret_cast<const char*>( currentOffset );

Before that it checks that the file size matches the counts in the header exactly. After it, it checks once that every index and name offset is in range, so the run time in Update #2 can trust them without checking every frame.

This is how MyGame uses it in cMyGame::Initialize():

// Load the guard's state machine
{
	if ( !( result = AI::cStateMachineDefinition::Load( s_path_stateMachine_guard, m_stateMachine_guard ) ) )
	{
		EAE6320_ASSERTF( false, "Can't initialize MyGame without the guard's state machine" );
		return result;
	}
	LogStateMachineDefinition( s_path_stateMachine_guard, *m_stateMachine_guard );

	// Names are looked up once, and the game then only uses the handles
	const auto chaseState = m_stateMachine_guard->GetStateHandle( "Chase" );
	const auto distanceParameter = m_stateMachine_guard->GetParameterHandle( "distanceToPlayer" );
	const auto canSeeParameter = m_stateMachine_guard->GetParameterHandle( "canSeePlayer" );
	...
}

and in CleanUp() the game releases its reference with DecrementReferenceCount(), like it does for meshes and effects.

The handles are small structs (sStateHandle, sParameterHandle) that wrap a uint16_t index. I made them two different types so that passing a state handle where a parameter is expected is a compile error.

LogStateMachineDefinition() only uses the public interface of the definition and prints what was loaded in the same words as the .fsm file. This is the log of the Direct3D Debug build (the OpenGL build prints the same):

Loaded the state machine data/statemachines/guard.fsmbin: 4 states, 4 parameters, 6 transitions (1 from any state), 6 conditions
	Initial state: "Patrol"
	Parameter 0 "distanceToPlayer": float, default 1000
	Parameter 1 "canSeePlayer": bool, default false
	Parameter 2 "health": float, default 100
	Parameter 3 "wasHit": trigger, default false
	Any state (1 stored at the start of the transition array):
		-> "Flee" if health < 20
	State 0 "Patrol" (transitions 1 to 2):
		-> "Chase" if canSeePlayer == true
		-> "Chase" if wasHit is set
	State 1 "Chase" (transitions 3 to 4):
		-> "Attack" if distanceToPlayer < 2
		-> "Patrol" if canSeePlayer == false and 3 s in state
	State 2 "Attack" (transitions 5 to 5):
		-> "Chase" if distanceToPlayer > 2.5
	State 3 "Flee" (no transitions of its own)
Handle lookups: state "Chase" = 1, parameter "distanceToPlayer" = 0, parameter "canSeePlayer" = 1
MyGame initialized successfully

Comparing this with the .fsm file above is the round trip test: Lua file, builder, binary file, loader, and back to the same description.

6. What’s next, and what was hard

Update #2

• cStateMachine, the per-agent instance: current state, time in state, parameter values (triggers reset after every update), SetFloat / SetBool / SetTrigger, and Update() that checks the “any state” transitions and then the current state’s transitions in order and fires at most one per update.
• Enter / update / exit callbacks as function pointers with a void* user-data pointer.
• The debug output of every state change, which will be the library’s only .win.cpp file (OutputDebugString).
• The demo: a guard in MyGame that reacts to the player object and changes its color or mesh with its state.

One thing I noticed while writing guard.fsm: “give up after 3 seconds without seeing the player” is not quite what minimumTimeInState = 3.0 means. It means “3 seconds since entering Chase”, so a guard that loses the player after 5 seconds of chasing would give up at once. For Update #2 I want to decide whether that is good enough or whether a condition needs a “for at least N seconds” option.

What was hard, and decisions I made

• Line numbers for errors that aren’t Lua errors. Lua tables don’t remember source lines, which I hadn’t thought about when I promised “with the file and line where possible” in the proposal. Searching the text was simple and works for every error in my test file.
• The keyed states table. I liked how it read in the proposal, but it made duplicate detection impossible and the state order unstable. Changing it now, before anyone else uses the format, was the cheap moment to do it.
• The header size. My first header had a 32-bit name block size plus a padding field and came out at 20 bytes instead of 16; the static_assert caught it on the first build. Since names are found by 16-bit offsets anyway, the block can’t be bigger than 64 KB, so the size became a uint16_t too.
• Keeping the file in memory. A mesh can free its file once the GPU buffers exist. A definition can’t, because its arrays point into the file, so the loaded data is moved into the definition and freed in its destructor.

Time spent so far: 3 hours