Engine System update: the state machine run time, debug output and a guard demo

Course: GAMES 6320-001 Game Engineering II (Fall 2026)

1. Where the system stands

My engine system is a finite state machine library for game AI. The states of an enemy, the transitions between them and the conditions for each transition are written in a human-readable .fsm file. A builder turns that file into a binary file, and at run time the game only sets parameters (like the distance to the player) and reacts to the state the machine is in.

This is the schedule from the proposal, updated:

What was added since Update #1:

• cStateMachine (namespace eae6320::AI), the per-agent run time: current state, time in state, parameter values, Update() and enter / update / exit callbacks.
• Debug output of every state change. This is the library’s only platform-specific code, in Windows/DebugOutput.win.cpp.
• A guard in MyGame that is driven by guard.fsm: it patrols, chases the player, attacks, and flees when it has been hit too often. It looks different in every state.
• All four configurations (Debug/Release, x64/x86) still build without warnings, and the Direct3D and OpenGL builds behave the same.

2. The cStateMachine interface

A cStateMachineDefinition is the shared, read-only part (one per .fsm file). A cStateMachine is the small part that every agent owns. This is the main part of the public header, Engine/StateMachine/cStateMachine.h:

// Called when a state is entered or exited.
using fStateCallback = void (*)( cStateMachine& io_stateMachine, void* i_userData );
// Called once per Update() for the current state, with the same elapsed time that Update() received
using fStateUpdateCallback = void (*)( cStateMachine& io_stateMachine, void* i_userData, const float i_elapsedSecondCount );

// Any of the three can be nullptr
struct sCallbacks
{
	fStateCallback onEnter = nullptr;
	fStateUpdateCallback onUpdate = nullptr;
	fStateCallback onExit = nullptr;
};

// Takes a reference on the definition (released in CleanUp()),
// sets every parameter to its default value, and starts in the definition's initial state.
cResult Initialize( cStateMachineDefinition& i_definition, const char* const i_debugName );
void CleanUp();

void SetCallbacks( const sStateHandle i_state, const sCallbacks& i_callbacks, void* const i_userData );

void SetFloat( const sParameterHandle i_parameter, const float i_value );
void SetBool( const sParameterHandle i_parameter, const bool i_value );
// A trigger stays set until the end of the next Update()
void SetTrigger( const sParameterHandle i_parameter );

void Update( const float i_elapsedSecondCount );

sStateHandle GetCurrentState() const;
float GetTimeInCurrentState() const;

There are getters for the parameters too (GetFloat(), GetBool(), IsTriggerSet()). Every setter and getter checks the parameter’s type in the definition and asserts in Debug builds if it is wrong, so SetBool() on the health float stops right there instead of silently doing nothing.

The callbacks are plain function pointers with a void* for the game’s own data. I thought about std::function, but a function pointer plus user data is what the rest of the engine uses, it can’t allocate, and it is easy to explain to someone who uses the library.

What Update() does

Update() is meant to be called once per simulation update, after the game has set the parameters. In this order it:

1. calls the initial state’s enter callback, but only the first time it is called;
2. adds the elapsed time to the time in the current state;
3. looks for a transition to fire: first the “any state” transitions, then the current state’s own transitions, each list in the order of the .fsm file. The first one whose conditions are all true, and whose minimumTimeInState has passed, fires. At most one transition fires per update, so a chain like Patrol to Chase to Attack takes two updates, and every state gets at least one update callback;
4. if a transition fired: calls the old state’s exit callback, changes the state, resets the time in state to zero, and calls the new state’s enter callback;
5. calls the current state’s update callback (so a state that was just entered is also updated in the same tick);
6. resets all triggers to false.

Because triggers are reset at the end of Update(), a trigger that the game sets is seen by exactly one round of transition checks. If no transition uses it in that update, it is gone, which is what I want for a one-shot event like “was hit”.

The conditions are compared as floats. Bools and triggers are stored as 0 or 1 (like in the binary file), so the check is one switch on the comparison with no special cases per type.

3. Debug output

Every time a transition fires the library writes one line:

<agent name>: <old state> -> <new state> (transition: <conditions>)

The text is built in DebugOutput.cpp, which is platform-independent and reuses the same wording as the .fsm file. Writing it to the debugger is the only platform-specific part. It is declared in DebugOutput.h and implemented in Windows/DebugOutput.win.cpp:

void eae6320::AI::DebugOutput::OutputToDebugger( const char* const i_line )
{
	// OutputDebugStringA() shows the text in Visual Studio's Output window
	// (and in tools like DebugView when no debugger is attached)
	OutputDebugStringA( i_line );
	OutputDebugStringA( "\n" );
}

I followed the pattern of the Asserts library: a Configuration.h defines EAE6320_AI_SHOULDOUTPUTSTATECHANGES only in Debug builds, and everything in the debug output files is inside #ifdef of it. A game that wants to see the state changes in a Release build can define it there. The same line also goes to the engine’s log file, which is where the lines below come from.

This is the log of one run of the Direct3D Debug build (the OpenGL Debug build printed exactly the same lines). I moved the player up to the guard, ran away to the right until the guard gave up, and then pressed H three times:

Guard: Patrol -> Chase (transition: canSeePlayer == true)
Guard: Chase -> Attack (transition: distanceToPlayer < 1.1)
Guard: Attack -> Chase (transition: distanceToPlayer > 1.4)
Guard: Chase -> Patrol (transition: canSeePlayer == false and 2 s in state)
The player hit the guard (health is now 70)
Guard: Patrol -> Chase (transition: wasHit is set)
The player hit the guard (health is now 40)
The player hit the guard (health is now 10)
Guard: Chase -> Flee (transition: any state, health < 20)

The lines starting with “The player hit the guard” come from MyGame, not from the library. The Release build of the same run writes only those three lines.

4. The demo

The guard walks back and forth above the player. The player moves with the arrow keys, and H hits the guard. The guard’s mesh and color show its state:

The colors are three small new fragment shaders. The “attack” one makes the red pulse with the simulation time, which is my stand-in for an attack animation. These are the states in the Direct3D build (the two Attack pictures are the same state half a pulse apart):

And in the OpenGL build:

Setting it up

In cMyGame::Initialize(), after the definition is loaded and the handles are looked up, the guard gets its own state machine and the callbacks:

// The guard gets its own instance of the state machine,
// and the callbacks change how it looks and moves
if ( !( result = m_guard.ai.Initialize( *m_stateMachine_guard, "Guard" ) ) )
{
	EAE6320_ASSERTF( false, "Can't initialize the guard's state machine" );
	return result;
}
m_guard.ai.SetCallbacks( patrolState, { OnEnterPatrol, OnUpdatePatrol, nullptr }, this );
m_guard.ai.SetCallbacks( chaseState, { OnEnterChase, OnUpdateChase, nullptr }, this );
m_guard.ai.SetCallbacks( attackState, { OnEnterAttack, nullptr, nullptr }, this );
m_guard.ai.SetCallbacks( fleeState, { OnEnterFlee, OnUpdateFlee, nullptr }, this );

The callbacks are static member functions of cMyGame, so the void* is the game and they can use its meshes and effects. An enter callback changes the look, and an update callback sets the velocity:

void eae6320::cMyGame::OnEnterChase( AI::cStateMachine&, void* i_userData )
{
	auto& game = *static_cast<cMyGame*>( i_userData );
	game.m_guard.object.SetMesh( game.m_mesh_cone );
	game.m_guard.object.SetEffect( game.m_effect_guardChase );
}

void eae6320::cMyGame::OnUpdateChase( AI::cStateMachine&, void* i_userData, const float )
{
	auto& game = *static_cast<cMyGame*>( i_userData );
	const auto offset = GetOffsetInXyPlane( game.m_guard.object.GetPosition(), game.m_player.GetPosition() );
	game.m_guard.object.SetVelocity( ( offset.GetLength() > 0.001f ) ? ( offset.GetNormalized() * s_guardSpeed_chase ) : Math::sVector() );
}

Every simulation update

In UpdateSimulationBasedOnTime() (the fixed-rate update) the game sets the parameters and then updates the state machine. The guard’s movement for this tick comes out of the update callback:

// Tell the state machine what the guard knows about the world...
const auto distanceToPlayer = GetOffsetInXyPlane( m_guard.object.GetPosition(), m_player.GetPosition() ).GetLength();
m_guard.ai.SetFloat( m_guardParameters.distanceToPlayer, distanceToPlayer );
m_guard.ai.SetBool( m_guardParameters.canSeePlayer, distanceToPlayer < s_guardSightRadius );
if ( m_wasGuardHit )
{
	m_wasGuardHit = false;
	m_guard.health = std::max( m_guard.health - s_guardDamagePerHit, 0.0f );
	m_guard.ai.SetTrigger( m_guardParameters.wasHit );
	Logging::OutputMessage( "The player hit the guard (health is now %g)", static_cast<double>( m_guard.health ) );
}
m_guard.ai.SetFloat( m_guardParameters.health, m_guard.health );
// ...and then let it change state if it needs to.
// The current state's update callback sets the guard's velocity.
m_guard.ai.Update( i_elapsedSecondCount_sinceLastUpdate );

The H key is read in UpdateSimulationBasedOnInput(), and only the moment it goes down counts as a hit, so holding it doesn’t hit the guard every tick.

The guard is a little in front of the plane the player moves in, so that the two meshes never go through each other. Because of that the distance is measured in the XY plane only.

Changes to guard.fsm

The numbers in guard.fsm from Update #1 were guesses. With the real scene they didn’t work: an attack distance of 2 is most of the screen. I changed the attack distance to 1.1, the “back to Chase” distance to 1.4, and the give-up time from 3 to 2 seconds. The structure of the file is the same. Only the data changed, and I didn’t have to touch any code for it, which is the point of the system.

5. Decisions

When the initial state is entered. The initial state’s enter callback is called in the first Update(), not in Initialize(). In Initialize() no callbacks can be registered yet, so the guard would never get its “enter Patrol” call and would have no mesh. Calling it in the first update means the order “initialize, register callbacks, update” just works.

An “any state” transition to the state the machine is already in is skipped. The guard’s anyState transition goes to Flee when health < 20. That stays true after the guard has started fleeing, so without a rule it would exit and re-enter Flee in every update, and the enter and exit callbacks would run in every simulation update. I decided that these transitions are skipped, and the next transitions in the list are still checked. A state’s own transitions are not affected, so a state can still have a transition to itself on purpose (for example to restart its timer).

minimumTimeInState stays as it is. In Update #1 I noticed that “give up after 3 seconds without seeing the player” is not what minimumTimeInState = 3.0 means. It means “at least 3 seconds since entering Chase, and right now the player isn’t visible”. I decided to keep it and document it instead of adding a “condition true for N seconds” option. In the demo it behaves well enough: the guard gives up 2 seconds after it started chasing if it can’t see the player at that moment. Doing it properly needs a timer per condition per agent, which makes the run time and the file format more complicated for one feature. A game that needs it can already do it with a float parameter, for example timeSinceSawPlayer, and a condition { "timeSinceSawPlayer", ">", 3.0 }. I will put that example in the README.

Initialize() takes a non-const reference. I wanted const cStateMachineDefinition&, because the state machine never changes the definition. But the reference count is changed through IncrementReferenceCount(), which isn’t const in the engine’s macro. Instead of a const_cast the parameter is non-const, and the comment says why.

Wrong parameter types. In Debug builds SetFloat() on a bool asserts. In Release builds the call is ignored and nothing is written. I preferred this over writing the value anyway, because a float written into a bool parameter would make the conditions compare against values other than 0 and 1.

What was hard

• Making the demo show every transition in one run took more tuning than writing the library. The first runs never got back to Patrol, because the guard always saw the player again before its 2 seconds in Chase were over. I made the guard slower than the player and tuned the sight radius until running away really works.
• In the first version the guard was behind the player, and in Attack the player’s sphere covered the red cone, so the most important state was the hardest to see. Moving the guard in front fixed it.
• Deciding the order inside Update(). I went back and forth on whether the update callback should run before or after the transition check. After is better: the game then always updates the state that the machine is really in at the end of the tick, and a new state starts moving in the same tick it was entered.

6. What is left for the final week (Nov 19)

• README for classmates: the .fsm format, how to add the builder and the library to their solution (asset type in AssetBuildFunctions.lua, build dependency, library reference), and the API with a short complete example, including the timeSinceSawPlayer pattern from above.
• A second example machine that uses the system differently from the guard, probably a door or turret with only bools and triggers, so that the README doesn’t only show one use.
• Cleanup: MyGame still has its own copy of the code that describes conditions as text for the definition log. The library now has the same code for the debug output, so the game should use the library’s version. I also want to look at the public header once more from the point of view of someone who hasn’t seen it.
• Helping classmates who want to use the system, and fixing what they run into.
• Stretch goal: let a game choose its own output function for the state changes (for example to show them on screen), with the debugger as the default. I think this is realistic because the output already goes through one function.

Time spent since Update #1: 5 hours