Should I use a state machine and what would its structure look like?

[gleaming bough]

Hey,

This is more of an advice question than a coding one, I wanted to ask how you guys have had the most success with organizing player behaviour in games. My projects tend to range on the smaller side, from jams to games I work on for a few months, and Ive been wondering how to go about organizng complex behaviours for things like players with lots of movement options or complex enemy AI.

Most sources have pointed me to a state machine, and in a recent game jam, I tried using a hierarchical state machine that basically built a state tree where parent states could run logic and have child states that ran more precise logic. Ex a parent has movement and cinematic state, with movement having child states grounded, airborne, and dash

Anyways, it ended up having a ton of bugs and it was a pain in the butt to navigate between states and manage parent/child node navigation. I wouldnt mind trying to remake it, but honestly it just seems like so much more of a hassle compared to a flat state machine.

I've also heard of using object hierarchies, with gameobjects that you enable/disable representing states, but idk if this is a good idea or not. Theres also Unity's built-in animation editor, but I dont have much experience with that and tutorials I've watched have made it look kinda tricky to use

Alternatively, I could just not have a state machine, but that usually ends up with large conditionals in movement scripts and unclear state handling. However, if thats not the case for your projects, please let me know how you went about it!

So how do you guys go about it? Do you use hierarchichal or flat state machines? Do you make them in code, use object hierarchies, or use Unity's animation cycles editor? Are certain methods different for different sized games? What kind of structure would be best for reuse in multiple projects while being scaleable? I just want something thats easy to implement/use while being useful in various games with more complex mechanics.

Thanks!

(I do need to go to bed so sorry if I dont reply until tmrw)

[visual zephyr]

Hierarchical state machines are my go to for at least my character controllers. If you're doing a game jam I don't think its necessary since the draw of a state machine is its maintainability over a long project for objects with complex state. There are a few useful things I like to do for debugging. First is using serialized strings for observing the state. Ill do something like this:

StateMachine {
    string ToString() {
        var cur = _root;
        var str = "";
        while (cur != null) {
             str += cur.GetType().ToString();
            if (cur.SubState != null)
                str += " -> ";
            cur = cur.SubState;
        }
        return str;
    }
}

Player {
    [SerializeProperty] string _curState;
    [SerializeProperty] string _lastStateChange;

    void Start() {
        _machine.OnStateChange += (from, to) => {
            _curState = _machine.ToString();
            _lastStateChange = from.GetType().ToString() + " -> " + to.GetType().ToString();
        };
    }
}

Heirarchical state machines are for when covering all youre states become a combinatorics problem. As in airborne and moving, airborne and idle, grounded and moving etc. If you don't need to deal with combinatorics, a flat state machine, even just using enums can easily be enough

[gleaming bough]

How do you manage state navigation? can parent states have null child states? can parent states run their own logic? When you enter a parent state does it auto-enter a default child state? How do you pass context?

[visual zephyr]

I say the "owner" of the state machine chooses when logic runs. Ill normally have 4 main callbacks in a state. OnEnter, OnExit, Update, and FixedUpdate. So for example, the player monobehaviour would have a StateMachine, and it would can its update and fixedUpdate accordingly:

Player {
    void Update() {
        var data = {
            // stuff needed by states such as a reference to the player and input
        }
     _machine.Update(data);
    }

    void FixedUpdate() {
        var data = ...
        _machine.FixedUpdate();
    }
}

StateMachine {
    void Update(Data data) {
        var cur = _root;
        while (cur != null) {
            cur.Update(data);
            cur = cur.SubState;
        }
    }

    Ditto for fixed update...
}

If you have some formal programming knowledge, it's useful to think of a hierarchical state machine as a linked list

So the "root" or first state in the chain has a null super state, a null sub state means its the end of the chain. And state changes is equivalent to swapping a node in a linked list, as you swap super and sub state links

[gleaming bough]

ah, makes sense, similar to how I had it, but can children navigate up the tree/(I guess in your analogy is it a singly- or doubly- linked list),

also how do you pass up info from great-grandchildren states or whatnot? Is it that all states make requests to a state manager, or can states themselves know of and navigate to other states?

Ex, say I have:

PlayerHFSM
 |- MovementState
 |   |- GroundedState
 |   |- AirborneState
 |   '- DashingState
 '- InteractingState
     |- DialogueState
     '- UseObjectState

How do I switch from GroundedState > DialogueState?

Also: can I switch to InteractingState without entering Dialogue or UseObject state?

Lastly, what if I want to be in 2 states simultaneously? Ex: (the InteractionState is poor for this example) but what if I wanted to be in GroundedState and in UseObjectState at the same time? Do I need 2 state machines?

Thanks for all your help so far btw!

[gleaming bough]

btw _machine is a reference to the root statemachine right? because how I set up my original hfsm is that every state that has child states has its own state machine that it uses to navigate the child states, but this means that the root doesnt know all its children so it needs to search for them. Did you implement hierarchy the same way or some other way?

[visual zephyr]

You won't need two state machines. What you have is closer to a tree. You can flatten that. If you're thinking of you're machine as a linked list of states. Think of each index in the linked list as a category of state, where each category describes a group of states that are generally mutually exclusive. For example, you can never be both grounded and airborne, so they would swap between each other at the same index. I'd assume you can be grounded and in dialog at the same time so that would go in a different index, but you can't be dashing and in dialog at the same time so they would go in the same index. If you end up in a situation where this doesn't work cleanly, I have a method on my StateMachine for searching for a state:

Crouching {
...
if (Machine.HasState<Sprinting>())
    Machine.SwapStates(this, Standing);
...
}

StateMachine {
    bool HasState<T>() {
        var cur = _root;
        while (cur != null) {
            if (cur is T) return true;
            cur = cur.SubState;
        }
        return false;
    }
}

A linked list iteration is slow so you can make better ways to check if a state exists but this is core idea.

[visual zephyr]

_machine is a StateMachine class i use to encompass my States. I put the operation logic in the StateMachine class such as state swapping, finding states, and event dispatching

[gleaming bough]

wait what? how are you representing hierarchy in a flat linked list? every item in the list has every permutation of simultaneously possible states? I dont fully understand what structure youre using

[visual zephyr]

Maybe this will make more sense. The hierarchy comes from layers of states. Each layer has multiple states but only one state can be active per layer.

Each layer is an index in the linked list

[gleaming bough]

Ohhhh, so its less of a state tree and more so that you have a list of active states that can be switched between, very smart actually, its like multiple flat statemachines organized into one. So im guessing you'll also have null or inactive layers for state layers youre not in entirely? Or does every layer have a state? Also, does switching on the layers happen by calling the state machine or is controlled by the states in the layer?

(The linked list makes a lot more sense now btw lol)

[visual zephyr]

In a sense you will always be in a state layer. Just not necessarily do anything with it. For example, the None state in the interaction layer is still in the layer, it just does nothing except check for the conditions needed to swap to either the Dialog or Interact states. As a rule of thumb I always keep state swapping logic contained to that layer. Like, the player shouldn't be able to move while in dialog, but ill put that logic in the Moving state to swap back to Idle. Such as what I showed above with the check if Sprinting code.

Instead of putting in the Dialog state as if its telling another layer to be something else.

The swapping is triggered by the individual states, but the actual swap logic is done in the state machine. I give States a reference the the StateMachine they're owned by so they call Machine.SwapStates(from, to) which will perform a typical linked list node swap which is the state change

[gleaming bough]

So there's no null states on any layer because you always need a state that can check the logic for when to swap to another state? Because assuming state A and B were on different layers and you wanted A to trigger a change in B (but the change logic was confined to A's update). Also if youre having parent states as different layers, can you disable layers such that youre forced to change the parent to reenable those layers to change them?

Ex,

1 [Move (active), Cinematic]
2 [Cutscene, Dialogue] (disabled)
3 [Grounded (active), Airborne]

So to change layer 2, you have to change from move to cinematic, to enable layer 2 and disable layer 3?

Btw checking for switching in states seems annoying if the change needs to be triggered by a state on another layer

[visual zephyr]

Since cutscene and dialog are only active if grounded and airborne arent they would really exist on the same layer. But I see what you're wanting to do by logically separating gameplay from cinematic states. Since any node in a linked list can be used as a root of another sub linked list, you can treat the Move and Cinematic states as two different roots. Instead of a typical swap which will update both the sub and super state links you can use a replace operation which will only update the super state link

The checks for other layers is really for minor edge cases like my if sprinting, then stand up example. Major functionality like switching between gameplay and cinematics would use this, or just straight up wouldn't use the character controller since it might just be an animation that plays completely independent of player control

[gleaming bough]

so its like instead of switching out one node, you switch out a chain of nodes in a linked list? does ordering matter in this case? thanks for all your help so far btw :D