Okay so need some help with deciding

[pliant turtle]

Every time I try to research ways this has been tackled (modular AI) I either run into how to make decision-making modular with different things like BT, FSMs, HTNs, and the likes (I already have my own decision making figured out already), or its an article/talk that just says "loosely couple everything" without explaining a good and manageable way to do that. If I go with a fully GameObject component based approach, would using GameObject.SendMessage while passing callbacks be both performant and sufficient enough for potentially a hundred agents with several components (for reference, most actions would not be making repeated messages. For example, a move action would send a message to initiate the move and be completed once the callback is called)? Alternatively, is making my own simple message system better, as I believe GameObjects use reflection for theirs? Another thing I've considered is having a sort of middle-man class that would take messages that have a type, lookup a dictionary with different types that are defined (locomotion, animation, common stuff used) that are tied to a component belonging to the agent, and then just route the message to that component to deal with as it chooses, and call the associated callback with that message after completion. I'd also have the option to just send the message to every component in the dictionary, or messages could have a list of types to send it to.

For reference, this is for refactoring an existing system where I basically just tossed everything into an agent superclass and tightly coupled it all. I want more rapid prototyping that loosely coupling would grant me, I just can't find an ideal way to do it that grants me the same level of control in my behaviors and actions and there's very little info out there on the architecture outside of decision making for game AI, at least that I can find.

[dense nacelle]

Every time I try to research ways this has been tackled (modular AI) I either run into how to make decision-making modular with different things like BT, FSMs, HTNs, and the likes
Because that's what AI means. And what you want to implement is just a modular character controller. It could be implemented the same way as for the player character. There's nothing "AI" about it.

[dense nacelle]

While a superclass might not be a good idea, you'd still need some kind of character controller class that orchestrates the work of all the "modules" that you might have. There's a limit of how modular you can make things without wasting too much time on it.

Maybe start from thinking of what kind of variants you need your game and derrive what systems you might need to isolate into a separate module.

Overthinking the complexity/modularity of your system might lead you to a refactoring hell, when you never have any actual progress in your project.

[pliant turtle]

Also probably should've specified here although I do intend on using this for a game, rn I'm just spending free time working on the AI system as a framework for fun and don't mind spending a lot of time on it making it robust (where it matters though, so if it really isnt necessary I'll reconsider my design intentions). Currently pursuing a degree and this is just a background project so that part doesn't bother me as much.

I do agree though, I'm more of just wanting to make it more flexible than it is now. For example, I don't have all my logic in the superclass, stuff like my sensors, deciders, knowledge blackboards, bark manager, etc. are all separate classes that the agent class has instances of. I currently don't use interfaces/abstraction for most of these though, but that's a pretty easy change. Issue is I want to be able to reuse my behaviors across multiple agent types who might not have certain modules (like animators for prototyping agents that just use capsules). I'd also like to be able to setup test harnesses for specific modules and behaviors so reducing the reliance on needing specific components would also be nice. I'll definitely look into modular player controllers and see if that helps out.

[dense nacelle]

Well, reducing reliance is pretty simple, just make a null check, before accessing the optional system.
Anything more complex than that might not be worth it.

[sturdy gulch]

I'm making an AI system that basically is this. So far loving it. 100% component based, and actually trying to completely decouple components from each other has worked very well so far. I think there's like one component that has a reference to another, but that's it.

IMO, this sort of design pattern I think is very valid instead of behaviour trees. When you think about what a Machine-Learned AI is, it's a totally 'if-less' behaviour mechanism so this pattern is closer to that than behaviour trees are.