Current best practice for using Agents with Unity?

[quartz axle]

Has anyone figured out a way for using agents properly with in-scene objects?
Specifically, adding UI, positioning it correctly and generally understanding the existing scene?
(Being able to wire up public properties and game objects with eachother in the scene)
It seems that currently bare bone agents without proper prompting lean towards generating everything at runtime, which isnt the optimal unity workflow.

[reef vale]

A ‘setup’ script at runtime is easier for a model to evaluate and reason about than a MCP that creates semi opaque assets that a model can’t properly ‘see’ anymore. If your workflow doesn’t involve humans in the loop that tune editor values, the script is a reasonable approach. You may have to inform the model exactly what kind of human checkpoint it needs to construct, otherwise it will always opt for its pre trained bias: defensive, locally smallest solution. Actual best practices don’t exist yet for anything related to agents/model use. Most of the stuff that you hear is anecdotally true but often not generalizing to your situation.

[quartz axle]

Il give it a try by instructing it to default to in-scene gameobjects with editor tunable values.
I wonder how well they are able to design UI for in-scene objects vs runtime creations

[reef vale]

That works quite well if you’re specific what you want. Works better for ui-toolkit/imgui than ugui

[near bobcat]

Ui depends on the model gpts tend to be worse with ui compared to Claude. Though, I haven't tested ui with gpt 5.4 yet. Also, in general llms were always bad with spatial reasoning, and while recent models are way better at that, I still wouldn't trust them with setting positions/rotations reliably.

[quaint willow]

I’ve been patching the assistant package for exactly this gap. I agree that plain runtime setup scripts are easier for a model to reason about than opaque editor mutations, but this fork is a lot less black-box than stock now.

It can now work with in-scene/UI state much more directly: inspect scene info, read UI layout/screen rects, get interactive regions, raycast authored UI, capture the Game view, ensure named UI hierarchies exist, set specific layout/display properties, and assign serialized scene or prefab references directly. So it’s not limited to “generate everything at runtime” anymore.

In practice that means it can do things like create or repair a Canvas subtree, wire public fields/GameObject refs together, adjust anchors/text/colors/button state, preview the changes first, then save them. That still doesn’t make the model magically good at spatial reasoning, but it pushes the workflow much closer to normal Unity editor-authored objects with tunable values instead of runtime-only setup scripts.

I also patched the MCP/relay startup side so it waits for stable editor states and recovers from broken empty-tool sessions, which makes agent loops a lot less flaky during compile/import/playmode transitions.