Working on a Private Fork of Unity AI Assistant Package with UI tools and more.

[eager lantern]

I’ve been working on a private fork of com.unity.ai.assistant because I kept running into the same issues people mention here around agents, in-scene objects, UI authoring, and flaky MCP/relay behavior during normal editor state changes.

The main goal was to make the workflow less “generate everything at runtime” and more “work with normal Unity editor-authored objects and tunable values.”

What I’ve added / patched so far:

• more reliable MCP + relay startup/reconnect behavior during compile/import/play mode transitions
• recovery from broken empty-tool MCP sessions
• scene/UI tooling so agents can inspect more of what already exists instead of operating as blindly
• UI helpers for things like reading layout state, interactive regions, raycasts, Game view capture, ensuring named UI hierarchies, and setting layout/display properties
• serialized scene/prefab property assignment so wiring references is less dependent on custom runtime setup scripts

It’s still not magic, and models still need good prompting/checkpoints, especially for spatial work. But it’s been a lot better for editor-driven workflows than stock behavior in my testing.

I’m keeping the fork private for now since it’s based on Unity’s package, but if people are interested in the approach or want to compare notes, ask here or DM me. If you want to try it, DM me your GitHub username and I can add you read-only.

[cloud pilot]

I have a question about this project, can it be modified to not call back to unity cloud and accept other providers? I can share a quick vid of something im working on that could be very complimentary

I'm using a connector I made that persists through domain reloads, I personally haven't noticed any issues with reliability.. But Im not using the ai assistant, Im using my own tooling connected to the mcp server

[deft coral]

The AI Assistant can already use other agents like Claude Code or Codex. Unless you mean something different.

[cloud pilot]

I mean local llm - like using lmstudio

[cloud pilot]

Id rather keep all ai/llm stuff internal, (no leaking data to providers)

In my case, Im using opencode, so i can control what goes to what provider. and i can keep requests that may have proprietary data going to lmstudio.

[deft coral]

It can't do local llm's at the moment. If you are in the beta server you can request that feature

[cloud pilot]

unfortunately im not in the beta channel. i wouldn't mind jumping in though if you could give me some direction where (not seeing in discord). the mcp server handles local llm fine until it hits a tool with a dependency on the "model" field being set. ive basically made my own version of the assistant side (dont want to hijack this thread tho)

[deft coral]

Its a separate server. You'll need to sign up for the new 2.0 beta to get access

[eager lantern]

i use the codex app. so it doesn't use the api, in order to use the assistant you need to use api. So i use the mcp bridge. I am keeping up to date when I get the chance every time they release another Unity Ai assisant version, I do a diff on the official patch and compare to what I have added, then i keep the MCP tools that are still useful and update to the ones they have added.
I am also aoditing the MCP tools for efficiency, i built an editor tool that tracks all the tool usage and keeps a running tab on all the amounts of data used.
My goal is to keep the MCP as efficient as possible and not get bloated and used too much model usage.

Sorry for not getting back to your questions. I will try to keep tabs on my thread more, feel free to ping me also.

[eager lantern]

in order to use the AI Assistant you need to use an API. So I mainly use codex APP which used a different usage that the openAI api.

It would be nice to have a native OpenAI login for the assistant instead of just api. I have sent and email to the beta team asking about it.

In the meantime as I work with the MCP bridge I am having the codex agent audit the tools and work I'm doing and suggest new tool or updates to the package, so I am still maintaning my own version of the package mainly to work with Native Windows Codex APP.

Quick update on the private fork for anyone following this thread:

Current base/version: com.unity.ai.assistant 2.3.0-pre.2.

I’m intentionally keeping Unity’s package version as the base and layering fork-specific patches on top, rather than inventing a separate package version.

What upstream 2.3.0-pre.2 already gives me as the base:

• Graceful relay shutdown/disposal
• Better domain reload handling
• Batch-mode MCP auto-approve support
• Unity.Web.Fetch
• Unity.GetDependency
• Official 2D capture and improved multi-angle scene capture

What I’m still patching on top of that:

• More reliable MCP + relay startup/reconnect during compile/import/play mode/domain reload transitions
• Recovery from broken “empty tool list” MCP sessions by preserving the last good tool snapshot
• Better bridge status/debug metadata around reload, recovery, and tool discovery state
• More tooling for working with normal editor-authored scene objects and UI instead of forcing everything through runtime-generated setup
• Serialized scene/prefab property assignment so references can be wired directly without depending on custom runtime setup scripts

Latest fork-only tools I’ve added:

• Unity.Asset.ConfigureSpriteImport
  Tune sprite import settings from MCP without manual importer clicking.
• Unity.Scene.SetSerializedProperties
  Set serialized properties directly on scene objects/components.
• Unity.Prefab.SetSerializedProperties
  Do the same for prefab assets.
• Unity.UI.EnsureNamedHierarchy
  Ensure a named UI hierarchy exists under a target root/canvas.
• Unity.UI.SetLayoutProperties
  Set authored UI layout/display values like anchors, pivots, size, active state, text/image values, button interactability, etc.
• Unity.UI.GetLayoutSnapshot
  Read a structured snapshot of current UI layout state.
• Unity.UI.Raycast
  Check what UI is actually hittable/blocking at a screen point.
• Unity.UI.GetInteractiveRegions
  Return stable screen-space interactive regions for authored UI.
• Unity.UI.CaptureGameView
  Capture the current Game view to disk for inspection/debugging.
• Unity.Runtime.GetVisualBoundsSnapshot
  Inspect runtime visual bounds/screen footprint for live scene objects.
• Unity.Project.ScanMissingScripts
  Scan scenes/prefabs for missing script references.
• Unity.Object.ValidateReferences
  Audit serialized object references for null/missing links.

Net result so far: this still is not magic, and models still need good prompts/checkpoints, especially for spatial work. But for editor-driven workflows, it has been a lot better in practice than stock behavior, because the agent can inspect more of what already exists and wire authored data directly instead of guessing or rebuilding everything at runtime.

I’m keeping the fork private for now since it’s based on Unity’s package. If people want to compare notes on the approach, ask here or DM me. If you want read-only access, DM me your GitHub username and I can add you.

[cloud pilot]

hey,,, one sec.. ill dm you

[dreamy crescent]

@eager lantern sounds very interesting, would love to test the repo. I DMd you