u/AnarchyDex

most "AI for game dev" tools either generate C# and hand it to you, or live inside the Editor as a chat plugin. both have the same problem they can't see runtime state, so they can't tell you whether what they intended actually happened. you paste in a script, something's off, and you don't know if it's the agent's fault, your project, or a serialized reference that didn't update.

i've been building a different approach. desktop app that holds a live websocket connection to the Editor. the agent reads console output, inspects actual component values, and verifies that the operations it executed produced the expected state. static project context, augmented by live runtime state.

stack:

• Electron + React on the client, LangGraph.js agent in the main process
• C# bridge package inside Unity that listens on a websocket and executes operations via Editor APIs
• Next.js control plane proxying LLM calls (Anthropic direct + OpenRouter)
• Qdrant for RAG, retrieved via tool call rather than system-prompt injection (system-prompt injection kills cache hits)

stuff that worked:

• consolidating 18 granular tool wrappers down to 5-7 workflow tools. way better tool-selection accuracy, fewer compounding errors across steps.
• two-tier model setup with prompt caching wired end-to-end. Haiku for the fast stuff, Sonnet for harder multi-step tasks. warm sessions are way cheaper than running this without caching.
• verifying every operation by reading back state. catches a lot of silent failures (component added to wrong object, ref not propagated, etc).

stuff that didn't:

• spatial reasoning is a model problem, not a tooling problem. perfect runtime visibility doesn't help the agent figure out why the camera clips through a wall.
• early attempts at giving the model lots of granular tools - more options just made it worse at picking.
• trying to jam RAG into the system prompt for "always-on context." killed caching, cold start cost dominated.

next up is play-mode integration so the agent can actually run the game, watch what happens, and iterate. right now runtime visibility is read-only.

curious what other people building editor-style agents (for any tool, not just Unity) are running into. the runtime-state-vs-static-context tradeoff feels general.

Full-stack dev here. C# was the easy part the language is fine, the tooling around it is fine. What got me was Unity as a development environment. The disconnect between code and runtime state. The way half the actual game logic lives in scene files and prefabs as serialized data instead of code, so reading the codebase tells you almost nothing about how the game actually behaves. Domain reload wiping your state every time you hit play. Scene and prefab merges in git being their own special hell. Tutorials that were correct in 2022 referencing APIs that don't exist anymore. The render pipeline mess picking URP vs HDRP vs Built-in feels like choosing which set of broken tutorials you want to use. AssetDatabase quirks where you touch a file the wrong way and corrupt your project's metadata.

None of this is "Unity is hard to learn." It's more like Unity has its own way of doing everything and a lot of it actively fights how I'd build software anywhere else.

Curious if other devs from other stacks (web, backend, mobile, ML) had the same experience. What was the moment you nearly bailed? Did you push through?

Full-stack dev here. C# was the easy part the language is fine, the tooling around it is fine. What got me was Unity as a development environment. The disconnect between code and runtime state. The way half the actual game logic lives in scene files and prefabs as serialized data instead of code, so reading the codebase tells you almost nothing about how the game actually behaves. Domain reload wiping your state every time you hit play. Scene and prefab merges in git being their own special hell. Tutorials that were correct in 2022 referencing APIs that don't exist anymore. The render pipeline mess picking URP vs HDRP vs Built-in feels like choosing which set of broken tutorials you want to use. AssetDatabase quirks where you touch a file the wrong way and corrupt your project's metadata.

None of this is "Unity is hard to learn." It's more like Unity has its own way of doing everything and a lot of it actively fights how I'd build software anywhere else.

Curious if other devs from other stacks (web, backend, mobile, ML) had the same experience. What was the moment you nearly bailed? Did you push through?

Full-stack dev here. C# was the easy part the language is fine, the tooling around it is fine. What got me was Unity as a development environment. The disconnect between code and runtime state. The way half the actual game logic lives in scene files and prefabs as serialized data instead of code, so reading the codebase tells you almost nothing about how the game actually behaves. Domain reload wiping your state every time you hit play. Scene and prefab merges in git being their own special hell. Tutorials that were correct in 2022 referencing APIs that don't exist anymore. The render pipeline mess picking URP vs HDRP vs Built-in feels like choosing which set of broken tutorials you want to use. AssetDatabase quirks where you touch a file the wrong way and corrupt your project's metadata.

None of this is "Unity is hard to learn." It's more like Unity has its own way of doing everything and a lot of it actively fights how I'd build software anywhere else.

Curious if other devs from other stacks (web, backend, mobile, ML) had the same experience. What was the moment you nearly bailed? Did you push through?

Full-stack dev here. C# was the easy part the language is fine, the tooling around it is fine. What got me was Unity as a development environment.

The disconnect between code and runtime state. The way half the actual game logic lives in scene files and prefabs as serialized data instead of code, so reading the codebase tells you almost nothing about how the game actually behaves. Domain reload wiping your state every time you hit play. Scene and prefab merges in git being their own special hell. Tutorials that were correct in 2022 referencing APIs that don't exist anymore. The render pipeline mess picking URP vs HDRP vs Built-in feels like choosing which set of broken tutorials you want to use. AssetDatabase quirks where you touch a file the wrong way and corrupt your project's metadata.

None of this is "Unity is hard to learn." It's more like Unity has its own way of doing everything and a lot of it actively fights how I'd build software anywhere else.

Curious if other devs from other stacks (web, backend, mobile, ML) had the same experience. What was the moment you nearly bailed? Did you push through?