AgentGarten: Programmable Worlds for Self-Evolving Agents

Meta description: AgentGarten combines simulators, game engines, and a neural renderer to create real-time environments where agents act, review rounds, and write playbooks.

Tags: AgentGarten, neural renderer, game engines, Adversarial Forcing, interactive training

Featured image: A stylized abstract network of light threads connecting a game-engine grid to a neural renderer, with no text or logos.

AgentGarten frames agent training around the environment itself. Agents learn through interaction, and the set of behaviors they can discover depends on the practice world they are placed inside. The system joins simulators and game engines with a shared neural renderer to create real-time interactive environments. In that setup, the engine keeps the world state and applies the rules, while the renderer converts geometry into observations that pretrained agents can act on. The result is a training loop where behavior, perception, and environment structure are connected from the first round onward.

Background: The Practice World Sets the Learning Boundary

The background idea is straightforward but consequential. If an agent learns by doing, then a narrow or inconsistent environment can cap what the agent learns, even if the agent itself is capable. A world that changes state unpredictably, fails to enforce its own rules, or provides observations that do not match the underlying geometry can produce behaviors that are brittle outside that setting.

This is why the environment layer matters as much as the agent layer. In many training pipelines, physics, rules, rendering, and policy learning are handled by separate components. The agent receives a stream of observations, but it may be unclear whether those observations faithfully reflect the state that the simulator is tracking. AgentGarten addresses that gap by making the practice environment a unified real-time system rather than a collection of disconnected pieces.

Development Process: Engines, Geometry, and a Shared Neural Renderer

The development path described for AgentGarten begins with a base world built in code. That world is not only simulated; it is also rendered through the same interface. Simulators and game engines are used to maintain state and run the rules, so the interaction remains grounded in a consistent set of possibilities. This gives agents a stable setting in which actions can have repeatable consequences.

The perceptual side is handled by a shared neural renderer. Instead of relying only on handcrafted visuals, the renderer is distilled with Adversarial Forcing and turns the geometry of the world into observations for the agents. Those observations are then fed to pretrained agents, which act inside the rendered frames while the engine continues to track the state behind them.

The loop does not stop at action. After each round, the agents review what happened and write playbooks for later agents. This adds a memory-like layer to the process: behavior learned in one round can be packaged and passed forward, giving subsequent agents a starting point rather than forcing them to begin from a blank slate.

Latest Progress: Round 4, Round 10, and Growing Worlds

The clearest timeline signal in the material comes from a hide-and-seek example, measured in rounds rather than calendar dates. By round 4, hiders build shelters. By round 10, seekers use ramps. Those milestones are small, but they show how behavior can change rapidly once agents are placed in an interactive world that preserves rules and state while supplying usable observations.

The shelter and ramp steps also illustrate why environment consistency matters. A shelter is only meaningful if the world keeps its state and rules intact, and a ramp is only useful if the agent can perceive and act on the geometry reliably. The shared rendering path links those two conditions: the geometry that defines the object becomes the observation that guides the behavior.

At the latest stage described, new worlds can be written as code and rendered through the same interface. That means the environment can expand alongside the agents. Instead of training only against a fixed map or a static task, AgentGarten allows the practice setting to grow as agents develop, keeping the learning boundary moving with the capabilities being tested.

What to watch next is whether this combined loop produces deeper compounding behavior across longer sequences of rounds. The current description points to a system where environment, perception, and review are tightly coupled, making each round both a test of the agent and a record that can be carried into the next.