Mosaic Advances Agentic Reinforcement Learning Across Multiple Sessions

Meta Description: Mosaic improves Multi-Session Agentic Reinforcement Learning for Qwen3.5-4B web research agents on BrowseComp, using fewer sessions than a baseline.

Tags: Multi-Session Agentic Reinforcement Learning, Mosaic, Qwen3.5-4B, BrowseComp, AI agents

Featured Image: Abstract technology style: layered session timelines and advantage curves over a web research graph, no people, no logos, no text.

The paper proposes Multi-Session Agentic Reinforcement Learning, a training setting for AI agents whose rollouts can span an arbitrary number of sessions, and introduces Mosaic, an algorithm that assigns credit more finely across those sessions. The change matters for web-research agents because a single task often unfolds through many interactions, and treating each rollout as one undifferentiated trajectory can obscure which intermediate steps helped or hurt final success.

Background: From single trajectory to many sessions

In agent training, a rollout is often evaluated as one trajectory: the agent acts, observes the environment, and receives an outcome-based advantage. The paper’s setting broadens this structure. Each rollout may contain an arbitrary number of sessions. Each session consists of agent–environment interactions within one continuously accumulated context. Each session ultimately corresponds to one training segment with its own assigned advantage.

This framing makes the training problem more granular. If an agent solves a web-research task over several sessions, the final answer does not reveal which session contributed decisive evidence, which consumed context, and which was relatively neutral. The proposed setting creates space for methods that can score sessions separately.

Development Process: How Mosaic reshapes session-level credit

Mosaic is presented as a general algorithmic improvement for Multi-Session Agentic Reinforcement Learning. It builds on trajectory-level, outcome-based advantages, but adjusts them using relative session counts within success/failure subgroups, balancing efficiency and exploration.

The method then measures each session’s marginal information gain through changes in the ground-truth answer’s log-likelihood. In this context, log-likelihood refers to how strongly the model assigns probability to the correct answer; a change across sessions suggests how much the session moved the model toward or away from that answer. Mosaic adds a type-conditioned normalization for different sessions within each prompt group, then combines the trajectory-level advantage and session-level shaping signal to estimate a finer-grained advantage for each session.

Latest Progress: Qwen3.5-4B on BrowseComp deep-search tasks

The evaluation uses BrowseComp, a benchmark of challenging web-research questions, and Qwen3.5-4B in a ReAct-style agent loop. ReAct-style means the agent interleaves reasoning with actions rather than only producing a final answer. Training uses 1,000 deep-search tasks from REDSearcher. Training and evaluation use both the context compaction or subagent forking execution loop.

In these experiments, Mosaic outperforms the broadcasting baseline in both context compaction and subagent forking scenarios while using fewer sessions on average. The result indicates that session-level credit assignment can improve the learning signal without requiring more sessions per rollout.

Current Stage: A framework for finer-grained agent training

The paper positions Multi-Session Agentic Reinforcement Learning as a more general setting, with Mosaic as one algorithmic response to the credit-assignment problem created by multiple sessions. The project homepage at https://lark-ai-lab.github.io/Mosaic/ includes a blog with an overview of the research.

The setting turns a rollout into a sequence of sessions, and Mosaic tests whether finer session-level advantage estimates can improve learning. What to watch next is how these session-level signals extend to longer tasks, different execution loops, and additional agent benchmarks.