Training to Learn from Context Using Synthetic Data Generated from Perturbed Public Documents
Synthetic perturbed docs generate filtered reasoning traces to train LLMs; a 35B student reaches 24.6% on CL-bench without human labels.
Key Takeaways
- LLMs often struggle with tasks that require reasoning over new inference-time context, because public web data is largely exhausted and can encourage memorization instead of genuine context learning.
- The research builds a synthetic training pipeline by perturbing high-quality public documents and generating context-dependent reasoning traces and answers without human annotation.
- The pipeline has four stages: rewrite documents to reduce memorization, generate questions and rubrics requiring document reasoning, answer using the document as context, and filter samples to keep only those that truly depend on the provided context.
- Training a Qwen3.6-35B-A3B student on about 10k synthesized samples improves CL-bench performance from 13.7% to 22.8% after SFT and 24.6% after rubric-reward reinforcement learning, placing it near a 2.4T-parameter frontier model.
- The gains mainly transfer to long-context understanding, instruction following, and reasoning, while code generation and general knowledge remain largely unchanged, leaving open questions about less public and more specialized domains.
Large language models often struggle when a task depends on complex context supplied at inference time rather than on knowledge baked into their weights. Collecting human annotations for such task-specific context is expensive and hard to scale, and the public web is largely exhausted: much of it was already seen during pretraining, so naive training on public documents rewards memorization instead of genuine context learning.
A new research effort attacks this problem by perturbing high-quality public documents and using a model to generate context-dependent reasoning traces and answers, which then train a smaller student model. No human annotators are involved.
How the synthesis pipeline works
The pipeline has four stages. First, it rewrites source documents to lower the risk that the model simply recalls them. Second, it generates questions and rubrics that require reasoning over the document. Third, it answers those questions with the document as context. Fourth, it admits only samples that genuinely depend on the document, filtering out cases the model could solve from parametric knowledge alone.
The result: roughly 10k samples drawn from 3.5k documents.
The numbers on CL-bench
Training a Qwen3.6-35B-A3B student on this data moves CL-bench performance from 13.7% to 22.8% after supervised fine-tuning. A subsequent rubric-reward reinforcement learning stage reaches 24.6%.
That places the 35B student in the same range as Qwen3.8-2.4T, a frontier model with over a trillion parameters, which scores 23.9% on the same benchmark.
Where gains transfer, and where they do not
Improvements were broad across long-context understanding, instruction following, and reasoning. Code generation and knowledge stayed mostly flat, suggesting the pipeline targets context-grounded reasoning specifically rather than general capability.
The authors position the work as a reproducible, scalable route to better context learning, and as a foundation for further research on context-grounded reasoning. The open question is whether the approach holds when documents are less public, domains are more specialized, or the filtering step admits too few samples to sustain training.
Comments
No comments yet. Be the first to comment.