Diffusion-Based Reward Models

Meta description: The thunlp/DRM repository presents a Diffusion Reward Model that treats reward modeling as conditional density estimation over p(r | x, y).

Tags: reward modeling, Diffusion Reward Model, conditional density estimation, human preferences, thunlp/DRM


The thunlp/DRM repository presents a Diffusion Reward Model that reframes reward modeling as conditional density estimation over p(r | x, y). The key idea is that human preference is not always a single clean score attached to a model output. It can have a multimodal structure: different reviewers, contexts, or evaluation criteria may imply different reward values for the same prompt and response. By treating reward as a distribution rather than a point estimate, the approach gives developers a way to model uncertainty and disagreement inside the reward signal itself.

From scalar rewards to preference structure

Reward modeling has long been a core part of training language systems that learn from human feedback. In many pipelines, a reward model takes a prompt, often denoted x, and a candidate response, often denoted y, and assigns a scalar value r that represents how desirable the response is. That scalar can then guide policy optimization, ranking, or filtering.

The repository description studies the multimodal structure of human preference. That phrase matters because preference data is frequently produced by humans who disagree in systematic ways. One person may value concision, another may value detail, another may prioritize safety, and another may prefer a different factual framing. A single average reward can hide those differences. If the goal is to understand what the preference signal actually contains, a distribution over r, conditioned on x and y, may be a more faithful object than a single number.

The conditional density formulation

The central formulation is p(r | x, y). In plain terms, this means: given a prompt x and a response y, what is the probability distribution over possible reward values r? This shifts reward modeling from “predict the score” to “estimate the score distribution.”

The proposed Diffusion Reward Model represents this distribution without committing to any parametric family. That is an important design point. Many reward models assume a simple functional form, such as a linear or neural scalar mapping, and may also rely on fixed distributional assumptions. By avoiding a hard parametric commitment, the model can better accommodate skewed, bimodal, or otherwise complex reward patterns that may arise in real human preference data.

The name Diffusion Reward Model suggests a generative density modeling approach. Diffusion is a class of generative models that learns a distribution by gradually removing noise from a corrupted sample. The repository description does not provide benchmark numbers or architecture details, so the exact implementation remains something to inspect in the code.

Repository as the current public artifact

The main verifiable object available from the source is the repository at https://github.com/thunlp/DRM. For readers, this is the place to check implementation details, dependencies, examples, and any later experimental results. The public material currently frames the work as a formulation and a proposed model, not as a fully detailed performance report.

That still gives the work practical value. Researchers working on reinforcement learning from human feedback, model alignment, ranking systems, or reward uncertainty can use p(r | x, y) as a design lens. Instead of asking only “What score should this response get?” they can ask “What distribution of scores does this response imply, and why?”

What to watch next

The next stage depends on what appears in the thunlp/DRM codebase: training procedures, evaluation settings, data formats, and any comparisons with scalar reward models. The formulation is the clearest contribution in the source: reward modeling as conditional density estimation over p(r | x, y), with a Diffusion Reward Model representing the distribution without committing to any parametric family. For now, the current stage is a public repository and a conceptual reframing of human preference modeling.