Reinforcement learning has become the defining ingredient of modern LLM post-training. GRPO, PPO, and their variants drive the reasoning capabilities that distinguish frontier models from their base checkpoints. Yet anyone who has actually run RL training on LLMs knows the dirty secret: it is fragile. Training curves collapse without warning, rewards oscillate, and models that performed well in training suddenly degrade in deployment. A recent paper identifies a structural cause hiding in plain sight — and proposes a fix that reframes what we should be optimizing.
The paper, from researchers at Tianjin University and Alibaba, identifies a problem they call training-inference mismatch. It is not a bug or a rounding error. It is a fundamental architectural reality of how modern LLM RL is implemented, and it means that every gradient update you compute might be optimizing the wrong objective.
The Split-Engine Architecture
Modern LLM RL separates rollout generation from gradient computation. During the rollout phase, an inference engine — typically vLLM or SGLang — generates candidate responses at high throughput, often using quantized models (FP8 or INT4) for speed. During the update phase, a separate training engine — FSDP or Megatron — computes log-probabilities and applies gradients in full precision.
These two engines use the same model weights, but they produce different probability distributions for the same token sequences. The causes are manifold: quantization precision differences, implementation-level numerical discrepancies in attention kernels, batching effects, and different handling of floating-point operations across backends. The training policy π and the inference policy μ diverge:
# The same model weights, two different probability engines
# Even after weight synchronization, these don't match:
#
# π_k(y|x) ≠ μ_k(y|x)
#
# where π = training policy, μ = inference policy
# This gap is persistent, not a one-time synchronization error.This gap creates a peculiar form of off-policyness. In standard RL, off-policyness arises when you learn from trajectories generated by a different policy than the one being optimized. Here, the off-policyness is structural — it exists even when the policies share identical weights, and it persists throughout training.
The Objective Misalignment Problem
Prior work recognized the mismatch and tried to mitigate it through importance sampling corrections, sample filtering, learning rate decay, or narrowing the system-level gap between engines. These approaches all share a common assumption: if you stabilize the training policy, the inference policy will follow.
The paper’s central insight is that this assumption is wrong. Improving the training policy does not guarantee improving the inference policy:
# Prior assumption (implicit in GRPO, PPO, importance sampling):
# J(π_{k+1}) - J(π_k) >= 0 ==> J(μ_{k+1}) - J(μ_k) >= 0
#
# Reality:
# The inference gap can absorb or reverse training gains.
# A "successful" training update may produce a WORSE inference policy.Think about what this means in practice. You run RL training, the reward curve goes up, checkpoints look good — but the deployed model underperforms. The training metrics are measuring the wrong thing. You are optimizing π, but what you deploy is μ. Under high mismatch (especially with FP8-quantized rollouts), the two can move in opposite directions.
MIPI: Reframing the Objective
The proposed solution starts with a reframing: instead of optimizing the training policy, optimize the inference policy directly. They call this principle Monotonic Inference Policy Improvement (MIPI). The objective becomes ensuring that each update actually improves μ, the policy your users interact with.
They decompose inference-policy improvement into three terms:
# MIPI decomposes: J(μ_{k+1}) - J(μ_k) into:
#
# Term ①: Post-update inference gap
# J(μ_{k+1}) - J(π_{k+1})
# How much do engines diverge AFTER the update?
#
# Term ②: Training-side update
# J(π_{k+1}) - J(π_k)
# The standard PPO/GRPO objective
#
# Term ③: Pre-update inference gap
# J(π_k) - J(μ_k)
# How much did engines diverge BEFORE the update?
#
# Terms ②+③ form a "sampler-referenced" update (Step 1)
# Term ① drives an acceptance check (Step 2)MIPU: A Two-Step Framework
MIPI is the principle; MIPU (Monotonic Inference Policy Update) is the algorithm. It works in two steps that address different parts of the decomposition:
Step 1 — Sampler-Referenced Policy Update. Instead of computing importance sampling ratios relative to the old training policy, MIPU references the update against the inference sampler μ. The key ratio factorizes into a pre-update mismatch weight and a current update ratio. Only the current-update ratio is clipped (standard PPO-style clipping), while the mismatch weight is truncated to control variance. This produces a candidate policy that is explicitly referenced to the deployment engine, not the training engine.
# Step 1: Truncated Importance Sampling (TIS)
#
# Standard PPO clips the full ratio: clip(π_θ(y|x) / π_k(y|x), 1-ε, 1+ε)
#
# MIPU factorizes the ratio:
# ρ_i(θ) = w_i * r_i(θ)
#
# w_i = π_k(y|x) / μ_k(y|x) # pre-update mismatch (fixed, truncated)
# r_i = π_θ(y|x) / π_k(y|x) # current update ratio (clipped)
#
# Only clip r_i. Truncate w_i at w_max=2 for variance control.
# This references updates to the SAMPLER, not the old training policy.Step 2 — Inference-Gap-Aware Acceptance. After synchronizing the candidate weights to the inference engine, MIPU estimates the post-update inference gap using a small set of validation rollouts from the updated μ. If the gap proxy indicates the update actually improved the inference policy, the candidate is accepted. If not, the update is rolled back.
This is not random rejection. The paper shows that random rollback with the same rejection rate still leads to training collapse. The inference-gap signal carries genuine information about whether training-side gains transfer to deployment.
Results: Stability Under High Mismatch
The experiments use FP8-quantized rollouts — a deliberately high-mismatch setting where the gap between π and μ is amplified. They evaluate on Qwen3-1.7B and Qwen3-4B across five mathematical reasoning benchmarks: MATH, AIME, Olympiad, Minerva, and AMC23.
The results tell a clear story. On Qwen3-4B, MIPU achieves the best average performance (66.71%) while all baselines (vanilla RL, MIS, LR-decay) collapse during training. The baseline starts strong but degrades as training instability compounds. MIPU maintains a stable trajectory throughout.
The ablation study confirms that both steps contribute: Step 1 alone improves the candidate update direction (65.36% average), but adding Step 2’s acceptance check pushes performance further (66.71%) by filtering out updates that would have harmed the inference policy.
Why This Matters Beyond Math Benchmarks
The training-inference mismatch is not specific to math reasoning. It affects every LLM RL pipeline that uses separate inference and training engines — which is essentially all of them in production. When you run GRPO with vLLM for rollout generation and PyTorch FSDP for gradient updates, you are operating under mismatch. When you use FP8 quantization for faster rollouts, you are amplifying it.
The standard practice of checkpoint selection — training for N steps, evaluating the deployed model on a held-out set, picking the best checkpoint — is implicitly an acknowledgment of this problem. You cannot trust training metrics because they measure π, not μ. MIPU makes this implicit acknowledgment explicit and builds it into the optimization loop itself.
There is a broader lesson here about RL systems in general. When your training environment and deployment environment differ — whether due to engine differences, quantization, simulator-reality gaps, or distribution shift — optimizing performance in the training environment is necessary but not sufficient. You need to measure and optimize against the deployment environment directly. The policy you train is not the policy you deploy.
The two-step pattern — construct a good candidate, then verify it against the deployment target — is a principled approach that extends beyond LLM RL. Any system with a train-serve gap can benefit from an acceptance step that validates updates against the real deployment metric, not a proxy computed in a different environment.
Practical Implications
For practitioners running RL post-training, this work suggests several concrete adjustments:
- Measure the gap. Log the probability divergence between your training and inference engines on the same sequences. If π and μ assign meaningfully different probabilities, mismatch is silently degrading your training.
- Be skeptical of training reward curves. A rising reward curve during RL training means π is improving. It says nothing about μ. Validate against the deployed model early and often.
- Quantized rollouts amplify the problem. FP8 or INT4 inference for faster generation widens the train-inference gap. The speed gains are real, but the training instability cost is hidden.
- Checkpoint selection is a partial fix, not a solution. Picking the best checkpoint by evaluation is better than trusting training metrics, but it throws away training compute on updates that were doomed from the start.
As RL post-training becomes the standard path for building reasoning models, addressing the train-inference gap will move from an academic concern to an engineering necessity. The split-engine architecture is here to stay — the computational economics demand it. The question is whether we optimize for the policy we train or the policy we deploy. MIPI makes a compelling case that we have been optimizing the wrong one.