Self-Evolving Vision-Language Models with Program Verification

Featured image: abstract technology style with geometric shapes, gradients, and network nodes; no real people, no company logos, no visible text.

Meta description: VQS cuts label errors in self-evolving vision-language training and lifts Qwen3-VL up to 3.18 points across ten benchmarks at 2B, 4B, and 8B.

VQS targets a weak link in self-evolving vision-language models: the labels they create for themselves. These models learn from questions generated from unlabeled images, but human checks show 24% of majority-vote labels and 18% of model-judge labels are incorrect. By making the image-to-question process more structured, VQS raises answer correctness to 94%, compared with 76% for majority voting, and improves Qwen3-VL by up to 3.18 points across ten benchmarks at 2B, 4B, and 8B.

Background: self-generated labels become the training bottleneck

Vision-language models pair images with language, so training data often needs paired questions and answers. Self-evolving methods try to reduce manual labeling by asking the model to inspect unlabeled images, create questions, and then assign labels. Majority voting uses repeated model outputs to choose a consensus answer. Model judging uses the model itself, or another model, to score candidate answers. Both approaches make training scalable, but they can also teach mistakes at scale. The human-checked error rates, 24% for majority-vote labels and 18% for model-judge labels, explain why better label construction matters: if the training signal is noisy, later model updates inherit that noise.

Development process: VQS turns each image into a structured record

VQS changes the sequence. Instead of asking the model to invent a full question-answer pair directly from an image, the model first converts each image into a structured record. The source describes this record as being like a scene graph or a chart table. A scene graph captures objects and relationships in a visual scene, while a chart table organizes the values shown in charts or tables into rows and columns.

After that record exists, fixed programs write questions and compute their answers. The model is not asked to freely create facts. It only confirms simple facts one at a time. This separation matters because programs can enforce consistency: once the image is represented as data, the question and answer can be generated from that data rather than from an unverified guess. The reported result is 94% correct answers, compared with 76% for majority voting.

Latest progress: Qwen3-VL gains at 2B, 4B, and 8B

The method's practical value appears when it is used to train Qwen3-VL, a vision-language model. Across ten benchmarks, VQS improves Qwen3-VL by up to 3.18 points at 2B, 4B, and 8B sizes. It also beats the best self-evolving baseline at every size in the report. The strongest reported gain is 3.84 points at 2B after three training rounds.

The size pattern is notable. Smaller models often benefit more from cleaner training data because they have less capacity to recover from noisy supervision. At 2B, the result after three training rounds is 3.84 points, suggesting that repeated rounds can add to the gain. At 4B and 8B, the gains remain up to 3.18 points, indicating that the method is not limited to the smallest model size.

Current stage: a clearer path for self-evolving training

The current stage is not a general claim that self-evolving training is solved. It is a narrower result: when questions are grounded in structured image records and answers are computed by fixed programs, label noise drops and model performance rises. For readers tracking vision-language model training, the key signal is that VQS connects two problems—label reliability and benchmark improvement—through one workflow. The next question is whether the same structured-record mechanism can be extended across more image types, more program templates, and more training rounds without giving up its accuracy advantage.

Tags: VQS, Qwen3-VL, vision-language models, self-evolving training, AI benchmarks