Fixed Models, Growing Expertise: Model-Agnostic Learning from Real-World Deployment for Multimodal Medical AI

Meta description: FMEE helps frozen language and vision models learn from deployment experience by storing skills, knowledge, and visual examples in a model-agnostic way.

Tags: FMEE, frozen models, LLMs, VLMs, deployment learning

Featured image: abstract technology style, neural memory pathways, no real people, no company logos, no text.

FMEE, short for Frozen Models, Evolving Expertise, presents a model-agnostic way to let frozen large language models (LLMs) and vision-language models (VLMs) keep learning from deployment experience without updating their weights. The issue it addresses is familiar to anyone who has watched a released model age in production: the weights stay fixed, while the questions, tools, and reference materials users bring to the system change. Instead of retraining the model, FMEE treats experience as external expertise that can be revised and checked.

Background: Frozen Models Meet Changing Workflows

A frozen model is one whose parameters are no longer trained after deployment. That stability is useful for predictable behavior and lower operational risk, but it also makes adaptation difficult. When a model encounters new task steps, recurring user errors, or reference documents that were not part of its training data, the usual response would be fine-tuning or a new release. FMEE changes the sequence: the model stays frozen, while the layer of guidance around it evolves. This is where deployment experience becomes useful. The framework is described as model-agnostic, meaning it is not tied to one particular architecture and can be paired with different LLMs and VLMs.

Development: Three External Expertise Layers and Batch Updates

FMEE organizes learned experience into three forms of external expertise. The first is Skill, a natural-language procedure for reasoning and tool use. It is rewritten after each batch, so the instructions themselves can become clearer, shorter, or better suited to observed failures. The second is Knowledge Memory, which stores reusable information drawn from earlier cases or trusted sources such as PubMed. This gives the system a place to hold facts and references that the frozen model cannot update internally. The third is MMKB, a collection of visual reference cases with source answers. When a new image arrives, the model compares the retrieved case with the current image before answering.

The update cycle begins after a batch of deployments has been scored. An optimizer then proposes a change based on scored trajectories, which are traces of the steps the model took and how they were evaluated. This is not a simple memory dump. The framework turns repeated interactions into candidate improvements: a revised skill, a new knowledge entry, or a visual example that may help future cases.

Latest Progress: Testing Updates Against New and Old Cases

The most important safeguard in the described process is how updates are accepted. FMEE does not rely on a fixed validation set. Instead, a proposed update is tested on both new cases and past cases. It is kept only if it improves performance on new cases while avoiding degradation on earlier ones. This dual check targets a common problem in continual learning: fixing a current failure by forgetting an old one.

The current stage is therefore a controlled external memory loop, not an autonomous model rewrite. The frozen model still depends on the quality of the scored trajectories, the reliability of stored sources, and the rules used to compare new and old cases. For readers, the practical takeaway is that FMEE shows how deployment learning can happen at the surface of a model: through skills, knowledge, and visual references that can be inspected, replaced, and tested before they become part of future responses.