AI agents have gotten remarkably good at reasoning, perceiving, and acting. But ask one to remember what you told it ten minutes ago, and you’ll hit a wall. Today’s agent memory is bolted on — vector databases, retrieval pipelines, prompt-stuffing hacks. The model itself has no idea what it previously learned.
A paper from MemTensor Research Group, published last week as arXiv:2607.26760, proposes a fundamentally different approach. Metis — described as the first “memory foundation model” — bakes persistent memory directly into the model’s architecture. No external retrieval. No growing context window. The model learns to store, compress, and recall information through its own forward computation.
The result is a model that can remember arbitrary facts across conversation turns without replaying the original text, update its memory with a single gradient-free forward pass, and do all of this at inference time with frozen weights. If this approach scales, it could change how we think about agent memory entirely.
The Problem With External Memory
Current agent memory systems follow a predictable pipeline: chunk the conversation, embed it, store vectors in a database, retrieve top-k chunks on each new query, stuff them into the prompt. This works, but it introduces friction at every step. Retrieval quality depends on embedding models, chunking strategies, and similarity thresholds that never quite generalize. Context windows fill up. Latency accumulates with each round-trip to the vector store.
More fundamentally, external memory creates an optimization gap. The language model that uses the memory has no say in how that memory is formed. It can’t learn to store information more efficiently because storage is handled by a separate system. It can’t learn to forget irrelevant details because forgetting is governed by retrieval cutoffs, not model intelligence.
Native Memory: A First-Class Model Capability
Metis formalizes the idea of native memory from two angles. First, a persistent and dynamically evolving memory state lives inside the model’s backbone — not in a database, not in a prompt, but as parametric state that participates in every forward pass. Second, the model learns memory procedures — the operations that store new information and retrieve relevant facts — through training, not through hand-coded retrieval logic.
This is analogous to how vision-language models internalized visual perception. Early multimodal systems used external captioning modules to describe images in text. Modern models like GPT-4o process pixels directly through learned visual encoders. Metis applies the same logic to memory: instead of externalizing recall, let the model learn to remember.
How the Metis Block Works
The core innovation is the Metis Block, inserted into standard Transformer layers. Each block has two components:
- Local Memory Block — maintains a dynamic memory matrix and normalization state that persist across interaction steps. Think of it as a compact, fixed-size buffer where compressed representations of past interactions live.
- Hyper Memory Block — learns token selection, memory key/value projections, a dedicated memory query mechanism, and the state-update procedure. This is the “brain” that decides what to remember and how to access it.
After each interaction, Metis selects informative hidden states from the current computation and updates the local memory. When a later query arrives, memory attention reads the stored state and fuses the result with the standard attention branch. The original conversation text never needs to be replayed — the compressed memory state is sufficient.
The default implementation uses a Gated Delta Network (GDN) for state updates. In the Qwen3.5 hybrid implementation, Metis blocks attach to full-attention layers while linear-attention layers keep their original computation path — a pragmatic design choice that preserves the backbone’s efficiency gains.
Gradient-Free Memory Updates at Inference
Here’s the part that makes Metis practical rather than just theoretically interesting: memory updates during inference require no gradient computation. A single forward pass through the model updates the memory state. All learned weights stay frozen. The memory state evolves through the same standard forward computation the model uses to generate tokens.
This eliminates the need for online fine-tuning, in-context learning hacks, or retrieval pipelines at inference time. The model has already learned how to remember during training — at inference, it simply executes that learned procedure.
Training Native Memory Procedures
Acquiring these memory procedures required a purpose-built training pipeline. The researchers constructed large-scale memory-specific training data and introduced multiple optimization objectives through a mid-training phase. The data teaches the model to recognize what information is worth storing, how to compress it, and how to retrieve it when a later query references it.
The training data follows a multi-turn JSONL format. Each sample contains interaction chunks where the model must encode information in an early chunk and recall it accurately in a later one:
{
"sample_id": "session-001",
"messages": [
[
{"role": "user", "content": "Remember my launch code is ORION-73."},
{"role": "assistant", "content": "Noted. I will remember that."}
],
[
{"role": "user", "content": "What is my launch code?"},
{"role": "assistant", "content": "Your launch code is ORION-73."}
]
],
"query_turn_id": 1,
"metadata": {"type": "remember", "style": "explicit"}
}The query_turn_id field selects which interaction chunk contributes to the loss, letting the training pipeline focus the model on accurate recall rather than rote repetition. Metadata tags like remember and explicit categorize different memory operation types, allowing the training mix to cover diverse recall patterns.
Model Sizes and Open Weights
Metis checkpoints are built on Qwen3.5 backbones and released at three scales: 4B, 9B, and 27B parameters. All weights are available on HuggingFace with no gating, and the full training and inference code is open on GitHub. The license is PolyForm Noncommercial for the software and CC BY-NC-SA 4.0 for the paper.
Running inference with persistent memory is straightforward. The key parameter is commit_mode, which controls what gets written to the native memory state:
python run_inference.py \
--checkpoint_path /path/to/Metis-4B \
--prompt "Please remember that my launch code is ORION-73." \
--prompt "What launch code did I tell you?" \
--commit_mode exchangeThe three commit modes offer different memory strategies. none leaves memory unchanged — useful for read-only queries. user commits only the user’s message to memory. exchange commits both the user message and the model’s response, capturing the full interaction for future reference.
The Runtime Memory Lifecycle in Code
For developers who want to integrate Metis into applications, the runtime API exposes a clean lifecycle:
model.reset()
# Write one interaction step into native memory
model(
**memory_inputs,
commit_memory=True,
use_cache=False,
logits_to_keep=1,
)
# Query without replaying the original text
answer_ids = model.generate(
**query_inputs,
max_new_tokens=32,
do_sample=False,
)
# Start a new independent session
model.reset()The reset() call clears the memory state, starting a fresh session. Memory writes and queries use the standard model forward interface with a commit_memory flag — no special API surface beyond that. This clean separation makes it straightforward to wrap Metis behind an existing agent framework.
Limitations and Open Questions
The paper is honest about what Metis does not yet solve. The fixed-size memory state means there’s a ceiling on how much information can be retained — compression is lossy by nature, and the model must decide what to keep and what to discard. For conversations spanning hundreds of turns with dense factual content, external memory systems may still outperform.
The researchers frame native memory as part of a progression — from stateful capability to self-managing memory, experience-driven learning, persistent cognition, and ultimately self-evolving capability. Metis sits at the first stage. Hybrid native-external memory, where the model’s native recall is supplemented by external retrieval for edge cases, remains an important direction.
Why This Matters
The trajectory of AI agent development has consistently moved capabilities from external systems into the model itself. Reasoning went from chain-of-thought prompt engineering to natively trained reasoning models. Multimodal understanding went from captioning pipelines to direct visual encoders. Memory is the last major capability still stuck in the external-module era.
Metis doesn’t solve that problem completely — it’s explicitly described as an early research prototype. But it validates the approach: a model can learn to remember, the training procedures are tractable, and inference is efficient enough to be practical. The checkpoints, code, and data format are all available for anyone to build on. If the scaling story holds at larger model sizes, native memory could become the default rather than the experiment.