How I implemented Self-Adapting Language Models with LoRA and QLoRA on an RTX 4060

Self-Adapting Language Models (SEAL): How I Taught an LLM to Teach Itself

Implement self-improving LLMs locally. A complete guide to ReST-EM, LoRA, and 4-bit quantization on a single 8GB VRAM GPU.

TL;DR: Regular language models are frozen after training: they can’t absorb new knowledge. SEAL (Self-Adapting Language Models) teaches a model to write its own fine-tuning data, synthetic “study notes”, keep only the notes that measurably improve its answers, and reinforce the habit of writing good ones. I implemented the full loop in a single Python file that runs in 8GB of VRAM using LoRA + 4-bit QLoRA, and got a Llama-3.2–1B model to generate self-edits that lifted its own QA accuracy on unseen passages. Here’s exactly how.

Why frozen language models can’t learn new facts

Picture a brilliant new hire who aced every exam in school but, the day after graduation, permanently loses the ability to remember anything new. Hand them a document and they’ll answer questions about it perfectly, while it’s in front of them. Take it away, and it’s as if they never read it.

That’s a large language model. After training, its billions of weights are frozen. It can *use* facts you paste into the prompt (the document in hand), but it never truly learns them. Once the text leaves the context window, the knowledge is gone.

The classic fix is fine-tuning: nudge the weights so the facts stick. But that surfaces a surprisingly deep question, and it’s the one SEAL answers:

> What is the best data to fine-tune on? The raw text? A summary? A list of facts? Practice questions?

Nobody knows the optimal format in advance. So instead of guessing, SEAL lets the model figure it out, and *learn to get better at it*. The paper is *Self-Adapting Language Models*; this post is my from-scratch, low-VRAM implementation of its knowledge-incorporation variant.

The core idea: an LLM that writes its own training data

SEAL’s insight is elegant. Given a new passage, the model generates a self-edit: synthetic study notes about that passage. Think of it as the model making its own flashcards. Then SEAL asks a measurable question:

> After studying its own flashcards, does the model actually get better at answering questions about the passage?

If yes, those were good flashcards, so we teach the model to make more like them. If no, we throw them away. Repeat, and the model becomes a better and better study-note writer.

This isn’t hand-waving. In my real run on a Wikipedia passage about *ctenophores* (comb jellies), the Llama-1B model started by answering **0 of 3** questions. After studying one of its own self-edits, *”Origin of ctenophores: Ctenophores are believed to have…”*, it answered 2 of 3. That’s a reward of +0.667. Meanwhile a lazier self-edit that just said *”This passage describes…”* changed nothing and was discarded. The model is literally being rewarded for writing dense facts and penalized for waffle.

How SEAL works: two nested loops, two kinds of learning

This is the most important concept, so let’s take it slowly. SEAL has two nested loops, and each performs a *different kind of weight update*. Keeping them strictly separate is what makes the whole thing correct.

The inner loop = measurement (temporary)

The inner loop exists only to score a single flashcard:

1. Take one self-edit.

2. Fine-tune a scratch copy of the model on it (“study this flashcard”).

3. Measure how well the studied model answers the passage’s questions.

4. Compute a reward = (accuracy after studying) − (accuracy before).

5. Throw the scratch copy away.

That last step is the whole point. The inner loop’s weight changes are disposable scaffolding. Their only product is one number, the reward, that tells us how good the flashcard was.

The outer loop = learning (permanent)

The outer loop is the only thing that changes the model for good:

1. For each passage, generate N self-edits.

2. Score each with the inner loop.

3. Keep only the ones with reward > 0, the flashcards that actually helped.

4. Fine-tune the flashcard generator on those winners.

5. Repeat.

The golden rule: the generator improves *only* through step 4, training on flashcards that proved themselves. If disposable inner-loop updates ever leaked into the generator, we’d be training on unfiltered junk. As you’ll see, most of the engineering exists to enforce that separation.

The math, in plain English

Two formulas, both simpler than they look.

Studying a flashcard is ordinary gradient descent, nudge the weights so the model finds the flashcard’s text more likely:

Read it as “new weights = old weights, nudged in the direction that makes the flashcard more probable.” Here η is the learning rate (the size of the nudge). We take a few such steps.

The reward is just a before-and-after comparison:

Positive reward = the flashcard helped. In my ctenophores example, reward = 0.667 − 0.000 = +0.667.

Why ReST-EM instead of PPO

To make the generator produce high-reward flashcards, the textbook approach is a heavy reinforcement-learning method, PPO, with value networks and clipping. SEAL uses something far simpler: ReST-EM, which is a two-word recipe:

Filter, then imitate.

• Filter (E-step): discard every flashcard with reward ≤ 0.
• Imitate (M-step): do plain supervised fine-tuning of the generator on the survivors.

That’s it. No value function, no clipping. “Keep the winners and imitate them” is a well-known, stable approximation to maximizing expected reward, which is exactly why it’s practical on a small GPU.

The engineering: fitting a self-adapting LLM into 8GB of VRAM

Here’s where a naive implementation dies. The obvious way to write SEAL is to copy or reload the whole model for every scratch experiment, impossible with a billion-parameter model on a laptop GPU. Making it efficient was the real work. Four tricks did the heavy lifting.

Trick 1, Load the model once, in 4-bit (QLoRA)

Each weight normally takes 16 or 32 bits. 4-bit quantization squeezes it to a quarter of the size with almost no quality loss (the QLoRA technique). I load this compressed model once, freeze it, and it never moves or reloads.

# Frozen base, loaded ONCE in 4-bit, the ~500MB that never changes.
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16)
base = AutoModelForCausalLM.from_pretrained(MODEL_NAME, quantization_config=bnb)

On my RTX 4060 this reports 4bit=True and the whole 1.24-billion-parameter model plus training fits comfortably in 8GB.

Trick 2, Two tiny LoRA adapters on one shared base

Instead of copying the giant model, I use LoRA (Low-Rank Adaptation): leave the frozen model untouched and bolt on a tiny set of steering weights. How tiny? My run trained 1.7 million parameters out of 1.24 billion, 0.14%.

The trick is attaching two adapters to the one shared frozen base:

• a generator adapter (the permanent flashcard-writer), and
• an inner adapter (the disposable scratch space).

Two experiments, zero extra copies of the big model:

# Adapter #1: the persistent generator. get_peft_model wraps the base once.
model = get_peft_model(base, LoraConfig(**lora_kwargs), adapter_name="generator")
# Adapter #2: the transient inner adapter, added onto the SAME base.
# No second copy of the 1B weights, just another few-MB set of LoRA matrices.
model.add_adapter("inner", LoraConfig(**lora_kwargs))

Trick 3, Reset the scratch adapter by zeroing it

Between experiments I need a clean scratch copy. Rebuilding it each time is wasteful. So I exploit a LoRA quirk: an adapter’s contribution is B × A, and B is initialized to zero. When B is zero, the adapter does nothing, the model behaves exactly like the untouched base. To reset, I just set B back to zero:

def reset_inner_adapter(model):
    for name, p in model.named_parameters():
        if "inner" in name and "lora_B" in name:
            p.zero_()   # B=0 → adapter is a no-op → back to base behavior

Trick 4, Hand-write the training loop and guard who can learn

For a 3-to-5-step “study” session, popular training libraries spend more time on setup than on training. So the inner loop is hand-rolled, a bare forward → backward → step:

def inner_adapt(model, tok, edit_text):
    reset_inner_adapter(model)          # start from base behavior (ΔW = 0)
    model.set_adapter("inner")          # route forward/backward through inner LoRA
    set_only_trainable(model, "inner")  # ONLY the inner adapter gets gradients
    model.train()

    params = [p for p in model.parameters() if p.requires_grad]
    opt = torch.optim.AdamW(params, lr=Config.INNER_LR)
    enc, labels = _lm_batch(tok, edit_text, device_of(model))

    for _ in range(Config.INNER_STEPS):     # just a few steps
        opt.zero_grad(set_to_none=True)
        out = model(**enc, labels=labels)   # forward
        out.loss.backward()                 # backward
        opt.step()                          # update
    # The next reset_inner_adapter() throws this adaptation away; it was transient.

And the mechanism that enforces the golden rule: before each phase I flip requires_grad so only the intended adapter can change. During studying, only the scratch adapter learns; the generator is locked. During the outer update, only the generator learns. This physically prevents disposable experiments from contaminating the real model.

def set_only_trainable(model, adapter_name):
    # Learning ON for exactly one adapter, OFF for everything else
    # (the other adapter AND the frozen base). Keeps the two updates separate.
    for name, p in model.named_parameters():
        p.requires_grad = ("lora_" in name) and (f".{adapter_name}." in name)

Real results: watching a Llama model improve itself

The entire scoring loop is about ten lines: sample the edits, then adapt-score-discard each one and keep only the winners.

edits = generate_self_edits(model, tok, passage, Config.N_SELF_EDITS)
for edit in edits:
    inner_adapt(model, tok, edit)            # transiently "study" this note
    adapted_acc = eval_qa(model, tok, qa)    # test the studied model
    reward = adapted_acc - base_acc          # the SEAL reward
    if reward > 0:
        winners.append((passage, edit))      # keep only what helped
    reset_inner_adapter(model)               # throw the adaptation away

And here’s what it prints, running Llama-3.2–1B in 4-bit on the 4060:

You can *see* the mechanism discriminating. The winners are clean, dense fact-lists. The losers are meta-commentary (“*These are some short, self-contained factual statements…*”), the model narrating instead of teaching. SEAL keeps the former and reinforces it.

Making a long run interpretable

Over many iterations, is the generator actually improving? To find out, I log a training curve, the fraction of flashcards kept per iteration. If SEAL is learning, this should trend upward: a better generator writes more useful notes, so more survive the filter.

Run it yourself

The whole thing is one heavily-commented Python file. Here is the Github Repo for the code : https://github.com/mithilai/SEAL

# 1. Isolated environment + dependencies
python -m venv .venv
.venv\Scripts\activate          # macOS/Linux: source .venv/bin/activate
pip install -r requirements.txt
# 2. Prove the loop in seconds on a tiny model (CPU is fine)
python seal_pipeline.py --smoke
# 3. A real run on real data (SQuAD), on an 8GB GPU
python seal_pipeline.py --squad --passages 10 --outer-iters 10 --n-edits 8 \
    --inner-steps 5 --edit-tokens 200 --max-seq-len 768 --save seal_generator

Two settings matter enormously for quality on real data, and cost me a wasted run to learn:

• --edit-tokens 200 and --max-seq-len 768. With the defaults too low, long passages get truncated and self-edits are cut off mid-sentence, the model never sees the full facts, so every reward is zero. Give it room and the winners appear.
• A GPU build of PyTorch. A CPU-only torch silently falls back to slow fp32 (4bit=False). Install the CUDA build so quantization actually engages (4bit=True).

Honest limitations

This implementation optimizes for understanding the mechanism, not benchmark records:

• Small models (0.5–1B) on short passages, so raw accuracy stays modest. Bigger models write better flashcards.
• The QA check is a forgiving text match, enough to demonstrate the loop, not a rigorous benchmark.
• Real SEAL research uses more samples, more iterations, and larger models. Scaling any of those is a config change, not a redesign, the machinery is identical.

The point isn’t the score. The point is that the loop is correct: it generates its own training data, keeps only what measurably helps, and improves the generator only through that filtered signal. That’s a self-adapting model, running on a laptop.

Key takeaways

• Frozen LLMs can’t learn new facts after training. SEAL is one answer to “how could they?”
• The trick is self-generated training data: the model writes its own study notes, tests them, and reinforces the good ones.
• Two strictly-separated loops: a transient one that measures a note’s usefulness, and a persistent one that learns from the notes that passed.
• ReST-EM (“filter, then imitate”) replaces heavyweight RL with something simple and stable.
• LoRA + 4-bit QLoRA + careful requires_grad gating make it all fit in 8GB.

If this helped you understand self-adapting language models, give it a few claps and follow for more hands-on LLM engineering write-ups. The full annotated source is on my GitHub, questions and pull requests welcome.

How I implemented Self-Adapting Language Models with LoRA and QLoRA on an RTX 4060 was originally published in Python in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.