June 27, 2026 · 21 min read

How to Build a Vision Language Model from Scratch Using Q-Former, Contrastive Learning, and LoRA

Train an image captioning and visual question answering model on 50,000 images in under 4 hours on a single GPU.

This post is a deep dive into the open source codebase built by Avishek Biswas. Huge shoutout to him for putting together such a clean and well-structured implementation. Check out the full repository here: github.com/avbiswas/vlm.

What You Will Learn

Vision Language Models (VLMs) like GPT-4V, LLaVA, and Google Gemini can look at a photo and answer questions about it, write captions, or extract structured data. They feel like magic but the training pipeline behind them is more accessible than most people think.

This post walks through a complete working VLM built from scratch using PyTorch. Instead of fine-tuning a massive 70B parameter model, we compose smaller pretrained components into a unified system inspired by the BLIP-2 paper. We train it on 50,000 image-caption pairs from the Conceptual Captions dataset and get a working image captioning system in about 4 hours on a single GPU.

By the end of this post you will understand:

The two-stage training pipeline that powers modern VLMs
How a Q-Former (Querying Transformer) bridges vision and language representations
The math behind CLIP-style contrastive learning
How LoRA (Low-Rank Adaptation) makes LLM fine-tuning possible on consumer hardware
How custom attention masking lets one model operate in multiple modes

1. What Are We Building?

The goal is simple. Given an image and a natural language question, generate a grounded, accurate answer.

Input:  [Photo of a dog at the beach] + "What is happening in this picture?"
Output: "A golden retriever is playing in the waves at a sandy beach."

The core challenge is bridging two very different representation spaces. A vision encoder produces a grid of patch embeddings from pixels. A language model expects a sequence of discrete token embeddings. These two spaces have nothing in common out of the box.

The solution is a Q-Former, a small transformer-based module that learns to translate visual patch embeddings into a compact set of visual tokens that the language model can read and understand.

**Figure 1: The VLM Generation Pipeline.** An input image is divided into 16x16 patches and processed by a vision encoder. A Q-Former then uses 32 learnable queries to extract the most relevant visual features, acting as a bridge to translate pixel data into visual tokens. Finally, an autoregressive language model reads these tokens to generate a coherent caption step-by-step.

2. Architecture Overview

The full system stacks four components on top of each other:

The key design principle here is freezing the vision encoder. The ViT is never updated. Only the Q-Former, the adapter, and a small set of LoRA parameters inside the language model are trained. This cuts memory usage and training time dramatically.

Here is the full breakdown of models and parameters:

Total trainable parameters: roughly 75 million out of a 300M+ total. This is what makes the project runnable on consumer hardware.

Figure 2: Color-coded architecture diagram illustrating a multi-stage Vision-Language Model. Features are extracted by a frozen ViT (grey), then aligned by a Q-Former trained in Stage 1 (blue). In Stage 2 (orange), an adapter projects features to the language model, and LoRA layers adapt the LLM itself. Tensor shapes label the flow of data.

3. Dataset: Conceptual Captions

The training data comes from the Conceptual Captions dataset, a large collection of image and caption pairs scraped from the web. We use 50,000 examples from the 200K subset, downloaded using the img2dataset tool.

# filter_dataset.py — download the parquet index
from datasets import load_dataset

ds = load_dataset(
    "flax-community/conceptual-captions-12",
    split="train"
)
df = ds.to_pandas()
df[:200_000].to_parquet("dataset/conceptual-captions-200k.parquet")

Each example is a URL pointing to an image and its associated text caption. After downloading, images are stored at dataset/cc_images/<shard>/<index>.jpg.

Dataset Pipeline for Stage 1

The CCImageCaptionDataset class handles loading for the Q-Former training stage. The most important design decision is that ViT encoding happens inside __getitem__. Every image is passed through the frozen ViT during loading and returned as patch embeddings rather than raw pixels. Since the ViT is frozen and deterministic, these outputs never change across epochs, so we avoid redundant computation.

class CCImageCaptionDataset(Dataset):
    def __init__(self, dataset_root="dataset", vit_model="google/vit-base-patch16-224", 
                 tokenizer=None):
        self.vit_processor = ViTImageProcessor.from_pretrained(vit_model)
        self.vit_model = ViTModel.from_pretrained(vit_model)
        self.vit_model.to(device)
        self._examples = self._build_index()

    def __getitem__(self, idx):
        ex = self._examples[idx]
        with Image.open(ex.image_path) as im:
            image = im.convert("RGB").copy()

        with torch.no_grad():
            image = self.vit_processor(images=image, return_tensors="pt").to(device)
            image = self.vit_model(**image).last_hidden_state
        # [1, num_patches, 768] → [num_patches, 768]
        image = image.squeeze(0)
        return image, ex.caption

Dataset Pipeline for Stage 2

The LMDataset class adds prompt templating to the image loading. To prevent the model from memorizing a single question phrasing, each training example randomly picks one of 12 different question variants:

self.prompts = [
    "Tell me about this image:",
    "Describe this picture.",
    "What do you see in this image?",
    "Provide a description of the photo.",
    "Can you explain what is shown in this image?",
    "What is in this picture?",
    # 6 more variants included in the full code
]

Each example is formatted using the model’s chat template with a system message, a user prompt, and an assistant response containing the ground truth caption:

random_prompt = random.choice(self.prompts)
user_prompt = tokenizer.apply_chat_template([
    {"role": "system", "content": "Answer the user's question truthfully"},
    {"role": "user", "content": random_prompt},
], return_tensors="pt")

assistant_prompt = tokenizer.apply_chat_template([
    {"role": "assistant", "content": caption},
], return_tensors="pt", add_generation_prompt=False)

Figure 3: A horizontal sequence diagram showing the structure of a training input for a multimodal model. The sequence combines a “System + User Prompt” (blue), “32 Visual Tokens from Q-Former” (orange), and the “Assistant Caption” (green). Corresponding badges indicate that the loss calculation is turned off for the first two components and turned on only for the generated caption.

4. Stage 1: Vision Language Alignment with Q-Former

What is a Q-Former?

The Querying Transformer (Q-Former) is the core innovation from the BLIP-2 paper. It is a transformer-based module that acts as a learnable information bottleneck between the frozen vision encoder and the language model.

The key idea is this. Instead of passing all 197 patch embeddings from ViT-base directly to the language model, we train a set of 32 learnable query vectors to extract the most relevant visual information from those patch embeddings. These 32 query tokens become the visual representation that gets passed to the LLM. They serve as a fixed-size, information-dense visual summary.

The Q-Former is built on top of DistilBERT, a 6-layer BERT variant. We copy the weights of a pretrained DistilBERT and add two new components:

Learnable query embeddings: 32 randomly initialized vectors that learn during training
Cross-attention blocks: inserted every 2 transformer layers to attend to ViT patch features

class QFormer(nn.Module):
    def __init__(self, bert_model, n_queries=32, cross_every=2, num_heads=12):
        super().__init__()
        self.n_queries = n_queries
        cfg = bert_model.config
        self.hidden_size = cfg.hidden_size  # 768 for DistilBERT

        # Copy pretrained DistilBERT layers
        self.embeddings = copy.deepcopy(bert_model.embeddings)
        self.encoder_layers = nn.ModuleList(
            [copy.deepcopy(layer) for layer in bert_model.transformer.layer]
        )
        
        # Add cross-attention every `cross_every` layers (layers 1, 3, 5)
        self.cross_blocks = nn.ModuleDict()
        for i in range(len(self.encoder_layers)):
            if (i % cross_every) == (cross_every - 1):
                self.cross_blocks[str(i)] = CrossAttentionBlock(self.hidden_size, num_heads)

        # The learnable queries — the heart of Q-Former
        self.query_embeddings = nn.Parameter(torch.randn(1, n_queries, self.hidden_size))

Each CrossAttentionBlock performs standard multi-head attention where:

Queries come from the current Q-Former activations
Keys and Values come from the ViT patch embeddings

class CrossAttentionBlock(nn.Module):
    def __init__(self, hidden_size, num_heads):
        super().__init__()
        self.cross_attn = nn.MultiheadAttention(hidden_size, num_heads, batch_first=True)
        self.layernorm = nn.LayerNorm(hidden_size)
        self.ffn = nn.Sequential(
            nn.Linear(hidden_size, hidden_size * 4),
            nn.GELU(),
            nn.Linear(hidden_size * 4, hidden_size),
        )
        self.ln2 = nn.LayerNorm(hidden_size)

    def forward(self, x_queries, kv):
        # x_queries: (B, 32, 768) — our learnable queries
        # kv: (B, 197, 768) — ViT patch features
        attn_out, _ = self.cross_attn(x_queries, kv, kv)
        x = self.layernorm(x_queries + attn_out)
        ffn_out = self.ffn(x)
        return self.ln2(x + ffn_out)

Figure 4: Inside the Q-Former. A layer-by-layer breakdown of the Querying Transformer. The architecture alternates standard DistilBERT transformer layers with custom cross-attention blocks (inserted at layers 1, 3, and 5). The 32 learnable queries (orange) act as an information bottleneck, iteratively extracting relevant visual features from the frozen ViT patch embeddings (blue) via cross-attention.

Contrastive Learning: The CLIP-style Loss

In Stage 1, we train the Q-Former using contrastive learning, the same training objective that made CLIP so effective. The intuition is straightforward. Given a batch of N image-caption pairs, we want matching pairs to have similar embeddings and mismatched pairs to have different embeddings.

The math behind it:

Given N images and N captions, compute an N x N similarity matrix S where:

S[i, j] = cosine_similarity(image_embedding_i, text_embedding_j)

The diagonal entries S[i, i] represent matching pairs. Off-diagonal entries are mismatches. We want the diagonal to be high and everything else to be low.

This is a classification problem. Each image must pick its matching caption out of N options, and each caption must pick its matching image.

def calculate_clip_loss(v, t, tau=0.07):
    N = v.size(0)
    v = F.normalize(v, dim=1)   # L2-normalize image embeddings
    t = F.normalize(t, dim=1)   # L2-normalize text embeddings
    
    logits = v @ t.t() / tau    # [N, N] similarity matrix, scaled by temperature
    labels = torch.arange(N, device=logits.device)   # Ground truth: diagonal
    
    loss_i2t = F.cross_entropy(logits, labels)      # Each image finds its text
    loss_t2i = F.cross_entropy(logits.t(), labels)  # Each text finds its image
    
    return 0.5 * (loss_i2t + loss_t2i)

The temperature parameter tau = 0.07 is critical. A small temperature makes the probability distribution sharper and helps the model learn to confidently separate similar embeddings. This value is taken directly from the original CLIP paper.

The symmetric loss averages both directions so that image-to-text and text-to-image retrieval both improve together.

Figure 5: A 4x4 heatmap similarity matrix illustrating the effect of a contrastive loss optimization strategy. Bright diagonal cells (yellow/green) indicate high cosine similarity, while dark off-diagonal cells (blue/purple) indicate low similarity. This visual pattern demonstrates how the loss function pulls matching image-caption pairs together in vector space and pushes mismatched pairs apart. Rows are labeled “Image Samples (1–4)” and columns are labeled “Caption Prompts (1–4)”

Attention Modes and Custom Masking

One of the most elegant features of this Q-Former implementation is support for three different attention modes using a single set of network weights. The mode is controlled by a custom boolean attention mask.

The create_attention_mask function builds a boolean mask of shape [B, 1, T+I, T+I] where T is the text length and I is the number of query tokens (32):

def create_attention_mask(B, I, text_presence_mask, mode):
    T = text_presence_mask.size(1)
    mask = torch.zeros(B, T + I, T + I, dtype=torch.bool)

    img_self = torch.ones(B, I, I, dtype=torch.bool)    # Queries attend to each other
    text_self = torch.ones(B, T, T, dtype=torch.bool)   # Text tokens attend to each other

    if mode == "multi_modal_causal":
        text_self = torch.tril(text_self)  # Lower triangular = causal mask

    # Cross-attention between modalities: enabled only in multi_modal modes
    cross_fn = torch.zeros if mode == "uni_modal" else torch.ones
    img_cross = cross_fn(B, T, I, dtype=torch.bool)   # Text attends to queries
    text_cross = cross_fn(B, I, T, dtype=torch.bool)   # Queries attend to text

    # Assemble the full attention mask
    mask[:, :T, :T] = text_self
    mask[:, -I:, -I:] = img_self
    mask[:, :T, -I:] = img_cross
    mask[:, -I:, :T] = text_cross

    # Apply padding mask (ignore padded text positions)
    presence_mask = torch.cat([text_presence_mask,
                               torch.ones(B, I, dtype=torch.bool)], dim=1)
    presence_mask = presence_mask.unsqueeze(2) & presence_mask.unsqueeze(1)
    return (mask & presence_mask).unsqueeze(1)

In uni_modal mode, the cross-attention between image queries and text is fully blocked. Image and text representations are computed independently. This is exactly what contrastive learning needs: separate, comparable embeddings for the similarity matrix.

Figure 6: Q-Former Attention Masking Strategies. A visual comparison of the three attention modes. White areas indicate permitted attention, while black areas are masked out. Left: uni_modal isolates modalities for contrastive learning. Center: multi_modal allows full bidirectional cross-attention. Right: multi_modal_causal applies a lower-triangular mask to the text-to-text block to enforce autoregressive generation.

The Stage 1 Training Loop

Stage 1 uses a grouped optimizer that applies different learning rates to different parts of the Q-Former. The cross-attention blocks and learnable queries get 10x more learning rate than the inherited DistilBERT weights. This is because the DistilBERT weights already carry useful language knowledge and need only small adjustments, while the new cross-attention blocks and queries must learn from scratch.

grouped_params = qformer.get_grouped_params()
optimizer = optim.Adam([
    {"params": grouped_params["default"],         "lr": 1e-5},  # DistilBERT weights
    {"params": grouped_params["cross_blocks"],    "lr": 1e-4},  # New cross-attention
    {"params": grouped_params["query_embeddings"],"lr": 1e-4},  # Learnable queries
])

for epoch in range(10):
    for (img, txt) in train_loader:
        img_emb, txt_emb = qformer(
            visual_feats=img,
            text_input_ids=txt["input_ids"],
            text_attention_mask=txt["attention_mask"],
            attention_mode="uni_modal"   # Separate image and text for contrastive loss
        )
        loss = calculate_clip_loss(img_emb, txt_emb)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

Every 10 steps, inference runs on the test set and the best checkpoint is saved. After Stage 1, the Q-Former has learned to encode images into 768-dimensional vectors that are semantically aligned with the text embedding space.

5. Stage 2: Language Model Fine-Tuning with LoRA

What is LoRA?

Low-Rank Adaptation (LoRA) solves a practical problem: how do you fine-tune a 135M parameter language model on a consumer GPU without running out of memory?

The core insight is that weight updates during fine-tuning tend to be low rank. Instead of updating the full weight matrix W of shape d x k, we decompose the update into two small matrices:

Delta W = B x A

where B has shape d x r  and  A has shape r x k  with  r much smaller than d and k

Only A and B are updated during training. The forward pass during inference becomes:

y = W x + B A x = (W + B A) x

The rank r controls how much capacity the update has. We use r=64 with alpha=128, which gives a scaling factor of alpha/r = 2. With these settings, LoRA adds roughly 8 million trainable parameters to the 135M-parameter SmolLM while still achieving strong performance.

peft_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    inference_mode=False,
    r=64,           # Rank of the low-rank decomposition
    lora_alpha=128, # Scaling factor = alpha divided by r = 2.0
    lora_dropout=0.1,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",  # Attention projections
        "gate_proj", "up_proj", "down_proj",       # FFN projections
    ],
)
self.llm = get_peft_model(self.llm, peft_config)

Figure 7: Diagram of the Low-Rank Adaptation (LoRA) workflow applied to a single weight matrix ‘W’. The original, massive, frozen matrix W is on the left, shown processing an input. On the right, the adaptation path trains two much smaller matrices, A and B, in a parallel stream. The final result is the sum of the two streams, guided by the equation: Output = W·input + B·A·input. LoRA freezes original weights and trains a low-rank update.

The Full VLM Wrapper

The LM_2_VLM class assembles all four components into a single nn.Module. The adapter is a two-layer MLP that projects the Q-Former's 768-dimensional query output into the LLM's hidden dimension (576 for SmolLM-135M):

class LM_2_VLM(nn.Module):
    def __init__(self, model_name, qformer_model_path, pad_token_id=None):
        super().__init__()
        self.llm = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)
        self.llm = get_peft_model(self.llm, peft_config)
        
        self.qformer = QFormer.from_pretrained(qformer_model_path)
        
        # Bridge: Q-Former dim (768) → LLM dim (576 for SmolLM)
        self.adapter = nn.Sequential(
            nn.Linear(self.qformer.hidden_size, self.llm.config.hidden_size),
            nn.ReLU(),
            nn.Linear(self.llm.config.hidden_size, self.llm.config.hidden_size),
        )

The forward pass builds a three-part embedding sequence and passes it directly to the language model:

def forward(self, img, prefix_ids, assistant_ids):
    # 1. Encode image through Q-Former → 32 visual tokens
    img_emb, _ = self.qformer.encode_image(img)  # [B, 32, 768]
    img_emb = self.adapter(img_emb)               # [B, 32, 576]
    img_emb = img_emb.to(dtype=self.llm.dtype)    # Cast to bfloat16

    # 2. Get LLM embeddings for text tokens
    prefix_emb = self.llm.get_input_embeddings()(prefix_ids)    # [B, T_p, 576]
    assistant_emb = self.llm.get_input_embeddings()(assistant_ids)  # [B, T_a, 576]

    # 3. Concatenate: [System+User Prompt | 32 Image Tokens | Assistant Response]
    input_embs = torch.cat([prefix_emb, img_emb, assistant_emb], dim=1)
    
    # ... (build attention mask, position ids, and labels — see next section)
    
    return self.llm(inputs_embeds=input_embs, attention_mask=attention_mask,
                    position_ids=position_ids, labels=labels)

Label Masking for Autoregressive Training

This is the detail that is easiest to get wrong. The language model is trained autoregressively, predicting each token given all the tokens before it. But we do not want the model to predict the system prompt or the image tokens. Those are inputs, not targets.

The solution is selective label masking using -100. PyTorch’s cross-entropy loss ignores any position labeled -100. So we assign:

Prefix tokens (system and user prompt): set to -100, excluded from loss
Image tokens (the 32 Q-Former outputs): set to -100, excluded from loss
Assistant tokens (the ground truth caption): use actual token IDs, included in loss

prefix_labels = torch.full_like(prefix_ids, -100)       # Mask prefix
image_labels = torch.full((B, 32), -100, device=device) # Mask image tokens
assistant_labels = assistant_ids.clone()
assistant_labels[assistant_ids == self.pad_token_id] = -100  # Mask padding

labels = torch.cat([prefix_labels, image_labels, assistant_labels], dim=1)

Position IDs are computed manually using a cumulative sum of the attention mask, which correctly handles variable-length padded sequences:

attention_mask = torch.cat([
    (prefix_ids != self.pad_token_id).long(),
    torch.ones(B, 32, device=device).long(),        # Image tokens always attend
    (assistant_ids != self.pad_token_id).long(),
], dim=1)

position_ids = attention_mask.cumsum(dim=1) - 1
position_ids.masked_fill_(attention_mask == 0, 0)  # Fill padded positions

Figure 8: A horizontal schematic illustrating the complete input structure provided to the language model during training. The sequence is color-coded into three main components: a “Prefix” containing the System and User Prompt (blue, LOSS OFF), followed by the “32 Image Tokens” derived from the Q-Former (orange, LOSS OFF), and concluding with the “Assistant Caption” (green, LOSS ON). The value labels below indicate the exact value assigned for loss calculation — the first two sections are masked with a “-100” label, while the third section uses the actual token IDs for the target caption.

Distributed Training with Accelerate

Stage 2 uses HuggingFace’s Accelerate library, which cleanly handles mixed precision, gradient accumulation, and multi-GPU distribution:

accelerator = Accelerator(
    gradient_accumulation_steps=4,   # Effective batch size = 8 × 4 = 32
    mixed_precision="bf16",           # bfloat16 for memory efficiency
    log_with="tensorboard",
    project_dir="logs",
)

# Prepare everything: model, optimizer, dataloaders, scheduler
model, optimizer, train_loader, test_loader, scheduler = accelerator.prepare(
    model, optimizer, train_loader, test_loader, scheduler
)

for data in train_loader:
    with accelerator.accumulate(model):   # Gradient accumulation
        with accelerator.autocast():       # Mixed precision
            output = model(img, prefix, assistant)
            loss = output.loss
        accelerator.backward(loss)
        
        if accelerator.sync_gradients:
            accelerator.clip_grad_norm_(model.parameters(), 1.0)  # Gradient clipping
        
        optimizer.step()
        scheduler.step()
        optimizer.zero_grad()

Full training configuration:

scheduler = get_cosine_schedule_with_warmup(
    optimizer,
    num_warmup_steps=100,
    num_training_steps=len(train_loader) * epochs // gradient_accumulation_steps,
)

6. Evaluation: Recall@K and Caption Generation Recall@K for Retrieval Quality

After Stage 1, we measure the Q-Former’s vision language alignment using Recall@K, a standard metric for cross-modal retrieval. It answers this question: given an image, is its matching caption ranked in the top K results out of N total candidates?

We measure both retrieval directions:

Image-to-Text (I2T): For each image, retrieve the top K captions
Text-to-Image (T2I): For each caption, retrieve the top K images

def calculate_recall(model, dataloader, device, k_values=[1, 5, 10]):
    image_feats_all, text_feats_all = [], []
    
    with torch.no_grad():
        for batch in dataloader:
            images, captions = batch
            q_out, t_out = model(
                visual_feats=images, text_input_ids=captions["input_ids"],
                text_attention_mask=captions["attention_mask"],
                attention_mode="uni_modal"
            )
            image_feats_all.append(F.normalize(q_out, dim=1).cpu())
            text_feats_all.append(F.normalize(t_out, dim=1).cpu())

    image_feats = torch.cat(image_feats_all)  # [N, 768]
    text_feats = torch.cat(text_feats_all)    # [N, 768]
    
    # [N, N] similarity matrix
    sim_matrix = image_feats @ text_feats.t()
    
    # I2T: For each image row, is the diagonal element in top-K?
    i2t_recall = {}
    for k in k_values:
        hits = sum(i in sim_matrix[i].topk(k).indices for i in range(len(image_feats)))
        i2t_recall[k] = hits / len(image_feats)
    
    return {"i2t": i2t_recall, "t2i": t2i_recall}

Figure 9: Vision-Language Alignment Evaluation (Recall@K). A 10x10 cosine similarity matrix heatmap demonstrating the semantic alignment achieved by the Q-Former. Rows represent specific images, while columns represent their associated ground-truth captions. The diagonal is bright green, indicating high similarity between the correct image-caption pairs. Off-diagonal cells (orange/red) show low similarity, confirming the model distinguishes incorrect pairs effectively. The top-right corner displays the resulting Recall@K scores (Recall@1, @5, @10), which are critical metrics for validating multi-modal models.

Caption Generation

After Stage 2, we generate captions by passing the visual prefix through the frozen Q-Former and adapter, then calling the language model’s generate method:

@torch.no_grad()
def generate(self, img, prefix_ids, max_new_tokens=100, temperature=0.7, top_p=0.95):
    img_emb, _ = self.qformer.encode_image(img)  # Shape: [B, 32, 768]
    img_emb = self.adapter(img_emb)               # Shape: [B, 32, 576]
    img_emb = img_emb.to(dtype=self.llm.dtype)

Figure 10: Qualitative result comparison grid showing four test images — a hiker, a sushi platter, a modern skyscraper, and a fox. Each panel displays the ground truth caption in green text and the model-generated caption in blue text, illustrating accurate object and context identification in varied scenes.

7. Results

Training on 50,000 image-caption pairs for roughly 4 hours on a single GPU produces these results:

Stage 1: Q-Former Retrieval Performance

These numbers may look modest. But a random baseline on 5,000 samples gives Recall@1 of only 0.02 percent. The Q-Former has learned real semantic alignment in just a few hours on consumer hardware.

Stage 2: Caption Generation Quality

The fine-tuned VLM generates coherent, image-grounded captions. It correctly identifies:

Object types and the actions they are performing
Scene context such as beach, office, or forest settings
People and their approximate activities
Dominant colors and spatial compositions

8. Key Takeaways

Two-stage training solves the representation gap

The hardest problem in multimodal AI is not caption generation. It is getting a vision encoder and a language model that were trained completely independently to share a common semantic space. The Q-Former with contrastive learning is a principled way to solve this alignment problem before any text generation training begins.

2. Learnable queries create an information bottleneck

The 32 query tokens do more than compress visual information. They learn to select what is most linguistically relevant in an image. The bottleneck structure forces the queries to encode semantically rich representations that can survive in the LLM’s embedding space.

3. LoRA makes LLM fine-tuning accessible

Without LoRA, fine-tuning even a 135M parameter model end to end would require far more GPU memory. By constraining weight updates to a low-rank subspace, LoRA adds only about 8 million trainable parameters while still achieving strong visual language adaptation.

4. Grouped learning rates protect pretrained knowledge

Applying the same learning rate to pretrained DistilBERT weights and newly initialized cross-attention blocks would cause catastrophic forgetting. The 10x learning rate differential between inherited weights and new components is important, not optional.

5. Label masking is the foundation of instruction tuning

The pattern of masking the prompt and image tokens with -100 and supervising only the assistant response is the core mechanism behind instruction following fine-tuning. Without this, the model wastes capacity trying to predict its own instructions.

6. bfloat16 plus gradient accumulation enables single GPU training

bfloat16 mixed precision cuts memory usage roughly in half. Gradient accumulation with 4 steps gives an effective batch size of 32 while keeping only 8 examples in GPU memory at once. Together these two techniques make the full training run feasible on a single 16GB consumer GPU.

Full Code Structure

All code discussed in this post comes from Avishek Biswas’s open-source repository: github.com/avbiswas/vlm. The repo is a complete, runnable implementation — everything shown below can be found there.

vlm/
├── vlm_train/
│   ├── networks/
│   │   ├── q_former.py         # Q-Former architecture + attention masking
│   │   └── lm_to_vlm.py        # Full VLM wrapper (Q-Former + Adapter + LLM + LoRA)
│   ├── datasets/
│   │   ├── cc_dataloader.py    # Stage 1 dataset (image → ViT patches + tokenized caption)
│   │   └── lm_dataloader.py    # Stage 2 dataset (with prompt templates + chat format)
│   ├── utils/
│   │   ├── calculate_recall.py # I2T and T2I Recall@K evaluation
│   │   └── utils.py            # Similarity grid visualization
│   ├── q_former_train.py       # Stage 1 training script
│   ├── lm_train.py             # Stage 2 training script (with Accelerate)
│   ├── basic_inference.py      # Recall@K evaluation + similarity visualization
│   └── test_generation.py      # Caption generation test on sample images
└── dataset/
    ├── conceptual-captions-200k.parquet
    └── cc_images/
        ├── 00000/
        │   ├── 0000001.jpg
        │   └── ...
        └── ...

Dependencies

[dependencies]
torch = ">=2.9.1"
torchvision = ">=0.24.1"
transformers = ">=4.57.3"
datasets = ">=4.4.1"
peft = ">=0.18.0"           # LoRA implementation
accelerate = ">=2025.11.11" # Distributed + mixed precision training
img2dataset = ">=1.47.0"    # Bulk image downloading
rich = ">=14.2.0"           # Terminal formatting

Theoretical Concepts Reference

For readers who want to go deeper, here are the primary papers behind each component:

Conclusion

We’ve walked through a complete, working VLM implementation that takes a frozen ViT, a Q-Former trained with CLIP-style contrastive learning, and a LoRA-adapted small language model, and combines them into a system that can understand and describe images.

The methodology — two-stage training with a lightweight alignment module — reflects the state of the art in efficient VLM construction. With just 50,000 image-caption pairs and consumer hardware, you can reproduce the core ideas behind models like BLIP-2 and LLaVA.

The codebase is clean and modular: each component has clear boundaries, the training scripts are straightforward, and the evaluation metrics give a principled measure of how well the alignment is working. It’s an excellent foundation to build on — whether you want to scale to larger models, experiment with different architectures, or apply the same pipeline to a domain-specific dataset.

The full code is available on GitHub: github.com/avbiswas/vlm — built and maintained by Avishek Biswas. A huge shoutout to him for putting together such a clean, well-documented implementation. Go give it a star! If you found this post helpful, feel free to share it or leave a comment.

Read the original on Medium