Abstract
Vision-language models (VLMs) have emerged as one of the most consequential developments in modern deep learning, enabling systems to reason jointly over visual and textual modalities. This paper examines the architectural foundations and training paradigms that underpin contemporary VLMs, with particular attention to contrastive alignment objectives (as pioneered by CLIP), cross-modal attention mechanisms, and the growing class of generative VLMs that autoregressively produce language conditioned on visual inputs. We analyze the mathematical structure of multimodal representation spaces, the role of large-scale paired datasets, and the tradeoffs between discriminative and generative objectives. Key empirical findings from the literature are reviewed, and open problems — including compositional generalization, grounding ambiguity, and hallucination in vision-language generation — are discussed in depth. The field has reached a critical juncture where scale alone is insufficient; architectural choices and training signal quality are increasingly dominant factors in downstream capability.
1. Introduction
The ability to connect visual perception with linguistic reasoning is central to many forms of intelligence. Humans effortlessly describe images, answer questions about visual scenes, and generate mental imagery from textual descriptions. Replicating this capacity in artificial systems has been a longstanding goal, yet progress was constrained for decades by the absence of suitable training data and computational infrastructure, as well as by the lack of unified architectures that could bridge vision and language without modality-specific engineering at every layer.
The landscape shifted dramatically with the introduction of large-scale contrastive training — most visibly through CLIP (Radford et al., 2021) — which demonstrated that a shared embedding space between image encoders and text encoders, trained on web-scale paired data with a simple contrastive loss, produced visual representations with remarkable zero-shot transfer capabilities. This work catalyzed a wave of subsequent research exploring richer forms of cross-modal fusion, generative conditioning, and instruction-following in multimodal settings.
Contemporary VLMs span a wide architectural spectrum. At one end sit dual-encoder models trained purely for alignment; at the other, fully generative systems that accept image tokens as input to autoregressive language decoders (e.g., Flamingo, LLaVA, GPT-4V). Between these extremes lies a continuum of approaches involving cross-attention fusion layers, perceiver-style resampling, and modality-specific projection heads. Understanding when each design choice is preferable — and why — requires careful analysis of the underlying mathematics and the nature of the downstream tasks.
This paper provides that analysis. Section 2 surveys foundational and recent work. Section 3 develops the technical core: contrastive objectives, cross-modal attention, and generative conditioning. Section 4 discusses key empirical findings and ongoing challenges. Section 5 concludes with open research directions.
2. Related Work
The development of VLMs builds on decades of work in both computer vision and NLP. Early approaches to visual question answering (VQA) and image captioning relied on separate, task-specific pipelines that extracted visual features with CNNs and fed them to RNNs or early transformer decoders (Vinyals et al., 2015; Antol et al., 2015). These systems were brittle and struggled with generalization beyond training distributions.
Radford et al. (2021) introduced CLIP (Contrastive Language-Image Pre-Training), training a dual-encoder model on 400 million image-text pairs from the web using a symmetric cross-entropy loss over cosine similarities. The resulting model demonstrated unprecedented zero-shot classification performance and strong transfer to a broad range of vision tasks without fine-tuning, establishing contrastive pretraining as a dominant paradigm.
Alayrac et al. (2022) presented Flamingo, a generative VLM that interleaves frozen visual features (from a pretrained NFNet encoder) with language decoder layers via cross-attention, trained on web-scraped interleaved image-text sequences. Flamingo demonstrated few-shot performance on VQA, captioning, and classification that matched or exceeded fine-tuned task-specific models, illustrating the power of in-context visual learning.
Li et al. (2023) proposed BLIP-2, which introduced a lightweight Querying Transformer (Q-Former) that bridges a frozen image encoder and a frozen large language model. The Q-Former is a small transformer that extracts a fixed number of learned query embeddings from image features, dramatically reducing the number of visual tokens passed to the language model and enabling efficient multimodal instruction following.
Liu et al. (2023) introduced LLaVA (Large Language and Vision Assistant), which employs a simple linear projection from CLIP visual features into the embedding space of LLaMA, followed by instruction-tuning on GPT-4-generated multimodal conversations. Despite its architectural simplicity, LLaVA demonstrated strong instruction following and visual reasoning, highlighting the importance of data quality over architectural complexity.
Zhai et al. (2022) conducted a systematic scaling study of image-text contrastive models (SigLIP), finding that model capacity in the vision encoder is a surprisingly critical bottleneck, and that sigmoid-based contrastive losses (replacing softmax normalization over full batch negatives) improve training stability at large scales.
3. Technical Analysis
3.1 Contrastive Alignment Objectives
The core of dual-encoder VLMs is a contrastive loss that encourages matched image-text pairs to have high cosine similarity while pushing mismatched pairs apart. Given a batch of $N$ image-text pairs $\{(v_i, t_i)\}_{i=1}^{N}$, image embeddings $\mathbf{f}_i = E_V(v_i) / \|E_V(v_i)\|$ and text embeddings $\mathbf{g}_i = E_T(t_i) / \|E_T(t_i)\|$ are computed by their respective encoders. The InfoNCE (noise-contrastive estimation) loss is:
$$\mathcal{L}_{\text{CLIP}} = -\frac{1}{2N} \sum_{i=1}^{N} \left[ \log \frac{\exp(\mathbf{f}_i^\top \mathbf{g}_i / \tau)}{\sum_{j=1}^{N} \exp(\mathbf{f}_i^\top \mathbf{g}_j / \tau)} + \log \frac{\exp(\mathbf{f}_i^\top \mathbf{g}_i / \tau)}{\sum_{j=1}^{N} \exp(\mathbf{f}_j^\top \mathbf{g}_i / \tau)} \right]$$
where $\tau$ is a learned temperature parameter. This loss is symmetric: it simultaneously optimizes image-to-text and text-to-image retrieval. The effective negative count is $N-1$ per anchor, making large batch sizes critical — CLIP used batches of 32,768 across distributed hardware.
The SigLIP formulation (Zhai et al., 2022) replaces the softmax with a sigmoid, treating each pair independently:
$$\mathcal{L}_{\text{SigLIP}} = -\frac{1}{N^2} \sum_{i=1}^{N} \sum_{j=1}^{N} \left[ y_{ij} \log \sigma(z_{ij}) + (1 – y_{ij}) \log(1 – \sigma(z_{ij})) \right]$$
where $z_{ij} = \mathbf{f}_i^\top \mathbf{g}_j / \tau + b$, $y_{ij} = \mathbf{1}[i = j]$, and $b$ is a learnable bias. This sigmoid formulation eliminates the need for global normalization across the batch, which is problematic at scale, and was shown to converge more stably for very large vision encoders.
3.2 Cross-Modal Attention in Generative VLMs
Generative VLMs must condition language generation on visual representations. Flamingo achieves this via gated cross-attention dense layers inserted between frozen pretrained LM layers. Each gated cross-attention layer computes:
$$\text{Attn}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$
where $Q$ derives from the language token representations and $K, V$ derive from visual tokens. A tanh-gated residual controls the contribution:
$$\mathbf{h}’ = \mathbf{h} + \tanh(\alpha) \cdot \text{Attn}(\mathbf{h}, \mathbf{z}_V, \mathbf{z}_V)$$
with $\alpha$ initialized to zero, ensuring that at the start of training the visual pathway contributes nothing and the frozen LM’s language modeling capability is preserved. This cold-start property is a crucial design choice that stabilizes training against catastrophic forgetting.
3.3 Q-Former and Perceiver-Based Resampling
A fundamental challenge in generative VLMs is the mismatch between the number of visual tokens (e.g., $14 \times 14 = 196$ for a standard ViT-L/14) and the context budget of a language model. High-resolution images with ViT-L/14 at 336px produce $24 \times 24 = 576$ patch tokens. Passing all these to an LLM is expensive.
BLIP-2’s Q-Former uses $K$ learned query embeddings $\mathbf{Q} \in \mathbb{R}^{K \times d}$ (with $K = 32$ typically) that attend to the full image feature map via cross-attention. These $K$ output representations then serve as a compressed, task-conditioned visual summary passed to the LLM. The Q-Former also applies self-attention among the queries and can be trained with multiple objectives: image-text contrastive (ITC), image-grounded text matching (ITM), and image-grounded text generation (ITG).
The compression ratio is dramatic: from 576 image tokens to 32 query embeddings. This is analogous to the perceiver resampler in Flamingo, which uses latent array cross-attention. The key distinction is that Q-Former queries attend to flattened spatial image features $\mathbf{Z}_V \in \mathbb{R}^{HW \times d_V}$, while perceiver resamplers may apply additional pooling or learned positional encoding.
3.4 Visual Token Integration in Instruction-Tuned VLMs
LLaVA and its successors adopt a simpler approach: a linear projection (or small MLP) maps CLIP image features to LLM embedding dimension, and the resulting visual tokens are prepended or interleaved with text tokens in the LLM’s input sequence. The LLM then attends to visual tokens natively via its own self-attention layers — no architectural modification is needed.
Let $\mathbf{Z}_V = E_V(v) \in \mathbb{R}^{L_V \times d_V}$ be the output of the vision encoder, and $\mathbf{W}_P \in \mathbb{R}^{d_V \times d_{LM}}$ be the projection matrix. The projected visual tokens are $\hat{\mathbf{Z}}_V = \mathbf{Z}_V \mathbf{W}_P \in \mathbb{R}^{L_V \times d_{LM}}$. These are concatenated with text token embeddings $\mathbf{Z}_T \in \mathbb{R}^{L_T \times d_{LM}}$ to form the full input, and the LLM autoregressively generates a response conditioned on both modalities.
The autoregressive language modeling objective for the generative step is:
$$\mathcal{L}_{\text{gen}} = -\sum_{t=1}^{T} \log P(x_t \mid \hat{\mathbf{Z}}_V, x_{1:t-1}; \theta)$$
where $x_t$ are output text tokens and $\theta$ denotes the (partially or fully trained) parameters. In instruction tuning, only the text generation tokens are included in the loss; visual tokens are used as context but not predicted.
3.5 Hallucination and Grounding Failures
One of the most studied failure modes in generative VLMs is hallucination: the model generates plausible but visually unsupported text. This manifests as fabrication of objects present in the training distribution but absent from the specific image (object hallucination), incorrect spatial relationships, or erroneous attribute assignments.
Formally, let $\mathcal{C}(v)$ denote the ground-truth visual content of image $v$ and $\hat{\mathcal{C}}(v, T)$ denote the model’s generated assertions in text $T$. Hallucination occurs when $\hat{\mathcal{C}}(v, T) \not\subseteq \mathcal{C}(v)$. The CHAIR metric quantifies object hallucination as the fraction of generated object mentions not present in the reference annotation. Hallucination in VLMs has a distinct character from pure-LLM hallucination: it arises not only from the language model’s statistical tendencies but from insufficiency of cross-modal alignment — the visual representation may fail to suppress linguistically probable but visually absent content.
4. Discussion
4.1 Discriminative vs. Generative Tradeoffs
Dual-encoder contrastive models (CLIP, SigLIP) produce highly transferable visual representations and excel at retrieval and zero-shot classification. However, they encode images into a single pooled vector (or a small set of patch tokens) that is not naturally suited to fine-grained spatial reasoning or generation. Generative VLMs, by contrast, preserve spatial structure and enable rich language grounding but require more complex training and are more prone to hallucination.
A middle ground is occupied by models like ALBEF (Li et al., 2021) and BLIP (Li et al., 2022), which combine contrastive, matching, and generative objectives. These multimodal training signals allow the encoder to learn both discriminative and generative representations. The empirical evidence suggests that combining objectives is beneficial up to a point, after which interference between objectives can degrade performance on each individual task.
4.2 Compositional Generalization
A persistent limitation of current VLMs is their difficulty with compositional queries that require binding attributes to specific objects or reasoning about relational structures (e.g., “the red cube to the left of the blue sphere”). Evaluation frameworks like ARO (Attribution, Relation, Order; Yuksekgonul et al., 2022) reveal that CLIP-style models often achieve near-random accuracy on negative captions that modify only relational terms, despite strong performance on standard benchmarks. This points to a distributional shortcut: models may match global scene statistics rather than performing structured compositional alignment.
Addressing this requires data augmentation with harder negatives, architectural changes that enforce structured alignment (e.g., scene graph supervision), or both. Current research explores structured latent representations, neuro-symbolic approaches, and dataset curation strategies to improve compositional robustness.
4.3 Scaling Dynamics and Data Quality
The scaling behavior of VLMs differs from pure language models. Zhai et al. (2022) found that increasing vision encoder size from ViT-B to ViT-22B yields substantial gains, but the returns are mediated by data quality — noisy alt-text annotations create a ceiling that additional parameters cannot overcome. Filtering strategies such as LAION-Aesthetics, CLIP-score thresholding, and semantic deduplication have been shown to improve downstream performance more than proportional increases in dataset size at the same parameter count.
This raises a fundamental question about the nature of multimodal learning: does scale in VLMs primarily improve the quality of visual representations, the depth of cross-modal alignment, or the breadth of world knowledge encoded in the language component? Ablation studies from Flamingo and BLIP-2 suggest that the language model’s prior knowledge is a dominant factor — better LLMs consistently produce better VLMs when the visual pathway is held constant.
4.4 High-Resolution and Multi-Scale Processing
Standard ViT-based encoders process images as fixed-size patch grids (e.g., 224px or 336px). This constrains fine-grained understanding — text in images, small objects, and dense scene graphs may be lost. Recent approaches like LLaVA-1.6 (Liu et al., 2024) employ dynamic tiling: the image is divided into sub-tiles matching the encoder’s native resolution, each encoded separately, and the resulting token sets are concatenated. This substantially improves OCR, chart understanding, and visual counting, at the cost of increased compute and context length, partially mitigated by aggressive token compression.
5. Conclusion
Vision-language models have matured from task-specific pipelines into general-purpose multimodal reasoning systems. The contrastive alignment paradigm established by CLIP remains the dominant approach for learning shared embedding spaces, with recent sigmoid-based variants improving scalability. Generative VLMs, particularly those employing cross-modal attention or learned query-based resampling, have demonstrated instruction-following and few-shot reasoning capabilities that approach human performance on structured benchmarks.
Yet significant challenges remain. Compositional generalization, hallucination under distributional shift, and fine-grained spatial understanding are not solved by scaling alone. The field is converging on several complementary strategies: higher-quality training data with hard negatives, more expressive cross-modal fusion mechanisms, and instruction-tuning procedures that explicitly target known failure modes.
The next frontier involves tighter integration between perception and reasoning — moving beyond static image understanding toward video, 3D, and embodied visual-linguistic reasoning. Achieving this will require not only architectural innovation but a deeper theoretical understanding of what contrastive and generative objectives actually optimize in the multimodal representation space.
References
- Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. ICML 2021.
- Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., & Zisserman, A. (2022). Flamingo: A visual language model for few-shot learning. NeurIPS 2022.
- Li, J., Li, D., Savarese, S., & Hoi, S. (2023). BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ICML 2023.
- Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023). Visual instruction tuning (LLaVA). NeurIPS 2023.
- Zhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022). Scaling vision transformers. CVPR 2022.
- Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019). VisualBERT: A simple and performant baseline for vision and language. arXiv:1908.03557.
- Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., & Zou, J. (2022). When and why vision-language models behave like bags-of-words, and what to do about it. ICLR 2023.
- Li, J., Li, D., Xiong, C., & Hoi, S. (2022). BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. ICML 2022.