‍

Text-based AI has already reshaped how we interact, work, and communicate—but human experience extends far beyond text alone. 

Multimodal models bridge this gap, empowering AI to see, hear, and sense the world more like humans do. Leveraging existing pre-trained language and vision models, these powerful systems blend different data types to solve complex real-world problems, from robotics and autonomous driving to advanced document understanding.

As we move into the era of AI agents, multimodal training strategies and model adaptation techniques will become increasingly relevant. These approaches enable models to leverage knowledge transfer between different data types – from images and text to audio and structured information – creating more versatile and robust systems.

We often see a multistage training paradigm combining pre-training, fine-tuning and instruction-tuning (or post-training), each stage contributing uniquely to the model's performance and adaptability. This multi step approach boosts performance by enabling the model to learn progressively more specific representations at each stage.

This article will look at recent techniques for using models in multimodal and cross-domain applications and explore the following:

1. What is Model Pretraining
2. Overview of the Model Training Process
3. What is Fine-Tuning
4. Overview of the Model Fine-Tuning Process
5. Pre-training vs Fine-tuning: Advantages and Limitations

What is Pre-Training?

Pre-training allows the models to learn fundamental representations about the underlying structure in a self-supervised manner. This phase is crucial for developing a robust understanding of complex visual and textual features by leveraging large-scale datasets, and allows the model to develop a versatile feature space without explicit instructions that forms the basis for subsequent task-specific learning. 

Objectives of Model Pretraining in Computer Vision

The primary objective of pre-training is to develop generalized representations and pattern recognition abilities by exposing the model to vast amounts of diverse data before fine-tuning it for specific downstream tasks. During pre-training, models typically learn through self-supervised objectives where the data provides its own supervision signals. 

For language models, this might involve predicting masked words or next tokens, while vision models might reconstruct partially obscured images or determine relative positions of image patches. These tasks force the model to develop rich internal representations that capture semantic, syntactic, and contextual information. This allows the model to:

• Learn the underlying structure, relationships and features present in the data by developing rich representations that learn semantic and syntactic features.
• Establish a strong foundational understanding which can then be used downstream for further fine-tuning to boost model performance.
• By exposing the model to large scale datasets during pre-training, the resulting model becomes robust to distribution shifts.

A common strategy used during model pre-training is contrastive learning. 

Techniques used in pre training

Pre training techniques have evolved significantly over the years with self supervised learning becoming the dominant paradigm. These methods allow us to define an unsupervised learning task (without needing labels) wherein the supervision or the signal is provided by the data itself. This becomes increasingly important in cases where data annotation is expensive such as medical imaging. Some of the key techniques used in vision and text pre training are:

• Contrastive Learning: Employed by techniques like SimCLR and MoCo contrastive, it learns to aim representations by grouping “similar” items together while pushing different items away from each other. 
• Masked Language Modelling (MLM): A common technique used for training large language models wherein random parts of sentences are masked and the model is trained to predict or fill in the missing parts. This allows the model to learn relationships between words.
• Next Token Prediction: A key focal point of the recent success of foundational models wherein the model is trained to simply predict the next token in a given sequence. 
• Masked Image Modelling (MIM): A simple extension of masked language modelling where instead of filling in missing parts of sentences, the model learns to fill the missing parts of images allowing the model to learn complex relationships between various parts of an image.

Pretraining in real-life computer vision applications

Now, let’s explore pretraining in various real-life computer vision use cases.

Pre Training for Autonomous Driving

Similar to recent trends, the Autonomous Driving domain has also seen an evolution from traditional supervised approaches to leveraging pre-trained models in novel ways to enhance perception tasks.

One such instance is the use of pre-trained semantic segmentation networks to guide geometric representation learning. As demonstrated in Guizilini et al.'s ICLR 2020 work Semantically-Guided Representation Learning for Self-Supervised Monocular Depth, these pre-trained networks can improve monocular depth prediction without requiring additional supervision, effectively transferring semantic knowledge to depth estimation tasks. 

In particular, they use pre-trained semantic segmentation networks to guide geometric representation learning and pixel-adaptive convolutions to learn semantic-dependent representations thereby exploiting latent information in the data.

‍

As seen in DepthPro (Bochkovski et al., 2024), the field now leverages pre-trained vision transformers for transfer learning, allowing for increased flexibility in model design. This approach uses a combination of pre-trained ViT encoders: a multi-scale patch encoder for scale invariance and an image encoder for global context anchoring. 

The effectiveness of pre-training is particularly evident in the mixing of datasets; using a combination of real and synthetic datasets during pre-training has been shown to increase generalization, as measured by zero-shot accuracy (Ranftl et al. 2019). This hybrid approach to pre-training helps models become more robust and adaptable across different scenarios.

In the context of ego-motion estimation, pre-training has evolved to incorporate multiple modalities. The two-stream network architecture proposed by Ambrus et al. (2019) demonstrates how pre-training can be enhanced by treating RGB images and predicted monocular depth as separate input modalities. 

This multi-modal pre-training approach enables the network to learn both appearance and geometry features effectively.

‍

For camera self-calibration tasks, pre-training has moved toward end-to-end frameworks that can learn from video sequences rather than static frames. The work by Fang et al. (2022) shows how pre-training can be done using view synthesis objectives alone, enabling models to adapt to various camera geometries, including perspective, fisheye, and catadioptric setups.

The trend in pre-training for autonomous driving is clearly moving towards the use of foundational models that can operate in zero-shot scenarios. This shift is exemplified by recent works that are moving away from requiring metadata such as camera intrinsics, instead focusing on developing models that can generalize across different scenarios and camera setups without explicit calibration or fine-tuning. 

The combination of multi-modal inputs and the use of both synthetic and real data during pre-training has enabled models to achieve better performance across various autonomous driving tasks while reducing the need for expensive labeled data.

Pro Tip: Are you labeling data? Check out 12 Best Data Annotation Tools for Computer Vision (Free & Paid).

Multitask Objectives

DeepMind introduced Gato in 2022 in the realm of agents and reinforcement learning. It is a generalist agent designed to perform a number of tasks across multiple modalities, including text, images, and robotic control. 

By leveraging a single neural network with consistent weights, Gato seamlessly switches between tasks such as playing Atari games, generating image captions, engaging in conversations, and controlling robotic arms to stack blocks.

‍

To effectively balance multiple objectives, Gato employed a unified training approach. Gato was trained on a diverse dataset encompassing over 600 tasks, including text generation, image captioning, and robotic control. This extensive training allowed it to learn shared representations applicable across tasks, facilitating efficient multitask learning. 

The model's architecture and training regimen enable it to generalise across tasks without requiring task-specific fine-tuning. Gato represented a significant step toward creating adaptable and efficient AI systems capable of performing diverse tasks.

Gato's architecture emphasises the use of general-purpose representations applicable across various tasks. The model can adapt to new tasks by learning shared representations during training without requiring task-specific modifications.

PaLI-X, Chen et al. (2023) is a multilingual vision and language model that significantly advanced benchmark performance across diverse tasks, including image captioning, visual question answering, document understanding, object detection, and video analysis.

‍

The authors employ an encoder-decoder architecture to process diverse data formats. Images are passed through a Vision Transformer (ViT) encoder, which processes visual data into embeddings. These embeddings are combined with textual inputs—such as questions, prompts, or captions—and fed into the decoder. This enables PaLI-X to handle tasks like image captioning, where the output is text describing the image, and visual question answering, where the output is a text response to a question about the image. Additionally, it can process multiple images simultaneously, facilitating tasks like video captioning and object detection.

Since the model balances multiple objectives, PaLI-X utilises a mixture-of-objectives training approach. This strategy combines prefix-completion and masked-token completion tasks, allowing the model to learn from both the context provided by preceding tokens and the structure of the masked tokens.

‍

What is Model Fine-Tuning

Fine-tuning is a method of transfer learning where a pre-trained AI model undergoes additional training on a specialized dataset to enhance its accuracy and effectiveness on specific tasks or within particular domains. 

This targeted retraining leverages the model’s existing knowledge, allowing it to rapidly adapt and achieve superior performance in tasks such as image classification, object detection, or natural language processing. 

Fine-tuning helps maximize the relevance of AI models in real-world scenarios, significantly reducing time and resources compared to training models from scratch.

Objectives of Fine-Tuning in Computer Vision

The primary objective of fine-tuning is to adapt a model's general knowledge to perform well on targeted applications while preserving the valuable representations learned during pre-training.

• During fine-tuning, models are typically exposed to task-specific labeled data and trained with supervised learning objectives directly aligned with the intended application. 
• This allows the model to achieve high performance on specialized tasks with minimal additional training data. 
• Fine-tuning further allows us to enhance model alignment and safety by incorporating human feedback to ensure the models behave appropriately in real-world use cases.

By leveraging the rich representations developed during pre-training, fine-tuning can achieve remarkable results with orders of magnitude less data than would be required for training from scratch, making specialized AI applications more accessible. 

Overview Of The Fine-Tuning Process

Now let’s look at various strategies and techniques used in various industries to improve downstream model accuracy and overall performance on relevant benchmarks. We will explore how iterative training, domain-specific adjustments, and algorithmic enhancements contribute to achieving SOTA performance.

‍

Training for Robotics, Decision Making and Embodied Reasoning

While LLMs have already become integral to many people’s workflows, their integration into our wider lives has been blocked by their inability to generalise to open-domain tasks and weak reasoning capabilities. To enable a world where intelligent agents and humans are in a shared environment, we need models capable of reasoning across text, vision, speech, and sensor modalities.

‍

The NeurIPS 2023 paper "Large Language Models are Visual Reasoning Coordinators" introduces Cola. This novel framework leverages large language models (LLMs) to coordinate multiple vision-language models (VLMs) for enhanced visual reasoning tasks. The authors propose that an LLM can effectively coordinate multiple VLMs by harnessing their individual strengths. They propose two primary variants:

1. Instruction Tuning (Cola-FT): In this variant, the LLM is fine-tuned with specific instructions to guide the VLMs in performing visual reasoning tasks.
2. In-Context Learning (Cola-Zero): This variant employs in-context learning, allowing the LLM to coordinate the VLMs without additional fine-tuning.

While VLMs have demonstrated proficiency in tasks like visual question answering with the help of methods like Cola, they often struggle with real-time processing and integrating multimodal data necessary for embodied tasks.

PaLM-E by Driess et al. (2023) attempts to create a single large embodied multimodal model capable of operating on multimodal sentences from a variety of observation modalities, on multiple embodiments, i.e., sequences of tokens, where inputs from arbitrary modalities (images or neural 3D representations) alongside text allow for a direct integration of the rich semantic knowledge stored in pre-trained LLMs into the planning process. 

They build upon Google's Pathways Language Model (PaLM) (A 540 B parameter LLM), incorporating sensor data from robotic agents, such as images and continuous state estimations, alongside textual inputs.

‍

PaLM-E employs an encoder that maps continuous observations into a sequence of vectors. These vectors are interleaved with text tokens, forming a combined input sequence for the model. The self-attention layers of the LLM backbone can then process these multimodal sentences in the same way as text. This allows the model to incorporate real-world continuous sensor modalities directly.

The model then generates sequences of actions for robots to perform complex tasks, considering physical constraints and environmental dynamics. Moreover, PaLM-E can also answer questions about visual scenes, integrate information from images and textual queries, and generate descriptive captions for images to showcase its understanding of visual content.

• The authors report strong benchmarks on tasks such as mobile manipulation environment (failure detection and affordance prediction)

• while maintaining strong performance in general visual-language tasks.

Zhai et al. (2024) propose a framework for training VLMs with Reinforcement Learning in their paper “Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning”. Given a task description, a VLM is prompted to generate chain-of-thought (CoT) reasoning based on the current states. This enables the efficient exploration of intermediate reasoning steps that lead to a final text-based action.

‍

The prompting step encourages the model to decompose complex tasks into manageable sub-tasks, facilitating structured decision-making. The model's output, an open-ended text response, is then parsed into an executable action, enabling interaction with the environment. The environment then provides feedback through rewards based on the model's actions. 

This iterative process allows the model to improve its performance over time, adapting to the specific requirements of the task.

• The authors report that their method can enable a backbone LLaVA-1.6-7b model to outperform commercial models (GPT4-V and Gemini) and supervised fine-tuned LLaVA-7b on tasks requiring arithmetic capabilities and visual semantic understanding.

• Moreover, they also empirically demonstrate that CoT reasoning is a crucial component, and the model’s performance suffers without it.

Improving Cross-domain Text Applications

Document understanding in Vision-Language Models (VLMs) involves integrating textual, visual, and structural elements to comprehend and process documents effectively. Models like DocumentCLIP (Liu et al. 2023) utilise contrastive learning to align images and their corresponding textual content within documents.

By training on large datasets, these models learn to associate visual elements with relevant text, enhancing their ability to understand the context and semantics of documents. They achieve this using multiple embeddings:

• Text embeddings are generated using a lower-cased byte pair encoding (BPE) to tokenize the sentences in documents.
• Visual embeddings are generated using a vision transformer to extract non-overlapping image patches into 1D tokens.
• Layout embeddings are used to understand the global context of the document.

‍

"LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding" by Luo et al. (CVPR 2024) introduces LayoutLLM, a method that enhances document comprehension by integrating large language models (LLMs) with layout-specific instruction tuning. This approach addresses the challenge of effectively utilising document layout information, which is crucial for accurate document understanding.

To capture the structural nuances of documents, LayoutLLM employs a pre-training strategy that focuses on three levels of information:

• Document-Level Information: This level captures the overall structure and organisation of the document.
• Region-Level Information: This level focuses on specific sections or regions within the document.
• Segment-Level Information: This level examines individual segments or blocks of text within the regions.

LayoutLLM also introduced a novel module named LayoutCoT, which enables the model to focus on regions relevant to a given question. This enhances the model's ability to generate accurate answers by directing attention to relevant sections of the document. Additionally, LayoutCoT provides interpretability, allowing for manual inspection and correction of the model's reasoning process. By training on document-level, region-level and segment-level tasks, LayoutLLM develops a hierarchical understanding of document layouts.

Pre-training vs Fine-tuning: Advantages and Limitations

• Computational Cost and Feasibility: Training from scratch is extremely resource-intensive, requiring massive amounts of compute power and time. Conversely, fine-tuning pre-trained models dramatically reduces computational cost, making high-performance AI accessible even to teams with limited hardware (though large models can still impose heavy demands during adaptation). In the current age where large language models are getting commoditized, fine-tuning is largely becoming the more accessible option.
• Data Quality and Domain Mismatch: Pre-trained models are often built on broad, generic datasets that may not represent the specific nuances of a target domain. When fine-tuning, this mismatch can lead to suboptimal performance if the data quality or domain characteristics differ significantly. In contrast, models trained from scratch can be optimized for the domain, but they require a high-quality, extensive dataset to avoid learning erroneous patterns.
• Catastrophic Forgetting: When fine-tuning, there is a risk that the model might “forget” the useful, general representations it learned during pre-training. This phenomenon—catastrophic forgetting—can be addressed by techniques like gradual unfreezing, using lower learning rates for earlier layers, or blending fine-tuned weights with the original pre-trained weights.
• Balancing Generalization and Specialization: The challenge is to tailor the model closely enough to the task (specialization) without losing the robust, generalized features learned during pre-training. This balance is critical: excessive specialization can impair performance on slightly varied inputs, while too much generalization might not fully capture task-specific nuances.
• Biases and Ethical Concerns: Pre-trained models may carry biases from their training data, which can be amplified during fine-tuning if not carefully managed. It’s crucial to conduct bias assessments and incorporate fairness checks, ensuring that the adapted model does not perpetuate harmful stereotypes or unethical outcomes.

Conclusion

The multimodal approaches outlined in this article represent different strategies for bridging the gap between language, vision, and embodied reasoning. While coordination frameworks like Cola and PaLM-E tackle the challenge through different architectural choices, document understanding models like LayoutLLM show how structural information can be effectively incorporated into model reasoning.

‍