Vision-and-Language Transformer for Visual Question Answering
Answers natural language questions about image content using unified vision-and-language Transformer.
What it does
ViLT (Vision-and-Language Transformer) is a unified transformer model that processes both images and text to answer questions about images. Given an image and a natural language question, the model generates an answer by understanding visual content and reasoning about the question.
Problem it solves
- Visual question answering (answering natural language questions about images)
- Image-text reasoning and understanding
- Accessibility: Provide textual answers about image content for visually impaired users
- Automated image understanding and analysis
- Content analysis and description generation with question guidance
- Educational and accessibility applications
- Image comprehension validation and testing
Input/Output
- Input: Image (RGB) + Natural language question
- Question format: Natural language text (any language supported by training)
- Example question: "How many cats are there?"
- Output: Predicted answer (text)
- Format: Short text answer
- Example outputs:
- "2" (counting)
- "yes" / "no" (yes/no questions)
- "red" (color/attribute questions)
- "playing" (action questions)
Accuracy & Speed
- Accuracy: Trained on VQAv2 dataset (standard VQA benchmark)
- Metrics:
- VQAv2 benchmark evaluation (accuracy varies by question type)
- Pure transformer architecture (no CNNs or region-based features)
- Efficient inference relative to CNN+LSTM approaches
Technical Details
- Architecture: Vision-and-Language Transformer (ViLT)
- Key Features:
- Pure transformer design—no convolutional layers
- Processes image patches and text tokens together in a single transformer
- No bounding boxes or region features needed
- Components:
- Vision Encoder (ViT-B/32): Converts images to patch embeddings
- Text Encoder: Tokenizes and embeds natural language
- Unified Transformer: Jointly processes vision and text via multi-head attention
- VQA Head: Generates answers from the unified representation
- Training:
- Pre-trained on vision-language datasets
- Fine-tuned on VQAv2 (visual question answering)
- Requirements:
- PyTorch (v1.7+)
- GPU recommended for inference
Compliance & Provenance
Provider | Open-source |
Provider type | Specialized |
License | Apache 2.0 |
EU AI Act risk class | Minimal Risk |
Art. 50 transparency | Not applicable |
Region availability | Available globally |
Training data summary | Pending — provider has not yet published per Art. 53(d) |
For more on how we classify models and mark outputs, see our AI Policy.
Model Source
- HuggingFace: https://huggingface.co/dandelin/vilt-b32-finetuned-vqa
- GitHub Repository: https://github.com/dandelin/vilt
- Paper: "ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision" ICML 2021 https://arxiv.org/pdf/2102.03334
- License: apache-2.0