APPLE AIMV2 · MULTIMODAL AUTOREGRESSIVE VISION ENCODERS

AIMv2: Apple’s open vision encoders for multimodal models

AIMv2 pre-trains a vision transformer to autoregressively reconstruct image patches and text tokens together. Apple reports that it outperforms CLIP and SigLIP on most multimodal understanding benchmarks while staying easy to train and scale. The checkpoints are open, and the reference code runs on PyTorch, JAX and MLX.

GitHub: apple/ml-aim Hugging Face collection Try image–text similarity (MobileCLIP)

Why AIMv2 matters for on-device vision

A VLM is only as good as its vision encoder. AIMv2 gives you a family of encoders with a simple training recipe and multiple resolutions, so you can pick a 0.3B model for a phone-class backbone or the 2.7B native-resolution model for document-heavy tasks. FastVLM uses its own FastViTHD encoder; AIMv2 is the general-purpose alternative when you build your own multimodal model.

Extract image embeddings with Transformers

The Hugging Face checkpoints load with AutoModel and trust_remote_code. Mean-pool the token features to get one vector per image for similarity search, clustering or as input to a small classifier.

Python · transformers

from PIL import Image
from transformers import AutoImageProcessor, AutoModel

model_id = "apple/aimv2-large-patch14-224"   # 0.3B; also 224/336/448, -native and -lit variants
processor = AutoImageProcessor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, trust_remote_code=True).eval()

inputs = processor(images=Image.open("photo.jpg").convert("RGB"), return_tensors="pt")
features = model(**inputs).last_hidden_state      # (1, tokens, hidden)
embedding = features.mean(dim=1)                  # one pooled vector per image

MLX on Apple Silicon

The official ml-aim package exposes the same checkpoints through an MLX backend, which is the fastest way to experiment on a Mac without CUDA. Check the repository README for the exact API of the release you install.

MLX · official package

pip install "git+https://github.com/apple/ml-aim.git#subdirectory=aim-v2"

# Python
from aim.v2.utils import load_pretrained
model = load_pretrained("aimv2-large-patch14-224", backend="mlx")   # or "torch" / "jax"

AIMv2 vs MobileCLIP2 vs SigLIP

MobileCLIP2 is a contrastive image–text model tuned for latency: use it for zero-shot classification and retrieval on phones (and in this site’s browser tools). SigLIP is the common contrastive encoder inside many VLMs. AIMv2 targets the encoder role with a generative objective and larger sizes; it is the one to fine-tune when you train your own VLM.

Questions

Can AIMv2 run in the browser?

There is no maintained Transformers.js port at the time of checking. This site’s browser similarity tool uses MobileCLIP-S0 instead; AIMv2 is best used from Python or MLX.

Which AIMv2 size should I start with?

aimv2-large-patch14-224 (0.3B) is the practical default. Move to 336/448 px or the native-resolution variants for text-heavy images, and to 1.2B–2.7B only when you can afford the compute.