MobileCLIP2 S0 vs S2: a reproducible CPU test

Measurement date: September 12, 2026 · one machine · six classes · no tuning on these examples

Measured by FastVLM.net

Test scope and limits

Apple M5 Pro host with 48 GiB unified memory, CPU only, float32, 4 PyTorch threads, batch size 1. Each model ran in its own process: 3 warmups, then 5 fixed-order repeats of all 24 images. The six candidate text embeddings were computed once. Timing includes image file decode, preprocessing, image encoding and scoring; excludes download, model loading and text encoding.

Peak RSS includes Python, library imports, checkpoint loading and initialization. Parameter-tensor bytes exclude buffers and runtime overhead. Neither figure is a minimum RAM requirement or GPU-memory measurement.

Six synthetic template families with four variants each, using six fixed English candidate labels. All three errors were small-text documents classified as interfaces. A different candidate-label set, photographs or real documents can change the result. Equal performance here does not mean equal general accuracy.

Apple-published results: a separate benchmark

Apple reports ImageNet zero-shot top-1 of 71.5% for S0 and 77.2% for S2. Those values are from a different dataset and measurement procedure; do not combine them with this 24-image result or compare our CPU latency to Apple’s device benchmarks. Apple S0 · Apple S2

How to choose

S0 is a reasonable starting point when these document-category distinctions and lower CPU latency fit the task. Choose S2 after testing whether it improves your own labels and images enough to justify the extra resources. Neither model generates a written answer.

Reproduce and inspect

python -m pip install "open_clip_torch==3.3.0" "timm==1.0.29" torch pillow
python scripts/evaluate-mobileclip.py --model S0
python scripts/evaluate-mobileclip.py --model S2

S0 JSONS2 JSONManifestProvenance

Inspect every case

CategoryAll categoriesReceiptsChartsInterfacesSmall English textChinese textShapes Only missing-answer / misclassified cases 24 cases shown

receipt-1

Expected class: a receipt

S0 Prediction: a receipt · 108.7 ms

S2 Prediction: a receipt · 270.8 ms

receipt-2

Expected class: a receipt

S0 Prediction: a receipt · 110.7 ms

S2 Prediction: a receipt · 267.6 ms

receipt-3

Expected class: a receipt

S0 Prediction: a receipt · 110.7 ms

S2 Prediction: a receipt · 263.7 ms

receipt-4

Expected class: a receipt

S0 Prediction: a receipt · 110.4 ms

S2 Prediction: a receipt · 268.9 ms

chart-1

Expected class: a bar chart

S0 Prediction: a bar chart · 109.9 ms

S2 Prediction: a bar chart · 264.2 ms

chart-2

Expected class: a bar chart

S0 Prediction: a bar chart · 114.8 ms

S2 Prediction: a bar chart · 285.3 ms

chart-3

Expected class: a bar chart

S0 Prediction: a bar chart · 112.7 ms

S2 Prediction: a bar chart · 280.6 ms

chart-4

Expected class: a bar chart

S0 Prediction: a bar chart · 109.5 ms

S2 Prediction: a bar chart · 268.2 ms

interface-1

Expected class: a user interface screenshot

S0 Prediction: a user interface screenshot · 110.3 ms

S2 Prediction: a user interface screenshot · 271.0 ms

interface-2

Expected class: a user interface screenshot

S0 Prediction: a user interface screenshot · 110.5 ms

S2 Prediction: a user interface screenshot · 271.5 ms

interface-3

Expected class: a user interface screenshot

S0 Prediction: a user interface screenshot · 110.6 ms

S2 Prediction: a user interface screenshot · 284.5 ms

interface-4

Expected class: a user interface screenshot

S0 Prediction: a user interface screenshot · 111.0 ms

S2 Prediction: a user interface screenshot · 267.8 ms

small-text-1

Expected class: an English text document

S0 Prediction: an English text document · 111.0 ms

S2 Prediction: an English text document · 270.9 ms

small-text-2

Expected class: an English text document

S0 Prediction: a user interface screenshot · 108.1 ms

S2 Prediction: a user interface screenshot · 284.5 ms

small-text-3

Expected class: an English text document

S0 Prediction: a user interface screenshot · 109.0 ms

S2 Prediction: a user interface screenshot · 282.1 ms

small-text-4

Expected class: an English text document

S0 Prediction: a user interface screenshot · 109.4 ms

S2 Prediction: a user interface screenshot · 277.1 ms

chinese-1

Expected class: a Chinese text document

S0 Prediction: a Chinese text document · 108.7 ms

S2 Prediction: a Chinese text document · 272.5 ms

chinese-2

Expected class: a Chinese text document

S0 Prediction: a Chinese text document · 108.1 ms

S2 Prediction: a Chinese text document · 271.8 ms

chinese-3

Expected class: a Chinese text document

S0 Prediction: a Chinese text document · 109.6 ms

S2 Prediction: a Chinese text document · 287.6 ms

chinese-4

Expected class: a Chinese text document

S0 Prediction: a Chinese text document · 109.0 ms

S2 Prediction: a Chinese text document · 274.4 ms

shapes-1

Expected class: an illustration of geometric shapes

S0 Prediction: an illustration of geometric shapes · 108.4 ms

S2 Prediction: an illustration of geometric shapes · 274.3 ms

shapes-2

Expected class: an illustration of geometric shapes

S0 Prediction: an illustration of geometric shapes · 110.7 ms

S2 Prediction: an illustration of geometric shapes · 264.1 ms

shapes-3

Expected class: an illustration of geometric shapes

S0 Prediction: an illustration of geometric shapes · 109.7 ms

S2 Prediction: an illustration of geometric shapes · 267.7 ms

shapes-4

Expected class: an illustration of geometric shapes

S0 Prediction: an illustration of geometric shapes · 108.1 ms

S2 Prediction: an illustration of geometric shapes · 265.2 ms