Skip to content
FastVLM.net

FASTVLM 指南 · Browser

构建 FastVLM WebGPU 图片问答应用

使用可下载的 Vite 完整项目构建本地图片问答，支持流式回答、取消和 JSON 导出。

全部 FastVLM 指南

操作步骤

  1. 1

    检查 WebGPU 和 shader-f16，准备 Node.js 22+。

  2. 2

    下载并解压项目，安装依赖，再启动 Vite。

  3. 3

    选择收据样例并运行，检查 $15.00 答案。

  4. 4

    阅读 Worker、验证取消与 JSON 导出，再构建到 HTTPS 站点。

示例

Browser 
curl -fLO https://fastvlm.net/examples/fastvlm-webgpu.zip
unzip fastvlm-webgpu.zip
cd fastvlm-webgpu
npm install
npm run dev
# Open the localhost URL printed by Vite.
# Production: npm run build, then host all of dist/ over HTTPS.

构建可运行的图片问答应用

下载 Vite + TypeScript 完整项目，使用与本站相同的推理 Worker。包含收据样例、本地图片输入、流式回答、下载进度、取消和 JSON 导出。

使用 Node.js 22+，以及同时提供 WebGPU 和 shader-f16 的浏览器。本地使用 localhost，部署后需要 HTTPS。首次运行从 Hugging Face 下载约 1.1 GB 文件，还需足够的空闲 GPU 内存。

下载完整示例 ZIP ↓体验实际应用 →

验证第一次运行

打开 Vite 输出的 localhost 地址，选择收据样例，再点击 Run。检查答案是否包含 $15.00；导出 JSON 查看原始答案和实测耗时。下载时点击 Cancel，再重新运行，检查恢复流程。

应用如何工作

  1. 主线程解码图片，将最长边限制为 1024 像素，再把像素和问题交给 Worker。模型运行时页面仍可响应操作。
  2. Worker 按需加载处理器和 ONNX 模型，逐步发送文字，最后返回完整答案与耗时。同一个 Worker 保留模型会话，供后续问题复用。
  3. 取消会终止 Worker。下次运行需要重建会话；浏览器模型缓存可能避免重新下载权重，是否可复用取决于来源站点和浏览器。

耗时如何定义

模型准备包含缓存读取、下载和会话初始化。“图片 → 回答”从图片处理前计时，到生成结束。首段文字截止于第一个非空流式文本片段，不等同于严格的首 token 基准。导出的结果不含图片像素。

阅读源码: src/main.ts
src/main.ts 
import { checkBrowserCapability } from './capabilities';
import type { WorkerResponse } from './protocol';

const input = document.querySelector<HTMLInputElement>('#image')!;
const prompt = document.querySelector<HTMLTextAreaElement>('#prompt')!;
const preview = document.querySelector<HTMLImageElement>('#preview')!;
const status = document.querySelector<HTMLElement>('#status')!;
const answer = document.querySelector<HTMLElement>('#answer')!;
const run = document.querySelector<HTMLButtonElement>('#run')!;
const cancel = document.querySelector<HTMLButtonElement>('#cancel')!;
const sample = document.querySelector<HTMLButtonElement>('#sample')!;
const exportButton = document.querySelector<HTMLButtonElement>('#export')!;
let worker: Worker | null = null;
let image = '';
let busy = false;
let inputVersion = 0;
let result: Extract<WorkerResponse, { type: 'result' }> | null = null;
let resultPrompt = '';

function setBusy(value: boolean) {
  busy = value;
  input.disabled = prompt.disabled = sample.disabled = value;
  run.disabled = value || !image || !prompt.value.trim();
  cancel.hidden = !value;
  exportButton.disabled = value || !result;
}
function stop() { worker?.terminate(); worker = null; setBusy(false); }

async function selectImage(blob: Blob) {
  const version = ++inputVersion;
  result = null; image = ''; preview.hidden = true;
  answer.textContent = ''; status.textContent = '';
  setBusy(true);
  const url = URL.createObjectURL(blob);
  try {
    const decoded = new Image(); decoded.src = url; await decoded.decode();
    if (version !== inputVersion) return;
    const scale = Math.min(1, 1024 / Math.max(decoded.width, decoded.height));
    const canvas = document.createElement('canvas');
    canvas.width = Math.max(1, Math.round(decoded.width * scale));
    canvas.height = Math.max(1, Math.round(decoded.height * scale));
    const context = canvas.getContext('2d')!;
    context.fillStyle = 'white'; context.fillRect(0, 0, canvas.width, canvas.height);
    context.drawImage(decoded, 0, 0, canvas.width, canvas.height);
    image = canvas.toDataURL('image/png'); preview.src = image; preview.hidden = false;
  } catch { status.textContent = 'Cannot read this image. Choose a JPEG, PNG or WebP.'; }
  finally { URL.revokeObjectURL(url); if (version === inputVersion) setBusy(false); }
}

input.onchange = () => {
  const file = input.files?.[0]; input.value = '';
  if (!file) return;
  if (!['image/jpeg','image/png','image/webp'].includes(file.type) || file.size > 10 * 1024 * 1024) {
    image = ''; preview.hidden = true; result = null; answer.textContent = ''; setBusy(false);
    status.textContent = 'Use a JPEG, PNG or WebP of at most 10 MB.'; return;
  }
  void selectImage(file);
};
sample.onclick = async () => {
  const version = ++inputVersion;
  setBusy(true);
  try {
    const response = await fetch('/sample.png'); if (!response.ok) throw new Error('Sample unavailable');
    const blob = await response.blob(); if (version !== inputVersion) return;
    await selectImage(blob);
  } catch { if (version === inputVersion) { setBusy(false); status.textContent = 'Sample unavailable. Choose a local image.'; } }
};
prompt.oninput = () => setBusy(busy);
cancel.onclick = () => { inputVersion++; stop(); status.textContent = 'Cancelled. Run again when ready.'; };

run.onclick = async () => {
  if (busy || !image || !prompt.value.trim()) return;
  result = null; answer.textContent = ''; setBusy(true);
  status.textContent = 'Checking this browser…';
  const version = ++inputVersion;
  const capability = await checkBrowserCapability();
  if (version !== inputVersion) return;
  if (!capability.supported) { setBusy(false); status.textContent = 'WebGPU with shader-f16 is required. See the Python guide below.'; return; }
  resultPrompt = prompt.value.trim();
  try {
    worker ??= new Worker(new URL('./worker.ts', import.meta.url), { type: 'module' });
    worker.onmessage = ({ data }: MessageEvent<WorkerResponse>) => {
      if (version !== inputVersion) return;
      if (data.type === 'loading') status.textContent = 'Downloading/preparing model…';
      if (data.type === 'progress') status.textContent = `Model files read: ${data.loadedMB} MB; preparing model…`;
      if (data.type === 'running') status.textContent = 'Reading image…';
      if (data.type === 'text') answer.textContent = data.text;
      if (data.type === 'result') {
        result = data; answer.textContent = data.text; setBusy(false);
        status.textContent = `Preparation: ${data.timing.load_ms} ms · First text: ${data.timing.first_text_ms ?? '—'} ms · Image to answer: ${data.timing.inference_ms} ms`;
      }
      if (data.type === 'error') { status.textContent = `Failed during ${data.phase}: ${data.message}`; stop(); }
    };
    worker.onerror = () => { if (version === inputVersion) { status.textContent = 'Worker failed. Retry or use local Python.'; stop(); } };
    worker.postMessage({ image, prompt: resultPrompt });
  } catch { status.textContent = 'Could not start the worker.'; stop(); }
};
exportButton.onclick = () => {
  if (!result) return;
  const url = URL.createObjectURL(new Blob([JSON.stringify({ ...result, prompt: resultPrompt }, null, 2)], { type: 'application/json' }));
  const link = document.createElement('a'); link.href = url; link.download = 'fastvlm-result.json'; link.click();
  setTimeout(() => URL.revokeObjectURL(url), 1000);
};
void checkBrowserCapability().then(value => {
  document.querySelector('#capability')!.textContent = value.supported ? 'WebGPU and shader-f16 available. Free GPU memory is also required.' : 'This browser does not meet the WebGPU / shader-f16 requirements. See the Python guide below.';
});
window.addEventListener('pagehide', () => { inputVersion++; stop(); });
阅读源码: src/worker.ts
src/worker.ts 
import { AutoModelForImageTextToText, AutoProcessor, RawImage, TextStreamer, env } from '@huggingface/transformers';
import type { LlavaProcessor, PreTrainedModel, Tensor } from '@huggingface/transformers';
import type { RunRequest, WorkerResponse } from './protocol';

env.allowLocalModels = false;
const modelId = 'onnx-community/FastVLM-0.5B-ONNX';
let model: PreTrainedModel | null = null;
let processor: LlavaProcessor | null = null;
let busy = false;
const send = (message: WorkerResponse) => self.postMessage(message);

self.onmessage = async ({ data }: MessageEvent<RunRequest>) => {
  if (busy) return;
  busy = true;
  let phase: 'load' | 'inference' = 'load';
  const loadStart = performance.now();
  try {
    if (!model || !processor) {
      send({ type: 'loading' });
      const files = new Map<string, number>();
      let lastMB = -1;
      const progress_callback = (info: { status: string; file?: string; loaded?: number }) => {
        if (info.status !== 'progress' || !info.file || typeof info.loaded !== 'number') return;
        files.set(info.file, info.loaded);
        const loadedMB = Math.floor([...files.values()].reduce((sum, bytes) => sum + bytes, 0) / 1_000_000);
        if (loadedMB !== lastMB) { lastMB = loadedMB; send({ type: 'progress', loadedMB }); }
      };
      processor = await AutoProcessor.from_pretrained(modelId, { progress_callback }) as LlavaProcessor;
      model = await AutoModelForImageTextToText.from_pretrained(modelId, {
        progress_callback,
        device: 'webgpu',
        dtype: { embed_tokens: 'fp16', vision_encoder: 'q4', decoder_model_merged: 'q4' },
      });
    }
    const loadMs = Math.round(performance.now() - loadStart);
    send({ type: 'loaded', duration: loadMs });
    phase = 'inference';
    send({ type: 'running' });
    const start = performance.now();
    const image = await RawImage.fromURL(data.image);
    const prompt = processor.apply_chat_template([
      { role: 'system', content: 'You are a helpful visual assistant. Answer the question accurately and concisely.' },
      { role: 'user', content: `<image>${data.prompt}` },
    ], { add_generation_prompt: true });
    const inputs = await processor(image, prompt, { add_special_tokens: false });
    let firstText: number | null = null;
    let streamed = '';
    const streamer = new TextStreamer(processor.tokenizer!, {
      skip_prompt: true,
      skip_special_tokens: true,
      callback_function: (chunk: string) => {
        if (chunk && firstText === null) firstText = performance.now() - start;
        streamed += chunk;
        send({ type: 'text', text: streamed });
      },
    });
    const output = await model.generate({ ...inputs, max_new_tokens: 192, do_sample: false, repetition_penalty: 1.2, streamer }) as Tensor;
    const inputLength = inputs.input_ids.dims.at(-1)!;
    const generated = output.slice(null, [inputLength, null]);
    const text = processor.batch_decode(generated, { skip_special_tokens: true })[0].trim();
    send({ type: 'result', model: modelId, text, timing: {
      load_ms: loadMs,
      inference_ms: Math.round(performance.now() - start),
      first_text_ms: firstText === null ? null : Math.round(firstText),
      generated_tokens: generated.dims.at(-1) ?? null,
    } });
  } catch (error) {
    // No input or raw exception text leaves the worker in telemetry.
    send({ type: 'error', phase, message: error instanceof Error ? error.message : 'Inference failed' });
    if (phase === 'load') { model = null; processor = null; }
  } finally { busy = false; }
};

构建与部署

npm run build 输出 dist/，将完整目录部署到 HTTPS 站点，保留 JavaScript 与 WASM 文件。权重由访问者的浏览器下载，Worker 与配套资源路径需要一起保留。

运行不成功时

  • WebGPU 可用但缺少 shader-f16：下载权重前就提供 Python 替代入口。
  • 模型准备很慢：区分文件读取与会话初始化；未知总量时不要显示虚构的百分比。
  • GPU 内存不足：关闭其他模型标签页。缩小上传图片不会缩小模型权重。
  • 部署后 Worker 或 WASM 404：确认 dist 资源上传完整，并检查站点基础路径。
Python 替代方式 →

一手文档: Transformers.js WebGPU · ONNX model · Example source

使用一手资料复核

API、模型文件和依赖版本都可能变化。请把本页作为实现路线，并在链接的官方文档中确认当前命令与许可。

打开官方来源