Skip to content
FastVLM.net

FOUNDATION MODELS FRAMEWORK · SWIFT · PYTHON

Foundation Models framework: on-device LLM sessions for your app

The Foundation Models framework is Apple’s native API for the model behind Apple Intelligence. You create a session, optionally describe the Swift type you want back, add tools the model may call, and stream the answer—no weights to ship, no server, no token bill. WWDC26 added images, Dynamic Profiles, a pluggable LanguageModel protocol, a Python SDK and evaluations.

Apple developer documentationWWDC26: What’s new in the Foundation Models frameworkCompare with Core ML and MLX

A first session in Swift

Check availability, create a session with instructions, then ask. Guided generation turns free text into typed Swift values, which is the feature that makes the framework practical for parsing receipts, forms and screenshots.

Swift · FoundationModels 
import FoundationModels

// Check that Apple Intelligence and the on-device model are available first.
guard case .available = SystemLanguageModel.default.availability else { return }

let session = LanguageModelSession(instructions: "You are a concise assistant.")
let answer = try await session.respond(to: "Summarise this receipt in one sentence.")
print(answer.content)

// Guided generation: the model fills a Swift type instead of free text.
@Generable
struct Receipt {
    @Guide(description: "Merchant name") var merchant: String
    @Guide(description: "Total amount in the receipt currency") var total: Double
}
let receipt = try await session.respond(to: "Extract the merchant and total.", generating: Receipt.self).content

Choosing a model backend (WWDC26)

Sessions are now backed by anything that conforms to the LanguageModel protocol. The on-device system model remains the default; Private Cloud Compute gives you Apple’s server models with the same privacy guarantees; CoreAILanguageModel and MLXLanguageModel run open models locally; and cloud providers such as Claude or Gemini can be plugged in behind one API.

Swift · LanguageModel protocol 
// WWDC26: any provider can back a session through the LanguageModel protocol.
let session = LanguageModelSession(model: SystemLanguageModel.default)          // on-device AFM 3 Core
// let session = LanguageModelSession(model: PrivateCloudComputeLanguageModel()) // Apple server models
// let session = LanguageModelSession(model: MLXLanguageModel(...))              // open-source local models
// Third-party cloud providers can conform to LanguageModel and plug in the same way.

Images, tools and Dynamic Profiles

Multimodal prompts let you pass images with text so the model can answer questions about a photo—the same task FastVLM performs, now with a system model. Vision framework tools such as OCR and barcode reading can be exposed to the model as callable tools. Dynamic Profiles switch the model, tools and instructions inside a continuing session, which is how an app moves from a quick on-device answer to a server model without losing context.

Python SDK, fm CLI and evaluations

WWDC26 session 334 covers the Python SDK and the fm command-line tool for scripting prompts against the system model on a Mac, and sessions 298/299/335 cover the Evaluations framework for testing prompts before release. Use them to prototype prompt designs quickly, then port the winning prompt into Swift.

When to use FastVLM instead

Use the framework when your app runs on Apple Intelligence devices and you want zero-download, system-managed inference. Use FastVLM (through MLX, Core ML or the browser) when you need a specific open model, cross-platform reach including the web and Linux, control over quantization and latency, or support for older devices.

Questions

Does the framework work offline?

Yes for SystemLanguageModel: inference runs on the device. Private Cloud Compute and third-party providers require a network connection.

Is the API the same as Core ML?

No. Core ML (and its WWDC26 successor Core AI) runs models you convert and ship yourself. The Foundation Models framework is a higher-level session API over models Apple provides—or, since WWDC26, over any provider that conforms to LanguageModel.