Compile Hugging Face models to run on Foundry Local

Use the Foundry Local C# SDK to load and run your compiled model with the native chat completions API. This approach doesn't require a REST server — the SDK communicates directly with the runtime.

Prerequisites

• .NET 8.0 SDK or later

Install packages

If you're developing or shipping on Windows, select the Windows tab. The Windows package integrates with the Windows ML runtime — it provides the same API surface area with a wider breadth of hardware acceleration.

dotnet add package Microsoft.AI.Foundry.Local.WinML
dotnet add package OpenAI

dotnet add package Microsoft.AI.Foundry.Local
dotnet add package OpenAI

The C# samples in the GitHub repository are preconfigured projects. If you're building from scratch, you should read the Foundry Local SDK reference for more details on how to set up your C# project with Foundry Local.

Run inference on the compiled model

Replace the contents of Program.cs with the following code:

using Microsoft.AI.Foundry.Local;
using Betalgo.Ranul.OpenAI.ObjectModels.RequestModels;
using Microsoft.Extensions.Logging;

CancellationToken ct = CancellationToken.None;

// Point ModelCacheDir at the directory containing your compiled model
var config = new Configuration
{
    AppName = "run-compiled-model",
    LogLevel = Microsoft.AI.Foundry.Local.LogLevel.Information,
    ModelCacheDir = "../models/llama"
};

using var loggerFactory = LoggerFactory.Create(builder =>
{
    builder.SetMinimumLevel(Microsoft.Extensions.Logging.LogLevel.Information);
});
var logger = loggerFactory.CreateLogger<Program>();

await FoundryLocalManager.CreateAsync(config, logger);
var mgr = FoundryLocalManager.Instance;

var catalog = await mgr.GetCatalogAsync();

// List cached models to find your compiled model
var cachedModels = await catalog.GetCachedModelsAsync();
Console.WriteLine("Cached models:");
foreach (var m in cachedModels)
{
    Console.WriteLine($"  {m.Id}");
}

// Select your compiled model from the cached list
var model = cachedModels.FirstOrDefault(m => m.Id.Contains("llama-3.2:1"))
    ?? throw new Exception("Compiled model not found. Verify the ModelCacheDir path.");

await model.LoadAsync();

// Use native chat completions
var chatClient = await model.GetChatClientAsync();

List<ChatMessage> messages = new()
{
    new ChatMessage { Role = "user", Content = "What is the golden ratio?" }
};

var streamingResponse = chatClient.CompleteChatStreamingAsync(messages, ct);
await foreach (var chunk in streamingResponse)
{
    Console.Write(chunk.Choices[0].Message.Content);
    Console.Out.Flush();
}
Console.WriteLine();

await model.UnloadAsync();

Run the application:

dotnet run

Use the Foundry Local JavaScript SDK to load and run your compiled model with the native chat completions API.

Prerequisites

• Node.js 20 or later installed.

Install packages

If you're developing or shipping on Windows, select the Windows tab. The Windows package integrates with the Windows ML runtime — it provides the same API surface area with a wider breadth of hardware acceleration.

npm install foundry-local-sdk-winml openai

npm install foundry-local-sdk openai

Run inference on the compiled model

Copy and paste the following code into a JavaScript file named app.js:

import { FoundryLocalManager } from 'foundry-local-sdk';

// Initialize the Foundry Local SDK with custom model cache directory
const manager = FoundryLocalManager.create({
    appName: 'run-compiled-model',
    logLevel: 'info',
    modelCacheDir: '../models/llama'
});

// List cached models to find your compiled model
const cachedModels = await manager.catalog.getCachedModels();
console.log('Cached models:');
for (const m of cachedModels) {
    console.log(`  ${m.id}`);
}

// Select your compiled model from the cached list
const model = cachedModels.find(m => m.id.includes('llama-3.2:1'));
if (!model) {
    throw new Error('Compiled model not found. Verify the modelCacheDir path.');
}

// Load the model
await model.load();

// Create a chat client
const chatClient = model.createChatClient();

// Generate a response
const completion = await chatClient.completeChat([
    { role: 'user', content: 'What is the golden ratio?' }
]);

console.log(completion.choices[0]?.message?.content);

// Unload the model
await model.unload();

Run the application:

node app.js

Use the Foundry Local Python SDK to load and run your compiled model with the native chat completions API.

Prerequisites

• Python 3.11 or later installed.

Install packages

If you're developing or shipping on Windows, select the Windows tab. The Windows package integrates with the Windows ML runtime — it provides the same API surface area with a wider breadth of hardware acceleration.

pip install foundry-local-sdk-winml openai

pip install foundry-local-sdk openai

Run inference on the compiled model

Copy and paste the following code into a Python file named app.py:

import asyncio
from foundry_local_sdk import Configuration, FoundryLocalManager


async def main():
    # Point model_cache_dir at the directory containing your compiled model
    config = Configuration(
        app_name="run-compiled-model",
        model_cache_dir="../models/llama",
    )
    FoundryLocalManager.initialize(config)
    manager = FoundryLocalManager.instance

    # List cached models to find your compiled model
    cached_models = manager.catalog.get_cached_models()
    print("Cached models:")
    for m in cached_models:
        print(f"  {m.id}")

    # Select your compiled model from the cached list
    model = next((m for m in cached_models if "llama-3.2:1" in m.id), None)
    if model is None:
        raise Exception("Compiled model not found. Verify the model_cache_dir path.")

    # Load the model
    model.load()

    # Get a chat client
    client = model.get_chat_client()

    # Stream the response
    messages = [{"role": "user", "content": "What is the golden ratio?"}]
    for chunk in client.complete_streaming_chat(messages):
        content = chunk.choices[0].message.content
        if content:
            print(content, end="", flush=True)
    print()

    # Tidy up - unload the model
    model.unload()


if __name__ == "__main__":
    asyncio.run(main())

Run the application:

python app.py

Use the Foundry Local Rust SDK to load and run your compiled model with the native chat completions API.

Prerequisites

• Rust and Cargo installed (Rust 1.70.0 or later).

Install packages

If you're developing or shipping on Windows, select the Windows tab. The Windows package integrates with the Windows ML runtime — it provides the same API surface area with a wider breadth of hardware acceleration.

cargo add foundry-local-sdk --features winml
cargo add tokio --features full
cargo add tokio-stream anyhow

cargo add foundry-local-sdk
cargo add tokio --features full
cargo add tokio-stream anyhow

Run inference on the compiled model

Replace the contents of src/main.rs with the following code:

use foundry_local_sdk::{
    ChatCompletionRequestMessage, ChatCompletionRequestUserMessage,
    FoundryLocalConfig, FoundryLocalManager,
};
use std::io::Write;
use tokio_stream::StreamExt;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Point model_cache_dir at the directory containing your compiled model
    let config = FoundryLocalConfig::new("run-compiled-model")
        .with_model_cache_dir("../models/llama");
    let manager = FoundryLocalManager::create(config)?;

    // List cached models to find your compiled model
    let cached_models = manager.catalog().get_cached_models().await?;
    println!("Cached models:");
    for m in &cached_models {
        println!("  {}", m.id());
    }

    // Select your compiled model from the cached list
    let model = cached_models
        .iter()
        .find(|m| m.id().contains("llama-3.2:1"))
        .ok_or_else(|| anyhow::anyhow!("Compiled model not found. Verify the model_cache_dir path."))?;

    // Load the model
    model.load().await?;

    // Create a chat client
    let client = model.create_chat_client().temperature(0.7).max_tokens(256);

    // Stream the response
    let messages: Vec<ChatCompletionRequestMessage> = vec![
        ChatCompletionRequestUserMessage::new("What is the golden ratio?").into(),
    ];

    let mut stream = client.complete_streaming_chat(&messages, None).await?;
    while let Some(chunk) = stream.next().await {
        let chunk = chunk?;
        if let Some(content) = &chunk.choices[0].message.content {
            print!("{}", content);
            std::io::stdout().flush()?;
        }
    }
    println!();

    // Tidy up - unload the model
    model.unload().await?;

    Ok(())
}

Run the application:

cargo run