apple on device openai

OpenAI-compatible API server for Apple on-device models

541

Swift

Apple On-Device OpenAI API

A SwiftUI application that creates an OpenAI-compatible API server using Apple’s on-device Foundation Models. This allows you to use Apple Intelligence models locally through familiar OpenAI API endpoints.

Screenshot

Use it in any OpenAI compatible app:

Features

OpenAI Compatible API: Drop-in replacement for OpenAI API with chat completions endpoint
Streaming Support: Real-time streaming responses compatible with OpenAI’s streaming format
On-Device Processing: Uses Apple’s Foundation Models for completely local AI processing
Model Availability Check: Automatically checks Apple Intelligence availability on startup
🚧 Tool Using (WIP): Function calling capabilities for extended AI functionality

Requirements

macOS: 26 beta 2
Apple Intelligence: Must be enabled in Settings > Apple Intelligence & Siri
Xcode: 26 beta 2 (for building)

Installation

Option 1: Download Pre-built App (Recommended)

Go to the Releases page
Download the latest .zip file
Extract and launch the app

Option 2: Build from Source

Clone the repository:

git clone https://github.com/gety-ai/apple-on-device-openai.git
cd apple-on-device-openai

Open the project in Xcode:

open AppleOnDeviceOpenAI.xcodeproj

Build and run the project in Xcode

Why a GUI App Instead of Command Line?

This project is implemented as a GUI application rather than a command-line tool due to Apple’s rate limiting policies for Foundation Models:

“An app that has UI and runs in the foreground doesn’t have a rate limit when using the models; a macOS command line tool, which doesn’t have UI, does.”

— Apple DTS Engineer (Source)

⚠️ Important Note: You may still encounter rate limits due to current limitations in Apple FoundationModels. If you experience rate limiting, please restart the server.

⚠️ 重要提醒: 由于苹果 FoundationModels 当前的限制，您仍然可能遇到速率限制。如果遇到这种情况，请重启服务器。

Usage

Starting the Server

Launch the app
Configure server settings (default: 127.0.0.1:11535)
Click “Start Server”
Server will be available at the configured address

Available Endpoints

Once the server is running, you can access these OpenAI-compatible endpoints:

GET /health - Health check
GET /status - Model availability and status
GET /v1/models - List available models
POST /v1/chat/completions - Chat completions (streaming and non-streaming)

Example Usage

Using curl:

curl -X POST http://127.0.0.1:11535/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "apple-on-device",
    "messages": [
      {"role": "user", "content": "Hello, how are you?"}
    ],
    "temperature": 0.7,
    "stream": false
  }'

Using OpenAI Python client:

from openai import OpenAI

# Point to your local server
client = OpenAI(
    base_url="http://127.0.0.1:11535/v1",
    api_key="not-needed"  # API key not required for local server
)

response = client.chat.completions.create(
    model="apple-on-device",
    messages=[
        {"role": "user", "content": "Hello, how are you?"}
    ],
    temperature=0.7,
    stream=True  # Enable streaming
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Testing

You can use the included test script to verify the server is working correctly and see example usage patterns:

python3 test_server.py

The test script will:

✅ Check server health and connectivity
✅ Verify model availability and status
✅ Test OpenAI SDK compatibility
✅ Run multi-turn conversations
✅ Test multilingual support (Chinese)
✅ Demonstrate streaming functionality

Make sure the server is running before executing the test script. The script provides comprehensive examples of how to interact with the API using both direct HTTP requests and the OpenAI Python SDK.

API Compatibility

This server implements the OpenAI Chat Completions API with the following supported parameters:

model - Model identifier (use “apple-on-device”)
messages - Array of conversation messages
temperature - Sampling temperature (0.0 to 2.0)
max_tokens - Maximum tokens in response
stream - Enable streaming responses

Development Notes

🤖 This project was mainly “vibe coded” using Cursor + Claude Sonnet 4 & ChatGPT o3.

License

This project is licensed under the MIT License - see the LICENSE file for details.