Compatible endpoints
OpenAI-compatible /v1/* and native Ollama /api/* endpoints let existing clients reach local engines through one proxy.
Project 01 / Inference orchestration
One endpoint for your local models.
A Python reverse proxy with OpenAI- and Ollama-compatible APIs. It routes requests to local inference engines by model and swaps backends as GPU memory requires.
One active backend at a time
The core idea
A client sends a model name to one endpoint. The router resolves its engine, manages the memory transition, and forwards the request.
Compatible APIs. One active backend.
The problem
Local inference often means several backends, each with its own port and lifecycle. On memory-constrained hardware, keeping multiple large engines loaded can exhaust the available GPU or unified memory.
I built local-engine-router to manage that boundary: clients keep a stable API, while the router selects the backend and handles the transition.
What I built
OpenAI-compatible /v1/* and native Ollama /api/* endpoints let existing clients reach local engines through one proxy.
Smart routing scores local models for the task, context fit, observed reliability, and the cost of switching away from an already loaded engine.
The router drains requests, frees the outgoing engine, waits for memory reclaim, and starts the target before forwarding the request.
Engineering decisions
Stopping an engine does not immediately make all its memory available. An explicit memory-settle step helps the next backend start after the operating system has reclaimed it.
During a cold start, the proxy emits keep-alive frames for streaming requests. Non-streaming requests wait for the swap to complete.
I implemented a GPU-free automated test suite with mock backends. The same suite runs in continuous integration, without needing a live inference server.
Try it locally
Install the router, detect your running engines, and start the proxy. The init wizard scaffolds a configuration from confirmed local backends.
For process-based engines, configure the start command so the router can restart them after a swap.
Read the full setup guidepip install local-engine-router
local-engine-router init
local-engine-router --config config.yaml