Magnitude targets faster local inference
Magnitude is an open-source inference engine for agents, built around compiling and tuning kernels for the user's device. Its launch drew attention on Hacker News, and the repository is licensed under Apache 2.0.
Hardware-specific tuning, not generic precompiled kernels
Magnitude says device-specific tuning can make open models run up to twice as fast as llama.cpp, citing 92% faster decoding on Metal and 19% on CUDA. These are project-reported comparisons; the supplied material does not give test configurations, models, or reproduction methods, so they should not be treated as guarantees across devices.
Supported setups and practical limits
Magnitude offers desktop apps for macOS, Windows, and Linux, with support for Apple Silicon, NVIDIA and AMD GPUs, as well as CPU-only machines. Users can download a recommended model in the app and connect agents such as Pi, OpenCode, Hermes, and Codex; other tools can use its OpenAI-compatible API.
The project also says prompts, files, and models stay on the device, and that no internet connection is needed after a model is downloaded. It highlights memory management and concurrent sessions as design priorities. Still, available hardware limits model size, and performance depends on the particular device and model. Developers considering adoption should check whether the benchmark gains hold for their workload and whether the supported model lineup meets their needs.