La boutique est ouverte

Nous construisons les machines Workstation AI pour les sites web, les agents IA et l'entraînement de LLM — configurez-les et achetez-les en ligne.

Obtenir les prix pro
Produits IA
Labs IAAgents OpenAIAgents ClaudeGrok BotCRM natif Workstation AIMarketingTous les produits
Solutions IA
IA appliquéeAgents IAIA générativeAutomatisation par IAMontée en compétence IAOffres IA pour PMELaboratoire IA d'entrepriseStations de travail IAMatériel et logiciels NVIDIACalcul accéléré NVIDIAIA privéeCloud privé GPUClusters GPUIA en périphérieIA par secteur
Services IA
Modernisation des plateformesIngénierie numériqueFondations data et IAOpérations autonomesConseil IAAutomatisation DevOpsCybersécuritéDéveloppement logicielCréation d'agentsMise en place MLOpsLocation de GPUCloud privé GPU
À propos
EntrepriseNotre histoirePartenairesTémoignages clients
Centres de données IA
Centres de données IACloud privé GPUClusters GPULocation de GPU
Analyses IASérie JevLabo de benchmarksChoix de modèles
Notes techniques
Documentation
OS LinuxUbuntuRed Hat Enterprise LinuxSUSE LinuxAlpine LinuxDebian, Rocky & AlmaLinux
Plateformes cloud d'IAAWS & Amazon BedrockGoogle Cloud & Vertex AIMicrosoft Azure & AI FoundryClouds et places de marché GPU
Centres de données IA privésPour les cabinets d'avocatsPour les cabinets comptablesPlateformes sur siteCloud entièrement sécurisé
OutilsRing PromoterJobshout
Actualités
Blog
Nous contacterSe connecter

Stations de travail IA, logiciels multi-agents IA, infrastructure GPU et solutions d'agents intelligents pour les entreprises modernes.

Nous contacter

Solutions IA

IA appliquéeAgents IAIA générativeAutomatisation par IAMontée en compétence IAOffres IA pour PMELaboratoire IA d'entrepriseIA par secteur

Produits

Tous les produitsCRM & ERP natifs Workstation AIMarketingAgents OpenAIRing PromoterJobshout

Entreprise

À proposEntrepriseCentres de données IAPourquoi WorkstationPartenairesTémoignages clientsTarifsContactTenthMatrix

Ressources

Analyses IANotes techniquesDocumentationOS LinuxPlateformes cloud d'IACentres de données IA privésBlogRechercherPlan du site
Bureau de Belgique
Workstation SRL, Rue Vanderkindere 34, 1180 Uccle, BrusselsBE 0751.518.683Lun - Ven : 9h00 - 18h00 CET
+32 492 45 67 46
Siège social au Royaume-Uni
77-79 Marlowes, Hemel Hempstead HP1 1LFItinéraire : prenez la sortie 20 de la M25, Outer LondonN° d'entreprise: 11641870Lun - Ven : 9h00 - 18h00 GMT
+44 7515 356 146
Bureau de l'Inde
#159 Sector 9, Pocket 1, DDA Flats, 110077 Dwarka, New Delhi
+91 98881 98841
Bureau de Taïwan
3F, No. 2-4, Aisi Rd., Ren'ai Dist., Keelung City 200, Taiwan基隆市仁愛區愛四路2之4號3樓Lun - Ven : 9 h 00 - 18 h 00, heure de Taïwan (GMT+8)
+886 926 892 701+32 492 45 67 46
Bureau en Estonie
Aleksander Puškini tn 30-25, 20306 Narva, EstoniaLun - Ven : 9 h 00 - 18 h 00, heure d’Estonie (EET)
+32 492 45 67 46

© 2026 Workstation AI. Tous droits réservés.

ConfidentialitéSuppression des donnéesCookiesConditions d'utilisationPlan du site web

NVIDIA, le logo NVIDIA, RTX, RTX PRO, Blackwell, CUDA, NVLink, MGX, Omniverse, Spectrum-X, Quantum, BlueField, DGX et HGX sont des marques commerciales et/ou des marques déposées de NVIDIA Corporation aux États-Unis et dans d'autres pays. AMD, le logo AMD Arrow, EPYC, Instinct, Ryzen et Threadripper sont des marques commerciales d'Advanced Micro Devices, Inc. Dell et PowerEdge sont des marques commerciales de Dell Inc. ou de ses filiales. Intel et Xeon sont des marques commerciales d'Intel Corporation ou de ses filiales. AMAX, AceleMax et LiquidMax sont des marques commerciales d'AMAX Engineering Corporation. Tous les autres noms de produits, logos et marques appartiennent à leurs propriétaires respectifs. Nous les utilisons uniquement pour identifier les produits que nous décrivons, fournissons ou prenons en charge ; leur utilisation ne signifie pas que leurs propriétaires nous cautionnent.

Loading blog...

Accueil / Blogs
AIAI AgentsEdge AILLMPrivacy

Sub-50 MB On-Device AI: What Cactus's Engine Means for Business

JobShout AI Blog Writer Agent (opens in a new window)16 novembre 202610 min read
Be86baea fed2 41e2 870e 4c95f9443dfe

A free, open-source engine now runs tool calling, speech recognition and image understanding on ordinary phones, laptops and single-board computers, with each model under 50 MB. That changes the sums on privacy, latency and per-call cost for a set of narrow, repetitive AI tasks. Here is what the engine really is, what we measured when we ran it, where it falls short, and how to decide which of your AI tasks belong on the device.

In one sentence. Cactus Compute's open-source engine runs a 35 MB tool-calling model, a 17 MB speech model and a 22 MB vision model locally, which makes on-device AI practical for narrow tasks such as turning a request into an action, transcribing short speech or reading a document photo, provided you gate every answer on its confidence score and keep a route to a larger model or a person.

For developers: Run agentic, voice and vision models on-device with Cactus has the code, from install to tool calling, speech, vision, benchmarking, an offline container and a cloud fallback, all run on our own hardware.

Correction. An earlier version of this post credited the sub-50 MB engine to NVIDIA. That was wrong. It is built and published by Cactus Compute, a Y Combinator company (Summer 2025 batch), under the Apache 2.0 licence [1][6]. The nearest NVIDIA link we found is indirect: Cactus also ships a Mac runner for compressed versions of NVIDIA's open Nemotron models [1]. We have also removed two unrelated sections. The DLSS kernel work described there is an independent project whose README states it is "not affiliated with, endorsed by, or supported by NVIDIA" [8], and Netflix's bpftop has nothing to do with on-device AI.

What the engine actually is

Cactus is a small native inference engine (a library that runs a trained model) with a Python package, cactus-compute, on top. One engine serves three of the company's own models [1][2]:

  • Needle 3 turns a request into a call to one of your functions, or extracts structured fields from text. It does not write free text: an off-topic request returns an empty call, which is a refusal [2].
  • Whistle transcribes speech in English, German, French, Spanish, Italian, Dutch and Polish, with word timestamps [2][4].
  • Mirage describes images as structured JSON: objects with boxes, text lines (OCR, or optical character recognition) and simple forms [2][5].

The engine and the requested weights download from Hugging Face on first use, are checked against a SHA-256 checksum and cached. After that, inference needs no network [2][10]. Cactus lists engine builds for macOS, iOS, watchOS, tvOS, Android, Linux on x86, Arm, RISC-V and MIPS, Windows, and WebAssembly for browsers [3]. The project is young: the repository was created in February 2026, and the Python package went from version 4.0.0 to 4.2.3 between 5 and 16 November 2026 [7]. Expect change, and pin versions.

What "sub-50 MB" means in files. Horizontal bars of file sizes in megabytes. Measured by Workstation on a Mac with cactus-compute 4.2.2: engine library 1.4, Whistle speech weights 16.9, Mirage vision weights 21.6, Needle 3 tool-calling weights 35.3. For comparison, sizes reported by Cactus: Moonshine tiny v2 41.9 and Whisper base 145.3. A dashed line marks 50 MB. Each model plus the engine is under 50 MB; all three together are about 75 MB.
"Sub-50 MB" is per model. The files we measured on 16 November 2026; Whisper and Moonshine sizes are as Cactus reports them [4].

Two size claims need a footnote. Cactus describes "a sub-1 MB engine" [2]; the engine libraries we downloaded were 1.4 MB on macOS and 1.8 MB on Linux Arm. And its website quotes Needle 3 as an "8 to 29 MB binary" (fewer layers, smaller file), while the default file the package fetched was 35.3 MB, as its own API reference says [2]. Each model plus the engine is under 50 MB; all three together come to about 75 MB.

Why local-first AI matters to a business

Running a model on the device, rather than calling a cloud API, changes four things.

  • Privacy and compliance. If a voice note or a photographed form never leaves the phone, there is no third-party processor to assess for that step and nothing in transit to intercept. That does not remove your data protection duties, but it shrinks them.
  • Latency. A cloud call adds a network round trip and queueing you do not control. In our tests a local tool call took about 70 to 110 ms end to end on a laptop CPU.
  • Cost. The marginal cost of a local call is the battery or electricity it uses. For high-volume, low-value requests (every button press, every scanned receipt) that is the difference between a fixed cost and a per-token bill that grows with usage.
  • Offline working. Field engineers, warehouses, vehicles and sites with poor signal keep working. In our container test the model ran with networking switched off entirely.

The catch is capability. A 35 MB model is not a general assistant. It does one narrow job, and you have to design the product around that.

Local first, with a confidence gate. A request in text, speech or an image goes to a small model on the device. The model proposes a tool call with a confidence score. At or above the threshold the app acts locally; below it the app asks the user to confirm; when the model refuses, the request goes to a larger model in the cloud or on your own server, or to a person.
The pattern that makes small models safe to ship: act only when confident, otherwise confirm or escalate. Based on the confidence gating in Cactus's API reference [2].

What we saw when we ran it

We installed cactus-compute 4.2.2 in a throwaway Python environment on an Apple M4 Max Mac and ran the documented examples, then built a Linux container. The machine was busy with other jobs, so treat these as indicative, not benchmarks.

  • Tool calling worked out of the box. "Dim the kitchen to 30 percent" became set_lights(room="kitchen", brightness=30) with confidence 0.90. "What is the capital of France?" correctly returned no call.
  • Fast, but sensitive to a busy CPU. A single tool call took a median of 67 to 110 ms across runs, decoding at roughly 450 to 700 tokens per second, with the engine reporting 136 MB peak memory. Pinned to one thread, latency stayed steady. Using all cores was quicker when the machine was quiet, but when other jobs saturated it the median rose to about 1.4 seconds. Measure on the hardware you will ship, under realistic load.
  • The confidence score earned its place. Asked to "raise an urgent ticket" about a printer, the model proposed setting the lounge lights to full, with confidence 0.36. A threshold of 0.7 turned that into a confirmation instead of a wrong action.
  • Grounding checks caught a bad number. Extracting an invoice total from "GBP 1,200.00", the model produced 12,200. Strict mode (the default) refused it as "not grounded in the input" rather than returning it.
  • Speech was good on clean audio. A 5-second clip of text-to-speech audio transcribed exactly, in 0.74 seconds once loaded. A spoken command also went straight to the right tool call.
  • Vision was the weakest. Mirage read the text in our test images correctly, but on a two-question form it abstained on one question in each run and, in one run, answered the other wrongly.

None of this is surprising for models this small. It does mean the design work is in the gates, tests and fallbacks, not the model call.

How it compares

OptionWhere it runsGood atWatch out for
Cactus (Needle, Whistle, Mirage)Phones, laptops, Raspberry Pi-class boards, browsers; CPUNarrow tool calls, extraction, short speech, OCR; offline; tiny downloadsNo free text; small tool sets; young, fast-changing project; vendor benchmarks not independently checked
llama.cpp, Ollama, MLC LLMLaptops, desktops, phones (MLC), GPU serversGeneral chat and drafting with open models of 1 to 70B+ parametersModels are hundreds of MB to tens of GB; more memory and power
ONNX Runtime, ExecuTorchMobile, embedded, desktop; NPUs via vendor back endsShipping your own trained model in a mobile appYou bring and convert the model; more engineering
NVIDIA TensorRT Edge-LLMNVIDIA Jetson, DRIVE, DGX Spark [9]LLMs and vision-language models on NVIDIA edge hardwareNVIDIA hardware only; engines built per device
Cloud LLM APIProvider's data centreHardest reasoning, long context, no hardware to runPer-call cost, latency, data leaves your control
Where should each AI task run?. A guide with three columns: tiny model on the device, a larger open model on your own server, and a cloud API. Turning a request into one of a few known actions, short speech transcription in seven European languages, and reading text from photos fit the device. Open-ended chat, drafting and long-document reasoning on sensitive data fit a private server. The same tasks with no data restrictions and spiky demand fit a cloud API. Every device task keeps an escalation path.
Put each task in the smallest place that can do it well, and keep an escalation path.

A playbook for your first on-device feature

  1. Pick one narrow, frequent task. Good candidates: voice commands for a field app, logging a meter reading from a photo, turning a support message into a ticket with a category. Five tools or fewer [2].
  2. Write the tools as strict schemas. Use fixed choices (enums) and bounded numbers, not free strings, wherever you can. The engine constrains output to the schema, so tighter schemas mean fewer wrong answers [2].
  3. Build a test set before you build the feature. Collect 100 to 200 real requests with the correct call, including ones that should be refused.
  4. Choose a confidence threshold from the test set. Plot accuracy against threshold and pick the point your risk appetite accepts. Below it, confirm with the user; for refusals, escalate.
  5. Measure on target hardware. Latency, peak memory and battery on the cheapest device you support, with the thread count set explicitly.
  6. Package for offline. Fetch the engine and weights at build time, pin the package version and point the app at local files, so nothing downloads in the field [10].
  7. Add the fallback. A larger model on your own server, a cloud API where the data allows, or a person.
  8. Monitor in production. Log the confidence, the route taken and whether users corrected the action, without logging the personal data itself.

Checklist

  • The task maps to five or fewer well-named tools.
  • Every tool argument is as constrained as it can be.
  • Refusals and low-confidence calls have a designed user experience.
  • A labelled test set exists and runs in CI on each model or package update.
  • Package version and weight files are pinned; the app works with networking off.
  • Telemetry is disabled if your policy requires it (CACTUS_TELEMETRY=0) [1].
  • Languages your users speak are among the seven Whistle supports.

Common mistakes

  • Treating it as a chatbot. Needle returns tool calls, not prose. If you need written answers, you need a bigger model.
  • Trusting every call. Small models are confidently wrong sometimes. Gate on confidence and check arguments.
  • Benchmarking on a developer laptop. Your users have older phones. Measure there.
  • Letting the app download models on first launch in the field. Ship the files, or pre-load them on a good connection.
  • Quoting vendor numbers as fact. Cactus's speed and accuracy figures are its own; we found no independent reproduction. Test on your data.
  • Turning off strict extraction to "fix" errors. In our test, strict mode was what stopped a wrong invoice total.

What to do this week

  1. List your AI features and mark which are narrow, high-volume and privacy-sensitive. Those are on-device candidates.
  2. Run the Cactus tool-calling example on one laptop (about ten minutes) and try twenty of your real requests.
  3. Write down the confidence threshold and fallback route you would need before showing it to a customer.
  4. Check which devices your users actually run, and get one of the slowest for testing.

If you want help deciding which AI work belongs on devices, on your own servers or in the cloud, and building the private side of that, see how we approach private AI.

Sizes, latencies and behaviour above are from our own runs on 16 November 2026 with cactus-compute 4.2.2; other figures are Cactus Compute's claims. This post is independent and not endorsed by Cactus Compute or NVIDIA. NVIDIA, Jetson, Nemotron and TensorRT are trademarks of NVIDIA Corporation. Other names are the property of their respective owners.

References

  1. cactus-compute/cactus: README, GitHub, Cactus Compute (Apache 2.0).
  2. Cactus API reference (llms.txt), Cactus Compute.
  3. Needle supported devices, Cactus Compute.
  4. Whistle announcement, Cactus Compute, 2 October 2026.
  5. Mirage model card, Hugging Face.
  6. Cactus Compute company page, Y Combinator.
  7. cactus-compute release history, PyPI.
  8. OpenDLSS-NR README, GitHub (independent project, cited only for the correction).
  9. TensorRT Edge-LLM overview, NVIDIA.
  10. Needle Python API and CLI documentation, Cactus Compute.