WinSTTCompute & Quantization
How WinSTT picks an ONNX Runtime execution provider and what the fp16 / int8 / q4 quantization labels mean for size, speed, and accuracy.
Two knobs decide how fast a model runs: the execution provider (which hardware ONNX Runtime targets) and the quantization (how compressed the model's weights are). The provider is fixed by the platform/build; the quantization is per-model in the picker.
Execution providers
WinSTT runs every model on ONNX Runtime. Which hardware backend — "execution provider" (EP) — it uses depends on the platform and build features. Every official package has a CPU fallback; Windows builds also include DirectML. Other EPs such as CoreML, CUDA, ROCm, and OpenVINO are custom/development build features until release packaging for those targets is finalized.
| Provider | Hardware | When to use |
|---|---|---|
| CPU | All supported desktop platforms | Universal fallback and the default path when no validated accelerator is bundled. |
| DirectML | Windows with any DirectX 12 GPU — AMD, Intel, or NVIDIA | Vendor-agnostic Windows GPU path; falls back to CPU when the GPU path is not viable. |
| CoreML | macOS builds with the CoreML feature | Apple GPU/Neural Engine path for custom or future signed macOS builds. |
| CUDA / ROCm / OpenVINO | Custom builds for matching hardware | Development or native-runner paths; not separate public alpha packages today. |
The Model tab's Device control offers Auto and CPU for official packages:
Auto uses the best bundled provider for the platform when one is viable, and CPU
forces the CPU path regardless.
Why no CUDA build on Windows?
On our workload DirectML is both faster and ~10× lighter than CUDA. On an RTX 3080 Ti running whisper-tiny-q4, DirectML measures p50 = 85 ms (p95 = 89 ms, stdev = 3 ms) versus CUDA's p50 = 120 ms (p95 = 151 ms, stdev = 37 ms) — DirectML wins on both the median and consistency, while the package is far smaller. CUDA remains a custom/development path for Linux NVIDIA machines.
Auto-fallback to CPU
GPU-capable builds degrade gracefully. At startup the backend probes the active provider, and if the requested GPU path isn't viable it silently falls back to CPU. You get a working app either way. The per-OS auto priority skips providers that were not bundled for that build.
When the GPU path is skipped
The backend falls back to CPU when there's no DirectX 12 GPU for DirectML, a custom EP is missing its driver/runtime libraries, or the model graph is unsupported on the requested provider.
Force CPU manually
Set the Model tab's Device control to CPU to pin the CPU path regardless of the
active build — useful when a GPU driver is flaky. OpenVINO's target can be steered with
the OPENVINO_DEVICE env var (AUTO / GPU / GPU.0 / CPU; default AUTO).
Quantization
Quantization compresses a model's weights to a lower numerical precision. Lower precision means a smaller download, less RAM/VRAM, and faster inference — at the cost of small, usually-imperceptible accuracy differences. WinSTT exposes it as the model.onnxQuantization setting, set per-model in the picker.
An empty value (Auto) uses the model's default precision. The picker only shows the quantizations the upstream repo actually publishes for a given model — you can't pick a variant that doesn't exist, and the server falls back to the default if a chosen file is missing.
| Quantization | Precision | Relative size | Relative speed | Best for |
|---|---|---|---|---|
| Auto | Model default (fp32) | 100% | 1.0× | Maximum accuracy; pick when unsure |
| fp16 | 16-bit float | ~50% | fast on GPU | DirectML / GPU — the fastest GPU path |
| int8 | 8-bit integer | ~25% | fast on CPU | CPU — best balance (NeMo, GigaAM, Vosk) |
| uint8 | 8-bit unsigned int | ~25% | fast on CPU | CPU alternative to int8 where published |
| q4 | 4-bit | ~13% | small + light | Smallest download, lowest RAM |
| q4f16 | Mixed 4-/16-bit | ~20% | small + light | Small with better quality than plain q4 |
| bnb4 | bitsandbytes 4-bit | ~13% | small + light | 4-bit alternative where q4 isn't published |
These seven values match the labels in the Model picker exactly. Availability is per-family: most Whisper and Lite-Whisper models publish fp16, q4, q4f16, and bnb4; NeMo, GigaAM, and Vosk/Kaldi models publish int8. A few models (e.g. large-v3, t-tech/t-one) ship the default precision only.
Picking a quantization
On a GPU build
Prefer fp16 when the model publishes it — it roughly halves memory with negligible quality loss and the GPU accelerates it directly. Fall back to Auto if fp16 isn't offered.
On the CPU build
Use int8 for NeMo / GigaAM / Vosk — best speed for minimal quality cost. For Whisper on a RAM-tight machine, q4 or q4f16 shrink the footprint the most.
When accuracy matters most
Leave it on Auto (full fp32). This is the safe default and the reference against which every other variant trades.
When disk or RAM is scarce
q4 / bnb4 are ~13% of the fp32 size. Great for laptops and small models you want resident with a long unload timeout.
int8 on a GPU isn't a free win
4-bit and 8-bit weight quantization is built for shrinking the download and easing CPU/RAM pressure — not for squeezing more speed out of a GPU. On a GPU the dequantization overhead can cancel the savings, and some int8/q4 ONNX exports are broken or unsupported on the GPU EP, so the session quietly falls back to CPU or errors. The reliable GPU lever is fp16; reach for int8/q4 when you're constrained on disk or memory, not chasing GPU throughput.
Per-quantization downloads
Each quantization is a separate download. In the picker you can download, pause, resume, cancel, or delete a single variant per model — so you can keep, say, fp16 for the GPU and delete the fp32 default to reclaim disk, without touching the other models.
Related
Model settings
Pick the model, quantization, language, and device in the app.
Browse the catalog
All 70+ models across families, with accuracy, speed, and language coverage.
Install
Pick the macOS, Linux, or Windows package for your machine.
Whisper models
The Whisper family and the quantizations each variant publishes.
Other Families
Moonshine, Cohere, Granite, Qwen3-ASR, Audio8-ASR, SenseVoice, Dolphin, GigaAM, Vosk/Kaldi, and T-One — the low-latency, multilingual, and Russian-focused models beyond Whisper and NeMo.
Command-Line and Headless Transcription
Launch flags, headless file transcription and benchmarking, and developer-only STT/TTS harnesses.