TinyCoder — a from-scratch inference engine for GGUF and EXL3
Dequantisation, the int8 MMA GEMM, attention and the gated-delta-net kernels are all implemented in the tree. Measured at 723.0 tok/s prefill on Qwen3.8-27B EXL3, on an 11 GB Turing card.
An inference engine, not a wrapper around one
TinyCoder is a from-scratch C++20/CUDA inference library for NVIDIA GPUs. The kernels are the product — there is no llama.cpp, ggml, Candle, PyTorch or ExLlamaV3 underneath.
Every kernel in the tree
GGUF and EXL3 dequantisation, the int8 mma.s8.m8n8k16 stream-K GEMM, warp flash attention, the gated-delta-net recurrent kernels and the samplers are implemented in this repository. No inference library is linked, embedded or called.
cuBLAS is only a GEMM
cuBLAS appears as a plain GEMM and nothing else. Prefill has two paths: cuBLAS fp16 tensor cores, and the in-tree int8 stream-K MMA that quantises activations to Q8_0 and reads GGUF blocks straight from VRAM.
Two first-class formats
GGUF v3 — K-quants Q2_K to Q6_K, Q5_0/Q5_1/Q8_0, the IQ series (IQ2_XXS/S/XS, IQ3_XXS/S, IQ4_NL/XS) and Prism pq2_0 ternary. EXL3 — exllamav3's trellis format, read directly from a Hugging Face checkpoint directory. Model::load sniffs the format; both share the same kernels, KV cache and offload path.
Model families
Qwen2.5, Qwen3.6 and Qwen3.8 — dense qwen35 and mixture-of-experts qwen35moe — plus Ornith, Gemma 4 and Prism Ternary-Bonsai.
Turing and newer
Turing, Ampere and Ada run native SASS; newer parts run through forward-compatible PTX JIT. sm_75 is the floor. The reference box is an RTX 2080 Ti with 11 GB — the MoE path streams experts through a layer arena, so the 34.4 GB Ornith-1.5-35B-A3B checkpoint runs on it.
A CPU path that actually ships
Runtime SIMD dispatch picks AMX, AVX-512, AVX2 or a scalar fallback, with OpenMP across cores. Without CUDA it still runs — several times to orders of magnitude slower, and a model that does not fit in RAM will not run at all.
A second first-class format — and a measured win on it
EXL3 is exllamav3's trellis-quantised checkpoint format. TinyCoder loads it natively, alongside GGUF, and is faster than the reference implementation on it.
What EXL3 is
EXL3 stores weights as a trellis code over 16×16 (k,n) tiles in a Hugging Face safetensors directory — config.json, *.safetensors, tokenizer.json — rather than in a GGUF file. At 1.60bpw, a 27B checkpoint is 8.6 GB.
Why native support matters
Before this, an EXL3 checkpoint meant running exllamav3: PyTorch, a JIT-compiled CUDA extension, Python per token. Now the same engine runs both formats — one loader, one KV cache, one offload path, and no conversion pass.
Model::loaddetects the format and selectsEXL3Loaderautomatically.- Trellis reconstruction, the EXL3 GEMV and the fused prefill GEMM live in
src/cpp/core/cuda/exl3/. - The autotuned cublasLt path is the default;
TINYCODER_EXL3_LT=0pins the older fixed-order k summation bit-for-bit. - Checkpoint shape and tokenizer are read from the directory — no GGUF conversion step.
against exllamav3 1.6.6 at 707.5 tok/s pp512 and 22.9 tok/s tg128 — same checkpoint, Qwen3.8-27B EXL3 1.60bpw, 8.6 GB.
Ahead in both columns, but the decode margin is narrow: exllamav3's tg128 includes Python per-token overhead, so its figure is a floor rather than a best case. Both columns were measured with the same pp512/tg128 protocol.
Run it as a local OpenAI-compatible provider
The engine tree ships a library, not a server: tinycoder_core, its headers, tests and benchmarks. The OpenAI-compatible endpoint is the TinyCoder OpenAI gateway — a Go HTTP server that links the engine — so existing OpenAI-style tooling has something to point at.
Embed the library
Pull tinycoder_core into your own CMake project, then load and generate through the C++ API.
FetchContent_Declare(tinycoder_inference
GIT_REPOSITORY https://github.com/mgorshkov/tinycoder-inference
GIT_TAG <tag or commit>)
FetchContent_MakeAvailable(tinycoder_inference)
target_link_libraries(your_target PRIVATE tinycoder_core)
#include "Model.hpp"
using namespace tinycoder;
Model model;
std::string err;
if (!model.load("/path/to/model.gguf", &err)) {
// handle error
}
InferenceParams params;
params.maxTokens = 512;
params.temperature = 0.7f;
params.topP = 0.9f;
model.generate("Write a C++ function to add two numbers.",
params, [](int32_t tokenId, const std::string &text) {
std::cout << text << std::flush;
return true;
});
Serve the API
Build the gateway with its native engine, then point it at a models directory.
./scripts/build.sh
./build/bin/tinycoder-openai \
--listen :8080 \
--models-dir /data/models
curl -s http://localhost:8080/v1/models
curl -s http://localhost:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"qwen3.6-35b-a3b",
"messages":[{"role":"user",
"content":"Write a C++ hello world."}]}'
Model ids come from the gateway's config registry — a GGUF file or an EXL3 checkpoint directory per entry — or from a scan of --models-dir.
Endpoints the gateway actually serves
| Method | Path | What it does |
|---|---|---|
| GET | /v1/models | List registry models, with tinycoder metadata |
| POST | /v1/chat/completions | Chat completion; SSE streaming; tool calling |
| POST | /v1/completions | Text completion; SSE streaming |
| POST | /v1/responses | OpenAI Responses API |
| POST | /v1/messages | Anthropic Messages API |
| POST | /v1/generate | Native endpoint; accepts prompt or messages |
| GET | /openai.json, /openai.yaml | OpenAPI 3.1 document |
| GET | /healthz | Liveness probe |
What it supports
- SSE streaming on the completion endpoints, a final
data: [DONE], and keep-alive comments while a model loads or a long prompt prefills. - Tool calling on
/v1/chat/completions: OpenAItools,tool_choiceandtool_calls, including streamed deltas. - Chat templates rendered inside the engine from the model's own template — send
messages, not a pre-formatted prompt. - One model registry for both formats: each entry points at a GGUF file or an EXL3 checkpoint directory, and the engine auto-detects which.
- Lazy loading with LRU idle eviction, optional bearer-token auth, CORS, request timeouts and a global concurrency cap.
What it is not
- Not full OpenAI parity. Those paths are the whole surface, and there are no unversioned or legacy aliases.
- The engine owns a single KV cache, so requests for the same model are serialized and queued rather than served concurrently.
- Tool calling is elicited through a text protocol, not constrained decoding: best effort, and only on
/v1/chat/completions. /v1/responsesand/v1/messagesaccept and ignore fields they do not model (store,previous_response_id,tools,metadata);max_tokensis required on/v1/messages.
Where TinyCoder is faster
A selection, not a survey: these are the four configurations where TinyCoder beats the baseline in both pp512 and tg128. The complete table — including the rows where llama.cpp is faster — is linked below.
| Model | pp512 (tok/s) | tg128 (tok/s) | ||
|---|---|---|---|---|
| TinyCoder | Baseline | TinyCoder | Baseline | |
Qwen2.5-Coder-1.5B q2_k
0.7 GB · full offload · vs llama.cpp
|
10,149.8 | 7,769.8 | 327.91 | 323.19 |
Qwen2.5-Coder-1.5B iq3_xxs
0.9 GB · full offload · vs llama.cpp
|
10,192.3 | 10,143.5 | 276.77 | 261.47 |
Ornith-1.5-35B-A3B q8_0 → q4_k experts
34.4 GB · hybrid, layer arena · vs llama.cpp -ncmoe 99
|
176.3 | 130.7 | 53.67 | 16.42 |
Qwen3.8-27B EXL3 1.60bpw
8.6 GB · full offload · vs exllamav3 1.6.6
|
723.0 | 707.5 | 23.26 | 22.9 |
TinyCoder: tinycoder_bench --n-prompts 512 --n-gen 128 --threads 36. Baselines: llama-bench -p 512 -n 128 -t 36 -r 1, interleaved with the TinyCoder run — three alternating pairs, means reported — and exllamav3 1.6.6 measured with the same pp512/tg128 protocol. The 35B row adds -ncmoe 99. A TinyCoder figure is printed only if that checkpoint passes the text gate (./scripts/text_gate.sh).
The complete table — every configuration, including the rows where llama.cpp is faster and the one with no comparable baseline — lives in BENCHMARKS.md in the engine repository, under the same protocol.
Formats the other engines can't run
Prism's Ternary-Bonsai-2-27B pq2_0 checkpoint — a 6.7 GB ternary model — runs fully offloaded on the same 11 GB card.
There is no llama.cpp baseline for it, and the full table marks it with a dash: llama.cpp has no loader for the pq2_0 ternary format, so there is no comparable run to publish. We are not going to invent one.
How we measure
Interleaved, not sequential
Each TinyCoder run alternates with its baseline run — three pairs — so clock drift lands on both columns equally. Means are reported.
Text before speed
A throughput number is only printed if the checkpoint passes the text gate. A skipped test counts as a failure, so a row cannot pass by omission.
One process at a time
Benchmarks run serially on the reference RTX 2080 Ti. Two processes on that GPU contend for bandwidth and describe neither run.
Implausible is a bug
Before publication, implied TFLOPS and GB/s are checked against the card's peaks. A number above the hardware is skipped work, not a win.
The VS Code extension is one consumer
The TinyCoder extension embeds tinycoder_core through its own N-API bridge — the engine's FetchContent path exists for exactly this. It adds a chat panel and editor commands on top, and it is how many people meet the engine day to day.
It is not the pitch. The library is the product; the extension is a downstream integration that proves the embedding path works.
TinyCoder extensionRun it on the GPU you already own
MIT-licensed C++20/CUDA. Clone the engine, build with CUDA, point TINYCODER_MODEL_PATH at a GGUF file or an EXL3 checkpoint directory.