C++20 / CUDA · no inference library linked

TinyCoder — a from-scratch inference engine for GGUF and EXL3

Dequantisation, the int8 MMA GEMM, attention and the gated-delta-net kernels are all implemented in the tree. Measured at 723.0 tok/s prefill on Qwen3.8-27B EXL3, on an 11 GB Turing card.

no inference library linked
GGUF + EXL3, both first-class
34.4 GB MoE on an 11 GB card
sm_75 and newer
What it is

An inference engine, not a wrapper around one

TinyCoder is a from-scratch C++20/CUDA inference library for NVIDIA GPUs. The kernels are the product — there is no llama.cpp, ggml, Candle, PyTorch or ExLlamaV3 underneath.

Every kernel in the tree

GGUF and EXL3 dequantisation, the int8 mma.s8.m8n8k16 stream-K GEMM, warp flash attention, the gated-delta-net recurrent kernels and the samplers are implemented in this repository. No inference library is linked, embedded or called.

C++20CUDAfrom scratch

cuBLAS is only a GEMM

cuBLAS appears as a plain GEMM and nothing else. Prefill has two paths: cuBLAS fp16 tensor cores, and the in-tree int8 stream-K MMA that quantises activations to Q8_0 and reads GGUF blocks straight from VRAM.

cuBLASint8 MMAQ8_0

Two first-class formats

GGUF v3 — K-quants Q2_K to Q6_K, Q5_0/Q5_1/Q8_0, the IQ series (IQ2_XXS/S/XS, IQ3_XXS/S, IQ4_NL/XS) and Prism pq2_0 ternary. EXL3 — exllamav3's trellis format, read directly from a Hugging Face checkpoint directory. Model::load sniffs the format; both share the same kernels, KV cache and offload path.

GGUFEXL3IQ-seriesPQ2_0

Model families

Qwen2.5, Qwen3.6 and Qwen3.8 — dense qwen35 and mixture-of-experts qwen35moe — plus Ornith, Gemma 4 and Prism Ternary-Bonsai.

QwenOrnithGemma 4Prism

Turing and newer

Turing, Ampere and Ada run native SASS; newer parts run through forward-compatible PTX JIT. sm_75 is the floor. The reference box is an RTX 2080 Ti with 11 GB — the MoE path streams experts through a layer arena, so the 34.4 GB Ornith-1.5-35B-A3B checkpoint runs on it.

sm_75sm_86sm_89PTX JIT

A CPU path that actually ships

Runtime SIMD dispatch picks AMX, AVX-512, AVX2 or a scalar fallback, with OpenMP across cores. Without CUDA it still runs — several times to orders of magnitude slower, and a model that does not fit in RAM will not run at all.

AMXAVX-512AVX2OpenMP
LoadGGUF or EXL3 — format sniffed by Model::load
PrefillcuBLAS fp16 GEMM, or in-tree int8 stream-K MMA
Decodeon-the-fly dequant GEMV · int8 KV pool
Attentionwarp flash attention + gated-delta-net recurrence
Sampletop-k / top-p / temperature / repetition penalty
New · EXL3

A second first-class format — and a measured win on it

EXL3 is exllamav3's trellis-quantised checkpoint format. TinyCoder loads it natively, alongside GGUF, and is faster than the reference implementation on it.

What EXL3 is

EXL3 stores weights as a trellis code over 16×16 (k,n) tiles in a Hugging Face safetensors directory — config.json, *.safetensors, tokenizer.json — rather than in a GGUF file. At 1.60bpw, a 27B checkpoint is 8.6 GB.

Why native support matters

Before this, an EXL3 checkpoint meant running exllamav3: PyTorch, a JIT-compiled CUDA extension, Python per token. Now the same engine runs both formats — one loader, one KV cache, one offload path, and no conversion pass.

  • Model::load detects the format and selects EXL3Loader automatically.
  • Trellis reconstruction, the EXL3 GEMV and the fused prefill GEMM live in src/cpp/core/cuda/exl3/.
  • The autotuned cublasLt path is the default; TINYCODER_EXL3_LT=0 pins the older fixed-order k summation bit-for-bit.
  • Checkpoint shape and tokenizer are read from the directory — no GGUF conversion step.
Measured · RTX 2080 Ti
723.0tok/s pp512
23.26tok/s tg128

against exllamav3 1.6.6 at 707.5 tok/s pp512 and 22.9 tok/s tg128 — same checkpoint, Qwen3.8-27B EXL3 1.60bpw, 8.6 GB.

Ahead in both columns, but the decode margin is narrow: exllamav3's tg128 includes Python per-token overhead, so its figure is a floor rather than a best case. Both columns were measured with the same pp512/tg128 protocol.

API

Run it as a local OpenAI-compatible provider

The engine tree ships a library, not a server: tinycoder_core, its headers, tests and benchmarks. The OpenAI-compatible endpoint is the TinyCoder OpenAI gateway — a Go HTTP server that links the engine — so existing OpenAI-style tooling has something to point at.

Embed the library

Pull tinycoder_core into your own CMake project, then load and generate through the C++ API.

cmakeFetchContent
FetchContent_Declare(tinycoder_inference
    GIT_REPOSITORY https://github.com/mgorshkov/tinycoder-inference
    GIT_TAG <tag or commit>)
FetchContent_MakeAvailable(tinycoder_inference)
target_link_libraries(your_target PRIVATE tinycoder_core)
C++load + generate
#include "Model.hpp"
using namespace tinycoder;

Model model;
std::string err;
if (!model.load("/path/to/model.gguf", &err)) {
    // handle error
}

InferenceParams params;
params.maxTokens   = 512;
params.temperature = 0.7f;
params.topP        = 0.9f;

model.generate("Write a C++ function to add two numbers.",
    params, [](int32_t tokenId, const std::string &text) {
        std::cout << text << std::flush;
        return true;
    });

Serve the API

Build the gateway with its native engine, then point it at a models directory.

shelltinycoder-openai
./scripts/build.sh
./build/bin/tinycoder-openai \
    --listen :8080 \
    --models-dir /data/models
shelllocalhost:8080
curl -s http://localhost:8080/v1/models

curl -s http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.6-35b-a3b",
       "messages":[{"role":"user",
                    "content":"Write a C++ hello world."}]}'

Model ids come from the gateway's config registry — a GGUF file or an EXL3 checkpoint directory per entry — or from a scan of --models-dir.

Endpoints the gateway actually serves

Only these paths are served — there are no unversioned or legacy aliases.
Method Path What it does
GET/v1/modelsList registry models, with tinycoder metadata
POST/v1/chat/completionsChat completion; SSE streaming; tool calling
POST/v1/completionsText completion; SSE streaming
POST/v1/responsesOpenAI Responses API
POST/v1/messagesAnthropic Messages API
POST/v1/generateNative endpoint; accepts prompt or messages
GET/openai.json, /openai.yamlOpenAPI 3.1 document
GET/healthzLiveness probe

What it supports

  • SSE streaming on the completion endpoints, a final data: [DONE], and keep-alive comments while a model loads or a long prompt prefills.
  • Tool calling on /v1/chat/completions: OpenAI tools, tool_choice and tool_calls, including streamed deltas.
  • Chat templates rendered inside the engine from the model's own template — send messages, not a pre-formatted prompt.
  • One model registry for both formats: each entry points at a GGUF file or an EXL3 checkpoint directory, and the engine auto-detects which.
  • Lazy loading with LRU idle eviction, optional bearer-token auth, CORS, request timeouts and a global concurrency cap.

What it is not

  • Not full OpenAI parity. Those paths are the whole surface, and there are no unversioned or legacy aliases.
  • The engine owns a single KV cache, so requests for the same model are serialized and queued rather than served concurrently.
  • Tool calling is elicited through a text protocol, not constrained decoding: best effort, and only on /v1/chat/completions.
  • /v1/responses and /v1/messages accept and ignore fields they do not model (store, previous_response_id, tools, metadata); max_tokens is required on /v1/messages.
Benchmarks · a selection

Where TinyCoder is faster

A selection, not a survey: these are the four configurations where TinyCoder beats the baseline in both pp512 and tg128. The complete table — including the rows where llama.cpp is faster — is linked below.

pp512 is prefill, tg128 is decode, values in tok/s. Bold marks the faster engine in each column. Warm mean-of-reps, RTX 2080 Ti.
Model pp512 (tok/s) tg128 (tok/s)
TinyCoder Baseline TinyCoder Baseline
Qwen2.5-Coder-1.5B q2_k 0.7 GB · full offload · vs llama.cpp 10,149.8 7,769.8 327.91 323.19
Qwen2.5-Coder-1.5B iq3_xxs 0.9 GB · full offload · vs llama.cpp 10,192.3 10,143.5 276.77 261.47
Ornith-1.5-35B-A3B q8_0 → q4_k experts 34.4 GB · hybrid, layer arena · vs llama.cpp -ncmoe 99 176.3 130.7 53.67 16.42
Qwen3.8-27B EXL3 1.60bpw 8.6 GB · full offload · vs exllamav3 1.6.6 723.0 707.5 23.26 22.9

TinyCoder: tinycoder_bench --n-prompts 512 --n-gen 128 --threads 36. Baselines: llama-bench -p 512 -n 128 -t 36 -r 1, interleaved with the TinyCoder run — three alternating pairs, means reported — and exllamav3 1.6.6 measured with the same pp512/tg128 protocol. The 35B row adds -ncmoe 99. A TinyCoder figure is printed only if that checkpoint passes the text gate (./scripts/text_gate.sh).

The complete table — every configuration, including the rows where llama.cpp is faster and the one with no comparable baseline — lives in BENCHMARKS.md in the engine repository, under the same protocol.

Format breadth

Formats the other engines can't run

Prism's Ternary-Bonsai-2-27B pq2_0 checkpoint — a 6.7 GB ternary model — runs fully offloaded on the same 11 GB card.

There is no llama.cpp baseline for it, and the full table marks it with a dash: llama.cpp has no loader for the pq2_0 ternary format, so there is no comparable run to publish. We are not going to invent one.

16.7tok/s pp512
15.24tok/s tg128
6.7 GBcheckpoint
EXL3 is not in this category. It is also a format llama.cpp does not load, but it does have a baseline — exllamav3 1.6.6 — and TinyCoder is faster on it: 723.0 / 23.26 against 707.5 / 22.9. That is a measured win, kept separate from an unmeasured capability.

How we measure

Interleaved, not sequential

Each TinyCoder run alternates with its baseline run — three pairs — so clock drift lands on both columns equally. Means are reported.

Text before speed

A throughput number is only printed if the checkpoint passes the text gate. A skipped test counts as a failure, so a row cannot pass by omission.

One process at a time

Benchmarks run serially on the reference RTX 2080 Ti. Two processes on that GPU contend for bandwidth and describe neither run.

Implausible is a bug

Before publication, implied TFLOPS and GB/s are checked against the card's peaks. A number above the hardware is skipped work, not a win.

Downstream integration

The VS Code extension is one consumer

The TinyCoder extension embeds tinycoder_core through its own N-API bridge — the engine's FetchContent path exists for exactly this. It adds a chat panel and editor commands on top, and it is how many people meet the engine day to day.

It is not the pitch. The library is the product; the extension is a downstream integration that proves the embedding path works.

TinyCoder extension
How it embeds
  1. FetchContent pulls the engine and links tinycoder_core.
  2. The extension adds an N-API bridge over the C++ Model.
  3. VS Code renders chat and commands over that bridge.

Run it on the GPU you already own

MIT-licensed C++20/CUDA. Clone the engine, build with CUDA, point TINYCODER_MODEL_PATH at a GGUF file or an EXL3 checkpoint directory.