Ember — Inference Engine

Ember (repo: otheru-ai/ember) is the inference engine that powers OtherU. It is a narrow, self-contained C-based runtime optimized for DeepSeek V4 Flash, not a general GGUF loader. It runs on the reference deployment in Docker with ROCm passthrough.

Source & Upstream

Ember is OtherU's own from-scratch engine, written against the ds4 architecture as an upstream reference and now at feature and performance parity with it. Like ds4 it builds on the foundations of llama.cpp / GGML.

Reference Deployment (Strix-Halo)

Start Command

docker run -d --restart unless-stopped --name ember-server \
  --device /dev/kfd --device /dev/dri \
  --security-opt seccomp=unconfined \
  --network host \
  -v /srv:/srv \
  ghcr.io/otheru-ai/ember:2026.8.23 \
  --port 8000 \
  -m /srv/models/DeepSeek-V4-Flash-0731-ablit1042-v2.gguf \
  --model-name deepseek-v4-flash \
  --model-card /ember/share/model_cards/deepseek-v4-flash-src.json \
  --kv-cache-dir /srv/ember/kvcache --kv-cache-mb 262144 \
  --ds4-expert-top-k 4 --default-temperature 0.6 \
  --prefix-cache-slots 6 --max-ctx 131072

Architecture

Ember is not a wrapper or generic runner — it is a purpose-built C program with its own graph execution engine. Key design choices:

• Single-model narrowness — only DeepSeek V4 Flash GGUFs with the expected tensor layout, quantization mix, and metadata
• Self-contained — no dependency on llama.cpp/GGML at runtime; kernels, quant formats, and prompt rendering are native
• Asymmetric quantization — routed MoE experts at IQ2_XXS, down at Q2_K; shared experts, projections, and routing left unquantized. This is what makes 2-bit quants viable: most of the model space is aggressively compressed but the critical paths stay accurate
• Disk-first KV cache — checkpoints are written to SSD at four moments (cold, continued, evict, shutdown), enabling fast session resume across restarts without re-prefilling from token zero
• Single-threaded inference — one graph worker serializes all requests; client threads handle HTTP parsing in parallel

Endpoints

The deployment serves a single model id: deepseek-v4-flash.

Key CLI Flags

Disk KV Cache Behavior

Checkpoints are written at four moments:

• cold — after a long first prompt reaches a stable prefix, before generation
• continued — when prefill or generation reaches the next aligned frontier
• evict — before an unrelated request replaces the live in-memory session
• shutdown — when the server exits cleanly

Cold saves trim the last 32 tokens and align down to a 2048-token chunk boundary to avoid BPE retokenization misses.

Model Quantization

Quantized GGUFs in use:

Download script: download_model.sh q2-imatrix fetches from HuggingFace.

Performance Benchmarks

Reference Deployment (AMD Strix-Halo) — Measured

Measured 2026-08-22 on one AMD Ryzen AI Max+ 395 with Radeon 8060S (gfx1151) and 128 GB unified memory, against the published artifacts, with DSpark speculative decoding enabled:

Acceptance is workload-dependent: mixed free-form sampling at temperature 0.6 measured \~0.52 mean acceptance on the same pair. Full harness, raw results and environment are in the benchmark bundle on the model card.

SSD Streaming

When the model does not fit in GPU-addressable memory, Ember can run in SSD streaming mode. Non-routed weights stay resident; routed MoE experts are cached and loaded from the GGUF file on miss. The automatic budget takes 80% of the GPU recommended working set, subtracts non-routed weights, then uses the rest for expert cache.

# Automatic cache budget
ember-server --ssd-streaming

# Explicit cache size
ember-server --ssd-streaming --ssd-streaming-cache-experts 32GB

Tooling

Additional tools under gguf-tools/: GGUF generation, imatrix collection, quantization, quality testing against official DeepSeek logits.