OtherU quantizes DeepSeek-V4-Flash itself rather than shipping a stock community GGUF. The pipeline takes the upstream FP8 safetensors and produces the 2-bit GGUF that Ember loads in production, in OtherU's own affine fp2 format.
The resulting model is published at otheru/DeepSeek-V4-Flash-Strix-Halo-GGUF on Hugging Face.
The GGUF uses custom ROCmFPx tensor types that mainline
llama.cpp, LM Studio and Ollama do not implement. Loading it with an incompatible runtime may fail or produce invalid output — it is built for Ember.
Target hardware
Everything assumes a single Strix Halo APU (**gfx1151**) — the host referred to as otheru. That box is also the production inference host, so the heavy stages must stop the ember-server unit first.
The pipeline at a glance
| Stage | Script | What it does |
|---|---|---|
| Download | scripts/00-download.sh |
Pull upstream FP8 safetensors (e.g. deepseek-ai/DeepSeek-V4-Flash-0731) |
| Abliterate (optional) | scripts/20-abliterate.sh |
wo_b refusal-direction surgery via tools/abliterate/ablate_wo_b.py |
| Convert | scripts/30-convert.sh |
FP8 → BF16 GGUF using the modular DeepSeek-V4 converter |
| Quantize | scripts/40-quantize.sh |
Affine quantization to Q2_0_ROCMFP2, recipe in recipes/deepseek-v4-flash-affine.env |
| Validate | scripts/50-validate.sh |
Ember differential validator |
| Deploy | scripts/60-deploy.sh |
Swap the ember-server model, with rollback |
The affine fp2 format
The production recipe carries every MoE expert at 2.58 bpw in OtherU's own affine fp2 format, Q2_0_ROCMFP2, where value = code · scale − offset.
Two format traps are worth knowing, because both have cost real debugging time:
- The on-box
deepseek4-quantizeunderstands the DeepSeek-V4 architecture but has no affine support — onlyiq2_xxs/q2_k, the older fallback recipe. - The prebuilt Strix
llama-quantizeexposesQ2_0_ROCMFPX, but that is a different codebook ({-4,-1,+1,+4}) from OtherU's affineQ2_0_ROCMFP2. Mixing them silently produces garbage output rather than an error.
The affine quantizer was built and verified on 2026-07-31 and is installed on otheru at /root/affine-quantize/. It accepts --tensor-type 'ffn.*exps=q2_0_rocmfp2'.
Published artifacts
Verify any local copy against the SHA-256 values on the model card.
| File | Size | Purpose |
|---|---|---|
DeepSeek-V4-Flash-0731-Abliterated-ROCMFPx-Strix-Lean-2.58bpw.gguf |
85.26 GiB | Target model, 2.58 bpw |
DeepSeek-V4-Flash-0731-Abliterated-DSpark-draft-4.25bpw.gguf |
10.15 GiB | DSpark drafter for speculative decoding |
Target plus matching drafter is 95.41 GiB resident.
Abliteration band
The refusal-direction edit is restricted to the band where the direction actually separates — blocks 10–42 plus the 3 DSpark stages, leaving the first ten blocks untouched.
| Edited tensors | 36 (blocks 10–42 + 3 MTP) |
| Strength (lambda) | 3.5 |
| SRA rank | 0 (none) |
| Row-norm preserve | yes |
| Direction | rank-1, published, SHA-256 pinned |
Early layers carry general features and separate harmful from harmless weakly, so editing there costs capability without adding bypass. Editing all 46 attention output-B tensors instead of 36 is known to damage agentic coherence.
Quality is not yet characterized
No perplexity, KL-divergence against BF16, or capability benchmark has been run on this artifact. Published numbers are throughput only. The abliteration is validated behaviourally and the tensor layout structurally, but the quality cost of the 2.58 bpw quantization itself is currently unmeasured — treat it as unknown rather than as acceptable.
Pipeline repository layout
The pipeline itself lives in an internal repository; its structure is:
| Path | Contents |
|---|---|
docs/CONSOLIDATION.md |
Manifest: every input, vendored vs on-box vs external |
docs/INVENTORY.md |
Where each tool and dependency lives, versions, build state |
docs/RECIPE.md |
The exact per-tensor affine recipe, plus the older fallback |
docs/PIPELINE.md |
End-to-end runbook, stage by stage |
docs/AFFINE-QUANTIZER-STATUS.md |
Build status and history of the affine quantizer |
recipes/*.env |
Machine-readable per-tensor type maps and imatrix path |
scripts/ |
Numbered stages 00–60, plus run-all-fallback.sh |
tools/abliterate/ |
Refusal-direction surgery, prompt sets, validation |
tools/dir-steering/ |
Refusal-direction extraction |
tools/deepseek4-quantize/ |
Bespoke DeepSeek-V4 quantizer (fallback recipe) |
tools/imatrix-dataset/ |
Imatrix regeneration set and build script |
port/ |
ROCmFPX sources and the affine-onto-master patch |
deploy/ |
Reference copy of the production ember-server systemd unit |
Status
- Portable code, prompts, recipe and full inventory are captured in the repository.
- The recipe and provenance are confirmed against the Hugging Face model card
otheru/DeepSeek-V4-Flash-Strix-Halo-GGUF. - The affine quantizer is built and verified.
- The convert and quantize stage scripts encode the intended commands but have not yet been run end-to-end on a full model. Treat this as a runbook to execute and then harden, not a proven one-shot.