Porting MTP to Ornith-1.0-35B: From GGUF Surgery to Hermes Agent

This guide walks through deploying Ornith-1.0-35B locally with llama.cpp, wiring it into Hermes Agent, and finally grafting on an MTP head to push inference speed even higher.

1. What Is Ornith-1.0-35B

#

Ornith-1.0-35B is a research model from DeepReinforce AI, continued-trained on top of Qwen3.5-35B-A3B with their in-house reinforcement-learning framework; it currently ships in four sizes — 9B / 31B / 35B / 397B. What sets it apart is that during training it doesn’t just learn to solve problems — it learns to generate its own scaffold to drive the solution, optimizing the scaffold and the final answer together. That gives Ornith-1.0-35B a clear edge over Qwen3.5-35B-A3B on the kind of long, multi-turn, tool-calling workflows found in Terminal-Bench and SWE-bench:

• Long-horizon agentic coding: officially positioned for long, multi-turn tool-calling workflows like Terminal-Bench and SWE-bench
• 262K context window: thanks to the Gated Delta Net hybrid linear-attention architecture, most layers keep a fixed state size that doesn’t grow linearly with context
• MIT license: free to self-host and fine-tune

Here are the official benchmark numbers, for reference:

This guide uses the Q4_K_M quant; the official GGUF page is deepreinforce-ai/Ornith-1.0-35B-GGUF.

2. Deploying Ornith-1.0-35B

#

Now that we have a rough picture of the model, let’s get it running locally.

2.1 Choosing an Inference Engine

#

There’s no shortage of options for running an LLM locally. A few common ones:

Ollama is convenient (ollama run ... and you’re off) and it’s llama.cpp underneath, but the extra layer of wrapping usually makes inference a touch slower than calling llama.cpp directly. vLLM, on the other hand, is Linux-only and doesn’t support GGUF, so it’s hungrier for VRAM — running a 35B model on a consumer card gets tight.

Weighing it all up, this guide goes with llama.cpp.

llama.cpp is an LLM inference engine written in C/C++. It supports the GGUF quantization format and can run fairly large models on consumer hardware — even pure CPU. See the official GitHub.

2.2 Installing llama.cpp

#

• macOS

  For macOS users, Homebrew is the easy route:

  brew install llama.cpp

  You can also grab the latest prebuilt binaries from llama.cpp’s GitHub or website.

• Windows

  Windows users can likewise download the official prebuilt binaries, or use WSL2 to install the Linux build.

• Linux

  Linux is the odd one out. Because CUDA driver installs and dependencies vary so much across distros, llama.cpp doesn’t ship a prebuilt CUDA binary for Linux. Install via Homebrew and you’ll get the Vulkan build — it runs, but it’s slower than the CUDA build:

  # Vulkan build
  brew install llama.cpp

  If you want the best inference speed, build the CUDA version yourself (there’s no official Linux CUDA prebuilt):

  # CUDA build (compile it yourself)
  
  # Install the build dependencies first (keep Vulkan around as a fallback)
  sudo apt-get install -y build-essential cmake git libvulkan-dev glslc spirv-headers
  
  # Clone llama.cpp and build the CUDA version
  git clone https://github.com/ggml-org/llama.cpp.git llama.cpp-cuda
  cd llama.cpp-cuda
  cmake -B build -DGGML_CUDA=ON -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
  cmake --build build --config Release -j $(nproc)

  Note

  A self-built binary won’t print load_backend: loaded ... on llama-server --version the way the Homebrew build does — a self-built binary links the backend in statically at link time. That’s expected and doesn’t mean CUDA was left out; just load a model to confirm.

2.3 Starting the Service

#

llama.cpp ships with llama-server, which can pull a model straight from Hugging Face and start serving — no need to download the GGUF by hand:

llama-server -hf deepreinforce-ai/Ornith-1.0-35B-GGUF --port 8000 -c 262144

Of course, if you want explicit control over every flag, here’s the full command:

llama.cpp-cuda/build/bin/llama-server \
  --hf-repo deepreinforce-ai/Ornith-1.0-35B-GGUF \
  --hf-file ornith-1.0-35b-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8002 \
  -c 65536 \
  --split-mode none --main-gpu 0 \
  --flash-attn on \
  --jinja

A few of the flags:

Once it’s up, a quick curl confirms it:

curl http://127.0.0.1:8002/v1/models

2.4 The Silent Downgrade: Don’t Let Auto-Config Slow You Down

#

Once the service is running, don’t rush ahead — scroll back through the startup log and you might spot these lines:

W tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance
W sched_reserve: layer 0 is assigned to device CPU but the fused Gated Delta Net tensor is assigned to device CUDA0
W sched_reserve: fused Gated Delta Net (chunked) not supported, set to disabled

This is an easy-to-miss behavior of llama.cpp’s automatic memory allocation: even if your command includes no CPU-offload flags at all, if it decides VRAM is tight it may still choose to keep a layer computing on the CPU.

Usually that doesn’t change the output — that layer just runs a bit slower. But if the layer happens to use a fused kernel like Gated Delta Net, which is currently only implemented for CUDA, the impact isn’t just “this one layer got slower” — the model’s fused kernel gets disabled entirely, falling back to an unoptimized compute path, and decode speed takes a measurable hit (see the ablation data in §5.3).

The fix is simple — use -ngl to put every layer on the GPU explicitly, together with --no-mmap:

-ngl 999 --no-mmap

The number in -ngl 999 just needs to exceed the actual layer count; llama.cpp clamps it, so the net effect is forcing all layers onto the GPU. Add these two flags back into the full command from 2.3 and restart the service:

llama.cpp-cuda/build/bin/llama-server \
  --hf-repo deepreinforce-ai/Ornith-1.0-35B-GGUF \
  --hf-file ornith-1.0-35b-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8002 \
  -c 65536 \
  --split-mode none --main-gpu 0 \
  --flash-attn on \
  -ngl 999 --no-mmap \
  --jinja

Check the startup log once more and those warnings should be gone — a sign the Gated Delta Net fused kernel is now running properly on the GPU.

3. Wiring In Hermes Agent

#

With the model running, the next step is wiring it into Hermes Agent so the agent framework can call the local model directly.

Hermes Agent is a local-first agent framework from Nous Research, with tool calling, long-term memory, and more. One thing to note: Hermes Agent only enables tool calling when the model’s context is at least 64K — which is where that -c 65536 came from.

3.1 Configuring a Custom Provider

#

Hermes Agent doesn’t take direct edits to config.yaml; you configure it through hermes config set (which supports dot notation):

hermes config set custom_providers.ornith-local.base_url "http://127.0.0.1:8002/v1"
hermes config set custom_providers.ornith-local.models."deepreinforce-ai/Ornith-1.0-35B-GGUF".context_length 65536

Once that’s set, switch to the new provider and you’re good to go. At this point Ornith-1.0-35B is reliably serving inference to Hermes Agent.

If you’d rather automate the steps above, just hand Hermes Agent this prompt and it’ll run the hermes config set commands for you:

I want to add a local llama.cpp service as a custom provider. Details:
- API endpoint: http://127.0.0.1:8002/v1
- Model name: deepreinforce-ai/Ornith-1.0-35B-GGUF
- context_length: 65536

Please use `hermes config set` to add this provider under custom_providers, and confirm that context_length was written correctly.

4. Going Further: Faster Inference with MTP

#

If you’re still not happy with the inference speed, consider MTP.

4.1 What Is MTP

#

MTP (Multi-Token Prediction) lets the model predict several tokens at once — every hit saves a round of computation, making it essentially free speedup. Ornith doesn’t yet ship an official build with an MTP head, so we have to port one from the community.

The MTP head used here comes from skinnyctax’s cross-model port:

4.2 llama.cpp Version Requirements

#

MTP speculative decoding needs llama.cpp b9180 or later (PR #22673, merged into mainline on 2026-05-16). On an older build, --spec-type won’t offer the draft-mtp option.

Check your current version:

llama-server --version

For a self-built binary, go back to the source dir, pull, and rebuild:

cd llama.cpp-cuda
git pull
cmake --build build --config Release -j $(nproc)

4.3 GGUF Graft Surgery

#

skinnyctax provides a donor file containing only the MTP head (20 tensors, named blk.40.*); our job is to splice it onto the official Ornith GGUF.

The graft logic:

1. Parse the GGUF headers of the target (the Ornith model itself, 733 tensors) and the donor (MTP head-only, 20 tensors)
2. Append the donor’s tensors to the end of the target (733 → 753)
3. Change two metadata fields: block_count: 40 → 41, and add nextn_predict_layers=1
4. Recompute tensor offsets and write out the merged file

The script and donor file are both in skinnyctax’s Hugging Face repo:

hf download skinnyctax/Ornith-1.0-35B-Q6_K-Frankenstein-MTP-GGUF \
  ornith-1.0-35b-Q4_K_M-MTP-donor-heads.gguf gguf_mtp_graft.py --local-dir .

python3 gguf_mtp_graft.py "$BASE_GGUF" \
  ornith-1.0-35b-Q4_K_M-MTP-donor-heads.gguf \
  ornith-1.0-35b-Q4_K_M-MTP-v3.gguf

A GGUF header bug worth watching for

#

The graft script has a subtle issue in how it handles the general.architecture KV pair: it decodes the value into a Python string, then re-encodes it on the way back out:

if key == "general.architecture":
    arch = rstr(fp)
    kv_pairs.append((key, 8, arch.encode("utf-8")))

arch.encode("utf-8") only re-encodes the string content itself, but the GGUF spec requires every string value to be preceded by an 8-byte length prefix — and this step drops it. general.architecture is usually one of the very first KV pairs in a GGUF file, so those missing 8 bytes shift every subsequent KV pair out of position; when llama.cpp reads it, it reads back an absurdly large number and refuses to load:

E read: string length 8029132205683210097 exceeds maximum 1073741824
E gguf_init_from_reader: failed to read key-value pairs

The fix is to preserve the full raw bytes instead of decoding and re-encoding:

if key == "general.architecture":
    arch = rstr(fp)
    val_end = fp.tell()
    fp.seek(val_start)
    val_raw = fp.read(val_end - val_start)
    fp.seek(val_end)
    kv_pairs.append((key, 8, val_raw))

After the fix, a small test (-c 4096) put draft acceptance at 0.87–0.96 — the cross-model MTP head is nicely compatible with the Ornith model.

4.4 Launching the MTP Build

#

Compared to the base setup, it’s just two extra flags — --spec-type draft-mtp and --spec-draft-n-max 2:

llama.cpp-cuda/build/bin/llama-server \
  --model /path/to/ornith-1.0-35b-Q4_K_M-MTP-v3.gguf \
  --alias deepreinforce-ai/Ornith-1.0-35B-GGUF \
  --host 127.0.0.1 --port 8002 \
  -c 65536 \
  --split-mode none --main-gpu 0 \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  --flash-attn on \
  -ngl 999 --no-mmap \
  --jinja

--spec-draft-n-max 2 is the cap on how many tokens MTP drafts per step — the next section’s experiments explain why 2 rather than 1 or 3.

5. Supporting Experiments

#

Every test below was run single-GPU, single-request, reading eval time and tokens per second straight from llama-server’s own output — not timed through a client app.

5.1 MTP Graft vs. Stock GGUF

#

Using the official original GGUF as the control. The base weights on both sides are identical (the port takes the official file as its target); the only difference is whether there’s an MTP head. Test conditions: -c 65536, forced full-GPU.

5.2 Long-Context Quality Check

#

A full -c 262144 exceeds what this card can run fully on the GPU, so this test ran with auto-offloading and the fused kernel disabled. Method: needle-in-a-haystack — bury a secret phrase in filler text, then ask the model to recall it at the very end.

5.3 Context Size Ablation

#

Fixing --spec-draft-n-max 1 and -ngl 999 --no-mmap, sweeping 5 values of -c, 5 repeats each.

5.4 spec-draft-n-max Ablation

#

Fixing -c 65536 and -ngl 999 --no-mmap, sweeping n-max 1–4, 5 repeats each.

5.5 Backend / Flash Attention Comparison

#

Same ported GGUF, -c 16384, n-max 1, all forced -ngl 999 --no-mmap, a fixed prompt averaged over 5 runs.

The FA on/off gap falls within the std dev — no significant effect. CUDA is about 22% faster than Vulkan, thanks to native Tensor Core and cuBLAS optimizations.

5.6 KV Cache Quantization

#

Fixing -c 65536 and n-max 2, all forced full-GPU, to test how quantizing the KV cache affects speed.

All three sit within noise — at this context length, the decode bottleneck is reading and computing the weight matrices, not KV-cache bandwidth.

Speed didn’t improve meaningfully, but q4_0 saved about 0.81 GB of VRAM — possibly enough to squeeze a 131K context into forced-full-GPU mode too, which is worth testing later.

6. Wiring the MTP Build into Hermes Agent

#

The MTP build serves on the same API endpoint and alias as the base build, so Hermes Agent needs no extra configuration. If you already did the setup in Chapter 3, just restart the service — Hermes still sees deepreinforce-ai/Ornith-1.0-35B-GGUF, but decode is now about 30% faster underneath.

The config commands are exactly the same as the base build:

hermes config set custom_providers.ornith-local.base_url "http://127.0.0.1:8002/v1"
hermes config set custom_providers.ornith-local.models."deepreinforce-ai/Ornith-1.0-35B-GGUF".context_length 65536

Hindsight is the long-term memory system that pairs with Hermes Agent; it extracts and recalls structured memories from conversations, and it calls the local model through this same provider too.

References

#

• Ornith-1.0-35B official model page
• Ornith-1.0-35B MTP port
• llama.cpp GitHub
• llama.cpp build docs
• llama.cpp Speculative Decoding
• GGUF format spec
• Hermes Agent docs

Appendix: Gotchas Cheat Sheet

#