Apple Silicon · Qwen3.8-Flash-Next · fork of antirez's ds4

dstar

A launcher and a set of Metal kernels that run a 126B mixture-of-experts model (6.7B active) on a 64 GB Mac with the full 262K context, MTP and vision, behind an OpenAI and Anthropic compatible server. Decode holds 44 tok/s at 4K and 35.4 at 398K tokens; with the Neural Engine in prefill it reaches 530 tok/s.

One line in the terminal

curl -fsSL https://raw.githubusercontent.com/paperniuk/ds4/m1-flash-next/install.sh | bash

Prebuilt binary · SHA-256 checked · nothing to compile

Read the write-up

macOS 15 or newer · Apple Silicon · 64 GB RAM for 262K · less with SSD streaming

Built on antirez's ds4 (DwarfStar)

The engine is antirez's ds4. This fork adds Metal kernels for the small quants, the Neural Engine prefill and the dstar launcher.

This is an unofficial fork. It is not affiliated with or endorsed by the upstream author. Kernel bug reports belong here; engine questions belong upstream.

Pick your setup

One line each. Not sure which row is yours? dstar doctor prints what your Mac fits. The Neural Engine is on in the first three; it is what takes prefill from 397 to 530 tok/s.

64 GB · best answers

recommended

IQ3_XXS, 44 GiB. Better LiveCodeBench, needs the memory limit raised for 262K.

sudo sysctl iogpu.wired_limit_mb=61440
dstar pull --quant iq3
dstar serve --quant iq3 --ctx 262k --ane

64 GB · fastest

Q2_0, 35 GiB of weights, the full 262K context, one sysctl above 131K.

dstar pull --quant q2
dstar serve --ane

96 GB and up

IQ3_S, 51 GiB, the best file ISTA publishes.

dstar pull --quant iq3s
dstar serve --quant iq3s --ane

Less than 64 GB

Experts stay on the SSD with a cache in memory: about 23 GiB planned at 8K context. Slower, identical output.

dstar pull --quant iq3
dstar serve --quant iq3 -- --ssd-streaming --ssd-streaming-cache-experts 10GB

Benchmarks

Prefill with the Neural Engine

While the GPU computes the MoE experts of each 512-2048 token chunk, the Neural Engine computes the dense projections: the GDN gate, half of the GDN qkv and attn_q, 40% of ssm_out and the shared expert. Two chunks run in pairs, one layer phase apart, so the ANE is never idle. The NLL moves inside the noise.

Prefill, Q2_0GPU aloneWith Neural Engine
8K prompt397 tok/s530 tok/s
32K prompt380 tok/s501 tok/s

Decode as the context fills up

One chat grown to its full context on an M1 Max with 64 GB, MTP speculative decoding on. Measured before 1.3; with 1.3 sampling IQ3_XXS at 4K is 38.2 tok/s.

ContextQ2_0IQ3_XXSDeepSeek V4 Flash
4K44.0 tok/s35.2 tok/s12.1 tok/s
128K43.0 tok/s34.2 tok/s9.1 tok/s
256K38.8 tok/s31.7 tok/snot run
398K35.4 tok/snot runnot run

The kernel work behind it

Every change is tested against the CPU reference; most give byte-identical output.

ChangeBeforeAfter
Indexer scorer, per layer at 32K blocks853 µs63 µs
Attention per token1.74 ms1.06 ms
Decode at 128K context23.8 tok/s33.8 tok/s
Decode at 262K context17.7 tok/s29.7 tok/s
Retried agent turn on a 31K prompt109 s0.3 s
Perplexity (NLL), before / after0.450440.45045

Against a DGX Spark, single stream

The most common single-Spark setup for this model is vLLM with the NVFP4 checkpoint and MTP, measured at 36.8 tok/s on prose and 45.8 on code.

MachineProseCode
DGX Spark (vLLM, NVFP4)36.8 tok/s45.8 tok/s
M1 Max, this fork43.2 tok/s44-45 tok/s

What changed in dstar 1.3

The latest release, in short.

Sampling now follows Qwen's model card

When a client sends no sampling settings (OpenCode sends none), ds4 used the DeepSeek defaults, which sample from a much wider tail than Qwen recommends. 1.3 uses the model card: temperature 1.0, top_p 0.95, top_k 20 while thinking; 0.7 / 0.8 / 20 with thinking off.

MTP drafts are sampled, not taken greedily

Up to two of every three tokens used to be chosen greedily, which Qwen advises against while thinking. MTP now runs with exact sampling; --fast-mtp brings the old behaviour back.

About 10% faster decode

IQ3_XXS on an M1 Max: 34.5 to 38.2 tok/s with the new default sampling. Hyper-connection weights are stored as 8-bit instead of BF16: 568 MiB less to read per token, perplexity moves by under 0.02%.

Experimental: DFlash2 drafter

dstar serve --dflash drafts up to 7 tokens at a time instead of one with MTP: 41.8 tok/s on code against 38.9. Costs 2.1 GB more memory; fetch it with dstar pull --dflash.

Experimental: SSD streaming for the ISTA quants

The experts can stay on the SSD with only a cache in memory, so the model runs on Macs with less RAM: IQ3_XXS at 16.8 tok/s with a 10 GiB cache on an M1 Max, against 33.4 fully in memory. Output is identical.

KV disk cache: about 6 times fewer writes

A checkpoint now waits until the context has grown by half, so filling 262K writes about 9.6 GiB instead of 63 GiB. A retried turn also resumes from its saved state instead of reprocessing the prompt.

116 upstream commits merged

Everything from antirez/ds4 since the last sync is in, including fixes so batched and one-token decoding give the same results.

Questions

Can a 64 GB Mac run a 126B model?

Yes, with the small quants: Q2_0 is 35 GiB and IQ3_XXS is 44 GiB, so both fit in 64 GB of unified memory with room for the context.

What does the Neural Engine do here?

In prefill, the ANE computes the dense projections of each chunk while the GPU computes the MoE experts, and paired chunks pipeline the two so neither waits. Prefill goes from 397 to 530 tok/s at 8K on an M1 Max. Turned on with --ane; if an evaluation fails, the chunk reruns on the GPU.

Why does the server start with a smaller context than expected?

macOS caps GPU-addressable memory at about three quarters of unified memory, 48 GiB of 64 GB, by default. Contexts above 131K with Q2_0 (262K with IQ3_XXS) need more than that, so the server takes the largest context that fits and prints the line that unlocks it: 57344 for Q2_0 at 262K, 61440 for iq3 at 262K. The setting resets at every reboot.

How far past the trained context can it go?

The model is trained on 262K; the fork applies YaRN with factor 2 above that. On 64 GB, 400K and 524K fit. A needle test with 20 facts found 20 of 20 at 400K and 18 of 20 at 524K.

What about a model larger than the RAM?

DeepSeek V4 Flash Q2 is 81 GiB. dstar keeps the dense weights and a cache of 5800 experts in memory and streams the rest from the SSD: 12.1 tok/s at 4K context on an M1 Max.

Does it work with coding agents?

Yes. The server speaks the OpenAI and Anthropic APIs, so OpenCode and Claude Code work against it. A retried turn on a 31K prompt takes 0.3 s instead of 109 s because the server keeps the recurrent state at the last turn marker.

Which Macs gain?

Everything is written and measured on an M1 Max, but almost nothing is M1-only: the kernels, the long-context work, MTP and YaRN run on any Apple Silicon. On M5 stock ds4 already uses the Metal 4 tensor path, so the gain there is unknown.