Apple Silicon · Qwen3.8-Flash-Next · fork of antirez's ds4
dstar
A launcher and a set of Metal kernels that run a 126B mixture-of-experts model (6.7B active) on a 64 GB Mac with the full 262K context, MTP and vision, behind an OpenAI and Anthropic compatible server. Decode holds 44 tok/s at 4K and 35.4 at 398K tokens; with the Neural Engine in prefill it reaches 530 tok/s.
This is an unofficial fork. It is not affiliated with or endorsed by the upstream author. Kernel bug reports belong here; engine questions belong upstream.
Pick your setup
One line each. Not sure which row is yours? dstar doctor prints what your Mac fits. The Neural Engine is on in the first three; it is what takes prefill from 397 to 530 tok/s.
64 GB · best answers
recommended
IQ3_XXS, 44 GiB. Better LiveCodeBench, needs the memory limit raised for 262K.
sudo sysctl iogpu.wired_limit_mb=61440
dstar pull --quant iq3
dstar serve --quant iq3 --ctx 262k --ane
64 GB · fastest
Q2_0, 35 GiB of weights, the full 262K context, one sysctl above 131K.
dstar pull --quant q2
dstar serve --ane
96 GB and up
IQ3_S, 51 GiB, the best file ISTA publishes.
dstar pull --quant iq3s
dstar serve --quant iq3s --ane
Less than 64 GB
Experts stay on the SSD with a cache in memory: about 23 GiB planned at 8K context. Slower, identical output.
Coding agents: dstar opencode prints the provider block for OpenCode. Claude Code works against the same server over the Anthropic API.
Benchmarks
Prefill with the Neural Engine
While the GPU computes the MoE experts of each 512-2048 token chunk, the Neural Engine computes the dense projections: the GDN gate, half of the GDN qkv and attn_q, 40% of ssm_out and the shared expert. Two chunks run in pairs, one layer phase apart, so the ANE is never idle. The NLL moves inside the noise.
Prefill, Q2_0
GPU alone
With Neural Engine
8K prompt
397 tok/s
530 tok/s
32K prompt
380 tok/s
501 tok/s
Decode as the context fills up
One chat grown to its full context on an M1 Max with 64 GB, MTP speculative decoding on. Measured before 1.3; with 1.3 sampling IQ3_XXS at 4K is 38.2 tok/s.
Context
Q2_0
IQ3_XXS
DeepSeek V4 Flash
4K
44.0 tok/s
35.2 tok/s
12.1 tok/s
128K
43.0 tok/s
34.2 tok/s
9.1 tok/s
256K
38.8 tok/s
31.7 tok/s
not run
398K
35.4 tok/s
not run
not run
The kernel work behind it
Every change is tested against the CPU reference; most give byte-identical output.
Change
Before
After
Indexer scorer, per layer at 32K blocks
853 µs
63 µs
Attention per token
1.74 ms
1.06 ms
Decode at 128K context
23.8 tok/s
33.8 tok/s
Decode at 262K context
17.7 tok/s
29.7 tok/s
Retried agent turn on a 31K prompt
109 s
0.3 s
Perplexity (NLL), before / after
0.45044
0.45045
Against a DGX Spark, single stream
The most common single-Spark setup for this model is vLLM with the NVFP4 checkpoint and MTP, measured at 36.8 tok/s on prose and 45.8 on code.
Machine
Prose
Code
DGX Spark (vLLM, NVFP4)
36.8 tok/s
45.8 tok/s
M1 Max, this fork
43.2 tok/s
44-45 tok/s
What changed in dstar 1.3
The latest release, in short.
Sampling now follows Qwen's model card
When a client sends no sampling settings (OpenCode sends none), ds4 used the DeepSeek defaults, which sample from a much wider tail than Qwen recommends. 1.3 uses the model card: temperature 1.0, top_p 0.95, top_k 20 while thinking; 0.7 / 0.8 / 20 with thinking off.
MTP drafts are sampled, not taken greedily
Up to two of every three tokens used to be chosen greedily, which Qwen advises against while thinking. MTP now runs with exact sampling; --fast-mtp brings the old behaviour back.
About 10% faster decode
IQ3_XXS on an M1 Max: 34.5 to 38.2 tok/s with the new default sampling. Hyper-connection weights are stored as 8-bit instead of BF16: 568 MiB less to read per token, perplexity moves by under 0.02%.
Experimental: DFlash2 drafter
dstar serve --dflash drafts up to 7 tokens at a time instead of one with MTP: 41.8 tok/s on code against 38.9. Costs 2.1 GB more memory; fetch it with dstar pull --dflash.
Experimental: SSD streaming for the ISTA quants
The experts can stay on the SSD with only a cache in memory, so the model runs on Macs with less RAM: IQ3_XXS at 16.8 tok/s with a 10 GiB cache on an M1 Max, against 33.4 fully in memory. Output is identical.
KV disk cache: about 6 times fewer writes
A checkpoint now waits until the context has grown by half, so filling 262K writes about 9.6 GiB instead of 63 GiB. A retried turn also resumes from its saved state instead of reprocessing the prompt.
116 upstream commits merged
Everything from antirez/ds4 since the last sync is in, including fixes so batched and one-token decoding give the same results.
Yes, with the small quants: Q2_0 is 35 GiB and IQ3_XXS is 44 GiB, so both fit in 64 GB of unified memory with room for the context.
What does the Neural Engine do here?
In prefill, the ANE computes the dense projections of each chunk while the GPU computes the MoE experts, and paired chunks pipeline the two so neither waits. Prefill goes from 397 to 530 tok/s at 8K on an M1 Max. Turned on with --ane; if an evaluation fails, the chunk reruns on the GPU.
Why does the server start with a smaller context than expected?
macOS caps GPU-addressable memory at about three quarters of unified memory, 48 GiB of 64 GB, by default. Contexts above 131K with Q2_0 (262K with IQ3_XXS) need more than that, so the server takes the largest context that fits and prints the line that unlocks it: 57344 for Q2_0 at 262K, 61440 for iq3 at 262K. The setting resets at every reboot.
How far past the trained context can it go?
The model is trained on 262K; the fork applies YaRN with factor 2 above that. On 64 GB, 400K and 524K fit. A needle test with 20 facts found 20 of 20 at 400K and 18 of 20 at 524K.
What about a model larger than the RAM?
DeepSeek V4 Flash Q2 is 81 GiB. dstar keeps the dense weights and a cache of 5800 experts in memory and streams the rest from the SSD: 12.1 tok/s at 4K context on an M1 Max.
Does it work with coding agents?
Yes. The server speaks the OpenAI and Anthropic APIs, so OpenCode and Claude Code work against it. A retried turn on a 31K prompt takes 0.3 s instead of 109 s because the server keeps the recurrent state at the last turn marker.
Which Macs gain?
Everything is written and measured on an M1 Max, but almost nothing is M1-only: the kernels, the long-context work, MTP and YaRN run on any Apple Silicon. On M5 stock ds4 already uses the Metal 4 tensor path, so the gain there is unknown.