# Spark-X2.6-1.7B WebGPU runtime

The chat app (`index.html`, `app/main.js`) runs Spark-X2.6-1.7B entirely in the browser through the
model-specific runtime in `lib/runtime/`. There is one inference path: a hand-written WGSL decoder for the
official Q4 weights. ONNX Runtime, Transformers.js and WebAssembly inference are not used or shipped.

## Requirements

- WebGPU and `shader-f16` on desktop Chrome or Edge.
- About 1.2 GB of downloaded weights and 1.8 GB of free GPU memory, including the KV cache.
- A 32,768-token context window. Prompt plus generated tokens never exceeds it.

## Model contract

| Name | Value |
|---|---:|
| Layers | 28 |
| Hidden size | 2048 |
| Intermediate size | 6656 |
| Query heads | 8 |
| KV heads | 2 |
| Head dimension | 256 |
| Q / K / V dimensions | 2048 / 512 / 512 |
| Fused QKV output | 3072 |
| Vocabulary | 131072 |
| Max positions | 32768 |
| Sliding window | 512 |
| Sliding KV capacity | 1024 |
| RMSNorm epsilon | 1e-6 |
| Attention scale | 0.0625 |
| EOS / BOS / pad | 1 / 0 / 2 |
| Full-attention layers | 3, 7, 11, 15, 19, 23, 27 |
| Sliding-attention layers | all other indices |

`layer_types[i] = (i % 4 === 3) ? "full" : "sliding"`. Activations are f16. K/V are stored as packed int8 with
f16 scales per 8-dimension group and per position; RMSNorm statistics, softmax, KV dequantization, and weight
dequantization accumulate in f32. Full-attention layers apply RoPE to 64 dimensions and copy the remaining 192
dimensions unchanged. GELU is the exact (`erf`) form, `erf` being the Abramowitz and Stegun 7.1.26
approximation. Attention has one sigmoid scalar gate per query head.

## Weight bundle

The published bundle lives in the model repository
[`leventkoldas/Spark-X2.6-1.7B-WebGPU-Runtime`](https://huggingface.co/leventkoldas/Spark-X2.6-1.7B-WebGPU-Runtime).
`index.html` sets the base URL in `<meta name="spark-model">`; `?model=<url>` overrides it, and a local
`spark/runtime/` directory is used when present.

```
runtime/
  manifest.json
  layer_00.bin … layer_27.bin
  lm_head_0.bin … lm_head_3.bin
  lm_head_scales.bin
  rope.bin
  final_norm.bin
  pack-report.json
```

Every file starts with 256 reserved bytes; every tensor starts at a 256-byte boundary. The manifest is the only
source of tensor offsets. Quantized payloads are copied byte-for-byte from the official Q4 export: most
projections are 4-bit with group-32 scales and zero points, `out_proj` and `lm_head` are 8-bit symmetric with
zero point 128, and `g_proj` is 8-bit asymmetric.

`runtime/manifest.json` uses format `spark-runtime-v1`:

```json
{
  "format": "spark-runtime-v1",
  "build": "1794304671",
  "model": "Spark-X2.6-1.7B",
  "quant": "q4",
  "context": 32768,
  "hidden": 2048,
  "intermediate": 6656,
  "vocab": 131072,
  "heads": 8,
  "kv_heads": 2,
  "head_dim": 256,
  "sliding_window": 512,
  "sliding_kv_capacity": 1024,
  "eos_token_id": 1,
  "layer_types": ["sliding", "sliding", "sliding", "full", "…"],
  "files": {
    "layer_00": { "url": "layer_00.bin", "bytes": 31762176 },
    "lm_head_0": { "url": "lm_head_0.bin", "bytes": 67108864 },
    "lm_head_scales": { "url": "lm_head_scales.bin", "bytes": 16777216 },
    "rope": { "url": "rope.bin", "bytes": 20971776 },
    "final_norm": { "url": "final_norm.bin", "bytes": 4352 }
  },
  "layers": [
    {
      "index": 0,
      "type": "sliding",
      "file": "layer_00",
      "tensors": {
        "input_norm": { "offset": 256, "bytes": 4096, "dtype": "f16", "source": "/l0/input_norm.weight", "n": 2048 },
        "qkv_q": { "offset": 4352, "bytes": 3145728, "dtype": "q4", "source": "/l0/qkv_proj.q4", "n": 3072, "k": 2048, "blocks": 64 },
        "qkv_scales": { "offset": 3150080, "bytes": 393216, "dtype": "f16", "source": "/l0/qkv_proj.scales", "n": 3072 },
        "qkv_zp": { "offset": 3543296, "bytes": 98304, "dtype": "q4zp", "source": "/l0/qkv_proj.zp", "n": 3072 }
      }
    }
  ],
  "lm_head": {
    "chunks": [
      { "file": "lm_head_0", "rows": 32768, "offset": 256, "bytes": 67108864 }
    ],
    "scales": { "file": "lm_head_scales", "offset": 256, "bytes": 16777216 },
    "symmetric": true,
    "zero_point": 128
  },
  "rope": {
    "sliding_cos": { "offset": 256, "bytes": 8388608, "dim": 128 },
    "sliding_sin": { "offset": 8388864, "bytes": 8388608, "dim": 128 },
    "full_cos": { "offset": 16777472, "bytes": 2097152, "dim": 32 },
    "full_sin": { "offset": 18874624, "bytes": 2097152, "dim": 32 }
  },
  "final_norm": { "file": "final_norm", "offset": 256, "bytes": 4096 }
}
```

Each layer carries `input_norm`, `qkv_q/scales/zp`, `g_q/scales/zp`, `out_q/scales`, `post_norm`,
`gate_q/scales/zp`, `up_q/scales/zp` and `down_q/scales/zp`. `out_proj` is symmetric, so it has no zero-point
tensor. `manifest.js` rejects a manifest with the wrong format, context, layer count or head geometry, a missing
tensor, or an `out_zp` entry.

`runtime/pack-report.json` records the source initializer, source range, output file/offset/length and SHA-256
of every tensor.

## Modules

| File | Responsibility |
|---|---|
| `lib/runtime/config.js` | Fixed model constants, layer types, scratch-buffer sizes, and adapter tuning knobs. |
| `lib/runtime/manifest.js` | Fetch and validate `runtime/manifest.json`. |
| `lib/runtime/weights.js` | Download the bundle through CacheStorage, create and upload one storage buffer per file. |
| `lib/runtime/kernels.js` | WGSL sources for embed, RMSNorm, Q4/Q8 GEMV and GEMM, RoPE, int8 KV packing, attention, fused GELU, exact top-k and fused argmax. |
| `lib/runtime/pipelines.js` | Create the selected compute pipelines (plus tuning variants under `?autotune`) with `createComputePipelineAsync`. |
| `lib/runtime/bindings.js` | Allocate scratch and int8 KV buffers, then create all bind groups and command encoders. |
| `lib/runtime/execute.js` | Submit a command buffer, bound the prefill queue, and perform the small readbacks. |
| `lib/runtime/index.js` | Public API, load sequence, measured tuning cache, prefill/decode loops, prefix reuse. |
| `lib/chat.js` | Tokenizer and chat template (`@huggingface/tokenizers`, `@huggingface/jinja`). |
| `app/compact.js` | Token counting, context compaction and prompt assembly. |
| `app/main.js` | Chat UI and generation loop. |
| `app/harness.js`, `app/bench.js` | Browser test hooks (`?harness`) and speed benchmark (`?bench`). |
| `tools/pack_weights.py` | Offline ONNX-to-runtime packer. |
| `tools/verify_pack.py` | Bundle hash, size, alignment and tensor-range verifier. |

## Execution

**Load.** `fetchManifest`, `loadChat` and `generation_config.json` are fetched in parallel. Weight files are
fetched with bounded concurrency (four by default), cached under `spark-runtime-<build>`, validated against their
manifest byte counts and uploaded to storage buffers. Pipeline compilation runs concurrently with the weight
transfer. Scratch buffers, KV buffers and every bind group are created once. Warm-up decode and prefill passes
compile driver first-use paths.

**Decode.** Pipelines and bind groups are static, but a WebGPU command buffer is single-use, so each token
records the same dispatch sequence into a fresh command buffer. Per-token JavaScript work is one uniform write,
one submit, and a 4-byte readback. The sequence is: embedding lookup fused with the first RMSNorm (tied to the
q8 `lm_head` table), QKV projection, attention gate, RoPE and packed-int8 KV write, GQA-fused tiled attention
over the KV cache, gated output projection with residual, RMSNorm, fused gate/up projection, GELU multiply, down
projection with residual. After 28 layers: final RMSNorm, four persistent `lm_head` chunks, then exact top-256
sampling or fused greedy argmax.

**Prefill.** Prompts are processed in 256-token GEMM chunks with 128-row × 16-column tiles (64-row tiles are
available for autotuning). Attention processes eight queries per workgroup over the cached K/V, with a 512-token
sliding window on sliding layers. Full-attention layers copy the unrotated K dimensions and use a 32,768-position
KV cache; sliding layers use a 1,024-position ring. Each 256-token chunk is submitted as four-layer groups and
awaited before the next chunk, which keeps the iGPU queue bounded on long prompts.

**Sampling.** Top-k is exact: every 256-value vocabulary block is sorted, then a 512-way merge keeps the true
top-256. Greedy decoding reduces one candidate per LM-head workgroup and never writes the full logits row. Both
paths return one token id (4 bytes). The GPU sampler applies temperature/top-p/top-k on device. The full
131,072-value logits row never reaches JavaScript in production; `onLogits` is a validation-only path.

**Autotuning.** `?autotune` measures decode rows-per-workgroup (8/16/32), prefill M tile (64/128), and full/sliding
split caps at 2K and 8K, then caches the selected values in `localStorage` keyed by adapter and build. A
candidate must beat the default by 3%, so noise does not replace the hand-tuned configuration. The split cap is
only a performance bound: the active key range is partitioned evenly across the active splits.

**Prefix reuse.** `generateIds` compares the new prompt with the cached token ids and reuses the common prefix
when `reuse` is true, falling back to `resetKv()` when there is no useful overlap.

## Public API

```js
import { loadSpark } from "./lib/runtime/index.js";

const spark = await loadSpark({
  base: "https://huggingface.co/leventkoldas/Spark-X2.6-1.7B-WebGPU-Runtime/resolve/main",
  dtype: "q4f16",               // optional; the only accepted value
  onProgress: ({ loaded, total }) => {},
  onDeviceLost: (info) => {},
});
```

`loadSpark` returns:

- `model`, `gpu`, `variant` (`"q4f16"`), `staticKV`, `maxPosition`, `context: { full, sliding }`
- `loadMs: { download, init, total }`, `memory()`, `kvCapacity()`, `tuning`
- `chat`: `{ tokenizer, render, encodeChat, decode }`
- `generate(messages, opts)`: async generator yielding decoded text chunks. `opts`: `thinking`, `effort`,
  `tools`, `temperature`, `topP`, `topK`, `maxNewTokens`, `signal`.
- `generateIds(tokenIds, opts)`: async generator yielding token ids. Adds `reuse`, `forced`, `batchPrefill`,
  `onLogits`, and a mutable `stats` object.
- `resetKv()`.

`stats` is populated with `promptTokens`, `newTokens`, `ttftMs`, `prefillMs`, `encodeMs`, `decodeMs`,
`decodeTokS`, `reusedTokens`, `maxNewTokens`, `contextLimit`, `done` and `totalMs`. `ttftMs` starts after
tokenization; user-visible time to first token is approximately `encodeMs + ttftMs`. `maxNewTokens` is always
clamped to `min(request, 32768 - promptTokens)`.

`generate` buffers decoded ids while the tokenizer returns a partial UTF-8 sequence, so consumers only see
complete text. Tokenizer encode and decode time are accumulated into `encodeMs` and `decodeMs`.

## CPU work and WebAssembly

The current tokenizer is pure JavaScript (`@huggingface/tokenizers`). Measured on the Intel Gen-12lp test
machine: 2,335-token prompt encode 8.2 ms, 9,292-token prompt encode 12.3 ms, and 16-token decode 0.7 ms.
That is under 0.1% of the 28 s (2K) and 104 s (8K) GPU prefill times, so no WebAssembly tokenizer is used.
The inference path is WebGPU compute; WebAssembly has no standard WebGPU bindings, and Wasm SIMD/threads,
memory64, GC, tail calls, and JSPI do not address the GPU bottleneck. The same measurement is the gate for
revisiting a Wasm tokenizer if tokenization ever becomes material.

## Type checking

The runtime is checked as JavaScript with JSDoc types and `@webgpu/types`; there is no build step and no
runtime dependency on TypeScript.

```bash
bun install
bun run typecheck
```

`jsconfig.json` checks the runtime and app with `checkJs`; `jsconfig.runtime.json` applies strict checking,
including implicit-any and null checks, to the runtime and chat layer. Per-token uniform, sample, prefill-id, and
verify-id staging use reused typed arrays; full-logits readback and exact top-k scratch buffers are allocated only
when needed.

## Chat app

`app/compact.js` keeps the full display history but summarizes older turns before building a prompt. Compaction
triggers at 28,672 prompt tokens (or when fewer than 2,048 tokens remain), keeps the latest 12,288 tokens
verbatim, targets 20,480 tokens, and caps the summary at 2,048 tokens. Long transcripts are summarized in
8,000-token chunks and merged. If summarization fails, the oldest messages are dropped and a fixed notice is
used, so sending still succeeds. The model's `maxNewTokens` is then the real remaining context.

## Packaging and verification

```bash
python tools/pack_weights.py --onnx /path/to/model_q4f16.onnx --out runtime
python tools/verify_pack.py --runtime runtime
```

The packer reads the ONNX graph without loading external data, copies tensor bytes exactly, aligns files and
tensors to 256 bytes, verifies q8 symmetric zero points, and writes the manifest and pack report. The verifier
checks file sizes, SHA-256 hashes, alignment and tensor ranges; with `--onnx` it also checks that every manifest
source tensor exists.

Browser verification hooks:

- `?harness` exposes `window.__harness.run(task)` for generation and logits parity checks, plus `memory()`.
- `?bench=N&prompts=short,long&runs=N` runs greedy speed tests and prints `window.__bench`.
- `?model=<url>` points the app at another bundle; `?dtype=q4f16` is the only accepted dtype.
- `?autotune` measures rows-per-workgroup, prefill M tile, and split caps, then caches the winner in `localStorage`.
