Skip to content

Investigations

Write-ups of work that was done to answer a question and where the answer wasn't a code change that lives in the repo. Specifically:

  • Optimizations that were considered, measured, and dropped (no signal / not worth the complexity).
  • Designs that were prototyped and rejected.
  • "I think we should do X" proposals where the analysis ruled out X (or scoped X to "later, when Y is true").

The point is to save the next person's time. If you find yourself reaching for the same idea, the entry tells you:

  1. It was already considered.
  2. What measurement / argument shut it down.
  3. Under what conditions it might become worth re-opening.

If you re-investigate anyway, update the entry with the new data — don't open a parallel doc.

What does NOT go here

  • Active bugs / unresolved problems → KNOWN_ISSUES.md (local) or a GitHub issue.
  • Problems with a known fix or workaround → docs/troubleshooting/.
  • Designs that were prototyped and shipped → the design doc lives with the subsystem (e.g. docs/dfx/<feature>.md).
  • Architectural decisions that constrain future code → if/when we adopt ADRs, those would live elsewhere; this folder is for things we didn't do.

File naming

YYYY-MM-<short-slug>.md — e.g. 2026-06-chip-swimlane-defer-wmb.md. The date is the month the investigation was done so entries sort chronologically and stale ones are easy to spot.

Template

# <Title — what was proposed, in one line>

**Date**: YYYY-MM-DD
**Verdict**: dropped / deferred-pending-X / superseded-by-Y

## Question

Brief statement of the proposal. Why it might be a good idea — the
intuition that would make a future contributor reach for the same
change.

## What was tried

Concrete actions. Commands, files touched, measurement setup. Enough
that someone can reproduce the measurement, not enough to retell the
whole codebase.

## Result

The numbers, the diff size, the bug found — whatever the actual output
of the investigation was.

## Why not (now)

The decision. Tie it to a specific signal in the result, not just
preference.

## When to reconsider

The condition under which this becomes worth re-opening. "If workload X
shows >Y µs in profile" / "after Z lands" / "if hardware changes such
that ...".

## References

- PRs, commits, issue links.
- Related rules (`.claude/rules/...`) or docs that informed the
  decision.

Index

Newest first.

  • 2026-10 — A5 host_build_graph resident scheduler: per-task argument capture — design only; the capability is deferred to its own change, and the runtime now says so rather than collecting nothing: args_dump_capture_supported_impl answers 0 for a5 host_build_graph and simpler_prepare_run refuses an enable_dump_args request with PTO_RUNTIME_ERR_UNSUPPORTED before the bind. The resident route's on-core schedulers both dispatch and complete, so no AICPU thread owns a task boundary and the two instants it does own are the run's — where a consumer's input reads its producer's buffer from before the producer ran, and several tasks writing one buffer all report the last write as their output. The storage side is already solved and bounded (256 KiB of metadata buffers plus a 128 MiB payload arena per dump lane, with per-payload host acknowledgement, truncation accounting and PARTIAL/HYBRID/FULL selection); what is missing is a producer that can call the dump at the boundary. Specifies a task-boundary rendezvous: all-active-participant BEFORE/AFTER barriers keyed on the graph's own active_mask so no subtask of a mix task executes before its input capture is sealed and no completion is published before every peer's writes are in (first-requester-wins is withdrawn as incorrect on both counts); a slot gate that is one 64-bit word carrying a state and a per-slot occupancy counter and no identity at all — the full run_epoch and TaskId travel untruncated in the 64-byte request and acknowledgement lines — claimable only from FREE or a SETTLED other occupancy, so a core arriving at CLAIMING or READY can never become a second claimer and re-clear the masks; a single access discipline in which every slot access after initialization is a 64-bit atomic or a device-direct store and the line is never written back or invalidated as a line, so no stale cached word can land on a peer's atomic update; per-phase released masks whose completion is the proof that every participant left and the only writer of SETTLED; each phase's own last arriver publishing its core id and sequence where every peer reads which acknowledgement line to poll and which sequence to require; one writer per cache line in each direction with (run_epoch, sequence) on the acknowledgement against stale seals; output publication by dcci(CACHELINE_OUT) over the written spans rather than by pipe_barrier alone. The second revision's "one argument per service step" is withdrawn too: dump_arg_record spins inside ensure_dump_arena_capacity and inside the metadata switch, and one argument can be the arena clamp (arena_size / 2 = 64 MiB) or a per-element strided gather — so the design names a resumable try-begin / copy-chunk / commit / abandon producer surface, a chunk as the service quantum, and a would-block outcome that is retried without being accounted as a refused record. One open reservation per lane, served FIFO by arrival: DumpMetaBuffer::count is both the allocation cursor and the publication watermark today, so two open reservations let a committer publish a peer's unwritten record — with one, the open record is always the one at count, an abandon rolls the arena offset back to where it was, and a metadata buffer is never published while a reservation is open. Prices worst-case FULL (≈96 MiB against a 128 MiB arena on the chained workload) rather than capping it, replaces three conflicting stop paths with one — a waiter flags the abort with the atomic that is its own publication and keeps polling, the supervisor settles the chunk in flight, abandons a reservation only if one is open, and acknowledges; participants leave only on that acknowledgement and go to run shutdown, and the occupancy is never re-claimed — because the barrier, not allocation lifetime, is what stops a successor task overwriting the bytes a chunk is copying; one immutable phase_seq allocated once by the phase's last arriver, separate from the request_tick its own progress loop advances, so a peer tests only identity plus a terminal Sealed/Aborted status and can neither accept partial progress nor poll forever on a superseded sequence; and observation-order service stated as what it is — true publication order across independent cores is not recoverable without a shared ticket, so the guarantee is that a core is overtaken at most once by a later publisher; counts the real budget at 18 KiB of GM plus ~2.2 KiB of AICPU state, the one-per-lane reservation included (correcting the 6.75 KiB the first draft promised from the request array alone), compares AICore-side staging and shows it is a larger ownership change rather than a cheaper variant, and names the four-row contract delta, one transition table with its adversarial and worked walks — including a two-argument multi-chunk phase with a would-block retry and an abort after a committed argument — and eleven acceptance conditions, including mixed subtasks, concurrent clusters, arena pressure, bounded service and diagnostics-off
  • 2026-09 — Reordering the scheduler loop's graph control pass ahead of the sync_start drain check — dropped; superseded by releasing a Graph shell's roots on the completion path: #2256's deadlock has an obvious fix — move the graph control block ahead of the Phase 2 drain check, a ~26-line block move that clears the hang for a bounded and small added drain latency (per retry, at most one thread pays a 4-task materialization slice; activation is once per graph, not once per retry). It was rejected because it leaves the load-bearing invariant — every path that can free a core another task waits on must sit before the drain check — held by nothing but the order of two blocks plus a comment, with no compile-time or test signal, and the same livelock returns as a zero-diagnostic hang the next time a control pass lands below the check. The archaeology says why it is likely rather than hypothetical: the check's position is inherited from a loop where the code after it was only dispatch, so "skip the rest" was exactly the drain's contract, and #1444 broke that equivalence silently while deleting the sentence that had stated it. Underneath sits an asymmetry — a shell's roots are staged through the early-dispatch publish chain like any candidate, but were released through a scheduler-loop queue, where every other release happens inline on the completion path. The shipped fix removes the asymmetry instead: push_ready_routed's GRAPH branch calls signal_graph_external_ready inline, so the scheduler loop stops being a release path at all. Reconsider if the resolution thread's doorbell loop becomes a measured bottleneck — and then with an explicit sync_start_pending guard, not a placement convention.

  • 2026-09 — The Runtime descriptor's input-visibility chain: what the code expresses and what remains unestablished — historical source audit, with a 2026-10-26 coherence clarification for #2745. At the named baselines, A5 HBG descriptor maintenance differed from A5 TMR; the host-input premise alone did not settle the clean's outbound role. The owner has since confirmed that A5 GM requires no AICPU-side clean/invalidate and that completed AICPU writes are visible to consumers. A5 descriptor teardown therefore retains required completion ordering without cache maintenance. The platform build selects empty A5/sim cache implementations; callers express protocol ordering separately. a2a3 Host DMA and conservative SDMA maintenance, including their completion barriers, remain. This is a project architecture premise, not new silicon evidence. The original per-field and first/last-access audits, withdrawn claims and independent ordering, ownership, atomicity and exceptional-lifetime limits remain in the entry at their frozen baselines. Coherence does not authorize concurrent reuse or remove AICore publication.

  • 2026-09 — Narrowing the AICore FIN-ordering dsb to tasks that have outputs — measured & dropped: #2324's dsb(DSB_DDR) sits on two per-task AICore paths and is required for correctness (#2233), which invites gating it on the task having outputs. It costs ~14 ns per task (95% CI ~2–26 ns), so up to two barriers per task put a single dsb at order 7–14 ns — cheaper than the branch that would skip it. The prior said otherwise: 2026-07-aicore-swimlane-switch-overhead-and-ack-gate.md's ~0.5 µs figure would have made the barrier 3% of sliding_window_deps, and it is off by roughly 50× for this site, because a dsb costs what it has to drain and a returning kernel leaves far less outstanding than a record write-back. The drift control is the whole result. On A-vs-B alone two of nine cases looked real — qwen3 at +33.9 ±10.5 µs (t = 3.11) and pa_unroll_manual_scope C2 at +4.9 ±5.5 — and a repeat of arm A run after B accounts for both: two runs of a byte-identical b9ba5d90 differ by −30.4 ±11.2 on qwen3, the same magnitude in the opposite direction, and C2's same-binary delta (+5.6) exceeds its cross-arm one. Only sliding_window_deps Dense16 survives (+14.0 ±12.3 across arms, −0.4 ±16.0 same-binary), and only because sliding_window_orch.cpp:76-103 submits one AIV task per step in a strict i → i-1 chain, putting 1001 tasks on the critical path — the other eight cases are near-chains and cannot resolve the effect at all. Two traps each produced a wrong answer first: build_runtimes does not track this header (#2331), so without rm -rf build/cache/a2a3/onboard both arms run the same binary and report no difference — that is how an earlier attempt measured a fake 18/20-vs-17/20 null on #2233 itself, and the md5 column is the positive control; and the box is shared, at 13/16 devices during A and B against 9/16 during A2, so the same-arm repeat is load-bearing rather than decorative. Carry-forward beyond the one decision: this corpus cannot resolve a per-task cost below roughly 10 ns × critical-path length, so a future per-task micro-cost question should go straight to a long-chain case with a same-binary repeat and should not expect an answer from the production workloads. a2a3 only — the identical a5 change is unmeasured

  • 2026-09 — What the per-run Runtime H2D reductions bought in prepare_execution — inconclusive: after #2297 removed a transfer per run and #2309 cut host_build_graph's per-run image 51,584 → 17,536 B, #2307's new chip.run.prepare_execution span makes the obvious next question "what did that buy". This length A/B did not identify a stable effect on total prepare time, and it constrains neither the real benefit nor the cost mechanism. Four batches (ABBA, 400 rounds each) at fixed HEAD flipping only runtime_device_copy_size(): batch medians 75.7 / 59.1 µs (old) against 77.1 / 65.0 µs (new), so the difference between arms is smaller than the spread between two batches of the same arm, in both arms. The arms also ran under different recorded occupancy — 6/16 devices during one pair (four held by a concurrent session of the same user) against 2/16 during another batch — so environmental confounding cannot be excluded; and B1's lag-1 autocorrelation of +0.815 against ~0.2 elsewhere shows strong temporal dependence whose source was not identified (contention is one candidate, thermal/frequency drift and the workload's own state evolution are others). The entry's main value is the analysis error it records. The first version pooled all 796 invocations per arm and bootstrapped to [−4.5, +3.6] µs, then used that to exclude an ~11 µs effect, conclude that fixed per-call rtMemcpy cost dominates the byte cost, and from there that #2297 was the better latency bet. All withdrawn: pooling invocations is pseudo-replication, because whatever moves between batches is shared by every invocation inside one, and #2297 was never an arm of this experiment. Note what is not claimed in the correction — aggregating to one number per batch makes the batch a sensible reporting unit without making it an independent replicate, so no interval on the true benefit is offered, and how far the original understated uncertainty is not quantified (the two statistics differ, so comparing their widths measures nothing). Two further corrections to the original framing: a controlled A/B on the total span is a valid way to measure overall benefit — sub-spans attribute a measured effect rather than enable measuring one, and cannot substitute for control — and the ~601 µs cold versus ~69 µs warm (~8.7×) first invocation is a cold/warm difference only, with the split across allocation, initialization and first upload unmeasured. What a conclusive version needs is experimental control — interleaved arms on an idle box, several batches each, batch-level analysis — not finer instrumentation. Nothing here retracts either PR: both reported removed allocations and transfers — counted, not timed — and explicitly declined a latency claim

  • 2026-09 — Rebuilding the orchestrator's host views every run instead of caching them per allocation — measured & dropped: the per-allocation view cache #2205 prescribes is needed, and the reason is a term #1848 could not see. #1848 read halHostRegister/halHostUnregister as essentially all per-byte (~6.7 ms/GiB) by inferring from the size asymmetry of one aggregate point, and its own text says the per-call split was never measured. Sweeping size at fixed call count finds a ~5.2 µs floor per register/unregister pair that is flat from 64 B to 4 KiB, alongside a 7.0 ms/GiB slope above ~1 MiB that does confirm #1848 for large buffers — both readings are right in their own regime. Register is per mapped page, and the page size is the allocation's, so 256 KiB on ordinary pages (64 pages, 11.34 µs) costs more than 4 MiB on huge pages (2 pages, 3.43 µs); that non-monotonicity is the mechanism, not noise, and it is #1531's page-configuration sensitivity again. A realistic paged_attention bind reads two control tensors for 24 µs/run, ~4% of a 0.6 ms warm chip.run, against a decision rule of 10 µs fixed before the number was seen. Two details beyond the ratio: the floor means cost tracks the number of accessed tensors rather than their size, so a 32-byte control tensor still costs 5.2 µs; and the slope makes #2205's stated trigger a cliff — rebuild would charge 1.77 ms/run for a 256 MiB orchestrator-accessed tensor, three times the whole run. Corrected 2026-09-14: the fallback's per-access cost was quoted from an estimate at ~1.5 µs and is measured at ~9.5 µs — a synchronous driver round trip, flat from 4 to 64 bytes. That moves the access-count crossover rather than deleting it: the pair is per mapped page, so against the 5.21 µs floor (~1 KiB) mapping wins from the first access, while against a 256 KiB block_table's 18.76 µs a single read is cheaper and mapping wins from the second. The crossover therefore sits at one-to-two accesses across this size range, and SVM-always holds because that is so low — the least-read tensor in the corpus is written 2–4 times — not because no crossover exists. Note the 9.5 µs is measured on a2a3 with 4 KiB pages, which is neither configuration that takes the fallback; a5 and 64 KiB-page hosts are unmeasured. The conclusion it supported (SVM-always) survives and is stronger than its original reasoning; the worst case gets worse, with block_table's 16,384 reads at ~156 ms rather than the ~16–33 ms estimated in #2205. The three arguments for rebuilding that hold regardless of cost — invalidation safety, the accessed set changing between runs, and pinned page-table entries — survive as design constraints on the cache rather than as reasons to skip it

  • 2026-09 — Widening the bounded WAIT-reduction bitmap past BL=64 — BL=64 shipped (#2009 / issue #1376), BL=128 and BL=256 deferred, and the larger result: BL=64 is end-to-end neutral on both production workloads — but amended 2026-09-07, that is a property of those graphs, not of the mechanism: both are near-chains (per-task fanin degree ≤ 1.5 across the whole benchmark corpus, and two of its cases have zero redundant edges), so there is almost nothing for reduction to remove. Under a degree-16 graph (sliding_window_deps, 15,865 WAIT edges of which BL=64 removes 100%) the same code is −20.47% Orch and −2.59% Effective, and Orch falls rather than rises because wire_fanin_task is charged to that span — ~26 ns saved per removed edge against a per-submit bitmap cost an order of magnitude smaller. Widening only moves one of the two production workloads — Qwen3-14B decode removes 1 of 40 redundant edges at every window because 39 of them are cross-ring long edges outside BL=256 too, while DeepSeek-V4 FLASH decode goes 10,065 → 20,214 of 21,698 (2.0x) — against WaitReachEntry growing 16 → 40 B per slot (1 → 2.5 MiB) and a 1-word shift-merge becoming 4-word on the AICPU submit path for every task. But the same campaign shows the removals buy no time to begin with: 10 pinned-die rounds put every row inside the ±1.1% run-to-run band, which is the expected result because bounded reduction preserves WAIT reachability exactly, so no task's earliest start time moves. Instrumentation says why so little is left to make cheaper — ~90% of the 9,495 cleared edges point at producers already CHIP_TASK_COMPLETED at wiring, which take the completed_fanin branch and never allocate a dep-pool entry, so the real saving is 990 of 19,114 entries (−5.2%) and −4.2% peak occupancy, not on the critical path of a ~30 ms AICore-bound step. Consequences worth carrying forward: the simulator's estimated_dep_pool_entries_removed is an edge-count upper bound that overstates the runtime saving ~10x (its edge counts are good — drop predicted exactly, total within 6%); and two DeepSeek-V4 aggregations produce wrong answers — keeping decode step 0 reports −37.8% from a single 7x launch-skew sample whose spike sits entirely in Sched, and the first step's slow rank cannot separate the arms at all (rank-sum U=5 against a critical value of 1), so only the fast rank is a clean first-step signal

  • 2026-09 — Sizing the host-log queue: the claim budget is not the constraint — measured & dropped: kProducerClaimAttempts = 1024 is not the knob it looks like. A producer wins its MPSC slot on the first attempt 57–78% of the time and the worst count across every shape tried is 86, so the bound has ~12× headroom; a bound of 16 would have turned 787 successful writes (0.06%) at 64 threads into drops for a worst-case CPU saving of ~1.6 µs against ~100 µs that never occurs. The constraint on loss is kQueueCapacity — the writer's drain rate — and every loss lands in queue_full at 4, 16 and 64 threads. Structural, not incidental: a full queue exits on difference < 0 before spending any attempt, so queue_full and claim_exhausted are nearly mutually exclusive. Records the first wrong conclusion too — a saturating benchmark measures the early-exit path and barely visits the claim loop, so the claim_exhausted == 0 it reports says nothing about the attempt distribution below the bound; pacing the producers is what puts the claim path under test. Also records two properties of this implementation that item 6 does not require and nothing tests: head-of-line blocking (a producer preempted between claiming position P and publishing it parks the writer on P, so later published records cannot drain and other producers start dropping — the writer sleeping rather than spinning is tested, the queue not backing up is not) and fairness (the losses fall on the slowest producers, the opposite of the useful bias for diagnostics). The 45–94% drop rates quoted come from deliberately pathological unpaced workloads and are not representative; what they establish is that the queue absorbs bursts, not sustained overload
  • 2026-09 — Porting the a2a3 AICore retirement protocol to a5: what was dropped on the way — ported (#2388): one a2a3 mechanism has no a5 counterpart, and the GM return gate is ported anyway. a5 has no fast-path window control anywhere in its software — no offset, no RegId, no caller — so there is no separately defined FAST_PATH close to copy; its dispatch/window close is the only window control. The earlier reading that this also made a GM return gate pointless is withdrawn: the gate independently orders the worker's return after the AICPU's last IDLE write, read-back and drain, so the port keeps isolated per-core gates, reset before window-open and released only to acknowledged cores. The HBG resident and legacy paths now use the same post-close gate. Historical measurements below describe earlier candidates, not the current per-core gated implementation: the claim is the entire cost (per core -12.35 us against -0.40 us for everything else combined, the sweep's broadcast paying for the read-back), and it is thread scatter proportional to how often the claim is taken: private cache lines recover half, relaxed ordering recovers nothing, one claim per owning thread takes 8.24 us to 0.98 us. Records two close micro-optimizations that measured nothing (store-burst then read-back-burst; closing on ACK instead of deferring), and four alternatives ruled out on architecture (a dsb for the read-back, one read-back for the group, reading back COND, and a shared deadline with a blocking wait). Supersedes the "dropped, no net gain" verdict proposed in #2300, which undercounted the portable mechanisms at three, treated the read-back as tied to fast-path close, and measured a shape using the per-core claim.

  • 2026-08 — hbg: rt_set_tensor_data's consumer-wait cannot observe device progress — fixed, found while retiring host-orchestration leftovers (#2068). wait_for_tensor_ready(wait_for_consumers=true) spun on the host mirror's completed_watermark, which only the device advances — in its own copy — so the comparison was -1 < last_consumer_local_id, unconditionally true, and the wait could only end at the 15 s TENSOR_DATA_TIMEOUT_MS. No subset worked: a producer completed inline on the host was not an exception, because the seed is the task's own id rather than -1, so even a producer with no consumers stalled (an earlier draft of #2068's description claimed that exception — it is wrong). The producer half was different and did have a working case, since it reads task_state, which alloc_tensors sets host-side. Left open at the time because two readings fit the code and implied opposite fixes — delete the wait as meaningless under host orchestration, or add the D2H it is missing. Resolved by the first: both halves are deleted and scalar access now rejects a tensor with a producer (owner_task_id or an overlapping TensorMap entry) with INVALID_ARGS at the call. The producer half went the same way rather than keeping its one working case, since a runtime allocation's buffer is uninitialized and has no host view either. last_consumer_local_id and completed_watermark existed only for that wait and are gone with it, which also takes update_completed_watermark()'s per-completion CAS prefix walk out of the scheduler's completion path

  • 2026-08 — The host-orchestration phase tail is page faults, not the code in the phase — root cause of the two shapes on every hbg host swimlane: 447 of the 449 record_node (now record_sub_task) calls above 10 µs took a minor fault, 19% of calls carry 79% of the phase, and a fault costs 14–33 µs here against 1.7 µs off-tree because the process's own mmap/munmap holds mmap_lock against every faulting thread — three 64 MiB unmaps in an off-tree reproducer recreate the whole distribution. The fault count is deterministic (1063/1065/1168 per two orchestrations) and drops to 29 when glibc keeps freed memory; the cost per fault varies 2.4× between runs of the same binary, which is the measurement noise that hid this for fourteen iterations. Refutes THP (PR_SET_THP_DISABLE leaves the count unchanged), preemption, and node shape; records why the tunables are not a fix (args regresses, and the probe build's −53% is the probe amplifying its own subject). Amended 2026-08-23: recording the Definition image into the retained upload staging (8 × ~126 KB per dsv4 bind, previously a vector per recording) moved graph_upload's faults 38 → 1 per bind but left host_orch's count unresolvable in both directions, because a freed 126 KB block is reused without re-faulting — size against glibc's mmap and trim thresholds, not byte count, decides what shows up in this tail. Amended 2026-08-25: retaining the 82 MB SM mirror on the runner (one buffer per pipeline slot instead of one per bind) removes an mmap + munmap of that size per bind — hblkhd stops returning to its pre-bind value on 6 of 6 binds, in both arms of two interleaved repetitions — and shows that a retained buffer must be handed over uninitialized: the first implementation used std::vector::resize, whose value-initialization faulted in all 20132 pages of the window on each rank's cold bind (~20k minflt against ~1100) and left the whole 82 MB resident. host_orch's warm-bind fault count resolves in neither direction (base [1160, 1256] over eight binds, retained [181, 1268]), since the mirror is ~6 THP faults of a ~1200-fault bind; an earlier attribution of a warm-bind rise to glibc's dynamic mmap threshold is retracted there. That amendment also closes the entry's "Where a fix would go" list — items 1–3 shipped as #1981, item 4 as #1988 plus #2013, and item 3's flat-region form as #2015 — and records that none of them reached the ~1100 faults the submitting thread takes per bind, which is what is left. Amended again the same day: the claim in that amendment that pre-sizing the recorder's node and tensor storage would only help a cold bind is retracted — a slot-creation counter shows warm dsv4 binds still creating 1336 node slots of 1679, because #1981's retention is per thread while the pool hands bodies out through one shared FIFO. Reserving each node's own buffer to the cap makes it exactly one page and is worse than main (minflt 1070 → 2540); packing every body's tensors into one never-grown bump region is what helps (record_node warm min 1702/3423 → 1239/1563 µs). Amended 2026-08-25 (last): the count × price framing this entry opened with is refuted by two arms pointing opposite ways — glibc keeping freed memory removes 86% of the faults and buys no time, while #2015 removes 6% and buys 29–43%. The count is not a lever; only the price is, and the in-tree mmap_lock writer that sets it is mprotect, which glibc uses to open a non-main arena (26 calls per bind in that band; those arenas only grow, MADV_DONTNEED there is 0). Every earlier strace here traced madvise/mmap/munmap/brk and not mprotect, which is why the in-tree source of the exclusion went unfound for three rounds. Consequence: the residual ~1100 resident-page re-faults per bind — survived #1981, #1988, #2013, #2015, mechanism undetermined — showed no measurable latency change when the tunable arm removed 86% of them, so they are not currently established as a performance defect; userspace tools are exhausted (mincore and pagemap both report presence, not writability), so pricing them at all needs bpftrace on handle_mm_fault. Corrected 2026-08-26: they are a warm-up cost and they end — at f40cacf30, host_orch's per-bind minflt runs 989, 983 (cold), then 114, 173, 54, 3, 13, 13, 11, 8, and reaches 0 by the sixth bind on three independent runs. Every per-bind count in this entry came from three-to-six-round runs divided by the bind count, so each averaged two cold binds and three or four still-decaying ones under a steady-state label; the quantity being divided was never per-bind. Nothing "survived" the four changes, and the mechanism left undetermined for three rounds turned out not to need determining. Control plane at that commit: 0.529 ms min / 0.570 median over 8 warm binds (host_orch 0.341/0.364). The reusable half of this — that dropping the cold bind does not reach the steady state — is now a trap in hbg-bind-phases.md. Also carries the per-site fault table (two of its top three sites were deleted by #2019 and #2015 within days — keep the method, not the numbers) and the three tooling traps that each produced a wrong conclusion first

  • 2026-08 — hbg: per-block Graph Definitions and cross-layer reuse — adopted at seven Definitions covering all 43 layers, after #1929 replaced the single recording slot the first attempt measured against (it demoted a Graph whose key differed from the in-flight recording's: 79 of 82 intended submissions recorded, host tasks rose 1131 → 1486). Now host submissions 1131 → 129, host_orch −44% and sm_h2d −85%, but graph_upload +207% for seven images instead of one, so the control plane nets −17% at the per-phase floor and nothing at the median — a predictable cost traded for a lower floor and a 133%-wide spread that depends on seven recording threads getting CPU. Keeps the structural map that made the reuse provable (367 kernels → 169 classes / 132 by code alone; which blocks can share a Definition and why the hash-routed MoE cannot), the two indices and the last layer's different hc_post destination, and the arithmetic that a recorded node costs about what a submitted task costs so break-even sits near three occurrences
  • 2026-08 — hbg: uploading Graph Definitions once as shared device objects — cut the per-replay 130 KB Definition re-serialization (image build 931→24 µs, orch total −54%), and confirmed the H2D stage is latency- not bandwidth-bound. Amended 2026-08-18: a --rounds 3 split shows 12.19 of the residual 12.9 ms is the one-time rtMalloc+memset of 40 execution-storage blocks (~53 MB), not per-call latency — real per-call is ~17 µs, so batching the reference submissions is worth ≤0.6 ms, and 88% of this change's own cold-start gain (−1.879 ms in that split) came from execution storage shrinking rather than from the byte reduction it targeted. Amended 2026-08-25: the one-time verify gate this change introduced no longer hashes anything — content_hash, verify_state and the whole-image zero-fill are removed, since graph_definition_array plus bind_graph_topology already bound every device-side read, so the three ~1 MB passes per dsv4 bind bought no safety the structural checks did not
  • 2026-07 — Why qwen3_14b_decode held a device for 406 s, and the four answers that were wrong — root cause was torch thread oversubscription in goldens, not any of the big things: the reference walks 3584 tiny slice ops per layer and torch sizes its pool from the core count, so a 320-core host paid 6.35 s/layer against 1.05 s at 4. Capping to 8 (#1601) cut the golden 359 s → ~40 s and let the case rejoin the sweep. Records the measured decomposition (compile 59 s / fixture 13 s / golden 359 s / device tens of ms), how to split queue wait from card-held work via the npu-lock timestamps, and why golden caching, vectorisation, a nightly split and skip_golden were each dropped
  • 2026-07 — sync_start drain retry ABA across reusable barrier state — reproduced #1455 deterministically by suspending an old-attempt scheduler between the ack barrier and election; retained a generation-tagged O(log N) tree with fixed thread-0 coordination after packed-atomic, no-root-broadcast, and rotating-coordinator experiments; removed the invasive runtime hook after hook-free UT/ST coverage was established
  • 2026-07 — Host worker dispatch latency: where the remaining ~50 µs goes — measured & dropped: after #1499 a Worker.run() costs 43.3 µs fixed + 8.08 µs per task, so the "94% of latency is our runtime, not the IPC" headline is an artifact of benchmarking a single task — at 256 tasks/run the fixed cost is 2%. Also settles #1498 against itself for latency: a pipe wake measures 4.4 µs one way against a 3.5 µs bare fork+shm round trip, so blocking adds more than the whole IPC floor to every dispatch; only the CPU-while-idle argument survives
  • 2026-07 — Containing A2/A3 SDMA stream teardown after an AICore fault — CANN exposes no remote-channel retirement fence: even device-confirmed stream frees followed by a soft-reset success that did not complete context teardown can add minutes to a later ordinary fault. Contained by making SDMA an explicit per-Worker opt-in (enable_sdma) so only opt-in Workers carry the slow teardown and ordinary Workers keep fast recovery; full closure needs CANN runtime + driver changes (#1425)
  • 2026-07 — AICore-side arg fill for ALL dispatches (not just the gated path) — measured & rejected for the ready path: making the AICore fill its own args[] for every task adds ~1.0 µs to each task's receive→start setup (paged_attention_unroll: 349 ns → 1356 ns), because a ready task has no idle doorbell gate to hide the fill. Offload is a win only on not_ready (early-dispatch), where the AICore already spins at the gate; shipped design keeps the AICPU filling ready tasks (#1328)
  • 2026-07 — chip swimlane AICore: switch-overhead source + FIN-early reorder & ACK-gate — measured: the ~0.8 µs inter-task switch is the record write-back dcci(record,OUT)+dsb (~0.5 µs) + payload setup (~0.28 µs), inherent and not reducible by moving FIN; the WAIT gap (p99 ~700 µs) dominates decode. Shipped: sample end_time after an early FIN, and an AICPU ACK-gate on buffer rotation (release the old buffer only when AICore ACKs the new buffer's first task) to close the FIN-before-record boundary race the reorder introduced
  • 2026-07 — Removing LocalReadyBuffer exposed a missing dcci in EP dispatch — RESOLVED in #1245: local-buffer removal changed dispatch timing and unmasked a latent kernel bug (dispatch never dcci'd recv_count_out to HBM → local_expert read count=0 → all-zero output); fixed with a one-line dcci in the example kernel
  • 2026-06 — Gating the two residual profiling enable() calls on the orch/scheduler hot path — gated under existing SIMPLER_DFX; magnitude unmeasured, no new macro
  • 2026-06 — Replacing COND with GM+dcci for AICore→AICPU notification
  • 2026-06 — Letting AICore directly read or write the SPR MMIO window
  • 2026-06 — PA-unroll 207001: an op-timeout-window issue fixed by #1035, not a launch-order bug
  • 2026-06 — Cross-task batched publish: hoist wmb across distinct tasks in one pop — also carries the root cause + fix for the spmd_sync_start_stress 507018 drain-barrier hang
  • 2026-06 — AICore first-task cold-start: pre-warm dispatch path
  • 2026-06 — a5 AICore op-timeout poisons the shared L2 worker (cascade)
  • 2026-06 — a5 AICPU filter gate: Scenario B fail-fast guard not added
  • 2026-06 — Sanitizer rollout scope: macOS, TSAN gating, LSan
  • 2026-06 — chip swimlane: defer per-task wmb to rotation