The Runtime descriptor's input-visibility chain: what the code expresses and what remains unestablished¶
Date: 2026-09-19
Historical verdict: host↔device coherence supplied as a project premise; the end-of-run
clean is audited per field — excluded as publisher for three of the four device-writer
groups with a named dependency each, and, since a second owner premise answered the
AICPU→AICore edge for the gate's access shape, excluded for the a2a3 teardown gate too;
the remaining ordering, atomicity, ownership and lifetime obligations stay open — no
defect established, no cache operation changed, no contract proposed or proved
Revision: the first version of this entry shipped in #2368. Static review found
that several of its conclusions promoted source control flow, or observations that
do not discriminate, into hardware guarantees. Those conclusions are withdrawn in
place below — each marked Withdrawn from the first version — rather than
deleted, so a reader arriving from #2368 can see which claim was retracted and
why. The #2368 and #2369 revisions share the 31da0560e baseline below; the later
outbound audit section pins its own, newer baseline and its code references and
extent figures belong to that commit alone.
Current disposition (2026-10-26)¶
The owner clarified the architecture contract for #2745: A5 GM accesses require no AICPU-side clean or invalidate, including Host DMA, SDMA and reads of device-written RDMA/URMA completion records (CQEs). Completed AICPU GM writes are visible to consumers without a cache clean. This supplies the outbound premise that the historical A5 audit below left open; it is a project architecture statement, not a vendor citation or a new silicon measurement.
Under that contract, A5 HBG resident and legacy descriptor teardown do not perform descriptor-wide cache maintenance. Required completion barriers, publication ordering, current-run retirement and exclusive reuse remain independent requirements. The platform build selects separate cache implementations: a2a3 retains cache maintenance and its completion barriers; A5 and sim implementations are empty. Shared callers supply protocol ordering with explicit barriers or existing acquire/release operations. A2/A3 Host DMA maintenance and conservative SDMA invalidation remain, as do AICore-side publication operations.
The source walkthroughs and withdrawn conclusions below are retained at their named historical baselines. Their statements that no operation was changed or that the A5 clean's outbound role was unestablished describe those audits, not the current implementation. The broader premise does not resolve their separate ownership, atomicity or exceptional-lifetime questions. See the current cache contract.
Question¶
The per-field inventory on #2254 noted an asymmetry: three of the four onboard
variants call cache_invalidate_range over the whole device descriptor, and a5
tensormap_and_ringbuffer does not. The tempting reading is "a5 TRB is missing
an invalidate."
That reading assumes the operation's only role is making the next invocation's host bytes visible. This entry traces what the code actually expresses, for each variant and each processor, and separates three things the earlier note ran together: what the code does, what that alone establishes, and what it merely expresses an intention about.
The outcome is that the asymmetry is real and is not evidence for any particular change. The host↔device coherence prerequisite is supplied as a project architectural premise (A3 not coherent, A5 coherent — see the premise section below) and is no longer open; the remaining obligations — ordering, publication atomicity, writer ownership, outbound publication, run lifetime and cross-stream dependencies — are not settled by it and stay open. Several statements in the first version of this entry are withdrawn below and marked as such.
Baseline¶
Every anchor was read at 31da0560e202b7bb378773f7dfc1d0074fb28e89
(main, 2026-09-19). Nothing here was built, run, simulated or measured; this is
a reading of source at one commit. No number from another commit is carried
forward as applying to this one.
The chain, stage by stage¶
1. Host source snapshot¶
The host mutates its own Runtime object throughout prepare_execution.
KernelArgsHelper::prepare_runtime_args
(src/common/platform/onboard/host/device_runner_helpers.cpp:80) snapshots exactly
the device-read prefix into host heap storage: RuntimeLaunchImage::prepare
(src/common/platform/include/host/runtime_launch_image.h:33) resizes a
std::vector<std::byte> to runtime_device_copy_size(runtime) and memcpys from
&runtime.
2. Allocation and reuse¶
The destination is not per-run. It is owned by the pipeline slot —
SlotPersistentArgs::runtime_args, one per slot in
RunResourceSlots<SlotPersistentArgs>, allocated lazily on a
slot's first run (device_runner_helpers.cpp:99-107) and released only in
finalize_common_impl. Per-slot rather than per-runner because the copy is not
stream-ordered, so a prepared successor would otherwise overwrite the image its
predecessor is executing against.
So the same device address is written by successive runs on that slot. Whether any device cache line covering it retains bytes from a previous invocation depends on the mapping and on cache behaviour across a kernel boundary, neither of which is established here.
Withdrawn from the first version: "a fresh allocation per run would make most of what follows moot." A new allocation is new ownership; it is not proof of new physical backing or of a cold cache, and this entry elsewhere records both as unknown. The sentence assumed what it was investigating.
What the code does establish about reuse. Two separate predicates, which the first version of this entry ran together:
- Same-slot exclusion is unconditional.
try_reserve_native_run(device_runner_base.cpp:2919) rejects a reservation outright if any live reservation names the samepipeline_slot— or the same owner — before any successor logic runs (:2931). A live reservation on a slot therefore blocks that slot entirely; there is no "unless". - The prepared-successor rule is about the predecessor, and needs a different
slot. The later condition (
:2940) additionally requiresactive == existing->owner, i.e. that the existing reservation is the currently-active run. It says nothing about the caller being active. Combined with the unconditional check above, a prepared successor necessarily occupies a distinct slot.
So same-slot exclusion does not rest on the successor-admission rule, and must not be argued from it.
Same-slot reuse follows completed prior device execution. On the success path
this is a device-execution boundary, not merely host call ordering.
drain_execution calls reap_run, which calls wait_run_fence
(device_runner_base.cpp:2539) before any teardown: when the run is fenced that
waits both run-completion events and then additionally calls
sync_stream_pair on both streams; when no boundary covers the submitted work it
falls back to sync_stream_pair on both streams directly. A non-zero result
returns early into recovery, so no completion is claimed for a failed
synchronization — the concrete contract holds on the success path only.
What that establishes is that the prior run's device execution has completed, and that its reservation is released, before the slot is prepared again. What it does not establish is anything about cache retention or input freshness across the boundary: neither an event boundary nor a stream synchronize is a statement about which lines a device cache still holds, or about when a device-side write-back retires. Execution completion and cache state are different properties, and this entry's open questions are about the second. Nor does this end-of-run boundary bear on the still-unproved startup marker / whole-line hand-off discussed later — that is a different edge of the same run.
3. Prepare/publish transfer¶
RuntimeLaunchImage::publish moves the snapshot into a function-local vector,
clears the member, and hands the local's data()/size() to a copy callback;
KernelArgsHelper::publish_runtime_args (device_runner_helpers.cpp:113) supplies
a synchronous rtMemcpy(..., RT_MEMCPY_HOST_TO_DEVICE).
- The moved-to local lives until
publishreturns, so a synchronous copy reads a valid source. An asynchronous copy would not: the source dies at return. - The snapshot is consumed exactly once; a repeated or reentrant publish is rejected rather than copying twice.
PR #2360 documents this host-side seam, including that the sole production caller
init_runtime_args_with_metadata (device_runner_base.cpp:2782) invokes prepare
and publish consecutively, and that a non-null args.runtime_args does not prove
publication happened. This entry starts where that stops, at the device side.
4. Launch ordering — two streams, no ordering between them¶
The descriptor's device address reaches both processors as a by-value launch argument, so the pointer is fresh per launch; the pointee is device memory.
Publication precedes both launches. The two kernels then go to distinct
streams, AICore submitted first: launch_aicore_kernel(streams.aicore, …)
followed by launch_aicpu_kernel(streams.aicpu, …)
(src/a2a3/platform/onboard/host/device_runner.cpp:664-687; a5 has the same
shape). Nothing orders one stream's execution against the other's.
This is the correction that reshapes step 5: there is no basis for a single chronological "first device read" across both processors. First reads must be given per processor.
5. First descriptor reads, per processor and mode¶
Withdrawn from the first version: "the earliest reader is the platform AICPU kernel entry … on all four variants." That is the earliest traced AICPU reader. It says nothing about the AICore, which is submitted first on another stream.
AICPU side, all variants. The earliest traced AICPU read is the platform
kernel's affinity gate, before aicpu_execute:
| arch | anchor | reads |
|---|---|---|
| a2a3 | src/a2a3/platform/onboard/aicpu/kernel.cpp:65,93 |
get_aicpu_allowed_cpu_count(), get_aicpu_launch_count(), get_aicpu_allowed_cpus() |
| a5 | src/a5/platform/onboard/aicpu/kernel.cpp:82,121 |
same three |
Neither file performs any cache operation. Because the gate decides which threads proceed, a stale read here changes the thread population rather than a value.
On a5 HBG a second AICPU read of the same kind follows: aicpu_execute
(src/a5/runtime/host_build_graph/aicpu/aicpu_executor.cpp:627) branches on
aicore_scheduler_runtime_enabled(runtime), reading the host-published
dev.workers[0].aicpu_ready (aicpu/aicore_scheduler_state.h:27) to choose the
resident or legacy executor.
AICore side. The first descriptor access differs by path, and on a5 HBG it is a read of host-published state that precedes any AICPU permission:
| path | first AICore descriptor access | before AICPU permission? |
|---|---|---|
| a5 HBG (mode select) | scheduler_observe_cache_line(handshake) then handshake->aicpu_ready mode checks (aicore/aicore_executor.cpp:533-535) |
yes — this read selects resident vs legacy |
| a5 HBG resident | then host-published handshake->task as SchedulerWorkerContext*, and that pointee, before READY (:553-561) |
yes |
| a2a3 TRB | write of physical_core_id/core_type/aicore_done to workers[block_idx], then dcci(…, CACHELINE_OUT) (aicore/aicore_executor.cpp:72-83); first explicit read is my_hank->task after window-open (:103-104) |
the first access is a write, not a read |
| a2a3 HBG | same shape (aicore/aicore_executor.cpp:72-83, read at :103-104) |
same |
| a5 TRB | same shape (aicore/aicore_executor.cpp report then window-open read) |
same |
| a5 HBG legacy | entered via the mode read above; thereafter the report-then-window-open shape | mode read yes |
So a5 HBG is the only path whose AICore consumes host-published descriptor content early; on the other paths the AICore's first descriptor access is its own report write, and its first read is the task pointer after window-open.
6. Last descriptor access and cache operations, per path¶
The only descriptor-wide cache operation is
cache_invalidate_range(runtime, sizeof(runtime->dev)), at four sites, all inside
deinit() — the end of a run, not before a read:
| site | variant |
|---|---|
a2a3/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp:1035 |
a2a3 TRB |
a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp:455 |
a2a3 HBG |
a5/runtime/host_build_graph/aicpu/aicpu_executor.cpp:563 |
a5 HBG resident |
a5/runtime/host_build_graph/aicpu/aicpu_legacy_executor.cpp:414 |
a5 HBG legacy |
a5 TRB's deinit takes Runtime * /*runtime*/ unnamed and performs none
(a5/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp:1029).
AICore-side final descriptor access differs per path and must not be pooled:
| path | last descriptor access on the AICore |
|---|---|
| a2a3 TRB | dcci(my_hank, SINGLE_CACHE_LINE, CACHELINE_OUT) (:276), then a read — bypass-load of dev.teardown_gates[block_idx].post_close_release (:280) |
| a2a3 HBG | same shape: dcci (:301), then teardown-gate read (:305) |
| a5 TRB | at baseline dcci(my_hank, SINGLE_CACHE_LINE, CACHELINE_OUT) (:255), with no gate member on this variant; see the post-baseline note below |
| a5 HBG legacy | final dcci(my_hank, SINGLE_CACHE_LINE, CACHELINE_OUT) before return, commented "Flush all dirty cache lines to HBM before kernel exit" (aicore_legacy_executor.cpp, end of function) |
| a5 HBG resident | no final handshake dcci; the function ends at write_reg(RegId::COND, AICORE_EXITED_VALUE) (aicore_executor.cpp:695) |
Post-baseline change (#2388). Everything here is stated at commit
31da0560e, before the a2a3 retirement port. a5 TRB has since gained ateardown_gatesmember and now matches the a2a3 exit shape:dcci(..., CACHELINE_OUT)(aicore_executor.cpp:268) then a bypass load ofdev.teardown_gates[block_idx].post_close_release(:269). Statements below that say a5 TRB "has no gate member" describe the baseline, not current code. The final #2388 revision also gates a5 HBG legacy and resident returns; resident startup/exit watchdog paths wait for the AICPU's actual EXIT before ACK so a reused descriptor's stale RELEASE cannot bypass the close.
The a2a3 paths' last descriptor access is a read placed after their own write-back. At this audit's baseline a5 TRB instead wrote back and returned, but post-baseline (#2388) it takes the a2a3 shape; a5 HBG resident still has no final flush. The resident path's exit must not be described using the legacy path's flush.
What the primitive was at the audit baseline (src/common/platform/onboard/aicpu/cache_ops.cpp:20):
dc civac— clean and invalidate to Point of Coherency, not a pure invalidate. It has two roles: it can make externally written bytes visible to this observer, and it can push this observer's modified bytes outward.- Per 64-byte line over
[addr, addr+size)rounded outward, so it can touch bytes outside the requested range — what thesizeof(DeviceRuntimeLaunchDesc) % 64 == 0static_assert protects. - Followed by
dsb syandisb. - The non-aarch64 branch (
:54) and the whole sim implementation (src/common/platform/sim/aicpu/cache_ops.cpp) are inert, so no simulation run exercises any of this.
The AICPU does store into descriptor lines — workers[i].task
(scheduler_cold_path.cpp) and, on a2a3 and post-baseline a5 TRB,
teardown_gates. Whether any given line is dirty at the dc civac call depends
on the mapping's write policy and on prior maintenance, neither of which is
established here; so the clean half of that call has a possible role in
publishing AICPU-authored bytes, not a demonstrated one. Either way, the call is
not reducible to "invalidate for the next DMA".
Withdrawn from the first version: "the
dc civacis performing a write-back of AICPU-authored bytes." Observed stores do not establish a dirty write-back cache, which unknown 3 says is unestablished.
7. Reuse, and what kernel mode does not do¶
Program mode rewrites the same slot block and repeats.
Kernel mode is scaffolding at this baseline, and its device-read chain is not
delivered. PersistentKernelArgs::prepare_once
(kernel_persistent_args.cpp:24, called once from device_runner_base.cpp:871)
does allocate and H2D the descriptor. But simpler_kernel_mode_supported returns
0 (c_api_shared.cpp:1561) and simpler_kernel_mode_launch
(c_api_shared.cpp:1700) validates its arguments and then unconditionally
returns PTO_RUNTIME_ERR_INVALID_STATE; the section header says so directly —
"Launch remains a rejecting stub, so supported() reports 0."
| kernel-mode stage | status at this baseline |
|---|---|
| allocation + single upload | implemented (prepare_once) |
| repeated launch through affinity gate / executor | not delivered — launch is a rejecting stub |
| repeated-run first/last device read | not delivered, therefore not traced |
| end-of-run maintenance over an unrewritten descriptor | not delivered |
Withdrawn from the first version: the claim that in kernel mode "every subsequent run reads descriptor bytes the host has not rewritten", and the matrix row asserting its first/last reads. No production path repeatedly launches this prepared descriptor, so no repeated-run cache contract can be inferred from the prepare-once helper.
Where the documented rule and the code differ¶
docs/hardware/cache-coherency.md:130-145 states the rule;
docs/hardware/chip-architecture.md:218-219 restates it in the end-to-end flow.
Two differences are plain matters of location and extent, independent of any
hardware question, and are corrected in those files by this change:
- Placement. The rule reads "before reading host-written Runtime" and the
flow puts it at AICPU entry. All four call sites are in
deinit(), at the end of the previous run. - Extent. The rule's snippet is
sizeof(Runtime); the code issizeof(runtime->dev), narrowed by #2309/#2315.
A third difference is not corrected, because settling it needs evidence this entry does not have: the rule says a5 does not call the operation for host-DMA reads, and its decision table says so for a5 generally, yet a5 HBG calls it at two sites while a5 TRB does not. Recorded as an open discrepancy.
cache-coherency.md now carries the verified history of that discrepancy: the call
was removed from both a5 runtimes by #1235, is present again in a5 HBG after the
migration in #1661 whose six enumerated deviations do not mention it, and was carried
into aicpu_legacy_executor.cpp from the point #2090 created that file. A migration
copy is one reading of that sequence, and the provenance note labels it an
inference: the history fixes when the call returned, not why it was kept. History
proves neither a hardware model nor its absence, so this lineage is a maintenance
fact and is independent of the coherence premise recorded above.
The a5 HBG resident handshake line: three writers, and a dependency the code expresses¶
Scope: a5 HBG resident startup. This section is about that path only. It is not a lifetime account of the other paths, whose exits are in step 6.
Handshake is exactly 64 B and 64 B-aligned (src/common/host_build_graph/runtime.h:98-112),
so one worker's handshake is one cache line. On a5 HBG that line is written by all
three tiers:
| writer | words | operation |
|---|---|---|
| host | aicpu_ready = SCHEDULER_RUNTIME_MODE_*, task = worker's SchedulerWorkerContext address |
part of the descriptor H2D (a5/runtime/host_build_graph/host/runtime_maker.cpp:977, :1275-1277) |
| AICore | physical_core_id, core_type, aicore_done |
store, then dcci(…, CACHELINE_OUT), then dsb |
| AICPU | task, aicpu_ready = RESIDENT_READY |
stores, then cache_flush_range = dc cvac over whole lines |
The sequence the code expresses, in order:
- AICore:
scheduler_observe_cache_line(handshake)(aicore_executor.cpp:533), readaicpu_readyto select mode (:534-535). - AICore: store
physical_core_id,core_type;OUT_OF_ORDER_STORE_BARRIER(); storeaicore_done;dcci(handshake, SINGLE_CACHE_LINE, CACHELINE_OUT);dsb(:541-546). - AICore: read host-published
handshake->task, then its pointee, then enter the READY wait, re-observing the line each iteration (:553-592). - AICPU: per worker,
cache_invalidate_range(handshake, sizeof(*handshake))then readaicore_done, spinning until non-zero, then read the reported payload (aicpu/aicore_lifecycle.cpp:110-128). - AICPU: only after report collection and configuration —
publish_context_partition(aicore_lifecycle.cpp:311) storestaskandRESIDENT_READY, thencache_flush_rangeover those lines, thenwmb()(:322-328).
What this establishes: the code expresses a dependency — the AICPU's
publication is placed after it has observed aicore_done from each core. That is
where the intended hand-off is written down, and it is a fact about control flow.
What this does not establish, and what the marker read alone cannot:
- That observing
aicore_doneimplies the producer's whole-line write-back has completed. Thedsbat step 2 orders the producer's own subsequent accesses; it does not make a remote load of the marker an observation that thedcciretired. Marker visibility and line-writeback completion are separate properties, and nothing here ties them. - That the AICPU's clean at step 5 —
cache_flush_range,dc cvac, outbound only — cannot write back an older cached copy of the line, overwriting words another tier published. The AICPU's inbound operation is separate: thecache_invalidate_range(dc civac) in its step 4 poll. Keeping the two apart matters, because the outbound and inbound edges of this line are what the open questions distinguish. - That the AICore's
CACHELINE_OUTat step 2 cannot write back a copy that predates a concurrent write by another tier.
These are precisely the publication and ownership assumptions this entry leaves unresolved elsewhere, so the dependency cannot be promoted into a correctness result using the same document's unknowns.
Withdrawn from the first version: "What keeps it correct is an ordering token", "the three whole-line writers are serialized", and the characterisation of this as a proved invariant ready to publish independently. No currently executing race is established by this finding either — the point is that the code's intent is legible while its hardware preconditions are not recorded.
What is worth carrying forward is narrower and still useful: a single 64-byte line carries words authored by three tiers, each publishing with a whole-line operation, and the only thing sequencing them is a marker dependency whose hardware preconditions are unwritten. Any change that adds a word to this line, alters who writes it, or relaxes the report dependency is changing something whose safety argument does not currently exist in written form.
Assumptions this entry explicitly does not make¶
- That an end-of-run invalidate establishes next-invocation visibility. It
completes before the next launch, but
dsb sy; isbafterdc civacorders the maintenance, not anything against a later host DMA. - That a slot's first run is covered on a non-coherent path. On a slot's first
run no previous
deinitran, so no descriptor-wide maintenance has ever executed for those lines. On A5 the premise removes the need; on A3 it does not, and what covers that first read is open. - That invalidation is an acquire. It is not.
- That stores imply a dirty write-back cache (see step 6).
- That passing CI bounds failure frequency. It is an observation of the exercised cases. Without a defined experiment and sampling basis it is not a quantitative bound, and the first version of this entry wrongly implied one.
Supplied premise: the host↔device coherence model¶
The project owner states the architecture: A3 host/device is not cache-coherent; A5 host/device is. This entry takes that as given and no longer lists it as an open question. It is a project architectural premise — not a vendor specification quoted here, and not something measured for this entry.
A second premise from the same source, added later and scoped to a different edge,
is used in the outbound audit below: on a2a3 an AICPU ordinary GM store is observable by
an AICore ld_dev bypass load of the same address without an AICPU-side clean. It is
recorded in
cache-coherency.md
with the same attribution and the same limits, and the two are not interchangeable: this
one says nothing about the host edge, and the host one says nothing about on-device
readers.
What it settles: the inbound host→AICPU freshness edge per architecture. On A5 a host-published descriptor byte becomes visible to an AICPU read without AICPU-side invalidation. On A3 it does not, so on that path some mechanism must make the host's bytes visible before the reader's first read. Which mechanism, and whether the descriptor call at its current end-of-previous-run placement supplies it, is not established here — the premise states the architecture, not the role of any particular call.
What it does not settle, and what the rest of this entry is about:
- Ordering. Coherence supplies no happens-before between a writer and a reader.
- Atomicity of a multi-field publication. A descriptor or a 64-byte handshake line updated field-by-field is not made a single observable event by coherence.
- Writer ownership. It grants no permission to overwrite a field another agent is live on, which is the whole subject of the three-writer section above.
- The outbound direction.
cache_invalidate_rangeisdc civac, which also cleans; what that clean publishes to on-device readers is a separate obligation. - Anything off the host↔device edge. On-device producers — AICore, SDMA — and the
question of how GM is mapped are not covered. Nor is A2: the owner named A3, the
a2a3tree serves both and takes the conservative path, and that is where this entry leaves it.
It authorises no code change. In particular it does not license deleting any cache operation now in the tree, nor a handshake redesign.
Remaining obligations, with the premise applied¶
| obligation | status with the premise |
|---|---|
| inbound host→AICPU freshness, A5 | supplied — coherent per the premise |
| inbound host→AICPU freshness, A3 | non-coherent per the premise, so maintenance is required there. Whether the existing end-of-previous-run placement discharges it for the next run's first read is a separate question this entry does not answer — nor is any defect claimed; see step 6 for where the calls sit, and note a slot's first run has no prior deinit |
| launch-time cache state | still open, and now only relevant where the premise leaves work to do: it bears on A3's placement question above, not on A5's inbound edge |
| outbound AICPU→AICore publication | audited per field. Excluded as the clean's beneficiary for the AICore-written trio (their own CACHELINE_OUT), for the a5 HBG hand-off (its own dc cvac), for workers[i].task on normal exits (the EXITED acknowledgment), and for a2a3's teardown_gates[i].post_close_release — the owner's AICPU→AICore premise says that store needs no clean to reach an ld_dev reader. What the premise does not supply for any of them is ordering; see the outbound audit below |
| outbound AICPU→host publication | narrowed — the only readers positioned after the clean are host D2H reads of workers[], and the only one on a production path is a LOG_DEBUG dump. See the outbound audit below |
| whole-line / multi-field publication atomicity | still open — see the three-writer section |
| ordering and happens-before between the three writers | still open — the marker dependency and its unproved preconditions |
| allocation and run lifetime, slot reuse | still open as described in steps 2 and 7 |
| cross-stream dependencies (AICore and AICPU launched independently) | still open as described in step 4 |
| GM mapping and write policy | still open, and not answerable from the premise |
Unknowns, and what each would and would not settle¶
The questions below remain unanswerable from this repository. Crucially, none of them alone licenses removing or relocating an existing operation, because the descriptor-wide call has more than one possible role (step 6).
| unknown | if resolved, addresses | does not address |
|---|---|---|
| 1. Does AICPU kernel launch leave the cache cold or perform maintenance? | on A3, whether the end-of-previous-run placement suffices for the next run's first read | whether prior device-authored bytes needed publishing to another observer before that point — a cold next cache says nothing about the previous run's outbound obligations. Moot for A5's inbound edge, which the premise settles |
2. Do the two unflushed AICPU stores reach memory for their readers? workers[i].task (store + OUT_OF_ORDER_STORE_BARRIER()) and teardown_gates[i].post_close_release (relaxed store after rmb()), both read on-device. |
Answered for the gate by the owner's AICPU→AICore premise: an ordinary GM store reaches an ld_dev reader with no clean, so the clean is not its publisher. Still open for task, whose reader takes a different access path |
for task this is separate from the end-of-run clean, which the EXITED acknowledgment places after that read on normal exits. For neither field does the premise supply ordering, atomicity, or anything about a reader left live by an earlier run |
3. Is GM cacheable for the AICPU, and under what write policy? The documented Device-nGnRE attribute covers the MMIO window only. |
whether cached descriptor lines can exist at all, and whether the clean half can have anything to write back | nothing about externally written staleness if the answer is write-through: a write-through cache still holds readable lines, and the write policy alone does not establish that host DMA invalidates or updates them |
Withdrawn from the first version: the claim that an uncached or write-through mapping "dissolves" the question. Write-through concerns writes; it is not a sufficient answer to input freshness. Also withdrawn: the "mutually exclusive candidate fixes" list, which presented each unknown as licensing a specific deletion.
Obligations any future change must account for¶
Rather than candidate fixes, the individual obligations the existing descriptor-wide call could be discharging. A change may only remove or relocate the call once every row it currently covers is accounted for by something:
| obligation | who needs it | currently discharged by |
|---|---|---|
| host-published bytes visible to the AICPU's first read | AICPU affinity gate, executor init, a5 HBG mode select | on A5, the coherence premise. On A3 the premise says maintenance is needed, and nothing in the same invocation supplies it — unknown 1 covers whether the end-of-previous-run placement discharges it |
| host-published bytes visible to the AICore's early reads | a5 HBG resident mode + context reads | the AICore's own scheduler_observe_cache_line. The host↔AICPU premise does not speak to an AICore reader |
| AICPU-authored descriptor bytes made visible outward | AICore readers of workers[i].task and teardown_gates[i]; host D2H diagnostics |
per field, not uniformly. For the trio and the a5 HBG hand-off, their own flush; for task, ordered before the clean by the EXITED acknowledgment on normal exits; for a2a3's teardown gate, by the store itself under the owner's AICPU→AICore premise. See the outbound audit below |
| a slot's first-run reads, AICPU observer, A5 | AICPU affinity gate, executor init | the coherence premise — a first run is not a special case for an edge that needs no maintenance |
| a slot's first-run reads, AICPU observer, A3 | same | nothing in that invocation, and no prior deinit has run for those lines. What the non-coherent path requires there is open, as above |
| a slot's first-run reads, AICore observer | a5 HBG resident mode + context reads | the AICore's own scheduler_observe_cache_line, on every run including the first |
| line-granular exclusivity on the a5 HBG handshake line | host, AICPU, AICore | a marker dependency with unwritten preconditions |
Relocating the call to before the first read would move it into the platform kernel entry ahead of the affinity gate, not into the runtime executors where the four sites sit today — and would not by itself discharge the outward-publication row.
Outbound audit: what the end-of-run clean actually publishes¶
Audit baseline: ad4bf20dbb207bdf7a892e887d943cb6a2cf94a4 — later than the
31da0560e baseline the rest of this entry pins. This section's code references and
its extent figures belong to that later commit and must not be read back onto the
earlier one; where the two differ is stated explicitly below. Nothing was built, run or
measured for it either.
cache_invalidate_range is dc civac, so it cleans as well as invalidates. The
obligations table above previously credited that clean with a possible outbound
publication role. This audit narrows that role field by field, across a2a3/a5 ×
TRB/HBG including a5 host_build_graph resident and legacy.
Six distinct descriptor members are written by the device, in four writer groups
(workers[i].task appears twice, written by two different producers). Everything else
in the descriptor is host-written, so the clean has nothing of theirs to publish.
| writer group | members | its own publication | on-device reader | is the clean excluded as publisher? |
|---|---|---|---|---|
AICore at entry (aicore_executor.cpp, all variants; a5 HBG legacy in aicore_legacy_executor.cpp) |
physical_core_id, core_type, aicore_done |
yes — the AICore's own dcci(…, SINGLE_CACHE_LINE, CACHELINE_OUT) whole-line write-back |
AICPU report poll, during the run | yes, on publication grounds — a different agent already flushes these; the AICPU's clean is not their publisher |
| AICPU scheduler cold path (all four variants) | workers[i].task |
partial — plain stores then OUT_OF_ORDER_STORE_BARRIER(), no clean. Consumer invalidates: AICore dcci(my_hank, SINGLE_CACHE_LINE) before the read |
AICore, after window-open (a2a3 TRB aicore_executor.cpp:104) |
yes, on the EXITED acknowledgment — the AICore writes COND = AICORE_EXITED_VALUE (:128, :215) only after that read, and the AICPU polls COND for EXITED in its quiesce pass before deinit. Not every COND write qualifies: the initial AICORE_IDLE_VALUE report at :97 precedes the read. Normal exits only; see the exceptional paths below |
AICPU a5 HBG resident hand-off (aicore_lifecycle.cpp:322,326) |
workers[i].task, aicpu_ready = RESIDENT_READY |
yes — its own cache_flush_range (dc cvac) over those lines, then wmb() |
AICore READY poll, during the run | yes, on publication grounds — the hand-off flushes them itself |
AICPU a2a3 window close (platform/shared/aicpu/platform_regs.cpp:149) |
teardown_gates[i].post_close_release |
none — a relaxed store after rmb(), with no flush |
AICore wait_for_post_close_release — a bypass load (ld_dev), a2a3 TRB aicore_executor.cpp:280, HBG :305 |
yes, on the owner's AICPU→AICore premise — this store reaches this reader with no clean. See below |
The teardown gate: the clean is not its publisher¶
The store loop at platform_regs.cpp:149 is followed immediately by return rc;. The
AICPU receives no acknowledgment that the AICore consumed the gate, and it has no
other way to observe that read — the AICore's return is visible to the host through
stream completion, not to the AICPU. So the AICPU can proceed from that return through
to deinit and its descriptor-wide dc civac (TRB aicpu_executor.cpp:1037,
HBG :455) while the AICore is still spinning. This interleaving is permitted by the
control flow:
Two arguments that look like they order this, and do not:
EXITEDprecedes the release gate, not the gate read. The quiesce pass waits forCOND == AICORE_EXITED_VALUEbefore closing windows and storing RELEASE. That orders the store after the ACK; it says nothing about when the subsequent GM gate read happens.- A bypass load is not a substitute for publication.
ld_devreads memory rather than the reader's cache, so on the read side it only proves the reader will not see its own stale line.
What settles it is a second owner premise, not either argument above. The project
owner has confirmed that on a2a3 an AICPU ordinary GM store is observable by an AICore
ld_dev bypass load of the same address with no AICPU-side clean
(cache-coherency.md).
The writer therefore needs no flush for this reader, and the clean is not this
field's publisher. Attribution is the same as for the host↔device premise: an
architectural statement from the project, not a vendor citation, not proven from the
public SDK, and not measured here.
The interleaving above is consequently benign rather than resolved-away: the gate read may still fall either side of the clean, and now nothing depends on which side.
Read the premise no wider than it is. It covers an a2a3 AICPU ordinary GM store
observed by an AICore ld_dev load of the same address. It is not the host↔device
premise, does not run in reverse, and does not transfer to a reader that reaches a
field through a different access path — which is why residual 2 below stays open for
workers[i].task. Nor is it ordering: the two edges this gate needs are ordered by
code, and are cited in the cache-coherency.md section above — reset then wmb()
before hs_setup_done_ and window-open (scheduler_cold_path.cpp:1183-1184 TRB,
:890-891 HBG), and the CLOSE pass with its readbacks then rmb() then the RELEASE
stores (platform_regs.cpp:82, :142, :149).
The earlier draft's universal exclusion and "no functional consumer" verdict stay withdrawn. Nothing here claims a race existed, and nothing here licenses removing the clean — see the host-direction row and residual 3.
Exceptional and partial exits, audited separately¶
The acknowledgment argument above is a normal-path argument and is not extended to these:
- A core that never ACKs
EXITEDleavesacknowledged[i]false, so no RELEASE is stored for it,rcbecomes-1, andreleased[i]reports it; the comment atplatform_regs.cpp:145-146hands it to the host recovery path. That core's gate read cannot succeed at all, so the question there is recovery, not publication. - A core released but still spinning when the AICPU tears down is the interleaving above.
- a5 at this audit's baseline had no gate member on TRB and does not use the
gates on HBG, so this whole row is a2a3-only. Post-baseline (#2388) a5 TRB
carries per-core
teardown_gates, and the final revision also uses them in a5 HBG. The same failure and partial-exit questions apply; the ordering and publication arguments above were written for a2a3 and are not restated here as validated on a5.
Limit of what is established about the failure path. retire_cores logs each
unreleased core and returns -1 (scheduler_cold_path.cpp:695-703 TRB); this entry
has not traced what the host recovery path then does, and no claim is made here
that an op-execute timeout, a device reset, or any other mechanism retires such a core
before a later launch touches the same gates. The coherence premise does not help with
this and slightly sharpens the question, since a still-live reader would observe a
later store promptly rather than eventually. Recorded as the boundary of what is
proven, not as a discovered defect: no such sequence is shown to be reachable.
Readers positioned after the clean¶
Two, both host D2H reads of workers[]:
print_handshake_results(device_runner_base.cpp:1008), called fromdrain_executionafterreap_run(a2a3device_runner.cpp:445, a5:641) — after both run boundaries, both stream synchronizes and the AICPU kernel. It readsaicore_done,aicpu_readyandtask; its only consumer isLOG_DEBUG, and it now performs no D2H at all unless the host DEBUG threshold is enabled (HostLogger::is_enabled). The obligations recorded here are unchanged for the debug-enabled path: when the copy does run it is the same read of the same bytes after the same clean, so whether that read needs the clean on A3, and what the clean's lifetime relationship to it is, stay exactly as open as before. Gating removes a copy; it settles nothing about the copy that remains.run_retention_probe(run_retention_probe.cpp:60), reachable only throughsimpler_probe_run_retention(c_api_shared.cpp:1284) — a fixture entry, not a production run path. It readsaicore_done, which the AICore already flushed.
So on the host direction the only identified consumer is a diagnostic log. That is a
statement about consumers found, not a survey that terminated: residual 3 below — whether
the host's D2H read of AICPU-written task needs the clean on A3 — is unexamined and is
in exactly this direction. It does not add up to "the clean has no consumer".
Two further facts¶
- a5 TRB performs no descriptor-wide clean at all, yet carries the same host debug reader as the other three variants. Evidence about a5 only; it says nothing about A3, where the premise makes the host↔device edge non-coherent.
- At this audit's baseline the two extents differ. The uploaded prefix stops at
offsetof(DeviceRuntimeLaunchDesc, teardown_gates)on a2a3 TRB and both HBG variants, while the clean still spanssizeof(runtime->dev), so the clean covers a device-only region. This is a post-31da0560echange: at the older baselineruntime_device_copy_sizereturnedsizeof(DeviceRuntimeLaunchDesc)and the two extents agreed. At baseline a5 TRB had no gate member, so its upload was the whole descriptor. Post-baseline (#2388) a5 TRB gainsteardown_gates, so itsruntime_device_initialized_prefix_sizestops atoffsetof(..., teardown_gates)whileruntime_device_extent_sizestayssizeof(DeviceRuntimeLaunchDesc)and the same two-extent split applies;runtime_device_copy_sizeis unchanged.
Evidence of no reader, versus unresolved reader¶
The reader statements rest on searching the call paths and the aliases these members are
reached through — dev.workers, get_workers(), all_handshakes, handshakes,
my_hank, handshake, get_teardown_gates(), dev.teardown_gates — across the AICPU,
AICore, platform and host trees for both arches and both runtimes, plus the two D2H
sites. What remains unresolved rather than absent:
- ~~Whether the clean is the gate's publisher on a2a3.~~ Answered: it is not.
The owner's AICPU→AICore premise covers exactly this store and this reader, so the
relaxed store needs no flush of its own. What the premise does not give is ordering,
which the reset and RELEASE edges get from
wmb()andrmb()respectively, or any statement about a reader left live by an earlier run. - Whether
workers[i].task's unflushed store reaches memory for the AICore's read. TheEXITEDacknowledgment orders that read beforedeinit, so the clean is not the candidate — but what publishes the store to the AICore in the first place is a separate, still-open question. The AICPU→AICore premise is stated for a bypassld_devread of the same address; this reader reachestaskby a different access path, so the premise is not extended to it here. - Whether the host's D2H read of AICPU-written
taskrequires the clean on A3. The premise addresses the inbound host→AICPU direction; this read is the opposite one.
What this licenses: nothing. All four writer groups are now excluded as the clean's
beneficiaries with a named dependency each — two by an explicit publication, one by the
EXITED acknowledgment on normal exits, and the a2a3 teardown gate by the AICPU→AICore
premise. That is still not a licence to remove the call: residual 3 is an unexamined
reader in the host direction, which the premise does not address. No operation is
changed and no defect is claimed.
Open questions for future measurement — not a proven probe plan¶
The first version presented these as probes with decisive outcomes. They are narrowed here to what each could observe and what would remain ambiguous. Nothing was run for this entry, and none of this is an actionable plan yet.
Q1 — on A3, does the end-of-previous-run placement discharge the inbound obligation? The coherence question this slot used to hold is answered by the premise; what is left is the A3 placement question, where the premise says maintenance is required. Observing it has the same difficulty the original framing did: a run must first establish that a stale readable line existed at the moment of the read — a verified initial cache state plus a read-back confirming the line's content before the host write — because fresh reads on their own cannot show which mechanism prevented staleness. The premise supplies the architecture, not the initial state of any particular line. Without that, a run is a comparison rather than a control.
Q2 — did the reader's cache survive the kernel boundary? The dirty-sentinel idea does not discriminate. Reading the old value is consistent with a surviving cache and with the old dirty line having been written back over the later host value and then read fresh from memory. Reading the host value is consistent with launch maintenance, with coherence, and with plain eviction before the read. Because the experiment introduces a write-back race into a read-freshness question, it needs observations that separate those four cases — at minimum a verified initial state and a way to distinguish "read from cache" from "read from memory" — before any verdict is assignable.
Q3 — the mapping's attributes: a question, not a probe. Timing against an MMIO reference cannot classify page attributes even with a second known-cached reference: access path, ordering attributes and latency all differ, so a ratio reports the sampled access behaviour rather than the mapping. The "flat ratio indicates uncached" rule from the first version is withdrawn. This question is answered by verified page attributes or a documented mapping contract, not by measurement from inside the runtime.
Provenance, for whenever any of this is attempted: record the CANN and driver
version, the exact libhost_runtime.so / AICPU .so md5, the commit, and device
occupancy. This repository has been misled by an incremental build producing
byte-identical artifacts across arms; the md5 is the control that the arms differ
at all.
What no measurement here can do: establish an architectural guarantee. It can demonstrate a violation. Retiring unknowns 1–3 as guarantees needs an SDK or architecture statement — the kind the host↔device premise supplies for its own edge, and the kind the AICPU→AICore premise now supplies for the gate half of unknown 2. The rest do not have one.
Why nothing is changed in the runtime¶
The coherence premises settle two visibility edges — inbound host→AICPU per architecture, and the a2a3 AICPU→AICore store the gate uses. Neither discharges the outbound host-direction question, nor atomicity, ordering or lifetime, and the remaining unknowns do not either. Adding an invalidate to a5 TRB, or deleting a5 HBG's two calls, or deleting all four, would each rest on the premises answering more than they do — on a2a3 the clean's host-direction role stays unaccounted for even with every on-device reader excluded.
What this hands to other work¶
- The a5 HBG resident handshake line carrying words from three tiers, with only a marker dependency sequencing them and no written hardware preconditions, is an input to any later redesign touching that line or worker publication. It is offered as an unresolved constraint, not as a proved invariant.
- Kernel mode's device-read chain is not delivered at this baseline and must
be traced when launch is implemented, not inferred from
prepare_once. - The per-path exit shapes in step 6 differ; a lifetime account for one path may not be reused for another.
Decisions for the maintainer, not taken here¶
- Which unknown to pursue first, and whether an SDK/architecture statement is the cheaper route than any device experiment.
- Whether to record the a5 HBG three-tier line and its marker dependency as a stated constraint — not an invariant — near the struct, given that its preconditions are unestablished.
- Whether the host↔device premise should also be recorded anywhere beyond
cache-coherency.mdand this entry — and, separately, whether the on-device coherency rows in that file (AICore, SDMA) warrant their own basis being written down, since the premise does not reach them.
References¶
-
2254 — the per-field inventory that raised the asymmetry.¶
-
2360 — the host-side prepare/publish seam; this entry begins at the device side.¶
-
2309, #2315 — narrowed the copy to the descriptor, hence the stale extent.¶
docs/hardware/cache-coherency.md,docs/hardware/chip-architecture.md.docs/aicpu-kernel-launch-mechanisms.md— launch methods and the seam..claude/rules/ascend.md— theDevice-nGnREMMIO attribute that does not extend to GM.