A2/A3 worker retirement: CLOSE must precede return

For deadline failure, late ACK and device ownership, see AICore retirement failure and device-owner recovery.

The A2/A3 persistent worker must not return after merely acknowledging EXIT. It must wait until the AICPU has closed the group's fast-path register windows and explicitly released the workers through global memory (GM).

This applies to both tensormap_and_ringbuffer and host_build_graph. It does not add borrowed-stream/L1 support or change the A5 exit protocol.

What fast path means here

The AICPU accesses each AICore's register window through MMIO. It writes DATA_MAIN_BASE to dispatch work and reads COND for ACK/FIN status. The AICore reads its own dispatch SPR and writes its own COND SPR. FAST_PATH_ENABLE (offset 0x18) is written with 0xE to open and 0xF to close the window. This is a device-side dispatch mechanism, not a torch allocator setting, host task-queue option, or ACL event.

The missing ordering edge

The old protocol established these two chains:

AICore: observe EXIT -> write COND=EXITED -> return
AICPU:                 observe EXITED   -> write IDLE -> CLOSE

They have a common predecessor, not an ordering between their tails. The following execution was therefore legal in the software protocol:

1. AICore observes EXIT and publishes EXITED.
2. AICore returns while its fast-path window is still open.
3. AICPU observes EXITED, writes IDLE, and closes the window.

EXITED proves that the worker has stopped accepting tasks. It cannot also prove that the AICPU has finished managing the window: that work occurs after the ACK. The missing edge is CLOSE -> worker return.

A host mutex, an event between streams, or waiting for an entire launch to complete cannot retroactively order these two operations inside that launch. Multiple in-flight graph executions are not required for this race. Nor does an asynchronous error reported at a later native operator prove that the native operator caused it.

Why the race is reachable

The AICPU program and the AICore kernel are two ops on two independent streams. DeviceRunnerBase creates stream_aicpu_ and stream_aicore_ separately, and DeviceRunner::launch_run submits one kernel to each. Stream order constrains only the ops within a stream, and the launch path inserts no cross-stream event — it states outright that it "intentionally performs no stream synchronization". The launch and reap halves are also separable, and a prepared successor is allowed to overlap its predecessor's execution (allow_prepared_successor).

So a worker that returns early makes its AICore-stream op complete early, while the AICPU-stream op of the same round is still writing that core's window. The AICore stream is then free to start the next round's kernel. That is what the return gate closes: it makes "the AICore op finished" imply "the AICPU closed every window it owned".

The neighbouring hazard already has a guard: launch_run clears each worker's aicore_done before the AICore kernel launches, because a prior run's report would otherwise "open a window on that run's physical_core_id". That guard covers a stale report being read by the new AICPU; the return gate covers the other direction, a stale window write landing on the new worker.

Retirement

Each AICPU thread retires the cores it owns as it leaves the dispatch loop, before it reaches the completion latch. Core ownership is a partition — assign_cores_to_threads hands every cluster to exactly one scheduler thread — so those retirements run concurrently and never name the same core.

Retirement must not hang off the completion latch. ThreadCompletionGate is a last-one-out counter, not a barrier: no participant waits, and a thread that returns early simply never arrives. Before this handoff existed, missing the latch cost a leaked runtime context; with workers blocked on gates, it would cost every core on the chip.

Retirement uses a current-response predicate, described in the freshness contract. Native and unstamped runs use platform_retire_aicore_group after report acceptance and exclusive claim. The claimant chooses a reserved nonce different from this physical core's observed COND; Core echoes it only after completing preceding writes, then performs no subsequent SPR writes. Native report epochs retain their startup checks. Both request and return gate are reset before READY or EXIT; the control remains one AICPU-owned 64-byte line. Fixed EXITED alone is insufficient.

The helper retires one claimed set in passes:

1. Publish each request before sending EXIT to every core in the set.
2. Poll for each target's matching current response against one shared deadline. A core that never responds remains unclosed and unreleased for host recovery.
3. Reset dispatch to IDLE and close every acknowledged window. Read back the MMIO windows, then drain those reads before publishing any GM return gate.
4. Store AICORE_POST_CLOSE_RELEASE to each acknowledged core's gate.
5. AICore observes that word, completes its final drain and only then returns.

The AICore wait is unbounded on purpose. A bounded wait cannot help: below the 45 s op-execute timeout it would return while the AICPU may still be writing — reopening exactly this race — and at or above it, STARS reaps the op first and the timeout is unreachable. The guarantee is instead that the release always happens, which is why retirement sits on every thread's exit path rather than behind a latch. An AICore has no logging path, so the AICPU device log is the only place a blocked worker is named: grep for AICore retirement: core N not released. The host's print_handshake_results covers workers[] only and does not read teardown_gates[], so there is no host-side view of the gate today; that dump also runs only when the host DEBUG threshold is enabled.

The A2/A3 onboard worker uses ld_dev for an uncached/bypass read and dsb(DSB_DDR) before return. The simulation uses an atomic acquire load. ld_dev is not being treated as a general-purpose atomic RMW or as a C++ acquire operation; this is a single-writer, per-core handoff. The platform's MMIO completion ordering and visibility of the GM publication are distinct parts of the protocol. A release store alone does not flush posted MMIO.

Emergency shutdown uses the same retirement helper and an ownership latch, so normal finalization does not subsequently write to already released workers' registers. A core that misses the deadline is not closed or released; responsive peers still retire. The existing host recovery path remains responsible for unresponsive cores and cores whose windows never opened. This patch does not introduce a new reset operation or recovery policy.

Cache-line and generation ownership

Requests during handshake

Retirement also covers a fatal exit while another thread is still publishing its cores. Each scheduler core has two atomic bits, READY and REQUESTED, reset by the boot leader before handshake. REQUESTED is retained even when the core has no published execution state yet; testing reg_addr == 0 before recording REQUESTED and skipping that core would lose the request.

READY is published after core ownership and payload initialization complete, matching the A5 TMR initialization boundary. TMR's barrier-free path publishes each owned group at the end of assign_own_clusters; the serial TMR and HBG paths publish from the leader at the end of post_handshake_init, after all handshake publishers have joined. Opening a window and storing CoreExecState alone do not publish READY. The normal dispatch loop does not read these bits.

Both READY and REQUESTED use __atomic_fetch_or with acquire-release ordering on a uint8_t array, as in A5 TMR. The boot leader clears that array before releasing any handshake thread. Whichever atomic operation adds the second bit owns retirement; the request path acquires the publisher's initialized state. Duplicate requests and normal/emergency overlap cannot elect another owner.

Initialization failure must also publish the successfully opened windows. The serial initializer does so after joining every handshake, before emergency retirement. An owned-cluster assignment failure publishes its fixed blocked partition, even if its tracker could not be populated. This fallback publishes only nonzero initialized addresses; a core with an invalid physical identity, zero register address, or incomplete handshake remains for host recovery.

An early REQUESTED therefore remains pending through initialization and is serviced by its publisher on success or failure. The retirement boundary retains the id/address checks after claiming, without turning an unpublished address into a discarded request. A silent core's claim remains consumed after timeout: a later request cannot close or release it without an ACK. Re-arming state and gates requires the preceding generation to have completed.

Serial initialization publishes its initialized worker-id set; the barrier-free path publishes the owning thread's blocked cluster set. This enumeration follows the existing A2/A3 startup layouts without adding an all-thread barrier. Both sets use the same per-core ownership rule and existing group deadline, ACK/CLOSE/readback/drain and per-core return gates. A5's software register definitions and platform-specific CLOSE sequence are unchanged.

The test_a2a3_tmr_scheduler_retirement and test_a2a3_hbg_scheduler_retirement tests run production handshake and scheduler retirement code with a recording platform sink. They pause after windows open but before execution state is published, and cover requests pending until ownership/payload initialization, initialization failure, ownership races, duplicate ids, partial timeout, zero-address rejection, and generation reset. These tests verify the scheduler handoff; the platform and worker tests below separately verify the ACK/CLOSE/return ordering.

The return gate lives in teardown_gates[], a separate array beside workers[] in the device-copied image — not inside Handshake:

Storage	Writer/access	Purpose
workers[i] — one 64-byte line	Existing cached report/task publication protocol; the AICore flushes the whole line with dcci(..., CACHELINE_OUT)	Startup identity, ready report, dispatch-payload pointer
teardown_gates[i] — one 64-byte line	AICPU atomic stores; AICore bypass reads	Post-close permission to return

The gate must not sit in a line the AICore writes back, or a stale cached report writeback would overwrite the AICPU's release. Keeping the two in separate arrays makes that hold by construction rather than by field offset: no cached write or cache maintenance targeting workers[i] can reach the gate line. static_assert(sizeof(Handshake) == 64) pins the report line, and AicoreTeardownControl is alignas(64) with its own size assertion.

Handshake therefore keeps its 64-byte layout and stride. HBG shares that definition with A5. A5 TMR and HBG use the separate gate array as implemented in PR #2388; this A2/A3 change leaves their handshake layout, execution protocol, and public Python API unchanged.

The boot leader zeroes the whole gate array before publishing handshake setup and before any register window opens; the write barrier that follows is what orders the reset ahead of both, so the preceding launch's value of 1 cannot release a new launch. Each cell is 4 bytes inside a 64-byte-aligned block, so a straggler still bypass-loading its gate cannot read a torn value. Reuse assumes the existing runtime lifecycle: the previous launch has completed before the same storage is re-armed. This is not a claim that one runtime image supports concurrent independent runs.

Evidence and reproduction boundaries

The executor regression links each production A2/A3 AICore execution loop against simulated register storage. With CLOSE deliberately withheld, all four AIC/AIV checks fail on the unmodified main implementation at fab1a41e2fd5bbefb9eb18e59609876e63297e98: the worker returns immediately. With the handoff, those checks pass. Companion tests check eventual return, 32 launches reusing the same gate, group ACK ordering, partial timeout, and invalid-target rejection.

cmake -S tests/ut/cpp -B tests/ut/cpp/build
cmake --build tests/ut/cpp/build --parallel 2 \
  --target test_a2a3_tmr_aicore_retirement test_a2a3_hbg_aicore_retirement \
  test_a2a3_tmr_scheduler_retirement test_a2a3_hbg_scheduler_retirement
ctest --test-dir tests/ut/cpp/build -R '^test_a2a3_.*_retirement$' --output-on-failure

These are software-ordering tests; they do not emulate silicon register retirement hazards. The motivating downstream L1 ACLGraph investigation separately found that removing only the post-close wait changed a passing 1280-replay run into a matching vector timeout at replay 2. The fault PC was at the persistent AIV wrapper's end, not inside the later native operator. That is downstream integration evidence, not a claim that upstream main already supports or reproduces that L1 graph workload. No undocumented internal hardware state-machine failure is asserted here.

The PR's validation record distinguishes new upstream tests from this historical downstream evidence and lists any unavailable toolchains.