Skip to content

Args Dump — Per-task Argument Capture

1. Background & Motivation

Numerical bugs (NaNs, wrong shapes, off-by-one offsets, mis-aligned strides) are notoriously hard to reason about by reading kernel code: the symptom shows up two tasks downstream, the suspect tensor is gone, and re-running with printf distorts the timing that triggered the bug in the first place.

Args Dump captures per-task tensor inputs/outputs plus scalar inputs during kernel execution and writes them to disk for offline inspection. The host pre-allocates the recording buffers, AICPU writes records during execution, and the host exports a JSON manifest plus any level-selected binary payload. The result is a stable, replayable record of every dumped argument a kernel saw, without the timing distortion of inline printing.

2. Overview

  • Per-task input/output capture. Inputs snapshotted before dispatch, outputs snapshotted after FIN; INOUT tensors at both stages.
  • Logical shape preserved. Records carry dtype, shape, strides, start_offset, and is_contiguous so logical views are reconstructable.
  • Manifest + level-dependent binary payload. Once at least one record is collected, every enabled level writes args_dump.json. Levels 1 and 2 then create args.bin eagerly; hybrid creates it lazily only when an Arg::dump(...)-selected tensor contributes payload. Tensor payload records carry bin_offset / bin_size, while scalar values remain manifest-only.
  • Unified scalar args. Scalar values are emitted as kind: scalar, stage: before_dispatch, zero-dim records in args_dump.json; there is no separate args-only manifest. Their final dtype is registered at submit time in a dump-only per-task side table and resolved by AICPU when writing each scalar dump record.
  • Cross-architecture metadata. The flags, manifest schema, and scalar metadata are wired through both runtime variants on a2a3 and a5. Real tensor-payload consumption is currently supported only on the a2a3 platform family; see the platform scope below.

Which runtimes capture

Capture needs a thread that observes each task's own dispatch and completion, so it is a property of the runtime rather than of the platform:

Platform Runtime Per-task capture Request for it
a2a3 host_build_graph yes — AICPU scheduler dispatches and reaps served
a2a3 tensormap_and_ringbuffer yes served
a5 tensormap_and_ringbuffer yes served
a5 host_build_graph no — on-core schedulers dispatch and reap refused at simpler_prepare_run with PTO_RUNTIME_ERR_UNSUPPORTED

The last row is the same on the onboard and the simulated backend. §3.1.1 is why, what the refusal looks like to a caller, and which entrypoints check it.

Enable in one line (full = every task, every arg):

python tests/st/<case>/test_<name>.py -p a2a3sim --dump-args full

3. How to Use

3.1 Enable Args Dump

--dump-args takes an optional mode. The modes are two independent choices — which tasks reach args_dump.json, and which of those also write payload bytes into args.bin — rather than a severity dial, so they are named rather than numbered:

Mode Manifest Payload
flag absent, or off — —
partial (bare --dump-args) args marked with CoreTaskArgs::dump(...) (see §3.2) the same marked args
hybrid every task marked args only
full every task every arg

Tensor-payload platform scope: payload is written on both platform families. a5 payload truth was broken by #1560 and is fixed; the scene test now asserts a known payload value in every mode that writes one, on a2a3 and a5 alike. Plain hybrid metadata, including inline scalar values, does not consume tensor payload and remains available.

hybrid is the capture mode used by python -m simpler_setup.tools.core_swimlane to build a Core swimlane simulator replay: it provides complete per-task argument metadata without dumping every tensor's element data. It reuses the exact task/argument mask produced by CoreTaskArgs::dump(...) for partial: every argument still gets a JSON record, while only marked tensors contribute bytes to args.bin. Scalar values already live inline in the manifest and never need payload copy. With no dump(...) markers, the AICPU skips all arena payload copies, the manifest's bin_file is null, and every bin_size is 0.

hybrid deliberately inherits partial's selector semantics; it does not add a second mask. tensormap_and_ringbuffer registers selection metadata in its AICPU per-task table. host_build_graph embeds the same mask, ambiguity flags, and scalar dtypes in each H2D task image, including cached sub-task definitions. The device collector consumes either source identically.

# Standalone runner
python tests/st/<case>/test_<name>.py -p a2a3sim --dump-args full
python tests/st/<case>/test_<name>.py -p a2a3 -d 0 --dump-args        # partial

# pytest
pytest tests/st/<case> --platform a5sim --dump-args full
# a5 host_build_graph has no per-task capture: every scene under it is refused
# with -1001 while this flag is set (§3.1.1). Use the a2a3 mirror for an HBG
# capture, or a5 tensormap_and_ringbuffer for an a5 one.
pytest tests/st/a2a3/host_build_graph/dump_args --platform a2a3sim --dump-args full
pytest tests/st/a5/tensormap_and_ringbuffer/dfx/args_dump --platform a5sim --dump-args hybrid
pytest tests/st/<case> --platform a2a3sim --dump-args full
pytest tests/st/a2a3/host_build_graph/vector_example --platform a2a3sim --manual include --dump-args full

The mode sets CallConfig::enable_dump_args (off/partial/hybrid/full, stored as 0/1/2/3). The host then allocates dump storage, publishes its base address through kernel_args.dump_data_base, and sets SIMPLER_DFX_FLAG_DUMP_ARGS (every mode but off) in each worker handshake's enable_profiling_flag for the enable/disable decision. The partial / full / hybrid distinction is carried in Runtime::dev.dump_args_level, published with this invocation's descriptor in stream order, rather than in the profiling-flag bitmask. The on-device AICPU reads the storage base via set_platform_dump_base(), the enable bit via set_dump_args_enabled(SIMPLER_GET_DFX_FLAG(...)), and latches the descriptor selection before task dispatch (PARTIAL → selective, HYBRID → all metadata, payload copy controlled by the same per-task mask as PARTIAL). Host admission freezes the export selection without writing or reading back a shared device level; descriptor publication failure stops kernel submission. Host admission only resets its local collection window. Device entry records the payload-credit baseline used by ordinary collectors and clears per-invocation drop accounting; transport counters and queue/arena cursors remain monotonic for both ordinary and retained collection. Device entry publishes the initialized epoch after the baseline is ready. Ordinary arena acknowledgements and leftover recovery require that epoch to match the host invocation; recovery reads must succeed before cached bytes can be consumed. Already collected records remain best-effort output; retained error reporting is unchanged. The runtime resolves the mask before each record reaches the platform writer, so the writer does not depend on a second task-id lookup. Because the level is decided host-side before any task is dispatched, it is latched up front — there is no dependence on task submission order. AICore executors read the same SIMPLER_DFX_FLAG_DUMP_ARGS bit to insert a pipe_barrier(PIPE_ALL) before FIN when dump is on, so AFTER_COMPLETION snapshots see the kernel's final writes.

3.1.1 The capture instant is the task's, and where that is unavailable

A record is taken at the task's own instant: BEFORE_DISPATCH as the scheduler hands the task to a core, and AFTER_COMPLETION as it reaps the FIN. That is what makes an intermediate tensor's recorded content the one that task actually read or wrote, which is what replay and localization need. Every AICPU-dispatching path captures there.

A5 host_build_graph has no such instant available to the AICPU, and therefore has no per-task argument capture at all. Its route is the AICore Scheduler, where the on-core schedulers dispatch and complete tasks, so no AICPU thread observes an individual task start or finish. The only instants the AICPU owns are the run's — before the DMB gate opens and after every core has retired — and a snapshot taken there is not the task's data:

  • task A produces X and task B consumes it, so B's input read at the pre-run boundary holds whatever X was before A ran;
  • two tasks write the same buffer in turn, so the post-run read gives the later write as both their outputs.

What a request for it does

args_dump_capture_supported_impl is the per-runtime answer, declared beside the other *_supported_impl capability predicates in the platform's c_api_shared.cpp and defined in each runtime's runtime_maker.cpp. a2a3 host_build_graph and both tensormap_and_ringbuffer runtimes answer 1; a5 host_build_graph answers 0.

simpler_prepare_run refuses a request this answer cannot serve, returning PTO_RUNTIME_ERR_UNSUPPORTED (-1001) before any state is read — so before the bind, before any device resource is provisioned, and on both the onboard and the simulated backend, which run the same AICPU and AICore sources. The two public entrypoints that carry a per-run CallConfig are simpler_prepare_run and simpler_run, and the second delegates to the first, so one check covers both; through the Python API that is every Worker.submit / run() whose CallConfig.enable_dump_args is non-zero. A caller sees the run fail with prepare_native_run failed with code -1001, nothing is written under its output_prefix, and the lane is not stopped — the next run is served normally.

Three properties are the point of refusing there rather than later:

  • an empty args_dump.json would be indistinguishable from a capture that found nothing there was to capture;
  • a run admitted and then found unable to settle this owner's accounting has already executed and already held a terminal bank and a pipeline slot;
  • only this owner is a term. A request that does not name it reaches the device unchanged, and a request that names it alongside the swimlane and PMU is refused for it — those two remain available on this runtime and are asserted there by tests/st/a5/host_build_graph/test_dfx_joined_launch_attribution.py, which also asserts this refusal.

The answer is the runtime's and not a bind's. a5 host_build_graph also keeps an AICPU-scheduled fallback, selected inside its bind from the graph's own task kinds and from nothing a caller states, so an answer that followed the selection would make the capture available or not depending on whether a graph happened to hold a task the resident route declines.

No default or bundled diagnostic set may include it there. --dump-args defaults to off and no scene, example or CI lane turns it on implicitly, so a request is always an explicit one — but a sweep is a request for every scene it covers: pytest tests/st/a5 --dump-args partial now fails every a5 host_build_graph scene instead of running them and collecting nothing. Select by path. Likewise, a suite that enables "every owner this backend has" must read that set per backend rather than per owner list; the joined-launch suites do this through their own DEVICE_OWNERS, and the a5 host_build_graph copy names the unsupported one in UNSUPPORTED_OWNERS instead of enabling it.

Closing the gap needs a capture at the real task boundary, which on that route means either an AICore↔AICPU rendezvous per task (the AICore publishes "about to execute N" and waits for the host's acknowledgement, and again after it completes) or AICore-side payload staging. The first adds a device round-trip to every task's dispatch path; the second adds per-task payload storage. The bounded design is docs/investigations/2026-10-a5-resident-per-task-args-capture.md; the capability is deferred to its own change, and until then the request is refused rather than partly served.

3.2 Partial Dump — Select Specific Args

partial captures only the tasks whose CoreTaskArgs is marked with dump(...); every unmarked task is skipped. Within a marked task, only the selected tensor/scalar args are recorded. Mark the arguments on the relevant CoreTaskArgs before submission:

CoreTaskArgs args;
args.add_input(x);
args.add_input(y);
args.add_output(z);
float scale = 1.0f;
args.add_scalar(scale);
args.dump(x, z, scale);
rt_submit_aiv_task(FUNC_ADD, args);

dump(...) selects arguments from the current CoreTaskArgs; it does not execute a dump immediately. The selected tensors and scalar lvalues must already belong to that CoreTaskArgs. Scalar selection is by the lvalue passed to add_scalar(...), not by scalar value. Temporaries such as args.dump(1.0f) are rejected because they do not identify a previously added scalar slot. If the same scalar lvalue is added more than once, dump(lvalue) selects the first matching scalar arg and marks that scalar entry with arg_index_ambiguous: true in args_dump.json; use distinct local variables when you need to select a later duplicate.

The runtime uses the tensor direction already provided by add_input(), add_output(), or add_inout() to decide when each selected tensor is captured:

  • input tensors are dumped before dispatch.
  • output tensors are dumped after completion.
  • inout tensors follow the existing inout dump behavior.
  • scalar args are dumped before dispatch.

Selective dump comes at two granularities, both expressed with the same dump(...) marker:

  • Arg granularity — dump(x, z, scale) selects specific tensor and scalar arguments of the task (the example above).
  • Task granularity — dump() with no arguments selects the whole task (every tensor and scalar argument on the CoreTaskArgs), without enumerating them:
CoreTaskArgs args;
args.add_input(x);
args.add_input(y);
args.add_output(z);
args.add_scalar(scale);
args.dump();        // whole task: every tensor/scalar arg on this CoreTaskArgs
rt_submit_aiv_task(FUNC_ADD, args);

Which tasks and args are captured is chosen by the mode (§3.1), latched host-side before any dispatch — not inferred from the markers, so it never depends on task submission order. Under partial, tasks without a marker are skipped and marked tasks dump only their selected arguments. Under full, the markers are ignored and every task's tensors/scalars are dumped. Under hybrid, every task's tensors/scalars are dumped as metadata and the same markers control which tensor records carry payload. With no --dump-args, dump is off entirely.

If you run partial but place no dump(...) markers anywhere, the collector receives no records and exports no manifest — that is the deliberate "only what I marked" contract. Use --dump-args full when you want everything.

3.3 Output

The dump artifacts land under the per-task output prefix (CallConfig::output_prefix, set by scene_test.py::_build_output_prefix to outputs/<ClassName>_<case>_<YYYYMMDD_HHMMSS>/ for SceneTest runs):

<output_prefix>/
└── args_dump/
    ├── args_dump.json  # unified argument manifest (`--dump-args`)
    └── [args.bin]      # eager for partial/full; present for hybrid only with selected tensor payload

Filenames are fixed (no per-file timestamp) — the directory is the per-task uniqueness boundary. Once at least one record reaches the collector, partial and full create args.bin, which may be empty when no tensor payload was selected or recorded. hybrid with records but no selected tensor payload emits only args_dump.json; otherwise it also emits args.bin.

args_dump.json — Unified manifest

args_dump.json is the manifest; its bin_file field points at the sibling binary payload. A hybrid capture with no marked tensors has bin_file: null, no .bin, and bin_size: 0 for every entry. A hybrid capture with marked tensors has bin_file: "args.bin"; only the CoreTaskArgs::dump(...)-marked tensor records have a non-zero bin_size.

Example manifest, from a full capture (one input tensor captured before dispatch):

{
  "run_dir": "args_dump",
  "bin_format": {
    "type": "logical_contiguous",
    "byte_order": "little_endian"
  },
  "dump_args_level": 3,
  "total_args": 1,
  "before_dispatch": 1,
  "after_completion": 0,
  "input_args": 1,
  "output_args": 0,
  "inout_args": 0,
  "truncated_args": 0,
  "dropped_records": 0,
  "dropped_records_complete": true,
  "bin_file": "args.bin",
  "args": [
    {
      "task_id": "0x0000000200000a00",
      "func_id": [0],
      "role": "input",
      "stage": "before_dispatch",
      "arg_index": 0,
      "kind": "tensor",
      "dtype": "float32",
      "shape": [16384],
      "strides": [1],
      "start_offset": 0,
      "is_contiguous": true,
      "truncated": false,
      "bin_offset": 0,
      "bin_size": 65536
    }
  ]
}

Key fields:

  • dropped_records_complete — ordinary manifests distinguish a confirmed device-reported loss count from an unread or incomplete one. It does not certify payload/file or whole-artifact completeness; see §7.3.
  • dump_args_level — the capture mode, as its wire value (0 off / 1 partial / 2 hybrid / 3 full).
  • task_id — runtime task identity. Use to correlate with swimlane / PMU output.
  • func_id — array of the task's active-subtask kernel ids (its mix membership). A single-kernel task is [N]; a cooperative or packed mix is [i, j, ...] (one entry per active subtask). The dump emits each payload tensor once with this array, so a mix is recoverable by grouping on task_id and reading the func set. [-1] when unknown (the host_build_graph dump path, which does not thread func_id). Note: the array is the task's mix set, not a claim that each listed kernel reads this specific slot.
  • arg_index — payload argument index (the slot this tensor occupies). Tensor entries use the payload tensor index; scalar entries use payload.tensor_count + scalar_index. When scalar lvalue selection matched multiple scalar slots, the selected first-match scalar entry carries arg_index_ambiguous: true.
  • role / stage — input / output / inout, captured before_dispatch / after_completion.
  • kind — tensor for arena-backed payloads, scalar for zero-dim before_dispatch scalar args stored directly in the manifest.
  • dtype / shape / strides / start_offset / is_contiguous — logical view geometry. Tensor bin_size is numel × elem_size of the logical view, gathered if non-contiguous; scalar entries have bin_size = 0.
  • bin_offset — byte offset into args.bin where the payload starts.
  • truncated — set when the tensor exceeded arena size; see §7.
  • Top-level dropped_records surfaces aggregate metadata-buffer loss — useful for spot-checking a run.

3.3 Inspect with dump_viewer

The viewer auto-picks the latest outputs/*/args_dump directory when invoked without arguments. It loads args_dump.json and uses its bin_file field to find the payload:

# List every dumped arg in the latest run
python -m simpler_setup.tools.dump_viewer

# Filter and save matching args to human-readable .txt files
python -m simpler_setup.tools.dump_viewer --stage before --role input --export

# Export one specific entry by its manifest index
python -m simpler_setup.tools.dump_viewer --index 42

# Pin to a specific dump directory
python -m simpler_setup.tools.dump_viewer outputs/<case>_<ts>/args_dump \
    --task 0x0000000200000a00 --export

Exported .txt files include metadata headers, a row-major overview with aligned columns, and a detail listing with multi-dim indices — diff-friendly against golden tensors and pasteable into a spreadsheet.

3.4 Add dump support to a new test

For tensormap_and_ringbuffer, each incore declares a signature (each tensor's ArgDirection) in its CALLABLE entry. The dump maps signature entry i to payload tensor slot i positionally — there is no arg_index. The signature gives every captured tensor a direction (which sets its capture stage); geometry (shape, dtype, strides) is derived automatically from the payload.

The signature must cover the task's full payload: for a mix task (multiple subtasks sharing one args[]) every cooperating incore declares the complete task signature, in payload order. A narrower task that dispatches only a prefix of that layout (e.g. a standalone kernel, or a 2-subtask mix of a 3-subtask shape) supplies a prefix — the dump records the slots present and skips the rest. The dump is driven by the widest active subtask's signature, so at least one cooperating incore must declare the full-width layout.

A signature usually lists only tensor directions — scalars are independent (added via add_scalar) and are typically omitted. If a SCALAR direction is listed it must come after every tensor entry; the dump skips SCALAR entries (they do not consume a positional tensor slot).

Both host_build_graph and tensormap_and_ringbuffer share the same code path: the AICPU dump-collector reads geometry directly from the payload's tensor region, which carries shape and offset info alongside the device-packed arguments. For allocated output tensors, alloc_tensors supplies a TensorCreateInfo at orchestration time that is embedded in the payload by submit. For tensors that already exist (inputs passed by the caller), the geometry travels directly on the Tensor object and needs no explicit per-task registration.

Full template: tests/st/a2a3/host_build_graph/dump_args. The a5 mirror at tests/st/a5/host_build_graph/dump_args demonstrates the same two registration APIs and asserts its computation, but collects nothing: that runtime has no capture (§3.1.1).

4. Capabilities

What you can read out of args_dump.json and, when present, args.bin:

  • Per-task input snapshots (role: input, stage: before_dispatch) — what each kernel was given.
  • Per-dispatch scalar args (kind: scalar, stage: before_dispatch) — the raw per-task scalar-slot values AICPU handed to the kernel, tagged with the dump-only dtype captured at submit time.
  • Per-task output snapshots (role: output, stage: after_completion) — what each kernel produced. The barrier ensures these reflect the kernel's final writes.
  • INOUT deltas — same arg captured at both stages; diff before vs after to see exactly what the kernel modified.
  • Logical view reconstruction — shape / strides / start_offset / is_contiguous plus the gathered logical-contiguous payload.
  • Per-task identity — task_id, stage, role, and arg_index identify each dumped argument within a task.
  • Loss accounting — a per-record truncated flag plus aggregate dropped_records in the summary.

4.1 Background output across repeated runs

By default a run's dump is finished on that run's own boundary: when run() returns, both args.bin and args_dump.json are complete. With Worker.init(collect_across_runs=True) — a level-3 worker with local chip children, the same gate Worker.flush_diagnostics enforces — ArgsDump instead finishes one run's payload file and manifest on a background writer while the next run executes on the device. The device still runs one op at a time; only the host-side output moves.

What changes for a caller:

  • run() returning no longer means the files are complete. Call the existing flush_diagnostics(timeout) before reading them. It returns normally only when, for every run closed before the call, both its payload bytes reached its payload file and its manifest was published.
  • Each run owns an exclusive file pair. The payload is args.e<run_epoch>.bin (with a .1, .2, … suffix if a name is already taken, which happens only when another process reaches the same destination with the same epoch number), and the manifest still names it through bin_file. A reader takes the payload's name from the manifest, as it already does.
  • The manifest is the signal, not the payload file. A payload file may exist and still be growing; the manifest appears only at publication, by atomic rename, so whichever args_dump.json a reader sees is always consistent with the payload file it names.
  • Nothing is deleted. A published payload file is kept so a reader holding an older manifest does not lose its bytes, and a failed run keeps its payload and its args_dump.json.<token>.tmp as evidence. A reused output prefix therefore accumulates files until you remove them, and a destination that still holds a failed run's .tmp refuses the next run rather than publishing beside evidence nobody has read.
  • A run's boundary still does finite work. Before it returns, the close reads this run's terminal lane state and waits — bounded — for proof that every buffer the run published has been received into host-owned storage. That proof is what makes the leftover decision sound and what keeps the successor from reusing an arena whose bytes nobody copied; a run whose proof does not land fails, rather than quietly publishing a file whose completeness nothing established.
  • Two runs may be unpublished at once. A third is refused before the device is handed anything, so a slow disk fails a launch instead of silently queueing behind it. The same refusal happens when a destination is owned by an open run, when no exclusive name pair is available, or when the retained host budget cannot admit the run's fixed state.
  • Retained host memory is bounded. One ArgsDump collector charges its retained metadata, payload copies and queue nodes against its own 256 MiB budget, charged before each allocation and including the transient peak while a metadata bucket grows. That figure is ArgsDump's alone: the total host cost of diagnostics is the sum over the collectors you enable. Device pool and arena sizes are unchanged.
  • Failures are explicit and sticky. A host-discarded record, a device-side dropped record, a write failure, or a completeness this collector cannot prove all fail flush_diagnostics, and the record survives for the runner's life — removing the evidence files does not clear it. A published manifest from any of those runs carries counts_unknown: true and a collection_verdict, so no artifact from a lossy run reads as complete. Device-side truncation keeps its existing non-failing status: it is configured behaviour (PLATFORM_DUMP_AVG_TENSOR_BYTES), reported by truncated and truncated_args, and is not host loss.
  • Extra manifest fields, diagnostic only — per-arg host_discarded, and per-run host_discarded_args, metadata_discarded_records, counts_unknown and collection_verdict. They explain an incomplete result; they never excuse one.

Two honest limits of this mode:

  • A run that dies without an observed device fence is not recovered. Ordinary collection also requires a stop proof before reading an un-flushed buffer: observed execution completion or independently joined producers. The onboard boundary supplies no independent producer-stop proof after a failed fence. Both modes therefore preserve host-owned best-effort output without recovering a buffer that a live producer may still modify.
  • A blocked disk is not a bounded wait. flush_diagnostics is bounded by the timeout you pass it, not by a promise that the writer finishes. It does not, however, hold the device: the producer's arena is released once the payload is in host-owned storage, so a stalled writer delays publication and nothing else. Only a payload the device actually handed over takes part in that release — a record recovered from a buffer it never published is output, not an acknowledgement, and contributes nothing to it.

The lane payload counters (published_payload_count, completed_payload_count) are monotonic for the collector's life on both paths. Retained collection uses lifetime host-ownership credits. Ordinary collection uses the device-entry run_published_payload_base plus this invocation's written payload count, so opening a new host window cannot erase or manufacture transport credits. The design assumes one collector's lifetime does not produce 2⁶⁴ payload records; retained admission checks the counters' headroom and refuses a run that is close to it, which lowers the risk of reaching that bound rather than proving no single run can.

5. Design Highlights

CoreTaskArgs::dump(...) selection state is compiled only when SIMPLER_DFX=1. With SIMPLER_DFX=0, the public API remains available but acts as a no-op: no dump-only CoreTaskArgs state is stored and submit does not propagate dump metadata.

5.1 Common device-side structures

Both architectures share the same device-side layout, published via kernel_args.dump_data_base:

DumpDataHeader                                  (host init, AICPU reads)
├── queues  [MAX_AICPU_THREADS][READYQUEUE_SIZE]
├── queue_heads / queue_tails (per-thread)
├── num_dump_threads
├── records_per_buffer
├── magic = 0x44554D50 ("DUMP")
└── reserved  (zero; the per-invocation level is in Runtime::dev.dump_args_level)

DumpBufferState[num_dump_threads]               (per-thread)
├── free_queue {buffer_ptrs[SLOT_COUNT], head, tail}
├── current_buf_ptr            (AICPU active DumpMetaBuffer*)
├── current_buf_seq
├── arena_base / arena_size    (per-thread arena pointers)
├── arena_write_offset         (AICPU cursor in the current arena reuse cycle)
├── published_payload_count    (payloads committed to Host through RQ entries)
├── completed_payload_count    (published watermark acknowledged by Host)
├── dropped_record_count       (lifetime, retained fallback accounting)
├── run_published_payload_base  (device-entry lifetime cursor)
├── run_dropped_record_count    (this invocation only)
└── run_epoch                   (initialized baseline identity)

DumpMetaBuffer pool (rotated)                   (BUFFERS_PER_THREAD per thread)
└── ArgsDumpRecord records[RECORDS_PER_BUFFER] + count   ← 128 B each

arena_data (per-thread, circular byte buffer)
  default = BUFFERS_PER_THREAD × RECORDS_PER_BUFFER × AVG_TENSOR_BYTES
          = 8 × 256 × 64 KiB = 128 MiB per thread

These structs are binary-identical between a2a3 and a5 (static_assert-checked). dump_data_base flows through KernelArgs, not Runtime — AICPU reads it from k_args->dump_data_base in kernel.cpp and passes it to set_platform_dump_base(). Dump enablement is propagated separately via the umbrella bitmask KernelArgs::enable_profiling_flag (bit0 = SIMPLER_DFX_FLAG_DUMP_ARGS); the AICPU kernel entry calls set_dump_args_enabled() with the decoded bit, so device-side code does not infer "dump enabled" from dump_data_base != 0.

Each record is fixed at 128 B (two cache lines) — see ArgsDumpRecord in args_dump.h.

5.2 Where dump calls are wired in

Each capturing runtime's AICPU scheduler dispatch code calls dump_args_for_task at two points in the per-task state machine (for tensormap_and_ringbuffer, this is in runtime/scheduler/scheduler_completion.cpp and runtime/scheduler/scheduler_dispatch.cpp). a5 host_build_graph has no such call site reachable on its route — the two points below belong to whichever thread owns the task, and there they are the cores' (§3.1.1):

┌──────────────────────────────────────┐
│ per-task dispatch:                   │
│   if enable_dump_args {              │
│     dump_args_for_task(              │
│         BEFORE_DISPATCH);            │
│   }                                  │
│   dispatch(task);                    │
│   wait FIN;                          │
│   if enable_dump_args {              │
│     dump_args_for_task(              │
│         AFTER_COMPLETION);           │
│   }                                  │
│   retire(task);                      │
└──────────────────────────────────────┘

dump_args_for_task first collects the task's active-subtask kernel ids (the func_id array), then walks the widest active subtask's callable signature once. Signature entry i maps to payload tensor slot i positionally; for each non-scalar entry it builds a ArgsDumpInfo (dtype + shape + strides + start offset + device address) for that slot and calls dump_arg_record for entries whose role matches the current stage — stamping every record with the whole func_id array. Each payload tensor is thus emitted once (not duplicated per subtask); a mix is recoverable as the func set carried on each slot. A signature entry beyond the payload is skipped (memory safety — a prefix-dispatched task); payload slots the signature does not reach are left undumped with a soft completeness warning. (The host_build_graph dump path is a separate overload that also maps positionally and stamps func_id = [-1].) Scalar values are dumped separately from the flat payload scalars[], once per task (with the same func_id array); their dtype table is registered at submit time and emitted as a dump-only metadata record at BEFORE_DISPATCH.

When dump is enabled, AICore executors also issue pipe_barrier(PIPE_ALL) after kernel execution and before writing the FIN handshake. This closes the ordering gap where AFTER_COMPLETION snapshots could observe output buffers before all device-side writes were globally visible. Older implementations could capture stale output data; the current implementation fixes this in the runtime, not in each individual kernel. The barrier is gated on SIMPLER_DFX_FLAG_DUMP_ARGS, so non-dump runs keep the original cheaper completion path.

5.3 Tensor metadata registration

AICPU has device addresses and sizes — the logical shape, dtype, and view geometry come from the runtime. Both host_build_graph and tensormap_and_ringbuffer share the same code path: the AICPU dump-collector reads geometry directly from the payload's tensor region, which carries shape and offset info alongside the device-packed arguments. For allocated output tensors, alloc_tensors supplies a TensorCreateInfo at orchestration time that is embedded in the payload by submit. For tensors that already exist (inputs passed by the caller), the geometry travels directly on the Tensor object and needs no explicit per-task registration.

When metadata is missing or inconsistent, the task is skipped for dump and a single LOG_WARN is emitted (guarded by try_log_dump_args_layout_mismatch to avoid log flooding); normal execution continues.

5.4 a2a3 — shared-memory streaming

halHostRegister maps device memory into host virtual address space so the host can read device buffers directly. ArgsDumpCollector runs split mgmt threads and collector shards on top of a BufferPoolManager<DumpModule>: drain/refill shards poll SPSC ready queues and refill free queues from shard-local recycled lanes while kernels are still executing. Collector shards drain the host hand-off queues into on_buffer_collected, then the replenish thread routes done buffers to same-kind lanes below their recycled watermarks. Modules that declare no watermark keep origin-shard routing. Any remaining watermark deficit is batch-allocated without writing device free queues.

Each collector appends metadata to its own vector, so collectors do not serialize on the manifest accumulator. Captured payloads share one args.bin: the narrow writer lock assigns the next binary offset and enqueues the payload in the same order, while metadata insertion stays outside that lock. Export joins the writer, folds the shard-local vectors together, and sorts the merged metadata by (task_id, stage, arg_index, role) before writing JSON.

        HOST                                         DEVICE
┌──────────────────────────┐               ┌──────────────────────────┐
│ ArgsDumpCollector      │               │ AICPU thread             │
│                          │               │                          │
│ initialize()             │  alloc +      │ dump_args_init()         │
│   rtMalloc + halRegister │──register────>│   read DumpDataHeader    │
│   build DumpDataHeader   │              │   cache per-thread ptrs  │
│                          │               │                          │
│ start()                  │               │ per-task run loop:       │
│   ┌────────────────────┐ │               │   BEFORE_DISPATCH        │
│   │ drain/refill shard │ │               │     dump_arg_record()    │
│   │ + replenish thread │ │ SPSC ready    │     → write to arena     │
│   │   poll ready queue │<┼──queues──────<│     → append record      │
│   │   refill freeQ     │─┼──free queue──>│     → push to ready_q    │
│   └────────────────────┘ │               │   dispatch kernel        │
│   ┌────────────────────┐ │               │   wait FIN               │
│   │ collector shard    │ │               │   AFTER_COMPLETION       │
│   │   reads arena via  │ │ shared mem    │     dump_arg_record()    │
│   │   host mapping     │<┼──mapping─────<│                          │
│   └────────────────────┘ │               │                          │
│                          │               │ dump_args_flush()        │
│ ordinary close(proof)    │               │   log per-thread stats   │
│   proved: quiesce        │               └──────────────────────────┘
│     reconcile counters   │
│   unknown: stop(false)   │
│     no held recovery     │
│                          │
│ export_dump_files()      │
│   → <output_prefix>/     │
│     args_dump/           │
│       args_dump.json     │
│       args.bin           │
└──────────────────────────┘

Lifecycle (device_runner.cpp):

init_args_dump()
  dump_collector_.initialize(..., output_prefix_)
  kernel_args_.args.dump_data_base = dump_collector_.get_dump_shm_device_ptr()
start()                          ← spawn split mgmt threads (drain/refill
                                   + replenish), then collector shards
launch AICPU / AICore
observe run completion
close_ordinary_run(completed || producers_joined)
  proof: quiesce() → reconcile_counters() → export_dump_files()
         collector threads remain resident
  unknown: stop(false) → export_dump_files()
           join host readers; skip final device drain and held recovery
           dropped_records_complete is false

ArgsDumpCollector on a2a3 inherits from profiling_common::ProfilerBase<ArgsDumpCollector, DumpModule>: the base class owns split mgmt threads, collector shards, and the BufferPoolManager<DumpModule> they share. ArgsDumpCollector only supplies the dump-specific pieces — the DumpModule trait that describes the shared-memory layout, initialize that allocates and pre-fills free queues, an on_buffer_collected callback that gathers payload bytes and appends metadata to a shard-local record list, plus reconcile_counters / export_dump_files / finalize. The mgmt/collector threading, buffer pooling, and Module trait pattern are shared with PMU and ChipSwimlane — see profiling-framework.md for the framework reference.

5.5 a5 — same framework, host-shadow transport

a5's ArgsDumpCollector derives from ProfilerBase<ArgsDumpCollector, DumpModule> and shares the split mgmt + collector shard structure with a2a3. The architectural deviation from §5.4 is the transport channel: a5 has no halHostRegister, so each device buffer is paired with a host-shadow malloc() and the framework synchronizes selected fields via profiling_copy.h (rtMemcpy onboard, memcpy in sim). MemoryOps therefore carries five callbacks (alloc / reg / free_ / copy_to_device / copy_from_device). The mgmt threads pull queue indices and ready entries on demand, then push back only the fields host modified (advanced queue_heads[q], refilled free_queue.tail and buffer_ptrs[slot]) via BufferPoolManager::write_range_to_device. Each popped DumpMetaBuffer is pulled inside ProfilerAlgorithms::process_entry. The per-thread arena lives outside the shm region, so on_buffer_collected separately refreshes arena_write_offset and copies the arena bytes. The arena-ack pass refreshes each thread's published_payload_count and the device-entry run_published_payload_base. For ordinary collection it compares the published cursor with that baseline plus the thread's current-window writer completion count. For each thread whose counts match, Host writes that published watermark back as completed_payload_count; AICPU requires this acknowledgement before reusing that thread's arena. The comparison and the write-back are both per-thread, so one lane's outstanding payload never holds up another's arena.

        HOST                                         DEVICE
┌──────────────────────────┐               ┌──────────────────────────┐
│ ArgsDumpCollector      │               │ AICPU thread             │
│   : ProfilerBase<...>    │               │                          │
│                          │               │                          │
│ initialize()             │  alloc + reg  │ dump_args_init()         │
│   rtMalloc shm           │──+ shadow────>│   read DumpDataHeader    │
│   per-thread arenas      │   memset 0    │   cache per-thread ptrs  │
│   per-thread             │   + push 0s   │                          │
│   DumpMetaBuffers        │               │ per-task run loop:       │
│   register_mapping(s)    │               │   BEFORE_DISPATCH        │
│                          │               │     dump_arg_record()    │
│ start(thread_factory)    │               │   dispatch kernel        │
│   split mgmt starts      │               │   wait FIN               │
│   collector shards start │               │   AFTER_COMPLETION       │
│                          │               │     dump_arg_record()    │
│ mgmt polling:            │               │   if buffer full:        │
│   refresh queue fields   │<──memcpy─────<│     push ready entry,    │
│   for each ready entry:  │               │     pop next from free_q │
│     copy buf from device │<──memcpy─────<│                          │
│     resolve host ptr     │               │ dump_args_flush():       │
│     push to host ready_q │               │   push remaining buffers │
│   advance queue_heads,   │               │   to ready_q             │
│     refill free_queues   │               │                          │
│   write_range_to_device  │──memcpy──────>│                          │
│     for each modified    │               │                          │
│     field                │               │                          │
│                          │               │                          │
│ collector shard:         │               │                          │
│   wait_pop_ready         │               │                          │
│   on_buffer_collected →  │               │                          │
│     copy arena slice     │<──memcpy─────<│                          │
│     extract DumpedArgs   │               │                          │
│     queue to writer thrd │               │                          │
│   notify_copy_done       │               │                          │
│                          │               │                          │
│ observe completion       │               │                          │
│ ordinary close(proof)    │               │                          │
│   proved: quiesce        │               │                          │
│     reconcile counters   │               │                          │
│   unknown: stop(false)   │               │                          │
│     no held recovery     │               │                          │
│ export_dump_files()      │               │                          │
│   → <output_prefix>/     │               │                          │
│     args_dump/...        │               │                          │
└──────────────────────────┘               └──────────────────────────┘

Lifecycle (device_runner.cpp):

init_args_dump()
  dump_collector_.initialize(num_dump_threads, ..., output_prefix_)
  kernel_args_.args.dump_data_base = dump_collector_.get_dump_shm_device_ptr()
dump_collector_.start(thread_factory)   ← split mgmt + collector shards
launch AICPU / AICore
observe run completion
dump_collector_.close_ordinary_run(completed || producers_joined)
  proof: quiesce → reconcile counters → export; keep host threads resident
  unknown: stop(false) → export host-owned records; no held recovery
dump_collector_.finalize()

ArgsDumpCollector on a5 inherits the same CRTP base (profiling_common::ProfilerBase) as a2a3 and parameterizes BufferPoolManager with DumpModule. The only a5-specific glue is the 5-callback MemoryOps, targeted shared-field refreshes, and the on-demand arena copy inside on_buffer_collected.

a5 normally relies on per-thread AICPU flush (dump_args_flush) to move the current buffer into the ready queue. Recovery of a non-empty current_buf_ptr requires proof that producers stopped, plus current-epoch and successful-read checks. A failed onboard fence supplies no independent stop proof: close exports host-owned records, skips held recovery and marks the dropped-record count incomplete.

5.6 a2a3 vs a5 at a glance

Aspect a2a3 a5
Device-side layout identical (same DumpDataHeader / DumpMetaBuffer / arena shape, static_assert-checked)
AICPU recording logic identical
Buffer model rotating pool (free + ready queues per thread) identical
Host threads split mgmt + collector shards, streams during execution identical
Host-class shape ProfilerBase<ArgsDumpCollector, DumpModule> identical
Host transport halHostRegister shared memory host-shadow malloc + targeted rtMemcpy/memcpy
MemoryOps callbacks 3 (alloc, reg, free_) 5 (+ copy_to_device, copy_from_device)
Arena access direct via SVM targeted copy_from_device inside on_buffer_collected
Tensor-payload support Supported Supported
Per-task capture by runtime both runtimes tensormap_and_ringbuffer only; host_build_graph has none and refuses the request (§3.1.1)
reconcile_counters recover held buffers and snapshot loss counts only after producer-stop proof and host quiescence identical
Ordinary close proof: quiesce → reconcile → export; unknown: stop(false) → export host-owned records identical

6. Overhead

Args Dump is opt-in and zero-overhead when disabled — without --dump-args the host does not allocate dump storage and AICPU / AICore skip the dump-specific code paths. The pipe_barrier(PIPE_ALL) before FIN is also gated on the same handshake bit.

With --dump-args, AICPU records mode-selected tensor/scalar metadata and copies tensor payload according to the level table in §3.1. Scalar values stay manifest-only. When tensor payload is selected, the per-task overhead is dominated by:

  • The BEFORE_DISPATCH / AFTER_COMPLETION payload memcpy into the per-thread arena (contiguous fast-path; logical traversal for non-contiguous views).
  • The completion pipe_barrier(PIPE_ALL) before writing FIN, which serializes all device-side writes for dumped tasks.
  • The arena and metadata writes themselves; host drain/replenish and collector work runs concurrently with the stream on both architectures. a5 additionally pays rtMemcpy/memcpy transport cost to keep host shadows in sync.

For interactive debugging, total memory pressure is what to watch: the default per-thread arena is 128 MiB (8 × 256 × 64 KiB), so a 7-thread run reserves ~896 MiB on device.

7. Limitations

The runtime-level one comes first: a5 host_build_graph has no per-task capture, and a request for it is refused at simpler_prepare_run rather than served partly or emptily. §3.1.1 is the full account; the table in §2 is the per-runtime answer.

Three pressure or failure conditions then exist, on the runtimes that do capture, when dump buffers run out of space. Successful arena backpressure preserves payload; truncation and record discard surface in the JSON manifest plus the dump_args_flush log line so users can detect and diagnose them.

7.1 Truncation (truncated = true)

Trigger: a single tensor's logical payload (numel × elem_size) exceeds the entire per-thread arena size.

Mechanism (identical on a2a3 and a5): before copying, AICPU compares bytes against arena_size. When bytes > arena_size, only arena_size / 2 bytes are copied and the record is flagged truncated = 1.

bytes = numel × elem_size
if bytes > arena_size:
    copy_bytes = arena_size / 2     ← half the arena
    truncated  = true

Effect: the tensor entry has "truncated": true and bin_size is smaller than the full tensor. The payload contains the first arena_size / 2 bytes of the logical layout (gathered or contiguous), enough for statistical sampling.

Tuning: raise PLATFORM_DUMP_AVG_TENSOR_BYTES (arena grows proportionally) so the arena is at least as large as the biggest tensor you need to inspect.

7.2 Arena payload acknowledgement

arena_write_offset remains monotonic and physical writes use % arena_size. Before reserving an offset or copying payload bytes, AICPU checks whether the arena is already one full physical cycle deep or whether the new payload would cross the physical end. If so, it seals and publishes the current metadata buffer, then waits for the host to acknowledge the payloads it has published.

At each RQ publish point, AICPU counts the non-empty tensor payload records in that metadata buffer and adds them to that thread's published_payload_count. The host's single writer increments the originating thread's completion count only after args.bin accepts a payload, and publish_arena_acks() — run once per replenish tick — writes that watermark back as the thread's completed_payload_count. The waiting thread proceeds once its own completed_payload_count reaches the value it published, then aligns its monotonic logical offset to the next physical arena boundary and writes the pending payload from the arena start.

The ordinary host window counts writes from zero. Device initialization snapshots the lifetime published cursor into run_published_payload_base; adding that baseline converts current-window disk completions to lifetime transport credits. The retained path continues using lifetime receipt/discard credits. Neither host admission nor device entry resets published/completed cursors.

Every quantity here is per-thread: each AICPU thread owns its own arena, so thread t may wrap as soon as thread t's own payloads are on disk. The cursor is never reset and no reclaimed/published arena offset is needed. The wait is bounded by the 30-second host-crash backstop; on expiry the record is dropped.

7.3 Record discard (dropped_record_count / dropped_records)

Trigger: the metadata record buffer (not the payload arena) is full and no replacement buffer is available.

Mechanism (identical on a2a3 and a5): each thread rotates through multiple metadata buffers via an SPSC free queue. When a buffer fills (256 records), AICPU tries to:

  1. Enqueue the full buffer to the per-thread ready queue (for the host mgmt thread to pick up).
  2. Pop a fresh buffer from the free queue.

If the ready queue is full or the free queue is empty, AICPU spins at that buffer-switch gate while the host drains or refills that thread's queue — no other lane is involved. If the host-crash timeout expires, the current records are accounted as dropped.

// Reuse current buffer — account for lost records
account_dropped_records(state, cur_buf.count)
cur_buf.count = 0          ← reset and reuse
dropped_record_count += N      ← saturating lifetime loss counter
run_dropped_record_count += N  ← saturating loss counter for this invocation

The same fallback applies during dump_args_flush() at end of execution if the ready queue is full.

Effect: dropped_records in the manifest summary counts device-reported record losses in this invocation. Ordinary manifests also include dropped_records_complete: it is true only after stopped-producer reconciliation has read the invocation's counters and resolved every held metadata buffer without an uncounted transport loss. A failed shared-state or held-buffer read, an unattributable held buffer, a retired unreadable ready buffer, or a saturated counter leaves it false. A held-buffer read failure leaves its potential record loss unknown even when the device's loss counter was readable. Without completed reconciliation, including when producer stop is unknown, it also remains false. In those cases the numeric field is an observed lower bound; zero does not prove zero loss. This flag describes record-loss accounting only: it does not certify payload reads, file writes, or the completeness of the entire artifact.

Ordinary collection retains its no-data convention: when no records were collected, it writes no manifest, including on an unproven close. Absence of an artifact is not evidence of zero device loss or complete collection. The completeness flag describes ordinary manifests that were actually written.

Device initialization clears run_dropped_record_count, independently of the lifetime counter used by retained collection. A saturated lifetime counter therefore does not hide a later ordinary invocation's losses. Dropped arguments do not appear in the args[] array at all.

Tuning: raise PLATFORM_DUMP_BUFFERS_PER_THREAD (more rotation buffers) and/or PLATFORM_DUMP_READYQUEUE_SIZE (deeper host hand-off queue).

7.4 Summary matrix

Condition Flag Metadata Payload a2a3 a5
Runtime has no per-task capture request refused -1001, nothing prepared None None — a5 host_build_graph (§3.1.1)
Tensor > arena truncated Preserved Partial (arena/2 bytes) Same Same
Arena host writer falls behind none on success Preserved Preserved after bounded wait for this thread's ack Same Same
Record buffer full, no free buffer dropped_records; dropped_records_complete states accounting confidence Lost Lost After gate timeout Same
Ordinary close without producer-stop proof dropped_records_complete: false when a manifest exists; no artifact does not prove zero loss Host-owned records preserved; held records not recovered Host-owned payload preserved; held payload not recovered Same Same

7.5 Configuration knobs

All defaults live in each platform's platform_config.h. The values below match between a2a3 and a5 except for the platform thread cap:

Constant Default Effect
PLATFORM_DUMP_RECORDS_PER_BUFFER 256 Max records per metadata buffer
PLATFORM_DUMP_BUFFERS_PER_THREAD 8 Arena size multiplier and SPSC free queue depth
PLATFORM_DUMP_AVG_TENSOR_BYTES 64 KiB Arena size multiplier
PLATFORM_DUMP_MAX_DIMS 5 Upper bound on shape / offset arrays
PLATFORM_MAX_AICPU_THREADS a2a3: 4; a5: 5 Maximum number of dump-producing threads

Per-thread arena = BUFFERS_PER_THREAD × RECORDS_PER_BUFFER × AVG_TENSOR_BYTES = 8 × 256 × 65536 = 128 MiB.

8. FAQ and Debug Guide

No args_dump/ directory or args_dump.json in the output. Check that --dump-args was passed; without it the host does not allocate dump storage. The collector exports files only after receiving at least one record, so the same result is expected when the kernel dispatches no arguments or when partial has no CoreTaskArgs::dump(...) markers. Add markers (§3.2), or pass --dump-args full.

args.bin fails known-value validation. The scene test asserts a known payload value in every mode that writes one, on both platform families, so a fresh failure here is a real capture defect rather than an accepted platform limitation.

Manifest has tasks but expected tensor records are missing. AICPU received a payload whose tensor count or metadata did not match what the orchestrator registered. Look for a LOG_WARN from try_log_dump_args_layout_mismatch — it identifies the first mismatched task, then is suppressed to avoid log flooding. A scalar-only task has no tensor records but still carries its scalar records in the manifest. For host_build_graph, ensure set_tensor_info_to_task (or add_task_with_tensor_info) was called for every task. For tensormap_and_ringbuffer, ensure every task that expects a tensor output calls alloc_tensors with a TensorCreateInfo, and every input tensor is added via add_input / add_output / add_inout.

AFTER_COMPLETION data looks stale or partially written. This should not happen with the runtime barrier in place — AICore issues pipe_barrier(PIPE_ALL) before FIN when dump is enabled. If you see it, verify the executor saw SIMPLER_DFX_FLAG_DUMP_ARGS set in the handshake (a missing handshake bit silently disables the barrier).

truncated_args > 0 in summary. A tensor exceeded the per-thread arena (default 128 MiB). Bump PLATFORM_DUMP_AVG_TENSOR_BYTES to extend the arena and rerun.

dropped_records > 0 in summary. Metadata-buffer pressure. On a5 raise PLATFORM_DUMP_RECORDS_PER_BUFFER; on a2a3 raise PLATFORM_DUMP_BUFFERS_PER_THREAD and/or PLATFORM_DUMP_READYQUEUE_SIZE.

Viewer reports "no outputs/*/args_dump directory found". Either the run did not produce one (see first question), or the viewer's working directory differs from the run's. Pass the explicit dump-dir path to the viewer: python -m simpler_setup.tools.dump_viewer outputs/<case>_<ts>/args_dump.

The run hung (AICore op-timeout / scheduler timeout) — is there still a dump? Yes. The host exports the manifest even though run() returns the timeout error. Within the selected level's record scope, it contains task inputs captured at BEFORE_DISPATCH, outputs from tasks that completed before the hang, and any recoverable partial output from an in-flight task. This rests on the opportunity for the AICPU to detect and dump the hang before a hardware watchdog or Host wait expires. The configured budgets express that intended relationship; they do not establish actual firing order:

SCHEDULER_TIMEOUT_MS (20 s)           <  PLATFORM_OP_EXECUTE_TIMEOUT_US (45 s)  <  PLATFORM_STREAM_SYNC_TIMEOUT_MS (50 s)
   AICPU declares hang,                   STARS reaps the AICore op              host stream sync gives up
   flushes + dumps in-flight              and poisons the context                and surfaces the error

These defaults can be overridden without rebuilding by setting SIMPLER_SCHEDULER_TIMEOUT_MS, SIMPLER_OP_EXECUTE_TIMEOUT_US, and SIMPLER_STREAM_SYNC_TIMEOUT_MS. Invalid requested values or requested onboard orderings are ignored with a warning and fall back to the defaults. Owned onboard initialization then checks the ACL-reported operator timeout. A successful setter with an invalid adjusted ordering emits a warning and continues with the original budgets; setter failure still rejects initialization. The host does not enlarge the stream budget automatically. Borrowed kernel contexts do not configure or validate the caller's external operator watchdog. The onboard host also requires stream-sync to cover the scheduler budget plus a 1.5 s scheduler-arming guard for cold init work before the no-progress timer starts. This guard covers fixed/cold costs such as kernel registration, orchestration SO dlopen, runtime init, and AICore handshake. It cannot know the graph-specific maximum orchestration producer wall time, so callers that raise scheduler/op timeouts must also size SIMPLER_STREAM_SYNC_TIMEOUT_MS for their worst-case orchestration window. Sim builds do not have STARS or ACL stream-sync timeouts, but scheduler overrides are still parsed and applied independently so slow CPU-sim kernels can raise the no-progress budget without onboard-only ordering limits. CI restores the fast-fail values through these env vars: 2 s scheduler, 3 s requested op-execute, and 6 s stream-sync for onboard jobs; 10 s scheduler for sim jobs. ACL may adjust the operator value; its reported setting is not a measured firing deadline.

  • Device-side graceful flush (primary). At 20 s of no progress the AICPU declares the hang, runs the end-of-loop flush, and dumps the partial output of every task still RUNNING on a core — written at the after_completion stage, reflecting current GM, so you can see how far a stuck kernel got and how much it wrote. Because this fires before STARS reaps the op (45 s), it normally completes while the kernel is still wedged. Only records whose publication reaches the host are available without a later stop proof. Best-effort: only AICore writes already drained to GM are visible (the stuck core issued no pipe_barrier, so writes still in its cache are not), and a read may be torn if the core is mid-write. To tell a running task's partial output apart from a genuine completion, cross-reference the [STALL …] state=RUNNING / SHUTDOWN_SNAPSHOT log lines — those task_ids are the in-flight ones.
  • Host-side recovery after proven stop. A producer that stops before flushing can leave its last buffer on the device (current_buf_ptr != 0). Only after observed execution completion or independent producer termination may reconcile_counters recover its records from the device buffer and arena. An onboard failed fence provides neither proof, so it does not enable this backstop. Recovery requires the current invocation's initialized lane and buffer epochs and successful shared-state and buffer reads; failed reads do not authorize cached-shadow recovery. Records already owned by the host retain best-effort export. Unread producer accounting or an unreadable held buffer leaves dropped_records_complete: false; dropped_records: 0 alone does not prove no loss. Recovery requires either observed execution completion or independently joined producers. Without either proof, ordinary close joins the host drain, replenish, and collector threads before exporting host-owned records, and skips the final device-queue drain and recovery of producer-held buffers. Entries already handed to host collectors finish; later device publications remain on the device. This host shutdown does not establish device completion or authorize freeing device storage. Epoch and read checks alone do not establish quiescence.

If STARS reaps the operator before the AICPU publishes its dump, recovery may be limited to records already available to Host collectors. Numeric timeout ordering alone does not prove that the device-side dump has run. Host waits use per-Event or per-Stream budgets rather than one deadline for the entire run; see local timeout boundaries. The configured chain lives in platform_config.h, which holds all three (PLATFORM_SCHEDULER_TIMEOUT_MS, surfaced as SCHEDULER_TIMEOUT_MS — 20 s, one value for onboard and sim, PLATFORM_OP_EXECUTE_TIMEOUT_US / PLATFORM_STREAM_SYNC_TIMEOUT_MS). The env overrides use those constants as their unset fallback and keep the #897 distributed-skew trade-off.