Scope Stats — Per-scope Resource Usage Peaks¶

Scope stats records the peak resource usage — task-window slots, heap bytes, dep-pool entries, and tensormap entries — for every SIMPLER_SCOPE region in an orchestration, so you can see which scope drove each resource to its high-water mark. When a model runs out of task windows, heap, or tensormap / dependency-list entries, the failure tells you which resource is exhausted but not where; scope stats gives you the where.

It is a diagnostic-only, opt-in feature for the tensormap_and_ringbuffer (T&R) runtime. When disabled (the default) it costs a single bool load per probe.

This guide also covers the background behind the feature, the T&R resource/ring_depth/scope model, and the data flow behind the HTML report.

1. Quick Start¶

The full workflow is three steps: enable the flag, run, then turn the resulting scope_stats/scope_stats.jsonl into an HTML report.

Step 1 — Run with `--enable-scope-stats`¶

Pass the flag to any T&R example or scene test:

CASE=...
NAME=...
python "tests/st/${CASE}/test_${NAME}.py" -p a2a3 -d 0 --enable-scope-stats

The flag is bit 4 of enable_profiling_flag; on a T&R run it turns on per-scope peak tracking. On other runtimes the flag is accepted but produces no records.

Step 2 — Locate the output¶

The run writes scope_stats/scope_stats.jsonl under the per-task output prefix. It is NDJSON: line 1 is run metadata, every subsequent line is one scope sample. See §3 for the schema.

The metadata line is also the quickest way to verify per-ring runtime sizing. For example, after running with:

# In the scene test's CASES entry -- ring sizing is per task, so it travels with
# the case rather than with the process:
"config": {
    "runtime_env": {
        "ring_task_window": [8192, 16384, 131072, 524288],
        "ring_heap": [134217728, 268435456, 402653184, 536870912],
        "ring_dep_pool": [4096, 8192, 16384, 32768],
    }
}

python "tests/st/${CASE}/test_${NAME}.py" -p a2a3 -d 0 --enable-scope-stats

inspect the first line:

python - <<'PY' path/to/output_prefix/scope_stats/scope_stats.jsonl
import json, sys
with open(sys.argv[1]) as f:
    meta = json.loads(f.readline())
print("task_window_max =", meta["task_window_max"])
print("heap_max        =", meta["heap_max"])
print("dep_pool_max    =", meta["dep_pool_max"])
PY

The three arrays are indexed by ring (0..3) and should match the effective runtime configuration. Per-sample ring values show which scope-depth rings were actually touched by the run; they are scope records, not task counts.

Step 3 — Visualize with `scope_stats_plot`¶

python -m simpler_setup.tools.tmr.scope_stats_plot "path/to/output_prefix"/scope_stats/scope_stats.jsonl
# writes "path/to/output_prefix"/scope_stats/scope_stats.html

# or send the report elsewhere:
python -m simpler_setup.tools.tmr.scope_stats_plot path/to/scope_stats.jsonl --out-dir /tmp/report

Argument	Required	Meaning
`jsonl`	yes	Path to a `scope_stats.jsonl` produced by Step 1
`--out-dir DIR`	no	Where to write `scope_stats.html` (default: next to the input)

The output is a single self-contained scope_stats.html — the SVG charts and a small chart-expansion script are inlined (no matplotlib, no external JS/CDN), so it opens offline and is trivial to share. Open it in any browser.

Reading the report¶

The report header keeps only orientation metadata: the JSONL name, the dominant scope site, a sampling-status chip, and small record-count chips. The Formula Guide defines the measured resources and chart metrics before the data tables. The Top Peaks table is the primary diagnostic summary: ring_depth-scoped resources list peaks by ring_depth, while TensorMap appears as one global row. The Use column shows which resource group is closest to capacity.

Below that, the report is grouped by resource. task_window, heap, and dep_pool are then split by ring_depth; TensorMap uses one global panel because it is not a per-ring_depth resource. Each panel has an inline SVG chart stack plus a right-side summary: Line colors lists each line's color, meaning, peak, and capacity use; Max use shows the selected risk metric, peak, capacity, use, peak scope, and peak site. For TensorMap, Max use also shows Peak context ring_depth, which is the scope/ring_depth context where the global TensorMap peak was observed. task_window, heap, and dep_pool put High water / Live at exit in the main resource-pressure chart, and render the per-scope allocation curve in a separate chart below it so small allocation changes stay readable. tensormap shows one global live-entry curve. Charts include scope-index ticks on x and observed-usage ticks on y. Percentages are rendered with two decimal places. The x-axis uses readable integer scope steps; the y-range is peak * 1.1, while y-grid ticks use readable integer or human-friendly steps. Hovering a point shows a highlighted dot plus its metric, scope index, y value, and source site. Clicking any chart opens a larger modal view of that chart; closing the modal releases the cloned SVG.

The sections below provide a more detailed walkthrough of the header metadata, Formula Guide, Top Peaks table, ring_depth panels, chart axes, and common patterns.

Metric	Formula	What it tells you
`scope_high_water`	`end.head − begin.tail`	Upper bound on the occupancy this scope reached — entry backlog plus everything this scope allocated, without subtracting what was released mid-scope. Not a realized peak, and not bounded by capacity: a scope that streams more than `cap` reports more than `cap`.
`real_occupancy`	`end.head − end.tail`	What is still occupied at scope exit (the live level on leaving). Always in `[0, cap]`.
`scope_alloc`	`end.head − begin.head`	How far the allocation frontier advanced over the scope — this scope's total allocation throughput (can exceed `cap`).

For task_window, heap, and dep_pool, head / tail mean that resource's allocation frontier and released boundary at the sampled scope boundary. All three are reported as monotonic, non-wrapping counters (byte totals for heap, entry counters for the others), so every delta above is an exact, non-negative subtraction regardless of how many times the underlying ring wrapped during the scope — no wrap correction is applied. tensormap is reported as a single in-use value (no head/tail), so it shows only the real_occupancy curve.

2. What gets captured¶

The whole orchestration, automatically. You do not mark a region to profile — instrumentation lives inside the SIMPLER_SCOPE macro itself, so every scope in the orchestration is recorded once the flag is on. The executor wraps the orchestration entry in a root SIMPLER_SCOPE (depth 0), and every user-written SIMPLER_SCOPE nests under it, so the report covers the entire orch scope tree end to end.
One sample per scope boundary. Each SIMPLER_SCOPE emits a begin record on entry and an end record on exit. The plot tool pairs them by site to compute the metrics above.
Per-ring. Each ring's task window, heap, and dep pool are tracked independently; a scope only ever touches its own ring (ring = min(depth, CHIP_MAX_RING_DEPTH−1)).
Capacity denominators. The *_max capacities in the metadata line are snapshots of what the runtime actually configured for that run, so used/cap ratios are exact.
Disk-bounded, no wrap. Full buffers stream off the device during the run, so the record count is bounded only by disk — there is no fixed-ring overwrite.

3. Output: `scope_stats.jsonl`¶

NDJSON. Line 1 is run metadata; each subsequent line is one scope sample (begin or end). heap_start/heap_end are monotonic cumulative bytes, not wrapping ring offsets:

{"fatal": false, "dropped": 0, "total": 4, "task_window_max": [8, 4], "heap_max": [268435456, 268435456], "dep_pool_max": [1024, 1024], "tensormap_max": 65536}
{"site": "example_orchestration.cpp:77", "phase": "begin", "depth": 1, "ring": 1, "task_window_start": 0, "task_window_end": 0, "heap_start": 0, "heap_end": 0, "dep_pool_start": 1, "dep_pool_end": 1, "tensormap": 0, "run_epoch": 7, "buf_seq": 0}
{"site": "example_orchestration.cpp:77", "phase": "end", "depth": 1, "ring": 1, "task_window_start": 0, "task_window_end": 4, "heap_start": 0, "heap_end": 8192, "dep_pool_start": 1, "dep_pool_end": 6, "tensormap": 5, "run_epoch": 7, "buf_seq": 0}

Metadata line (line 1):

Field	Type	Meaning
`fatal`	bool or null	`true` iff a fatal was latched during the run; records past it are diagnostic-only
`dropped`	uint or null	Records dropped on device (free_queue empty / ready_queue full); `0` on a healthy run
`total`	uint or null	Total records the device attempted (collected + dropped)
`task_window_max`	int[] or null	Per-ring task-window capacity (indexed by `ring`)
`heap_max`	int[] or null	Per-ring heap-byte capacity (indexed by `ring`)
`dep_pool_max`	int[] or null	Per-ring dependency-list pool capacity (indexed by `ring`)
`tensormap_max`	int or null	Tensormap entry capacity (scalar)

Retained TMR runs carry these capacities from their resolved runtime layout into their own export slot, including when binding uses a cached arena. A delayed export therefore keeps its run's configuration after a successor initializes. Missing required capacity metadata refuses admission; numeric zero is not an unknown-capacity sentinel. The serialized, retention-off path freezes checked shared metadata after producer completion or an independent producer-stop proof.

At the counting limit these two numbers saturate rather than wrap: the device counters clamp at 4294967295, and a run that reaches it is reported with unknown counts instead of a small wrapped number that would look complete. The figure does not certify that every attempted sample reached the host.

Ordinary close without producer-stop proof¶

A failed execution fence does not prove that the device producer stopped. Without either execution completion or an independent producer-stop proof, the ordinary boundary stops chasing device publications, finishes entries already handed to host ownership, and joins its host readers. It does not read final shared accounting or recover the producer's unpublished buffer. Host-thread exit does not authorize freeing device allocations. Published buffers left in the device queue retain their original epoch; after restart, an admitted ordinary collector rejects and reports records from another epoch rather than importing them into its new run.

If this run has host-owned samples, they are exported with this metadata:

{"fatal": null, "dropped": null, "total": null, "task_window_max": null, "heap_max": null, "dep_pool_max": null, "tensormap_max": null, "collection_verdict": "partial_cut_unknown", "counts_unknown": true, "host_received_records": 2, "host_retained_records": 2}

fatal, dropped and total are unknown, not false or zero. The two host counts name received and retained samples; their difference records a host storage refusal and is not a device drop count. Capacities come only from this run's host binding, saved by value at admission. The binding supplies all four capacity fields together, or all four are null; numeric zero is never an unknown sentinel.

A begin sample without its end remains in JSONL. The plot labels incomplete statistics and unpaired samples; it does not fabricate pairs, report unknown counts as healthy, or compute percentages from missing capacities. With no host sample, an unproven close emits no artifact; artifact absence proves neither zero samples nor zero loss. When reusing an output prefix, an empty unproven close removes only the old scope_stats/scope_stats.jsonl, so it cannot leave a predecessor's healthy artifact at the current result path; unrelated files remain. A removal failure is logged and returned. It does not export prior-run metadata.

With a stop proof, ordinary collection keeps its resident threads, recovers held records and preserves its seven-key metadata line. A direct export without successful checked metadata still refuses, including read failure or a required producer that never initialized. Retained runs keep their stricter quarantine/no-artifact contract on incomplete execution.

Background mode (`collect_across_runs=True`)¶

With Worker(collect_across_runs=True) on a level-3 worker, the run boundary keeps the receive drain and the terminal read and hands only the rendering and the file write to a background writer, so the next run's device work can start while this run's file is still being written. run() returning no longer means scope_stats.jsonl exists — Worker.flush_diagnostics() is the barrier that says the files up to that point are published. TMR only; on host_build_graph there is no producer and the boundary behaves as it always has. The default (collect_across_runs=False) path continues to attempt export at its run boundary.

The four collection-quality keys below are used by background exports and by unproven ordinary closes. Proven ordinary exports keep their original seven-key metadata line:

Field	Type	Meaning
`collection_verdict`	string	`published`, `partial_safe` (loss, host refusal or a latched fatal, counts still known) or `partial_cut_unknown` (unproved producer stop, a saturated counter or an accounting mismatch)
`counts_unknown`	bool	`true` only when the counts themselves cannot be trusted; a device fatal alone leaves this `false`
`host_received_records`	uint	Records the host took delivery of; held-buffer recovery is included only when producer stop is proved
`host_retained_records`	uint	Of those, how many host storage kept; the 256 MiB per-collector budget applies only to background collection

A run that ends without the producer handing its last buffer over leaves that buffer named in current_buf_ptr, possibly containing unsettled records and still stamped with the run that acquired it. Before handshake and orchestration configuration, the next enabled run's AICPU leader resets the four producer counters and re-stamps the held buffer's count, run_epoch and local_seq. This device entry runs after the preceding producer has stopped. Ordinary/unmapped collection settles the predecessor's host boundary before reuse; mapped retained collection uses its device terminal's live_at_close, even when host sealing happens later. The rule applies both with and without a retained terminal bank; host admission clears only its host collection window and performs no producer-state readback or header upload. The pointer and the free queue are left alone: the drain shard is the sole runtime writer of the free queue, and clearing the pointer would orphan a pool buffer. A mapped terminal bank supplies per-run accounting publication; it does not select who initializes the producer. Disabled invocations leave the resident producer state and terminal untouched, even when sim retains the pool address. The shared accounting epoch is published after initialization, using existing padding without growing the shared region. Ordinary collection checks it against the admitted run before recovery or export. A run that never reaches device initialization, or whose shared-region read fails, cannot export the predecessor's accounting. This marker identifies initialization; it does not prove completion. A producerless runtime (CapacityMetadata.required=false, as in HBG) still exports its metadata-only artifact; it has no producer epoch to validate. Unknown/required producers retain the epoch check.

Ordinary collection is best effort: a rejected snapshot is logged and no artifact is written; the diagnostic rejection does not replace the computation result. Retained collection uses its existing collection-error reporting below.

In retained mode, a published file is not by itself a success: loss, a fatal and unknown counts each make flush_diagnostics() raise, and the error is sticky for the runner's life. A file already occupying the destination is never removed or overwritten — the run fails instead. And a run whose device completion could not be proved writes no file at all: nothing shared is read on that path, its host copies are discarded once the collector threads are joined, and every later flush_diagnostics() and close() reports the failure. While such a run is held, a later run that enables scope stats is refused before it is submitted.

Per-sample lines, oldest-first:

Field	Type	Description
`site`	`"basename:line"`	Source location of the `SIMPLER_SCOPE()` call
`phase`	`"begin"`/`"end"`	Scope entry or exit sample
`depth`	int	Nesting depth (0 = root scope inside the executor)
`ring`	int	Ring this scope used; indexes `task_window_max` / `heap_max` / `dep_pool_max`
`task_window_start`	int	Task-window ring tail at this boundary
`task_window_end`	int	Task-window ring head at this boundary
`heap_start`	uint	Heap reclaim boundary — monotonic cumulative bytes reclaimed (non-wrapping)
`heap_end`	uint	Heap allocation frontier — monotonic cumulative bytes allocated (non-wrapping)
`dep_pool_start`	int	Scheduler-published dependency-list pool tail at this boundary
`dep_pool_end`	int	Scheduler-published dependency-list pool top at this boundary
`tensormap`	int	Tensormap entries in use
`run_epoch`	uint	Which run produced this record. Copied out of the buffer the record came from, so it stays correct after that buffer is returned to the pool and reused by a later run. `0` means the producer had no run identity to stamp
`buf_seq`	uint	Generation from the producer's persistent rotation counter, so it identifies a buffer only together with `run_epoch` — not a cross-run ordering key

start/end are the ring's tail/head pointers at that boundary — see the metric formulas in §1 for how a scope's peak and occupancy are derived from a paired begin/end.

4. Internals¶

This section is for maintainers wiring scope stats into a runtime; users do not need it. There is no public C/C++ API — the only external interfaces are the --enable-scope-stats flag and the plot tool above.

4.1 Layering¶

Scope stats uses a platform-provides / runtime-calls pattern:

platform/include/aicpu/scope_stats_collector_aicpu.h
    Pure-value API declarations. No runtime types cross this boundary.

platform/shared/aicpu/scope_stats_collector_aicpu.cpp
    Owns all collector state (depth stack, peak arrays, shared buffer).
    Scope lifecycle, peak comparison, capacity registration, record writes.

platform/shared/host/scope_stats_collector.cpp
    Host side: allocates the shared header/buffer pool, streams full
    buffers off the device, reconciles counters, writes
    scope_stats/scope_stats.jsonl.

runtime (orchestrator.cpp, scheduler.h)
    Calls platform APIs at instrumentation points, passing extracted
    values (ring_id, task head/tail, heap head/tail, dep-pool top/tail, ...)
    as plain integers. No scope_stats source files live in the runtime directory.

4.2 AICPU platform API¶

Header: src/common/platform/include/aicpu/scope_stats_collector_aicpu.h

All entry points are extern "C" and take primitive types only, so the same collector links into any runtime that wires it up. Symbol resolution is unconditional (see §4.4), so call sites need no guards.

Single-producer contract: all *_peaks updates use non-atomic read-max-write and assume the orchestrator thread is the only writer. Concurrent callers may lose peaks silently — acceptable for diagnostic data, and it saves an atomic on the hot path.

// Host → AICPU init (called from kernel.cpp at kernel entry)
void set_scope_stats_enabled(bool enable);
void set_platform_scope_stats_base(uint64_t scope_stats_data_base);

// Runtime → AICPU init (once per ring at orchestrator init / scheduler attach)
void scope_stats_set_ring_capacity(
    int ring_id, int32_t window_cap, uint64_t heap_cap, int32_t dep_pool_cap
);
void scope_stats_set_tensormap_capacity(int32_t cap);

// Runtime → AICPU per-scope
void scope_stats_set_pending_site(const char *file, int line);
void scope_stats_begin(
    int ring_id, int32_t task_start, int32_t task_end,
    uint64_t heap_start, uint64_t heap_end,
    int32_t dep_pool_start, int32_t dep_pool_end,
    int32_t tensormap_used
);
void scope_stats_end(
    int ring_id, int32_t task_start, int32_t task_end,
    uint64_t heap_start, uint64_t heap_end,
    int32_t dep_pool_start, int32_t dep_pool_end,
    int32_t tensormap_used
);
void scope_stats_on_fatal();

enable mirrors the host's --enable-scope-stats flag; scope_stats_data_base is the device-visible address of the shared header the host allocates. set_platform_scope_stats_base doubles as device-side init (maps the header + per-instance buffer state, resets collector-local state). When enable=false every probe early-returns after one bool load.

A scope costs exactly two collector calls — begin and end — each carrying that boundary's sample for the scope's own ring. The dep-pool values come from orchestrator-published snapshots taken during Orch-side wiring, so they track the submit path directly. The runtime gates both on the local weak is_scope_stats_enabled() stub first, so a disabled run pays neither the cross-.so calls nor the cross-agent active_count() read (same idiom as is_dep_gen_enabled). SIMPLER_SCOPE() expansion calls set_pending_site(__FILE__, __LINE__) before begin; copy_basename keeps the JSON readable without forcing the host to chase a device pointer into the orchestration .so's string table. on_fatal sets header.fatal_latched, surfacing as "fatal": true; the host still emits whatever records made it.

ring_id outside [0, SCOPE_STATS_MAX_RING_DEPTH) is silently dropped. Caps are copied verbatim into the buffer header so the host can render used/cap without a second device→host query.

4.3 Comparison with other profiling subsystems¶

Feature	Layer	Runtime scope	Why
PMU	platform only	all runtimes	reads hardware registers
chip swimlane	platform only	all runtimes	reads AICore ring buffers
dep_gen	platform only	all runtimes	traces `submit_task`
args dump	platform only	all runtimes	dumps argument data
scope stats	platform API + runtime call sites	T&R only	runtime extracts values, platform tracks peaks

4.4 Symbol resolution¶

kernel.cpp (platform, shared by all runtimes) always calls set_scope_stats_enabled / set_platform_scope_stats_base, so the collector symbols resolve into every AICPU .so. Only the T&R runtime adds the begin/end/capacity call sites, so only it produces records; host_build_graph links the collector but never invokes it.

4.5 Data flow¶

Host                              AICPU (T&R runtime)
─────                             ─────────────────────
ScopeStatsCollector                platform scope_stats_collector_aicpu.cpp
  allocate header + buffer pool      set_platform_scope_stats_base(addr)
  pre-fill free_queue                set_scope_stats_enabled(true)
  set kernel_args fields             runtime: scope_stats_set_ring_capacity()
  launch kernel                      runtime: scope_stats_set_tensormap_capacity()
      │                                  │
  collector shard(s):                on SIMPLER_SCOPE begin/end:
   append records to memory  ◀──┐      runtime samples task/heap/dep_pool/tensormap
      │                         │      runtime: scope_stats_begin()/end()
      │                         │         └─ emit record, append to buffer;
  proved close: quiesce        └──────────── push full buffer to ready_queue
    recover held + snapshot          orch exit: flush remaining buffers
  unproven close: stop(false)
    join; host-owned samples only
  write_jsonl() (frozen host values)

A worked example is in tests/st/a2a3/tensormap_and_ringbuffer/dfx/scope_stats/test_scope_stats.py — it runs the vector_example orchestration with --enable-scope-stats and asserts the resulting NDJSON.

4.6 Future: cross-runtime support¶

If host_build_graph adds scope-like concepts, extending scope_stats only requires adding the same platform call sites in HBG — no platform changes. The collector is already runtime-agnostic: it accepts plain values and has no knowledge of T&R types.