Simulator Trace Cleaning

clean_sim_trace converts the MindStudio Insight binary dump (visualize_data.bin) emitted by the operator simulator into a de-cluttered, Perfetto-viewable AI-core pipeline trace (Chrome Trace Event JSON).

The simulator writes, per kernel run, an OPPROF_*/simulator/ directory with two profiling artifacts:

• trace.json — the official simulator-generated Perfetto/Chrome trace.
• visualize_data.bin — the binary container consumed by MindStudio Insight.

The official trace.json is lossy — it carries strictly less information than visualize_data.bin. The binary container holds the full set of MindStudio Insight blocks: the trace events plus per-instruction metrics (API_INSTR), source mapping, and other detail blocks. trace.json exports only the trace events. So clean_sim_trace reads the richer binary file directly, recovering the per-instruction metrics — emitted as an instr_metrics.json sidecar — that the official Perfetto export drops.

It also de-clutters the trace itself: opening trace.json in the Perfetto UI is hard to read because SET_FLAG / WAIT_FLAG synchronization slices and scalar address-arithmetic instructions bury the actual AI-core pipeline. clean_sim_trace rebuilds it into a clean pipeline view (see Rebuild rules).

Producing the dump

The OPPROF_*/simulator/visualize_data.bin this tool consumes is generated by the Ascend msprof op simulator (the cycle-accurate camodel), not by a normal PyPTO run. The incore-profiling skill drives it per kernel — for each PTOAS kernel in a build_output/<case>/ it generates a standalone testcase, builds it with ccec/bisheng, and runs the op-simulator:

python .claude/skills/incore-profiling/incore_profile.py \
  --build-dir build_output/<case> --func <kernel> --target a2a3

This writes .../kernel_insight_all_funcs_<ts>/funcs/<kernel>/collect/out/OPPROF_*/; point clean_sim_trace at that OPPROF_* directory.

Data-dependent kernels — those whose loop trip-count or work-table size is read from an input tensor — profile as a near-empty trace under the skill's auto-generated zeroed inputs (CUBE/VECTOR book ~0 cycles). That is a synthetic-input artifact, not a fast kernel; see the skill's Caveats for how to wire full-size real intermediates.

Usage

python -m pypto.tools.clean_sim_trace <path> [-o OUTPUT_DIR] [--keep-scalar] [--raw-metrics] [--no-copy-raw]

<path> is a visualize_data.bin file or an OPPROF_* directory (the tool locates simulator/visualize_data.bin inside it). Two files are written next to the input, or to -o if given. When -o points at a separate target directory, the raw binary trace is also copied alongside so the folder is self-contained:

Output	Content
trace.clean.json	Rebuilt Chrome Trace Event JSON — opens in chrome://tracing and the Perfetto UI
instr_metrics.json	Per-core instruction metrics from the API_INSTR block
raw_simulator/	Copy of the raw visualize_data.bin + per-core trace data (only with -o; the source OPPROF_* dir is left intact)

Flag	Effect
--keep-scalar	Keep the SCALAR setup lane (dropped by default)
--raw-metrics	Dump the API_INSTR block verbatim instead of reshaping it
--no-copy-raw	Skip copying the raw binary trace into <output-dir>/raw_simulator/

The visualize_data.bin format

The file is a flat sequence of length-prefixed blocks. Each block has a 12-byte, 4-byte-aligned header:

Offset	Size	Field	Meaning
0	uint64 LE	contentSize	payload length, including trailing padding
8	uint8	type	block type (2 = TRACE, 4 = API_INSTR, 1 = SOURCE, ...)
9	uint8	paddingLength	trailing zero bytes for 4-byte alignment
10	uint8	instrVersion	API_INSTR version marker
11	uint8	reserve	always 0x5a (binary-format magic)

Each block payload is plain JSON. SOURCE blocks prefix the payload with a fixed 4096-byte file path. The tool consumes the TRACE and API_INSTR blocks; other block types are skipped.

Rebuild rules

1. Lane selection — keep the pipeline lanes (MTE2, MTE1, CUBE, VECTOR, FIXPIPE, MTE3); drop CACHEMISS, FLOWCTRL, ALL. The SCALAR lane is dropped by default (--keep-scalar restores it).
2. Event filtering — keep X (complete) instruction events; drop SET_FLAG / WAIT_FLAG / BAR slices.
3. Lane ordering — emit process_* / thread_* metadata so cores and pipeline lanes render in dataflow order (load -> compute -> store).
4. Sub-lane packing — software-pipelined instructions on a pipe are often several in flight at once and only partially overlap. Chrome-trace X events on one tid must be disjoint or strictly nested, so each pipe's overlapping instructions are greedily split into sub-lanes (MTE1, MTE1#1, ...) — one row per concurrently-live instruction — preserving the true pipeline depth in the viewer (otherwise the depth collapses to ~2).
5. Sync as arrows — each SET_FLAG -> WAIT_FLAG pair becomes one flow arrow, re-anchored from the producing instruction to the consuming one (on its actual sub-lane).
6. Coloring — slices are recolored per pipeline lane.
7. Timestamps — kept verbatim, so the cleaned trace lines up with the raw trace.json.

Metrics sidecar

instr_metrics.json reshapes the API_INSTR block: each metric, stored in the raw block as an array indexed by core, is flattened into a per-core list of instruction records (address, pipe, cycles, vector_utilization_percentage, ...). The original Instructions Dtype map is preserved under column_types.