Debugging

A practical guide to debugging pypto-lib kernels — from compile errors through runtime hangs to precision mismatches. It pairs with compile-runtime-workflow.md (what each phase does), performance-tuning.md (perf), and precision-tuning.md (numerical fidelity — cast modes, dtype alignment, the error_distribution sweep).

The harness exposes most of these as both a run / run_jit kwarg and a CLI flag; a typical model __main__ wires them up like:

parser.add_argument("--runtime-dir", type=str, default=None)
parser.add_argument(
    "--dump-args", nargs="?", const=1, default=0, type=int,
    choices=(0, 1, 2, 3),
)
parser.add_argument("--enable-dep-gen", action="store_true")
...
result = run_jit(
    fn=indexer_test,
    specs=build_tensor_specs(...),
    golden_fn=golden_indexer,
    runtime_dir=args.runtime_dir,                     # reuse a compile (§3)
    runtime_cfg=dict(
        platform=args.platform, device_id=args.device,
        log_level="v5",                               # runtime log level; raise to v0 for hangs (§4)
        enable_dump_args=args.dump_args,              # argument dump (§5)
        enable_dep_gen=args.enable_dep_gen,           # dependency graph (§6)
    ),
    rtol=1e-3, atol=1e-3,
)

---

1. Read the pypto / ptoas error first

pypto's IR verifier and ptoas's assembler emit direct, actionable errors — they name the offending op, the bad shape / layout, and a source location. Most compile failures are fixed at the cited site without any further tooling; read the message before reaching for the heavier mechanisms below.

• Compile failure — scripts using golden.run call ir.compile directly, whose default dump_passes=True writes per-pass IR under build_output/<...>/passes_dump/. golden.run_jit constructs a RunConfig, whose default is dump_passes=False; pass compile_cfg=dict(dump_passes=True) when you need the same files. Diff the last clean pass against the first failing one to see which pass rejected the IR. report/ holds scheduling diagnostics.
• PTOAS failure — the error quotes the .pto op. With golden.run, compile_cfg=dict(skip_ptoas=True) keeps the raw .pto MLIR and stops before the C++ wrapper, isolating whether the regression is in PyPTO's IR-to-MLIR path or in PTOAS. skip_ptoas is not a RunConfig field and therefore is not accepted by golden.run_jit.
• Runtime crash — rerun on the matching simulator (-p a2a3sim / a5sim); it gives more diagnostic output than the device backend and reproduces most lowering bugs.

---

2. Replay failing data with golden_data

A run only snapshots its inputs to data/in/<name>.pt and its golden outputs to data/out/<name>.pt inside the build directory when it is called with save_data=True (full-model kernels expose this as --save-data). save_data is off by default, so without it nothing is saved and there is nothing to replay. When the snapshot exists, reproduce a failure on the exact same tensors instead of re-rolling random data by pointing a re-run at that directory:

python models/deepseek_v4_flash_mtp/decode_csa.py -p a2a3 -d 0 \
    --golden-data build_output/_jit_attention_csa_test_20260602_020256/data

golden_data="<dir>" loads <dir>/in/*.pt as inputs and <dir>/out/*.pt as the reference, skipping both input generation and golden_fn — it wins over golden_fn. This makes a mismatch deterministic and is the starting point for every precision investigation.

---

3. Reuse a compile with runtime_dir; edit .cpp / .pto and retest

runtime_dir="<build_output dir>" (CLI --runtime-dir) skips compile and codegen and runs the existing artifacts straight through the runtime/simpler. This is the tight loop for hand-editing generated code — a generated kernel or the orchestration — and re-testing in seconds:

1. Edit any kernels/aic/*.cpp / kernels/aiv/*.cpp or orchestration/*.cpp. You can also edit the raw ptoas/*.pto MLIR — the harness splices .pto edits back into the owning .cpp (rebuild_kernel_cpp_from_pto) and bumps its mtime.
2. Re-run with --runtime-dir pointing at the same directory (note: the build directory itself, not its data/ subdir):

python models/deepseek_v4_flash_mtp/decode_csa.py -p a2a3 -d 0 \
    --runtime-dir build_output/_jit_attention_csa_test_20260602_020256

The harness flags every .cpp whose .so/.o is missing or older than the .cpp, drops the cached binaries for the build, and the runtime rebuilds them. You do not need to rm the .o/.so yourself — editing the .cpp (mtime bump) is the signal.

The log states which path it took:

[cpp->.so] cpp edits or missing binaries detected (2 file(s)): kernels/aiv/foo.cpp, ...; rebuilding
[cpp->.so] no cpp edits since last build; reusing cached binaries

A single stale .cpp invalidates the cached binaries for the whole build directory, so batch all your edits before one run rather than re-running per file.

---

4. Runtime hang / deadlock — device log via log_level + ASCEND_PROCESS_LOG_PATH

When a run hangs (no progress, followed by AICPU sync-timeout messages) rather than raising a clean error, the Python side has nothing to show — the stall is on the device. Raise the runtime log verbosity and read the device log:

1. Set log_level in runtime_cfg:

runtime_cfg=dict(platform=..., device_id=..., log_level="v0")

log_level is a harness-only key — it is consumed up front (configure_log) and not forwarded to the runtime call. Accepted values: debug, v0..v9, info, warn, error, null. The runtime default is v5 (= INFO); lower is more verbose, so v0 (or debug) raises the detail above the default to surface the most runtime tracing.

1. Point the CANN / simpler runtime at a device-log directory before running:

mkdir -p build_output/device_logs
export ASCEND_PROCESS_LOG_PATH="$PWD/build_output/device_logs"
python models/deepseek_v4_flash_mtp/moe.py -p a2a3 --ep 2 -d 0,1

1. Read the logs under build_output/device_logs/ to find the last task that dispatched and which core stalled — that pins the kernel or dependency the schedule is waiting on.

ASCEND_PROCESS_LOG_PATH is a runtime environment variable, not a harness kwarg, so it is set in the shell. Create a directory writable by the current user before running; build_output/ is suitable local generated state. Hangs under high host concurrency are often false timeouts — run the suspect test serially before deep-diving.

---

5. Localize a precision mismatch with args dump

enable_dump_args writes dfx_outputs/args_dump/{args_dump.json,args.bin} — per-task tensor inputs/outputs and scalar inputs captured at kernel-task boundaries. Use it to turn a "the whole kernel is wrong" mismatch into "this one op is wrong".

Dump levels (runtime_cfg["enable_dump_args"]): 0 off · 1 partial — only tensor arguments you mark · 2 full — every task's tensor payloads and scalar values (heavy; can saturate the host collector / trip AICPU timeouts on large workloads) · 3 full metadata only — every task's tensor/scalar metadata, but no tensor payload or args.bin.

The usual flow — tag the tensor, dump at level 1. Mark the tensor of interest with pl.dump_tag(t) right where it's produced, then run with level 1:

h_tile_i8 = pl.create_tensor([RECV_TILE, MOE_INTER], dtype=pl.INT8)
pl.dump_tag(h_tile_i8)          # capture this one tensor under partial dump

run_jit(..., runtime_cfg=dict(platform=..., enable_dump_args=1))

pl.dump_tag works on plain function args and on internal pl.create_tensor GM tensors (incl. inside @pl.jit.inline); it returns the tensor unchanged and is a no-op when dump is off. Equivalent per-scope form: pl.at(..., dumps=[t]) / pl.submit(..., dumps=[t]). Prefer level 1 + tags over level 2 — it keeps the dump small and avoids the full-dump timeouts.

Inspect the result with the viewer — with no filters it lists every captured tensor (task_id / stage / role / dtype / shape); add filters (--task, --stage before|after, --role, --arg, -i N) + --export to decode the chosen tensors to args_dump/txt/ for element-wise comparison against torch:

python -m simpler_setup.tools.dump_viewer <build_output/.../dfx_outputs/args_dump>

Pass the dump dir explicitly — with no argument the viewer looks under ./outputs/*/args_dump, but run_jit writes to build_output/<...>/dfx_outputs/args_dump.

This section is the dump mechanism. For the end-to-end precision-localization workflow — pairing this dump with the error_distribution comparator to find the first stage whose distribution blows up, and the §1–§5 precision rules that stage likely violated — see precision-tuning.md §7.

---

6. Find missing dependencies with gen-deps

enable_dep_gen=True (CLI --enable-dep-gen) writes dfx_outputs/deps.json. Rendering is an offline step; the runtime does not create an HTML file automatically:

cd build_output/<ProgramName>_<timestamp>/dfx_outputs
python -m simpler_setup.tools.deps_viewer deps.json \
  --format text -o deps_viewer.txt
python -m simpler_setup.tools.deps_viewer deps.json \
  --format html --engine sfdp -o deps_viewer.html

Open deps_viewer.html when results are non-deterministic (values shift run to run, or shift with location) or a GM write→read looks raced. A dropped edge — a consumer that does not wait on its producer — shows up as a missing arrow. That points straight at the orchestration dependency that was lost (a classic cause is add_output instead of add_inout on a write-then-read GM round-trip, which drops the read-dep and lets the downstream task race).

---

Quick reference

Symptom	Tool	Kwarg / flag
Compile / PTOAS error	passes_dump/; skip_ptoas for golden.run only	compile_cfg=dict(dump_passes=True); with run, compile_cfg=dict(skip_ptoas=True)
Need to reproduce on the same inputs	golden-data replay (§2)	golden_data= / --golden-data
Iterating on generated .cpp / .pto	runtime-dir reuse (§3)	runtime_dir= / --runtime-dir
Run hangs / deadlocks (§4)	device log	runtime_cfg["log_level"]="v0" + ASCEND_PROCESS_LOG_PATH
Precision mismatch, unknown stage (§5)	args dump	enable_dump_args= / --dump-args [LEVEL]
Non-deterministic / raced result (§6)	dependency graph	enable_dep_gen= / --enable-dep-gen