Performance Tuning

A practical guide for tuning pypto-lib kernels on Ascend NPU (A3 / 910C). The flow is two-tiered: first balance the inter-kernel schedule on the AICPU side (L2 swimlane), then optimize each kernel's internal pipeline (L1/L0 swimlane + PMU).

For the underlying levels see simpler's hierarchical_level_runtime.md: L2 = one chip (AICPU + AIC/AIV cores), L1 = die / L2 cache, L0 = single compute core.

---

Measuring — the benchmark loop (PYPTO_BENCH)

Tuning needs a number before and after. Set PYPTO_BENCH=1 and every run / run_jit call in the process times the kernel on device after its correctness dispatch — no --benchmark flag, no edit to the model file:

PYPTO_BENCH=1 python models/qwen3_14b/decode_fwd.py -p a2a3 -d 0

[RUN]   effective_us (100 rounds) min=520.1 median=538.4 mean=539.9 max=602.0

Effective is the framework's post-graph-build execution window on device (orch ∪ sched — the old device-log "Total"), recovered from the runtime's [STRACE] markers. Quote mean=: daily CI's per-case perf number is exactly this field of exactly this line (daily_ci.yml), so a local mean is directly comparable to the dashboard.

Requirements: a real device — a *sim platform prints effective_us unavailable: no device-domain spans — and a runtime built with SIMPLER_PROFILING. A runtime_dir= replay has no live CompiledProgram and skips benchmarking with a [RUN] benchmark skipped note.

Multi-card (L3) output

A distributed program adds a per-rank breakdown and a context line:

[RUN]   effective_us (100 rounds) min=520.1 median=538.4 mean=539.9 max=602.0
[RUN]     rank 10: eff_us min=500.0 median=510.0 mean=511.0 max=520.0
[RUN]     rank 11: eff_us min=520.1 median=538.4 mean=539.9 max=602.0
[RUN] benchmark kernel=moe_ep2 l3_resident=1 rounds=100 ranks=2 host_union_mean_us=900 host_mean_us=950

• The headline is the per-round max across ranks — the round ends when the slowest card finishes. The eff_us lines expose the cross-card imbalance that max hides; a persistent gap between ranks is a load-balance problem, not a kernel problem.
• A rank N: eff_us line sums that card's dispatches within a round (a card runs them serially). When a card dispatches more than once per round, each rank line gains a nested slot line per dispatch so you can see which dispatch owns the time:

[RUN]     rank 10: eff_us min=500.0 median=510.0 mean=510.0 max=520.0
[RUN]       slot 0 (prefill_orch): eff_us min=200.0 median=205.0 mean=205.0 max=210.0
[RUN]       slot 1 (decode_orch): eff_us min=300.0 median=305.0 mean=305.0 max=310.0
[RUN]     rank 11: eff_us min=520.1 median=561.0 mean=561.0 max=602.0
[RUN]       slot 0 (decode_orch): eff_us min=520.1 median=561.0 mean=561.0 max=602.0

slot is the dispatch's position within its rank's round (slot 1 is the same dispatch in every round), and the name in parentheses is the orchestration function it runs. Once the slot lines appear, every rank's dispatches are listed — including single-dispatch ranks like 11 above, whose slot line necessarily restates its rank line — so the breakdown stays a complete tree. The slot lines are omitted entirely when every card dispatches exactly once per round, and also when a card's dispatch order varies between rounds — a slot then names no single callable, so pypto reports no per-dispatch view rather than mislabelling it. - host_union_mean_us is the cross-rank host-timeline window (max(end) - min(start)), so it captures start skew and overlap, but includes host dispatch overhead. - fallback_flattened=1 means a rank's dispatch count was not divisible by warmup + rounds (a non-deterministic dispatch shape), so per-round segmentation was abandoned and the numbers are a pooled per-dispatch sample — treat them as indicative only.

Knobs

Env	Default	Effect
PYPTO_BENCH	off	Enables the timed loop. Any value except "" / 0 / false / False is on.
PYPTO_BENCH_ROUNDS	100	Timed rounds. 100 rounds is ~0.1 s of device time for a decode step but minutes for a long prefill or a multi-card run — drop it while iterating.
PYPTO_BENCH_WARMUP	5	Leading launches discarded before measurement. The resident L3 path always keeps ≥ 1 (its first warmup launch doubles as the validation dispatch).
PYPTO_BENCH_RAW	off	Prints every measured dispatch's Effective sample, one line per rank, in dispatch order. Use it when a summary looks suspicious — start-up drift, a bimodal rank, one card lagging.

A malformed or out-of-range value warns and falls back to the default rather than failing the run. Daily CI sets none of the three, so its numbers always come from the 100 / 5 baseline; if you change the loop sizes locally, compare only against other runs with the same sizes.

# Quick iteration on a long prefill, with the raw per-dispatch samples.
PYPTO_BENCH=1 PYPTO_BENCH_ROUNDS=10 PYPTO_BENCH_WARMUP=2 PYPTO_BENCH_RAW=1 \
  python models/deepseek_v4_flash_mtp/prefill_fwd.py -p a2a3 -d 0

When only the timing changes between iterations — not the numerics — save the golden once and replay it via golden_data=, cutting the torch recompute out of every later run. See Save and Replay Golden Data.

---

Part 1 — L2 tuning (inter-kernel schedule)

Capture

Run the case with --enable-l2-swimlane. The runtime writes raw per-task L2 records under the build directory and, on a real-device platform, converts them to a merged swimlane:

python models/qwen3_14b/decode_fwd.py -p a2a3 -d 0 --enable-l2-swimlane

build_output/<ProgramName>_<ts>/dfx_outputs/
├── l2_swimlane_records.json
├── deps.json                    # real-device graph pass
└── merged_swimlane_<ts>.json   # real device only; open this

Two viewers work:

• Open merged_swimlane_<ts>.json in https://ui.perfetto.dev/.
• Or open l2_swimlane_records.json directly with the pypto-toolkit VSCode extension.

Simulator platforms emit l2_swimlane_records.json but intentionally skip the merged conversion because their records do not yet include the task metadata the converter requires. Use a real-device capture when you need the merged Perfetto view and dependency arrows.

The trace shows one lane per AICPU / AIC / AIV with task name, duration and dependency edges — gaps and stalls are visible directly.

What to look for

Look for these shapes on the swimlane that indicate a problem:

Symptom	Likely cause	Fix
Cores idle while AICPU lane is solid	Kernels too small; AICPU scheduling is the bottleneck	Make kernels larger (item 2)
Long tail on a single AIC/AIV	One kernel is too big and serializes	Split it (item 3)
Cube / vector unit utilization low even though kernel is busy	Tile size under-fills the user-visible on-chip buffers	Re-tile against Mat / Acc for cube or Vec for vector work (item 4)
Cube lane busy while vector lane idle (or vice versa)	Vec/cube epilogue is split into separate kernels	Merge into a mixed kernel (item 2c)
Sequential AICPU dispatch trail per region	Region issues one kernel per iteration	Use pl.spmd to dispatch a block fan-out once (item 5)

Tuning rules

1. Use pl.range vs. pl.parallel correctly

pl.parallel declares iterations are independent — the compiler may distribute them across cores. pl.range is strict sequential and forces a dependency chain. Use pl.parallel whenever there is no carried state, and reserve pl.range for accumulators or stateful loops.

# decode_fwd.py: batch tile is independent — pl.parallel
for b0 in pl.parallel(0, batch_padded, BATCH_TILE):
    ...

A pl.range over an independent dimension forces the swimlane into a single lane; switching to pl.parallel is usually the largest single win at this stage.

2. Kernels too small — make each kernel do more

When the swimlane shows cores idling while the AICPU lane is fully saturated, the AICPU dispatcher is the bottleneck. Target ~50 µs per kernel on A3 / 910C (smaller kernels add dispatch overhead that the AICPU can't hide). Three ways to grow each kernel:

a. Fold outer iterations into the core. Move part of an outer pl.range / pl.parallel's iterations into the pl.at region as an inner pl.range, so each dispatched kernel processes a tile of iterations instead of one:

# Before: one kernel per outer iteration — many tiny dispatches
for b in pl.parallel(0, BATCH):
    with pl.at(level=pl.Level.CORE_GROUP, name_hint="step"):
        ...

# After: fold BATCH_TILE iterations into each kernel via an inner pl.range
for b0 in pl.parallel(0, BATCH, BATCH_TILE):
    with pl.at(level=pl.Level.CORE_GROUP, name_hint="step"):
        for b in pl.range(b0, b0 + BATCH_TILE):
            ...

b. Merge consecutive pl.at blocks. Adjacent pl.at regions in the same scope each become a separate kernel with an AICPU hand-off between them. Fuse back-to-back regions into one pl.at so a single kernel covers the whole sequence:

# Before: two adjacent regions → two kernels + a hand-off
with pl.at(level=pl.Level.CORE_GROUP, name_hint="rmsnorm"):
    ...
with pl.at(level=pl.Level.CORE_GROUP, name_hint="q_proj"):
    ...

# After: one region → one kernel
with pl.at(level=pl.Level.CORE_GROUP, name_hint="rmsnorm_q_proj"):
    ...   # rmsnorm, then q_proj

c. Merge cube + vector into a mixed kernel. When a matmul (cube) and its epilogue (cast / add / norm — vector) sit in separate pl.at regions, every projection generates two kernels and an AICPU hand-off between them. Place both inside the same pl.at and the compiler co-schedules cube and vector on the right unit internally, removing the hand-off:

with pl.at(level=pl.Level.CORE_GROUP, name_hint="q_proj"):
    for kb in pl.pipeline(0, input_proj_k_blocks, stage=2):
        ...
        q_acc = pl.matmul_acc(q_acc, tile_a, tile_b)     # cube
    q_bf16 = pl.cast(q_acc, target_type=pl.BF16)         # vector
    q_proj[b0:b0 + BATCH_TILE, q0:q0 + Q_OUT_CHUNK] = q_bf16

3. Kernels too big — split and parallelize

When one kernel dominates the swimlane and the rest of the chip waits on it, the kernel is too coarse. Pull a pl.range out of the pl.at and convert it to a pl.parallel chunk loop so each chunk becomes its own InCore kernel scheduled across cores:

# Before: one giant InCore region over all q_out blocks
with pl.at(level=pl.Level.CORE_GROUP, name_hint="q_proj"):
    for q0 in pl.range(0, hidden, Q_OUT_CHUNK):
        ...

# After: each q-chunk is its own kernel, parallel across cores
for q0 in pl.parallel(0, hidden, Q_OUT_CHUNK):
    with pl.at(level=pl.Level.CORE_GROUP, name_hint="q_proj"):
        ...

4. Tiling — fill the core-internal buffers

Each AIC / AIV core has fixed on-chip buffers. At the DSL level, cube tiles directly consume Mat (L1 operand storage) and Acc (L0C accumulator storage), while vector tiles consume Vec (UB). The tile sizes declared in your pl.slice / pl.matmul (typically BATCH_TILE, K_STEP, Q_OUT_CHUNK, …) control these spaces.

• Too small → buffers are under-utilized, cube/vector throughput drops proportionally, MTE2 issues many small loads.
• Too large → tile spills, the compiler falls back to smaller transfer units, or compile-time shape checks fail.

Check actual occupancy. Every compile writes a per-kernel buffer report to

build_output/<ProgramName>_<ts>/report/memory_after_AllocateMemoryAddr.txt

listing, for each compute function, how full each on-chip space runs against its hardware limit (on the illustrated 910C configuration: vector Vec has a 184 KB compiler-safe limit within the 192 KB physical UB; cube Mat is 512 KB, Left / Right are 64 KB each, and Acc is 128 KB):

--- gather_kv ---
  Space  |  Used       |  Limit      |  Usage   |  MemRefs
  -------+-------------+-------------+----------+---------
  Vec    |   129.0 KB  |   184.0 KB  |   70.1%  |  2

--- kv_proj_matmul ---
  Space  |  Used       |  Limit      |  Usage   |  MemRefs
  -------+-------------+-------------+----------+---------
  Mat    |    80.0 KB  |   512.0 KB  |   15.6%  |  4
  Left   |    32.0 KB  |    64.0 KB  |   50.0%  |  1
  Right  |    16.0 KB  |    64.0 KB  |   25.0%  |  1
  Acc    |     4.0 KB  |   128.0 KB  |    3.1%  |  1

Scan the Usage column for Mat, Acc, and Vec. These are the user-visible constraints affected by the M/N/K or vector fragment. Left and Right report the L0A/L0B staging chosen by the compiler for the L1 fragment; they can routinely read close to 100% and are not independent DSL tile budgets. Do not shrink a tile merely to reduce a Left or Right percentage.

Grow the space that limits the task: Mat for operand fragments, Acc for the output fragment, or Vec for vector working data. An oversized plain matmul output may be compiler-subtiled through L0C, so Acc is often a performance boundary rather than an immediate compile failure; extra tiles still add FIXPIPE drains. The exact build report and compile result are authoritative.

Practical procedure:

1. Start from the natural problem dimensions (BATCH, HIDDEN, …).
2. Pick K_STEP and the output-chunk size so Mat and Acc stay within the intended bounds without forcing inefficient compiler sub-tiling.
3. Sweep one tile dim up/down by 2× and re-measure with PMU — keep the size that pushes the cube (or vector) unit closer to 100 %.

The K loop is then driven by pl.pipeline(stage=2 or 4) so the next tile's MTE2 overlaps the current tile's compute (see Part 2 item 2). For the complete M/N/K constraint model and empirical sweep method, see Cube Tile Tuning.

5. pl.spmd for parallel sub-kernel dispatch

pl.spmd(N) dispatches N blocks of an InCore body in parallel from one AICPU schedule entry, instead of N successive pl.parallel + pl.at dispatches. When a region has many parallel chunks and AICPU overhead is visible per iteration on the swimlane, replace the explicit for ... in pl.parallel: with pl.at: ... pattern with pl.spmd:

# qwen3_32b/decode.py: one AICPU dispatch fans out Q_OUT_BLOCKS blocks
for qi in pl.spmd(Q_OUT_BLOCKS, name_hint="q_proj"):
    ...

Collapsing N dispatches into one schedule entry cuts AICPU scheduling overhead sharply. The win is largest for MPMD-shaped regions with heavy fan-in / fan-out — where each block depends on (or feeds) many others, the AICPU would otherwise track a dependency edge per block, and pl.spmd replaces that whole fan with a single dispatch and its barrier.

Use pl.spmd once the per-iteration body is self-contained and the AICPU lane shows a dispatch trail; keep the explicit form when you need to nest named sub-regions inside the chunk.

---

Part 2 — L1 / L0 tuning (intra-kernel)

Once L2 is balanced, individual kernels become the bottleneck. Two artifacts drive intra-kernel tuning:

Capture

PMU counters per kernel:

python models/deepseek_v4_flash_mtp/decode_sparse_attn.py -p a2a3 -d 0 --enable-pmu 2
# → build_output/<...>/dfx_outputs/pmu.csv

Not every kernel exposes --enable-pmu; a kernel that does not can still be captured by passing runtime_cfg={"enable_pmu": 2} to its run / run_jit call (the harness bundles it into the runtime's DFX options).

For a per-kernel intra-core swimlane, use In-Core Simulator Profiling. It explains how the repository workflow builds a standalone single-core testcase from the generated .cpp and sibling .pto, runs it under msprof op simulator, validates that data-dependent work actually executed, and cleans the Insight trace for Perfetto.

For phase timing inside a multi-core extern on real hardware, use cce-incore-profiling.md. It covers per-core on-device timestamps, collective-barrier interpretation, and exact partitions that reconcile internal phases with the L2 task total.

Tuning rules

1. Fix tile-shape MTE hints from perf_hints.log

Every compile writes a perf-hint log next to the memory report:

build_output/<ProgramName>_<ts>/report/perf_hints.log

The compiler flags every tile.load / tile.store whose innermost (trailing) dimension is smaller than the 512 B L2 cache line — the case that forces MTE into many short, cache-line-straddling transfers. Each hint carries the exact source location:

[perf_hint PH001] TileInnermostDimGranularity: tile.load has innermost
dim = 256B; recommended >= 512B for backend a2a3 (L2 cache line = 512B).
Consider increasing tile shape on the innermost axis.
at models/deepseek_v4_flash_mtp/qkv_proj_rope.py:68:4

Walk the log and widen the trailing tile dimension at each flagged site so the innermost slice is a multiple of 512 B (item 3 gives the per-dtype element counts). Bringing every flagged tile.load / tile.store up to ≥ 512 B is usually the single biggest MTE-efficiency win at this level.

2. pl.pipeline for ping-pong on the K loop

Inside a pl.at region, the reduction loop of a matmul (the K loop) should be pl.pipeline(..., stage=2 or 4). The compiler replicates the loop body stage times for ping-pong buffering, so MTE2 (load) overlaps with cube/vec compute on alternating tiles.

# decode_fwd.py — stage=4 used for the largest input-proj K dim
for kb in pl.pipeline(input_proj_k_blocks, stage=4):
    ...

# stage=2 is the common default
for kb in pl.pipeline(0, hidden_blocks, stage=2):
    ...

A pl.range here forces strictly serial K iterations — the cube unit will stall on every load. Always prefer pl.pipeline in the K loop.

3. Watch pl.slice / pl.assemble granularity

MTE transfers prefer 512-byte aligned addresses and lengths on A3 / 910C. Pick the trailing-dim tile size so the slice is a multiple of 512 B:

• BF16 (2 B/element) → trailing dim multiple of 256 elements
• FP32 (4 B/element) → trailing dim multiple of 128 elements
• INT8 (1 B/element) → trailing dim multiple of 512 elements

Misaligned slices fall back to slower paths visible as long MTE2 bars in the kernel-insight swimlane. In the qwen3-14b kernels, all K_STEP / Q_OUT_CHUNK constants are picked to keep the inner load 512 B aligned.

4. Read PMU utilization

Recommended PMU counters to collect per kernel:

pmu_total_cycles
vec_busy_cycles        cube_busy_cycles        scalar_busy_cycles
mte1_busy_cycles       mte2_busy_cycles        mte3_busy_cycles
fixpipe_cycles

What each pipe means in context:

Counter	Cube kernel (AIC)	Vector kernel (AIV)
mte1_busy_cycles	L1 → L0 (operand staging into cube)	—
mte2_busy_cycles	GM → L1 (operand load from device memory)	GM → UB (input load)
mte3_busy_cycles	—	UB → GM (output store)
fixpipe_cycles	L0C → GM (cube result write-out)	—
cube_busy_cycles	cube compute	—
vec_busy_cycles	—	vector compute

The bottleneck pipe should sit near 100 % of pmu_total_cycles; the others run overlapped underneath it. Targets:

• Cube kernel: max(mte2_busy_cycles, cube_busy_cycles) / pmu_total_cycles ≈ 100 %. Either the L1 load or the cube compute is saturated — whichever the shape is bound by.
• Vector kernel: max(mte2_busy_cycles, vec_busy_cycles) / pmu_total_cycles ≈ 100 %. Either the GM→UB load or the vector compute is saturated. For very store-heavy kernels, mte3_busy_cycles can be the bottleneck instead.

If both compute and MTE2 are well below 100 %, open the kernel-insight swimlane: gaps usually mean (a) a missing pl.pipeline on the K loop, (b) suboptimal instruction scheduling, or (c) incorrectly placed synchronization barriers.