Types¶

Legacy Python API examples: platform= calls on this page are retained for compatibility. They emit DeprecationWarning; new code should use backend_type and execution_mode. See the target mapping.

Every value in a PyPTO program carries a type that says where it lives and how wide its elements are. Getting the annotation right is how you tell the compiler what to allocate and what it is allowed to do.

Prerequisites: Programming Model § memory hierarchy.

Concept¶

Three things are encoded in a type annotation, and it is worth separating them because they fail differently.

Where the value lives. pl.Tensor is in DDR, pl.Tile is an on-chip buffer, pl.Scalar is a register-width value. This is not a hint — a tensor operation on the execution plane and a tile operation on the control plane are both rejected, and the container type is how the compiler tells which is which.

How wide the elements are. The dtype constants (pl.FP16, pl.INT32, …) name a hardware element format. Mixing them is legal but never implicit: there is no promotion, so a pl.cast is required wherever the widths differ.

How the caller may use it. Parameter direction — In (the default), pl.Out[...], pl.InOut[...] — is part of the signature, not a convention. The compiler derives task dependencies from directions, so a mis-declared direction produces a wrong dependency graph rather than a compile error.

Shapes are static by default and checked at parse time. pl.dynamic() opts a dimension out of that, at the cost of everything the compiler could have concluded from knowing it.

Register annotations (IR infrastructure)¶

pl.Reg[L, dtype] describes a register value with a static positive logical lane count L. The optional third parameter is pl.RegKind.VecReg (the default) or pl.RegKind.MaskReg. MaskReg requires pl.BOOL; VecReg forbids it. For example, pl.Reg[64, pl.FP32] and pl.Reg[64, pl.BOOL, pl.RegKind.MaskReg] are valid annotations. pl.reg provides P0 register operations inside pl.reg_scope(); inputs are captured implicitly and Reg/local Scalar values cannot escape. IR construction, validation, binary serialization and DSL text roundtrips are supported. VF backend compilation remains unavailable until PR3. See register operations.

Quickstart: reading a signature¶

import pypto.language as pl
import torch
from pypto.runtime import RunConfig

CFG = RunConfig(platform="__PLATFORM__")
torch.manual_seed(0)

M = pl.dynamic("M")                       # an axis whose extent varies at run time


@pl.jit.incore
def scale_rows(
    x: pl.Tensor[[M, 128], pl.FP16],                    # In (default): read-only, DDR
    acc: pl.InOut[pl.Tensor[[M, 128], pl.FP32]],        # read-write, DDR
    out: pl.Out[pl.Tensor[[M, 128], pl.FP32]],          # write-only, DDR
    factor: pl.Scalar[pl.FP32],                         # scalar, passed by value
):
    tx = pl.load(x, [0, 0], [64, 128])
    scaled = pl.mul(pl.cast(tx, pl.FP32), factor)
    acc = pl.store(pl.add(pl.load(acc, [0, 0], [64, 128]), scaled), [0, 0], acc)
    out = pl.store(scaled, [0, 0], out)
    return acc, out


@pl.jit
def apply_scale(
    x: pl.Tensor[[64, 128], pl.FP16],
    acc: pl.InOut[pl.Tensor[[64, 128], pl.FP32]],
    out: pl.Out[pl.Tensor[[64, 128], pl.FP32]],
):
    acc, out = scale_rows(x, acc, out, 2.0)             # the dynamic axis binds to 64 here
    return acc, out


x = torch.randn(64, 128, dtype=torch.float16)
acc = torch.randn(64, 128, dtype=torch.float32)
out = torch.zeros(64, 128, dtype=torch.float32)
acc_before = acc.clone()

apply_scale(x, acc, out, config=CFG)

scaled = x.float() * 2.0
torch.testing.assert_close(out, scaled, rtol=1e-2, atol=1e-2)               # Out was written
torch.testing.assert_close(acc, acc_before + scaled, rtol=1e-2, atol=1e-2)  # InOut was read *and* written

Element	Reads as
`pl.Tensor[[M, 128], pl.FP16]`	2D DDR array, `M` rows (runtime value), 128 columns, half precision
`pl.InOut[...]`	The kernel both reads and writes it — the compiler orders it against both earlier writers and earlier readers
`pl.Out[...]`	The kernel only writes it. Reading an `Out` parameter before writing it reads undefined memory
`pl.Scalar[pl.FP32]`	A single value, not a buffer
`M = pl.dynamic("M")`	The dimension is unknown at compile time and bound per launch

Mechanics¶

Data types¶

Constant	Bits	Notes
`pl.BOOL`	1
`pl.INT4` / `pl.UINT4`	4
`pl.INT8` / `pl.UINT8`	8
`pl.INT16` / `pl.UINT16`	16
`pl.INT32` / `pl.UINT32`	32
`pl.INT64` / `pl.UINT64`	64
`pl.FP16`	16	IEEE half
`pl.BF16`	16	Brain float
`pl.FP32`	32	IEEE single
`pl.fp4e2m1x2` / `pl.FP4E2M1X2`	8	Packed pair of E2M1 values; shapes count byte carriers
`pl.fp4e1m2x2` / `pl.FP4E1M2X2`	8	Packed pair of E1M2 values; shapes count byte carriers
`pl.FP8E4M3FN` / `pl.FP8E5M2`	8	MXFP8 data formats
`pl.FP8E8M0`	8	MX block-scale exponent
`pl.HF4` / `pl.HF8`	4 / 8	Hisilicon float formats
`pl.INDEX`	64	Index arithmetic — loop variables, dimensions
`pl.TASK_ID`	—	Producer handle for a launched task

dtype.get_byte() returns the byte-addressable element size, rounded up to one byte. Use it for byte-addressable dtypes whenever a byte count is computed rather than written as a literal. Do not use logical_elements * dtype.get_byte() for a 4-bit buffer: PyPTO packs every semantic 4-bit dtype two logical elements per byte, so the physical size is ceil(logical_elements / 2).

nbytes = 256 * pl.FP32.get_byte()          # 1024, not 256

pl.fp4e2m1x2 and pl.fp4e1m2x2 are distinct packed floating-point formats, with uppercase aliases pl.FP4E2M1X2 and pl.FP4E1M2X2. Each element occupies one byte and contains two FP4 values. Shapes, valid_shape, strides, and offsets count these carriers throughout IR, JIT specialization, runtime calls, and serialized argument metadata; there is no implicit dimension scaling. The old pl.FP4, DataType.FP4, and DT_FP4 names are removed.

Explicit packed annotations accept torch.uint8. E2M1 also accepts torch.float4_e2m1fn_x2 with its physical shape unchanged; E1M2 uses UINT8 because Torch has no matching native dtype. Unannotated UINT8 remains UINT8. Within PyPTO, calls and submits require matching argument and parameter dtypes whenever either is packed FP4. Pass an explicit reinterpret_view for supported UINT8/INT8 byte aliases; a callee annotation cannot create an implicit alias.

On Ascend950, both row-major packed formats convert to/from BF16: [4,32] carriers decode to [4,64] values. E2M1↔E1M2 conversion uses BF16 and keeps the carrier shape. Packing requires an even source extent, including a provably even dynamic valid_shape such as 2 * n. Supported modes are rint, round, floor, ceil, and trunc; odd is rejected. Implicit encoding chains must preserve values through BF16; write an explicit BF16 cast when intermediate rounding is intended.

ND/DN carrier transfers and ND padding are supported. DN casts require full static dense 2D BF16/packed endpoints and PYPTO/DSA_RP static addresses. pl.quant_mx(..., dtype=pl.fp4e2m1x2) supports FP16/BF16 input; E1M2 quantization and native packed FP4 matrix multiplication remain unsupported. Ascend910/A2A3 execution remains unsupported for both formats. See FP4 for layout, alias, and memory-planner constraints.

Annotating a scalar binding¶

Scalar arithmetic normalizes its operands to pl.INDEX, so the result is INDEX-typed even when the inputs were not:

v = pl.cast(i, pl.INT32) + 1                        # pl.Scalar[pl.INDEX]

An annotation decides the dtype the name is bound at, and the value is converted to match it — so the two sides of the assignment always agree:

v: pl.Scalar[pl.INT32] = pl.cast(i, pl.INT32) + 1   # binds INT32; the cast is inserted for you
v: pl.Scalar[pl.INDEX] = pl.cast(i, pl.INT32) + 1   # binds INDEX; nothing to convert

Annotate the dtype you want the value stored at: a narrower annotation converts the value, it does not reinterpret it.

INDEX-returning calls follow the same rule, including pl.tile.get_block_idx():

v: pl.Scalar[pl.INT32] = pl.tile.get_block_idx()    # converts the INDEX result to INT32

This compatibility applies only to integer/INDEX pairs. An FP32 annotation on an INDEX expression is rejected; use an explicit pl.cast for a float conversion.

Container types¶

Type	Lives in	Written as
`pl.Tensor[[shape], dtype]`	DDR	`x: pl.Tensor[[64, 128], pl.FP32]`
`pl.Tile[[shape], dtype]`	On-chip buffer (Vec by default)	`t: pl.Tile[[64, 64], pl.FP32]`
`pl.Scalar[dtype]`	Value, not a buffer	`s: pl.Scalar[pl.FP32]`
`pl.Array[extent, dtype]`	On-core array	`a: pl.Array[16, pl.INT32]`
`pl.Tuple[T1, T2]`	—	Multi-value return annotation

pl.TaskId is a convenience alias for pl.Scalar[pl.TASK_ID].

pl.Array is normally created rather than annotated — arrays do not cross function boundaries, so the annotation form is rare. The update is functional — it produces a new array value and rebinds the name, so an array assignment inside a loop is a carried value like any other.

arr = pl.array.create(16, pl.INT32)
arr[i] = value          # array.update_element — functional, rebinds arr
x = arr[i]              # array.get_element

Layouts¶

Write pl.Tensor annotations with the runtime row-major shape and no layout marker. Layout is an IR-internal concern; passes derive it from the operations that produce and consume each view.

b: pl.Tensor[[N, K], pl.FP32]              # ✅ source shape, no marker

The layout-only shorthand pl.Tensor[..., pl.DN] is not supported: it raises ParserTypeError. For a transposed matmul operand, pass a_trans=True / b_trans=True to pl.matmul, or derive the transposed view at the use site with pl.transpose(x, -2, -1). A slice or reshape of a DN-producing operation inherits DN automatically. A slice keeps the source layout whenever its trailing two axes survive intact, so a leading-axis shard (w[r] of a [N_RANKS, R, C] weight) stays on the parent's layout.

A layout annotation must agree on both sides of a call. It is a claim about byte order in memory, not a conversion request, so passing an ND-typed argument to a parameter declared pl.NZ (or an MX scale layout) is rejected by the type checker before the first pass runs: Layout mismatch at argument 1 of call to 'nz_helper': parameter 'b' is declared NZ but the argument is ND. Annotate both ends, or neither. DN is exempt, because a parameter cannot declare it: a DN value is derived at the use site with pl.transpose, so handing one to an ND-declared parameter is the intended workflow, not a disagreement.

A device= dispatch is the other exception: the host only says which card a buffer goes to, so an ND host tensor holding NZ-packed bytes may bind a pl.NZ parameter of the device program. That holds only for whole matrices — the host tensor itself, or a slice that keeps its last two axes whole, such as the rank shard w[r] of a [N_RANKS, R, C] stack. A window inside the matrix (w[r * R : (r + 1) * R] of a once-packed [N_RANKS * R, C] weight) is rejected: those rows are contiguous, but they are runs of several fractal column blocks, not an NZ-packed [R, C]. Pack each shard separately and stack the shards instead. An argument the checker cannot trace back to a host parameter through such slices — a reshape, a newly created tensor — is rejected too. The trace stays within the dispatching function, so a window passed into a helper function arrives there as a parameter and is not followed back to its caller.

pl.ND is the default row-major layout and never needs writing. pl.NZ declares PTO-native NZ fractal byte order in global memory. A matmul operand load expects already packed bytes and can skip online ND→NZ conversion; a supported Acc store produces NZ bytes directly. The annotation itself does not convert existing bytes: shape and slicing remain logical, and the compiler derives the blocked physical descriptor. The tensor needs a static, fractal-aligned trailing plane with a whole-byte dtype (shape[-2] % 16 == 0, shape[-1] % (256 / dtype bits) == 0) and logical rank at least 2. Leading axes fold into the descriptor's single batch slot.

On A2/A3, pl.store(acc, offsets, out) also supports a fully valid, static 2-D FP32 Acc tile and an FP32 pl.NZ destination. Rows and columns must be multiples of 16; the tile must cover the destination's entire row axis at row offset 0. Each store writes one panel, with non-negative, provably aligned column and batch offsets. Stack independent panels as [PANELS, R, C] and write them separately. FP32 NZ uses C0=8; it does not change the tile's mathematical values. Supported matmul-to-store chains are automatically tiled along N while keeping the complete M axis; ordinary K reduction tiling remains available. The logical output can exceed L0C or a store instruction's column limit. Each emitted Acc tile must fit the backend's L0C capacity (currently 128 KiB on A2/A3) and have at most 4095 columns. The GM destination may be larger. If no legal full-M plan exists, the automatic tiler reports a compile-time diagnostic. Producer/use patterns outside the supported rewrite retain the allocation and final store checks. Explicit column tiling is also supported, with each chunk spanning the destination's full row axis. Atomic stores, FIXPIPE epilogue options (pre_quant, pre_relu), dtype conversion, A5, Vec/Mat sources, and dynamic or partially valid tiles are rejected. The full scope and the underlying row-window restriction are documented in BlockNzTensorViews.

When a tensor's rows are not contiguous — a window into a larger buffer, a strided slice handed in from outside — describe it with pl.TensorView, which makes the strides explicit instead of leaving them to be inferred:

view = pl.TensorView(stride=[1024, 1], layout=pl.TensorLayout.ND, valid_shape=[16, 64])

layout= is required whenever any of stride, valid_shape, or pad is given. pl.TensorLayout is the enum those layout constants come from — pl.ND is pl.TensorLayout.ND.

pl.MX_A_ZZ and pl.MX_B_NN are the two remaining layout constants. They tag the GM scale tensor of an MX (microscaling) operand on Ascend950 — MX_A_ZZ for the left/A scale pack, MX_B_NN for the right/B one — so that a Mat-to-scale pl.move can check the source layout instead of byte-copying incompatible data into LeftScale / RightScale. They are the one case where a layout marker on a pl.Tensor annotation is required rather than discouraged. An MX pl.load may omit target_memory; the Python API selects pl.Mem.Mat, and matmul_mx operand placement inserts the required moves. Ordinary MX subviews (slice, reshape, transpose, reinterpret_view) and MX remote_load are rejected. Exception: FP8E8M0 pl.tensor.view may alias packed ND backing to MX_A_ZZ / MX_B_NN as a logical rank-2 view (layout=mx_*; PTOAS v0.60 packs physically). The matmul itself is pl.matmul_mx and its _acc / _bias variants, which take a data tile and a scale tile per operand. Both data tiles reaching the op must be FP8E4M3FN. Either packed FP4 format may supply the left operand after conversion to FP8: write pl.cast(fp4_tile, pl.FP8E4M3FN) before matmul_mx. On A5 the cast legalization pass expands that request to packed FP4→BF16→FP32→FP8E4M3FN. Native FP4×FP4 and the reverse FP8×FP4 form are not supported. pl.quant_mx supports MXFP8 and native E2M1 MXFP4, with group_axis matching PTOAS grpAxis. FP8 results can feed matmul_mx in one mixed InCore task through V2C, including the generated Vec-to-Mat-to-scale-memory path.

With the default FP8 dtype, tensor quantization returns [M,K] data and MX_A_ZZ[M,K/32] scales on axis1, or [K,N] data and MX_B_NN[K/32,N] scales for axis0 input [N,K]. Selecting dtype=pl.fp4e2m1x2 halves the payload's final extent; scales are unchanged. FP4 requires FP16/BF16 input, full validity, K%64==0, axis1 M%16==0, and axis0 N%64==0. OCP is the default; NV FP16/BF16 is limited to axis0. E1M2 quantization is unsupported. PYPTO/DSA_RP static allocation is required for packed output. pl.tile.quant_mx_raw exposes raw UINT8 exponents and explicit workspaces. Only FP8 data/scale pairs feed tensor pl.matmul_mx, which is 2-D-only and returns FP32; tensor MX accumulation, bias, transpose flags, batches, dynamic shapes, and native packed FP4 matrix operands remain unsupported.

Dynamic shapes¶

pl.dynamic(name) marks an axis whose extent is not known when the kernel is compiled and may differ from one launch to the next — a batch that varies with the request, a sequence length that grows as decoding proceeds. The extent becomes a run-time value, so one compiled program serves every size that axis takes: dynamic dimensions collapse to None in the JIT cache key, and changing the extent does not trigger a recompile.

N = pl.dynamic("N")
TILE = 32                       # the physical tile the kernel moves per call


@pl.jit.incore
def rows(x: pl.Tensor[[N, 64], pl.FP32], out: pl.Out[pl.Tensor[[N, 64], pl.FP32]]):
    out = pl.store(pl.mul(pl.load(x, [0, 0], [TILE, 64]), 2.0), [0, 0], out)
    return out


@pl.jit
def drive(x: pl.Tensor[[N, 64], pl.FP32], out: pl.Out[pl.Tensor[[N, 64], pl.FP32]]):
    return rows(x, out)         # the entry is dynamic too, so both extents share it


# Two extents through one program: the dynamic dim collapses to None in the JIT
# cache key, so the second call is not a recompile.
for extent in (TILE, 3 * TILE):
    x = torch.randn(extent, 64, dtype=torch.float32)
    out = torch.zeros(extent, 64, dtype=torch.float32)
    drive(x, out, config=CFG)
    torch.testing.assert_close(out[:TILE], x[:TILE] * 2.0, rtol=1e-4, atol=1e-4)

The entry has to stay dynamic for that to hold. Giving it a concrete shape pins the program to one extent, and doing orchestration-level work on a dynamic-shaped tensor (rather than handing it to an InCore kernel) fails earlier still, in InitMemRef, which needs a constant dim. The kernel above moves a fixed TILE; covering the whole of a larger input is the chunking loop from Control Flow.

The same DynVar object used in several annotations refers to the same dimension — reuse the object, do not create a second one with the same name if you mean the same value.

Keep a dimension static when it really is fixed. A static extent is a number the compiler can plan around — tiling choices, unroll factors, static bound checks — and a dynamic one withholds it.

Parameter directions¶

Direction	Syntax	The compiler concludes
In (default)	`x: pl.Tensor[...]`	Read-only. Orders after producers
Out	`x: pl.Out[pl.Tensor[...]]`	Written, not read. Orders after prior readers and writers
InOut	`x: pl.InOut[pl.Tensor[...]]`	Both. Orders against everything touching it

Directions are what the compiler reads to order tasks against each other. Declaring an InOut buffer as Out tells the runtime nothing needs to finish before this task writes it — which is a race, not a diagnostic.

Edge Cases¶

Fatal pitfall: a byte count written as an element count silently under-allocates. pld.alloc_window_buffer(256) reserves 256 bytes — room for 64 FP32 values, not 256. Any non-literal size must be spelled n * pl.<DTYPE>.get_byte(). Nothing warns; the symptom is corrupted data past the first 64 elements.

Symptom	Likely cause	Fix
`ParserTypeError` about the DN layout-only shorthand	`pl.Tensor[..., pl.DN]` — removed, it forced two coordinate systems onto one annotation	Write the source shape with no marker; derive DN at the use site with `pl.transpose(x, -2, -1)`; or inherit it through a slice/reshape of a DN-producing op
Results wrong only when two tasks overlap	A read-write buffer declared `In` or `Out` instead of `InOut`	Declare the direction the kernel actually performs
Reading an `Out` parameter returns garbage	`Out` promises write-before-read	Use `pl.InOut[...]` if the prior contents matter
`pl.cast` where you expected implicit promotion	There is no implicit promotion	Insert the cast; check LegalizeTileCast for multi-hop pairs
Two dimensions that should match are treated as independent	Two separate `pl.dynamic("M")` calls	Create the `DynVar` once and reuse the object

Not every pl.cast is one instruction. Whether a (src, dst) pair maps to a single hardware pto.tcvt or expands into a chain depends on the target: INT32 -> FP16 is one instruction on Ascend910 and lowers to INT32 -> FP32 -> FP16 on Ascend950. Each hop costs a tcvt, and where an intermediate is narrower than the source the result can differ from a directly rounded conversion by one ULP of the destination. This is expected behaviour, not a defect — see LegalizeTileCast for the per-architecture tables.

pl.cast also takes a keyword-only saturation_mode ("on" / "off", or 1 / 0) for Tensor and Tile inputs. "on" clamps an out-of-range result to the destination range; "off" keeps the target's non-saturating conversion, whose overflow behaviour is architecture-defined. "on" is the default when the destination is an integer type — nothing standard fixes what an overflowing conversion to an integer produces, clamping is the safer thing to get by accident, and on A2/A3 it is also the conversion the hardware performs natively. A float destination is left alone: IEEE says an out-of-range narrowing yields an infinity, and PyPTO matches that unless you ask for "on". A multi-hop cast applies the mode to its final hop, which is the one that reaches the dtype you named.