R1 Endpoint abstraction keeps local mailbox local-only and routes remote L3 through WorkerEndpoint / RemoteL3Endpoint |
remote-l3-worker-design.md §Target Architecture; implementation-plan.md step 1; worker-manager.md §4 |
required in this PR cut |
WorkerEndpoint, LocalMailboxEndpoint, RemoteL3Endpoint, WorkerManager::add_next_level_endpoint, Worker::add_remote_l3_socket |
test_remote_endpoint, test_scheduler, test_remote_sim_noop_task_roundtrip |
None found |
PR 4 / PR 5 |
| R2 Endpoint outcomes distinguish success, task failure, endpoint failure, and skipped group members |
remote-l3-worker-design.md §Failure Semantics; implementation-plan.md steps 1, 4 and Failure Poisoning Contract; scheduler.md §§2,6,9 |
required in this PR cut |
EndpointOutcome; endpoint completion mapping; scheduler failure poisoning and group skip state |
RemoteTaskErrorMapsToTaskFailure, FailedProducerPoisonsDependentTask, GroupFailureWaitsForRunningMembersThenConsumes, remote host_tcp error/exit tests |
None found |
PR 4 / PR 5 |
R3 Scheduler dispatch preserves stable worker ids and worker= is validated against final eligibility |
remote-l3-worker-design.md §Worker Identity and Callable Routing; implementation-plan.md step 2; scheduler.md §3 |
required in this PR cut |
Orchestrator::validate_worker_eligibility, directed NEXT_LEVEL queues, Python worker-id set intersection |
TargetMustBeInEligibleEndpointSet, target-ID mapping tests, Python remote callable worker-id intersection tests |
None found |
PR 4 |
| R4 Mixed local/remote pools are allowed only when callable and tensor representations are consumable by the selected worker |
remote-l3-worker-design.md §Worker Identity and Callable Routing; buffers-and-transports.md §TaskArgs Sidecar Contract |
required in this PR cut |
Python Orchestrator only allows RemoteTensorRef for RemoteCallable; C++ rejects remote sidecars on local workers and non-owner remote-device dispatch without import |
RemoteSidecarRejectsLocalEndpointEligibility, RemoteSidecarRejectsNonOwnerEligibleEndpointWithoutImport, Python RemoteCallable sidecar tests |
No blocking drift; add a future end-to-end mixed local+remote smoke during split validation |
PR 4 |
| R5 Remote sidecars are hidden metadata aligned by tensor index; local endpoints reject sidecars; remote endpoints reject bare host pointers |
remote-l3-worker-design.md §Remote TaskArgs Representation; buffers-and-transports.md §§Public Memory API, TaskArgs Sidecar Contract; protocol.md §TASK Payload |
required in this PR cut |
RemoteTaskArgsSidecar, Python _remote_sidecar_for, C++ validate_remote_sidecars, LocalMailboxEndpoint::run, RemoteL3Endpoint::build_task_payload |
TestRemoteTaskArgsSidecar, RemoteBarePayloadFailsBeforeSlotCommit, BareHostPointerWithoutSidecarIsEndpointFailure |
None found |
PR 4 / PR 6 |
R6 Remote null OUTPUT tensors fail fast unless the caller supplies an explicit RemoteTensorRef |
remote-l3-worker-design.md §Remote TaskArgs Representation; buffers-and-transports.md §Remote OUTPUT Allocation Policy; implementation-plan.md step 3 |
required in this PR cut |
Orchestrator::validate_remote_sidecars requires sidecar for remote OUTPUT; reserve_outputs_and_slot skips local HeapRing only when sidecar present |
RemoteOutputSidecarSkipsLocalAutoAllocAndRegistersRemoteKey, remote host_tcp OUTPUT buffer tests |
No code drift; add a narrow negative test for null remote OUTPUT without sidecar when carving PR 4 |
PR 4 / PR 6 |
R7 Remote dependency keys use (address_kind, owner_worker_id, buffer_id, generation, offset) and preserve exact-start semantics |
remote-l3-worker-design.md §Remote TaskArgs Representation; buffers-and-transports.md §Dependency Keys; orchestrator.md §2 |
required in this PR cut |
TensorKey::remote_buffer, Orchestrator::infer_deps remote path, TensorMap remote keys |
RemoteInputSidecarUsesRemoteTensorMapKey, test_remote_sim_failed_dependency_skips_consumer |
Range-overlap support remains future/open decision |
PR 4 |
| R8 Remote callable identity uses canonical hashid descriptors, not target-private slots or cross-worker integer ids |
remote-l3-worker-design.md §Worker Identity and Callable Routing; task-flow.md §Callable Identity; implementation-plan.md step 6 |
required in this PR cut |
callable_identity.py, CallableHandle, descriptor builders, Worker._identity_registry |
hash stability, public export, forged/mutated handle, cleanup-uncertain tests |
None found |
PR 2 / PR 6 |
R9 RemoteCallable("module:qualname") is the required baseline with explicit non-empty workers=[...] |
remote-l3-worker-design.md §Worker Identity and Callable Routing; protocol.md §CONTROL Payload; implementation-plan.md step 6 |
required in this PR cut |
RemoteCallable, parse_python_import_target, _build_callable_registration, explicit remote worker validation |
target validation and explicit remote worker tests |
None found |
PR 2 / PR 6 |
| R10 Multi-worker remote registration is all-or-nothing with prepare, commit, abort, cleanup-uncertain blocking, and final-unregister tombstones |
remote-l3-worker-design.md §Worker Identity and Callable Routing; protocol.md §CONTROL Payload; implementation-plan.md step 6 |
required in this PR cut |
Worker._post_start_register_remote, remote_prepare/commit/abort/unregister, pending unregister tombstones, uncertain hashid guard |
post-init register, unregister/reregister, inner register/unregister, uncertain cleanup guard |
Partial-failure cleanup path is complex; keep as high-risk review item |
PR 2 / PR 5 / PR 6 |
R11 INNER_L3_WORKER is remote-internal; parent TASK frames resolve only in REMOTE_TASK_DISPATCHER |
remote-l3-worker-design.md §§Worker Identity and Callable Routing, Remote Worker Session; protocol.md §CONTROL Payload; task-flow.md §Callable Identity |
required in this PR cut |
session dispatcher registry vs inner registry; _prepare_register_callable; get_inner_handle |
dispatcher rejects chip target, inner Python import/chip callable install, inner sub-task integration |
None found |
PR 5 / PR 6 |
R12 PYTHON_SERIALIZED remote callables reject unless serialized support is explicitly negotiated |
remote-l3-worker-design.md §§Scope, Worker Identity and Callable Routing; protocol.md §CONTROL Payload; implementation-plan.md step 6 |
required explicit unsupported behavior |
_prepare_register_callable rejects CallableKind.PYTHON_SERIALIZED before install |
test_remote_register_rejects_python_serialized_without_negotiation |
Phase 3 added missing coverage |
PR 6 |
R13 CHIP_CALLABLE is valid only for INNER_L3_WORKER; inline blobs are supported by host_tcp, staged blobs reject unless negotiated |
protocol.md §CONTROL Payload; implementation-plan.md step 6; implementation-record.md §Completed Items |
required in this PR cut plus required explicit unsupported behavior for staged blobs |
_prepare_inner_chip_callable; dispatcher rejects chip target; staged blob reject |
inner chip manifest install, dispatcher reject test, test_remote_inner_chip_callable_rejects_staged_blob_without_negotiation |
Phase 3 added missing staged-blob coverage |
PR 5 / PR 6 |
R14 Bootstrap manifest installs dispatcher and inner registries before HELLO READY; unsupported negotiated manifest extensions reject rather than partially install |
remote-l3-worker-design.md §Remote Worker Session; protocol.md §CONTROL Payload; implementation-plan.md steps 6-7 |
required in this PR cut |
remote_l3_session.run_session, _install_manifest_inner_registry, daemon manifest validation |
manifest inner Python/chip install tests; remote host_tcp roundtrip tests |
None found |
PR 6 |
R15 Fork ordering preserves prestart before command/health transport threads; HELLO READY is a scheduling barrier |
remote-l3-worker-design.md §Fork-Safe Remote Process Model; protocol.md §HELLO Payload; hierarchical-level-runtime.md §Process Model |
required in this PR cut |
daemon writes manifest, session prestarts inner_worker, then binds command/health and sends HELLO READY; parent add_remote_l3_socket waits for READY |
remote unreachable daemon test; remote host_tcp roundtrip/prep tests |
None found |
PR 6 |
| R16 Remote frame protocol uses versioned canonical little-endian encoding and never memcpy's C++ POD structs |
remote-l3-worker-design.md §Protocol; protocol.md §§Frames, Wire Encoding; implementation-plan.md step 5 |
required in this PR cut |
remote_wire.cpp explicit put/get helpers; Python _Reader; local mailbox remains same-binary POD IPC only |
FrameRoundTripValidatesHeader, Python decode tests |
Found and fixed CallConfigWire.enable_scope_stats drift in Phase 3 |
PR 3 |
| R17 Frame codec rejects bad magic/version/type/flags, oversized/truncated payloads, unknown enums, non-zero reserved fields, and malformed counts |
protocol.md §§Frames, Wire Encoding, Bounds and Fuzz Tests; implementation-plan.md step 5 |
required in this PR cut |
C++ and Python decode validation for headers, counts, enums, reserved fields, payload sizes |
remote wire bad header/truncation/reserved tests; Python materialization/decode tests |
No blocking drift found |
PR 3 |
R18 TASK payload carries digest, CallConfigWire, and RemoteTaskArgsWire; tensor wire data must be zero; HOST_INLINE must be descriptor-backed and bounds-checked |
protocol.md §§TASK Payload, RemoteTensorDesc, Bounds and Fuzz Tests; buffers-and-transports.md §TaskArgs Sidecar Contract |
required in this PR cut |
encode_task_payload, decode_task_payload, RemoteL3Endpoint::build_task_payload, Python _materialize_task_args |
non-zero tensor data rejection, HOST_INLINE descriptor bounds, host-inline materialization/roundtrip |
Phase 3 fixed and tested missing enable_scope_stats field |
PR 3 / PR 5 |
| R19 COMPLETION and CONTROL_REPLY match sequence/name/version, bound error text, and fabricate failures on health expiry or process exit |
protocol.md §§COMPLETION Payload, CONTROL_REPLY Payload, Ordering; remote-l3-worker-design.md §Failure Semantics |
required in this PR cut |
decode_completion, decode_control_reply, remote endpoint reply validation, session _format_remote_error, health monitor |
sequence/name mismatch tests, remote host_tcp error completion and process exit tests |
None found |
PR 3 / PR 5 / PR 6 |
| R20 Each endpoint has one ordered command lane; TASK, state-changing CONTROL, SHUTDOWN, replies, and visibility are sequence ordered |
remote-l3-worker-design.md §§Remote Worker Session, Protocol; protocol.md §Ordering; buffers-and-transports.md §HCOMM Adapter Contract |
required in this PR cut |
OrderedCommandLane, endpoint command mutex, session command loop, SHUTDOWN frame path |
OrderedCommandLaneIsSingleFlight, remote prepare/register tests |
None found |
PR 3 / PR 5 / PR 6 |
| R21 Health/liveness is independent from command-lane progress; health expiry removes endpoint eligibility and fails pending/in-flight work |
remote-l3-worker-design.md §Remote Worker Session; protocol.md §Ordering; implementation-plan.md steps 5,7 |
required in this PR cut |
separate health socket/thread, remote endpoint health monitor, process-exit endpoint failure |
long-task health-lane test, process-exit endpoint failure test |
Broader multi-endpoint health-removal coverage should remain in PR 5/6 verification |
PR 5 / PR 6 |
| R22 Remote memory APIs expose opaque handles and host_tcp alloc/free/copy/export/import/release-import controls |
buffers-and-transports.md §§Buffer Handles, Public Memory API, Required Controls; implementation-plan.md steps 8-9 |
required in this PR cut |
Python RemoteBufferHandle, Worker remote memory APIs, C++ controls, session host buffer registry |
opaque handle tests, buffer copy roundtrip, export/import controls roundtrip |
None found |
PR 5 / PR 6 |
| R23 Owner/imported handle semantics preserve owner identity, worker eligibility, access flags, and import rollback behavior |
remote-l3-worker-design.md §Buffer Lifecycle; buffers-and-transports.md §§Export/Import Handle Semantics, Release Policy; protocol.md §Remote Buffer Export and Import Controls |
required in this PR cut |
export/import descriptors, access flag validation, owner/import worker routing, imported address-space materialization |
remote buffer export/import wire tests, imported buffer runs on peer worker |
Import rollback is implementation-reviewed but should get split-specific failure injection |
PR 5 / PR 6 |
| R24 Owner free and release-import defer physical cleanup until slot refs and imports drain; failed runs use the same post-drain cleanup path |
remote-l3-worker-design.md §Buffer Lifecycle; buffers-and-transports.md §Release Policy; implementation-plan.md step 9 |
required in this PR cut |
Python slot refs, pending remote frees/import releases, run/close cleanup flush |
owner-free-waits-for-import-release, input-free-deferred-until-slot-refs-drop |
None found |
PR 4 / PR 5 / PR 6 |
R25 Remote CommDomain controls COMM_INIT, ALLOC_DOMAIN, and RELEASE_DOMAIN are reserved and rejected as unsupported in this cut |
remote-l3-worker-design.md §Scope; protocol.md §CONTROL Payload; implementation-plan.md steps 8,13 |
required explicit unsupported behavior |
session control loop returns error for reserved domain controls; local C++ mailbox CommDomain controls remain local-only |
Implementation inspected; no direct UT found |
Add direct reserved-control negative in PR 6 verification |
PR 5 / PR 6 |
| R26 A2 RoCE, A3 HCCS, and A5 UB HCOMM profiles are documented contracts but pending hardware-profile work, not part of this split cut |
remote-l3-worker-design.md §§Current Implementation Status, Rollout; buffers-and-transports.md §§HCOMM Adapter Contract, A2/A3/A5 Profiles; implementation-plan.md steps 10-12 |
reserved or future work |
daemon accepts only transport="host_tcp"; export/import host_tcp rejects unsupported profiles |
Implementation inspected; remote worker manifest validation covers host_tcp-only |
Future only; do not include HCOMM adapters in child PRs |
Future |
| R27 Exact HCCS/UB HAL names, daemon auth/isolation, serialized compatibility metadata, and future CommContext split remain open decisions |
implementation-plan.md §Open Decisions |
open decision |
Documented only |
Not applicable |
Future only |
Future |
R28 Top-of-stack verification excludes unavailable hardware except through task-submit, covers Python UT, C++ UT, pre-commit docs, sim ST, and remote host_tcp integration |
pr-split-and-audit-plan.md §Review and Verification; .claude/rules/task-submit-isolation.md |
required in this PR cut |
Audit follows local UT only here; hardware remains task-submit only |
Current run log below |
Full child PR verification remains to be run after split |
All |