Qwen3-14B

PyPTO Serving supports Qwen3-14B through the bundled Qwen model loader, NPU executor, and PyPTO kernels. Use this path for single-device validation, offline tensor-parallel runs, and HTTP serving with one or more data-parallel replicas.

Checkpoint

Use a local Hugging Face style Qwen3-14B checkpoint directory. The directory must contain config.json, tokenizer files, and model weight shards readable by the active Python environment.

Offline Generation

pypto-serving \
  --model /path/to/Qwen3-14B \
  --prompt 'Huawei is' \
  --platform a2a3 \
  --device 0 \
  --max-model-len 512 \
  --generate-config '{"max_new_tokens": 32}'

For one tensor-parallel worker group, provide --devices and --tp:

pypto-serving \
  --model /path/to/Qwen3-14B \
  --prompt 'Huawei is' \
  --platform a2a3 \
  --devices 0,1 \
  --tp 2 \
  --max-model-len 512 \
  --generate-config '{"max_new_tokens": 32}'

Offline generate mode shares the serving engine. --dp creates independent replica engines, and repeated --prompt values are routed across them by the least_pending_tokens policy.

HTTP Serving

pypto-serving \
  --model /path/to/Qwen3-14B \
  --backend npu \
  --platform a2a3 \
  --device 0 \
  --max-model-len 512 \
  --port 8899

Send a request after startup:

curl --noproxy "*" http://127.0.0.1:8899/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Huawei is","max_tokens":32,"temperature":0.0}'

DP=2 Serving

Data parallel serving creates independent replicas and routes requests by the least_pending_tokens policy:

pypto-serving \
  --model /path/to/Qwen3-14B \
  --backend npu \
  --platform a2a3 \
  --devices 0,1 \
  --dp 2 \
  --tp 1 \
  --max-model-len 512 \
  --port 8899

Weight Staging

The loader stages per-layer weights without keeping a second full model copy. Its host slabs use shared memory because upload reads them from a forked child. A checkpoint without QK norms is supported; the missing gamma weights default to ones. See Weight Staging for the shared loading flow.

Runtime Notes

Qwen uses the standard pypto-serving runtime capacity, generation, prefix caching, chunked prefill, and compile-cache controls. See pypto-serving for the command-line reference.

Prefix caching and chunked prefill are enabled by default for Qwen serving.