GB200 A4X 部署指南
适用于 GCP A4X (GB200/GB300 NVL72) 自建 Kubernetes 环境。支持多集群、多区域部署。
全局变量
修改以下变量后,后续所有命令可直接复制粘贴执行。本指南支持在不同区域部署不同规模的集群,请根据当前实际交付的集群(GB200 或 GB300)取消对应区块的注释,并注释掉另一个区块。
# ======================== 项目 ========================
export PROJECT="tencent-gcp-taiji"
# ======================== 区域、规模与机器配置 ========================
# 【选项 A】: GB200 集群 (1296 GPU = 324 VM, us-east1)
export REGION="us-east1"
export ZONE="us-east1-d"
export TOTAL_GPUS=1296
export MACHINE_TYPE="a4x-highgpu-4g"
export CLUSTER_PREFIX="gb200-east"
# GVNIC 网段 (假设客户主 VPC 为 taiji-net-0)
export GVNIC_NET="taiji-net-0"
export GVNIC_SUB="${CLUSTER_PREFIX}-gvnic-sub"
export GVNIC_CIDR="10.0.0.0/22" # /22 提供 1020 个可用 IP
export GVNIC_NET_1="${CLUSTER_PREFIX}-gvnic-net-1"
export GVNIC_SUB_1="${CLUSTER_PREFIX}-gvnic-sub-1"
export GVNIC_1_CIDR="10.0.4.0/22" # /22 提供 1020 个可用 IP
# RDMA 网段 (独立的区域性 RoCE VPC)
export RDMA_NET="${CLUSTER_PREFIX}-rdma-net"
export RDMA_CIDRS=("10.0.16.0/22" "10.0.20.0/22" "10.0.24.0/22" "10.0.28.0/22")
# ======================== 集群 B: GB300 (us-central1) ========================
# 【选项 B】: GB300 集群 (1800 GPU = 450 VM, us-central1)
# export REGION="us-central1"
# export ZONE="us-central1-b"
# export TOTAL_GPUS=1800
# export MACHINE_TYPE="a4x-maxgpu-4g-metal" # GB300 裸金属机型
# export CLUSTER_PREFIX="gb300-central"
# GVNIC 网段 (共享同一个客户主 VPC taiji-net-0,使用不重叠的 CIDR)
# export GVNIC_NET="taiji-net-0"
# export GVNIC_SUB="${CLUSTER_PREFIX}-gvnic-sub"
# export GVNIC_CIDR="10.0.32.0/22" # /22 提供 1020 个可用 IP
# export GVNIC_NET_1="${CLUSTER_PREFIX}-gvnic-net-1"
# export GVNIC_SUB_1="${CLUSTER_PREFIX}-gvnic-sub-1"
# export GVNIC_1_CIDR="10.0.36.0/22" # /22 提供 1020 个可用 IP
# RDMA 网段 (独立的区域性 RoCE VPC,使用不重叠的名称和 CIDR)
# export RDMA_NET="${CLUSTER_PREFIX}-rdma-net"
# export RDMA_CIDRS=("10.0.48.0/22" "10.0.52.0/22" "10.0.56.0/22" "10.0.60.0/22")
# ======================== 镜像与预留 ========================
export IMAGE="your-tlinux4-gb200-image" # 客户提供的 OS 镜像名 (GB300 裸金属需确保为 ARM64 构建)
export IMAGE_PROJECT="your-image-project" # 镜像所在项目
export RESERVATION="your-reservation-name" # 预留实例名称
# ======================== 拓扑计算 ========================
export GPUS_PER_NODE=4
export NODES_PER_DOMAIN=18 # NVL72 = 18 节点/域
export NUM_DOMAINS=$((TOTAL_GPUS / GPUS_PER_NODE / NODES_PER_DOMAIN))
export CP_NAME="${CLUSTER_PREFIX}-cp"
export WORKER_PREFIX="${CLUSTER_PREFIX}" # Worker: ${WORKER_PREFIX}-d${DOMAIN}-w${INDEX}
export PLACEMENT_PREFIX="${CLUSTER_PREFIX}-nvl72-policy"
export SUPERNET="10.0.0.0/16" # 防火墙规则用
# ======================== Kubernetes ========================
export K8S_VERSION="1.34"
export K8S_PATCH="1.34.9" # >= 1.34.5 (修复 kubelet DRA gRPC 死锁)
export POD_CIDR="10.244.0.0/16"
export CALICO_VERSION="3.29.3"Part 1: 网络基础设施
1.1 创建 VPC
注意:两个集群如果共享主管理网络,第二次部署时已有的 GVNIC VPC 可能会提示已存在,此为正常现象,脚本中的
|| true会忽略该报错并继续执行。RDMA 网络在不同区域内是完全隔离的,每个集群均独立创建。
# 主 GVNIC 网络 — MTU 8896 (Jumbo Frames, 必须)
gcloud compute networks create $GVNIC_NET \
--subnet-mode=custom --mtu=8896 --project=$PROJECT || true
# 辅 GVNIC 网络
gcloud compute networks create $GVNIC_NET_1 \
--subnet-mode=custom --mtu=8896 --project=$PROJECT || true
# RDMA HPC 网络 (必须基于当前 Zone 的网络配置文件)
gcloud beta compute networks create $RDMA_NET \
--network-profile=${ZONE}-vpc-roce \
--subnet-mode=custom --project=$PROJECT1.2 创建子网
# 主 GVNIC 子网
gcloud compute networks subnets create $GVNIC_SUB \
--network=$GVNIC_NET --region=$REGION \
--range=$GVNIC_CIDR --project=$PROJECT
# 辅 GVNIC 子网
gcloud compute networks subnets create $GVNIC_SUB_1 \
--network=$GVNIC_NET_1 --region=$REGION \
--range=$GVNIC_1_CIDR --project=$PROJECT
# RDMA 子网 (采用 Rail-optimized 架构,每张网卡接入一个独立的子网)
for n in 0 1 2 3; do
gcloud compute networks subnets create ${RDMA_NET}-sub-${n} \
--network=$RDMA_NET --region=$REGION \
--range="${RDMA_CIDRS[$n]}" --project=$PROJECT
done1.3 创建防火墙规则
注意: 为了防止不同集群在共享 VPC 时防火墙规则同名报错,以下规则名称中已加入
${CLUSTER_PREFIX}前缀。
# IAP SSH (用于 gcloud compute ssh --tunnel-through-iap)
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-iap-ssh \
--project=$PROJECT --network=$GVNIC_NET \
--direction=INGRESS --priority=900 --action=ALLOW \
--rules=tcp:22,tcp:56000 --source-ranges=35.235.240.0/20
# 主 GVNIC 内部通信 (节点间 + Pod CIDR)
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-internal-gvnic \
--project=$PROJECT --network=$GVNIC_NET \
--direction=INGRESS --priority=1000 --action=ALLOW \
--rules=tcp,udp,icmp --source-ranges="${GVNIC_CIDR},${POD_CIDR}"
# 辅 GVNIC 内部通信
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-internal-gvnic1 \
--project=$PROJECT --network=$GVNIC_NET_1 \
--direction=INGRESS --priority=1000 --action=ALLOW \
--rules=tcp,udp,icmp --source-ranges="${GVNIC_1_CIDR}"注意: 如果组织策略会自动删除防火墙规则,需在每次 SSH 前检查规则是否存在。RDMA 网络的防火墙由 Network Profile 管理,无需手动创建。
1.4 创建 Placement Policy
每个 NVL72 Domain 需要一个独立的 Placement Policy。
for d in $(seq 1 $NUM_DOMAINS); do
gcloud beta compute resource-policies create group-placement \
${PLACEMENT_PREFIX}-${d} \
--collocation=collocated \
--gpu-topology=1x72 \
--project=$PROJECT --region=$REGION
done
# 验证
gcloud compute resource-policies list --project=$PROJECT --filter="name~${PLACEMENT_PREFIX}"Part 2: VM 创建
2.1 Startup Script
将以下脚本保存为 tlinux4-customer-init.sh。v5.3
镜像已内置 NVIDIA driver、GVE、IMEX、GDRCopy、nvidia-persistenced 等所有
GPU 相关组件,startup script 仅负责: 1. 文件系统初始化(Boot disk 分区
+ Local SSD RAID0) 2. sshd port 22 + 56000 配置 3. RDMA IB
设备改名(mlx5_X → bondY)
注意: - v5.3 镜像不再需要重启。脚本一次运行即可完成所有配置。 - TLinux 4 基础镜像的 sshd 默认端口为 36000(Port 22 被注释掉),startup script 会显式启用 Port 22 和 Port 56000。
#!/bin/bash
# TLinux 4 (TencentOS Server 4.0) — v5.3 Image Startup Script
# Compatible with: A4X/GB200, GB300, H100, CPU-only instances
#
# v5.3 镜像已内置: NVIDIA driver, GVE, IMEX, GDRCopy, nvidia-persistenced,
# SELinux/firewalld disabled, memlock unlimited, sudoers, SSH
#
# 本脚本仅负责: 文件系统初始化 + sshd 22/56000 端口 + 网卡改名 (bond0-bond5)
set -euo pipefail
exec > >(tee /var/log/tlinux4-customer-init.log) 2>&1
INIT_STAMP=/etc/.customer-init-done
###############################################################################
# Network interface rename (A4X only: bond0-bond5) — runs every boot
# GVNIC: enp0s3→bond0, enp192s2→bond1 (persisted via udev after first boot)
# RDMA: mlx5_0→bond2, mlx5_1→bond3, mlx5_2→bond4, mlx5_3→bond5
###############################################################################
if rdma link show 2>/dev/null | grep -q .; then
echo "=== Network interface rename (A4X) ==="
# GVNIC rename (Linux netdev: ip link set name)
for pair in "enp0s3:bond0" "enp192s2:bond1"; do
OLD="${pair%%:*}"
NEW="${pair##*:}"
if ip link show "$OLD" &>/dev/null; then
ip link set dev "$OLD" down
ip link set dev "$OLD" name "$NEW"
ip link set dev "$NEW" up
echo " $OLD → $NEW"
elif ip link show "$NEW" &>/dev/null; then
echo " $NEW already exists (skip)"
fi
done
# Update NM connection profile to match new name
NM_CONN="/etc/NetworkManager/system-connections/enp0s3-gvnic.nmconnection"
if [ -f "$NM_CONN" ] && grep -q "interface-name=enp0s3" "$NM_CONN"; then
sed -i 's/interface-name=enp0s3/interface-name=bond0/' "$NM_CONN"
sed -i 's/id=enp0s3-gvnic/id=bond0-gvnic/' "$NM_CONN"
nmcli connection reload 2>/dev/null || true
echo " Updated NM profile: enp0s3-gvnic → bond0-gvnic"
fi
# Create udev rule for persistent GVNIC naming across reboots (one-time)
UDEV_RULE="/etc/udev/rules.d/70-a4x-bond-rename.rules"
if [ ! -f "$UDEV_RULE" ]; then
PCI0=$(basename "$(readlink -f /sys/class/net/bond0/device)")
PCI1=$(basename "$(readlink -f /sys/class/net/bond1/device)")
cat > "$UDEV_RULE" << UDEVEOF
SUBSYSTEM=="net", ACTION=="add", KERNELS=="$PCI0", NAME="bond0"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="$PCI1", NAME="bond1"
UDEVEOF
echo " Created udev rule for persistent GVNIC naming"
fi
# RDMA IB device rename (rdma dev set — runtime only, needs every boot)
for i in 0 1 2 3; do
if rdma dev set mlx5_${i} name bond$((i+2)) 2>/dev/null; then
echo " mlx5_${i} → bond$((i+2))"
fi
done
echo " --- Result ---"
ip -br link show type ether 2>/dev/null || true
rdma link show 2>/dev/null || true
fi
if [ -f "$INIT_STAMP" ]; then
echo "Customer initialization already completed. Skipping."
exit 0
fi
echo "================================================================"
echo " TLinux 4 v5.3 — Startup Script"
echo " $(date)"
echo "================================================================"
###############################################################################
# [1] Boot disk partitioning: /usr/local (20G) + /data (remaining)
###############################################################################
echo "=== [1] Boot disk partitioning ==="
BOOT_DEV=$(findmnt -n -o SOURCE / | sed 's/p[0-9]*$//')
PARTITION_STAMP=/etc/.partitions-created
if [ ! -f "$PARTITION_STAMP" ]; then
echo "--- Creating boot disk partitions (1TB disk) ---"
# Expand GPT to full disk (image GPT backup may be at original image size)
sgdisk -e "$BOOT_DEV" 2>/dev/null || true
sgdisk -n 3:0:+20G -t 3:8300 "$BOOT_DEV"
sgdisk -n 4:0:0 -t 4:8300 "$BOOT_DEV"
partprobe "$BOOT_DEV" 2>/dev/null || true
sleep 2
mkfs.xfs -f "${BOOT_DEV}p3"
mkfs.xfs -f "${BOOT_DEV}p4"
TMPMNT=$(mktemp -d)
mount "${BOOT_DEV}p3" "$TMPMNT"
cp -a /usr/local/. "$TMPMNT/" 2>/dev/null || true
umount "$TMPMNT"
rmdir "$TMPMNT"
mount "${BOOT_DEV}p3" /usr/local
mkdir -p /data
mount "${BOOT_DEV}p4" /data
P3_UUID=$(blkid -s UUID -o value "${BOOT_DEV}p3")
P4_UUID=$(blkid -s UUID -o value "${BOOT_DEV}p4")
if [ -n "$P3_UUID" ] && ! grep -q "$P3_UUID" /etc/fstab; then
echo "UUID=$P3_UUID /usr/local xfs defaults,nofail 0 2" >> /etc/fstab
fi
if [ -n "$P4_UUID" ] && ! grep -q "$P4_UUID" /etc/fstab; then
echo "UUID=$P4_UUID /data xfs defaults,nofail 0 2" >> /etc/fstab
fi
touch "$PARTITION_STAMP"
echo "Boot disk partitioned: p3=/usr/local(20G) p4=/data(remaining)"
fi
###############################################################################
# [2] Local SSD RAID0 → /mnt/stateful_partition
###############################################################################
echo "=== [2] Local SSD RAID0 setup ==="
SSD_DEVICES=()
for disk in /dev/nvme*n1; do
[ -b "$disk" ] || continue
PART_COUNT=$(lsblk -n -o NAME "$disk" 2>/dev/null | wc -l)
if [ "$PART_COUNT" -le 1 ]; then
SSD_DEVICES+=("$disk")
echo " Detected local SSD: $disk ($(lsblk -n -d -o SIZE "$disk"))"
fi
done
NUM_SSDS=${#SSD_DEVICES[@]}
echo " Found $NUM_SSDS local SSD device(s)"
if [ "$NUM_SSDS" -gt 0 ]; then
MOUNT_POINT="/mnt/stateful_partition"
mkdir -p "$MOUNT_POINT"
if [ "$NUM_SSDS" -eq 1 ]; then
TARGET_DEV="${SSD_DEVICES[0]}"
mkfs.xfs -f "$TARGET_DEV"
mount "$TARGET_DEV" "$MOUNT_POINT"
DEV_UUID=$(blkid -s UUID -o value "$TARGET_DEV")
if [ -n "$DEV_UUID" ] && ! grep -q "$DEV_UUID" /etc/fstab; then
echo "UUID=$DEV_UUID $MOUNT_POINT xfs defaults,nofail 0 2" >> /etc/fstab
fi
else
mdadm --stop /dev/md0 2>/dev/null || true
mdadm --create /dev/md0 --level=0 --raid-devices="$NUM_SSDS" "${SSD_DEVICES[@]}" --force --run
mkfs.xfs -f /dev/md0
mount /dev/md0 "$MOUNT_POINT"
mkdir -p /etc/mdadm
mdadm --detail --scan >> /etc/mdadm/mdadm.conf
DEV_UUID=$(blkid -s UUID -o value /dev/md0)
if [ -n "$DEV_UUID" ] && ! grep -q "$DEV_UUID" /etc/fstab; then
echo "UUID=$DEV_UUID $MOUNT_POINT xfs defaults,nofail 0 2" >> /etc/fstab
fi
fi
mkdir -p "$MOUNT_POINT"/{gcsfuse-cache,scratch-data}
echo "$MOUNT_POINT mounted (total: $(df -h "$MOUNT_POINT" | awk 'NR==2{print $2}'))"
else
echo "WARNING: No local SSD devices found"
mkdir -p /mnt/stateful_partition
fi
###############################################################################
# [3] sshd port 56000
###############################################################################
echo "=== [3] sshd port 22 + 56000 ==="
# TLinux 4 base image uses Port 36000 with Port 22 commented out.
# Ensure both Port 22 and Port 56000 are explicitly configured.
SSHD_CHANGED=0
if ! grep -q '^Port 22' /etc/ssh/sshd_config; then
sed -i '1i Port 22' /etc/ssh/sshd_config
SSHD_CHANGED=1
fi
if ! grep -q '^Port 56000' /etc/ssh/sshd_config; then
sed -i '/^Port 22/a Port 56000' /etc/ssh/sshd_config
SSHD_CHANGED=1
fi
if [ "$SSHD_CHANGED" -eq 1 ]; then
systemctl restart sshd
echo "sshd now listening on port 22 and 56000"
else
echo "sshd ports 22 and 56000 already configured"
fi
###############################################################################
# Verification
###############################################################################
echo ""
echo "================================================================"
echo " Verification"
echo "================================================================"
echo "--- OS ---"
cat /etc/os-release | head -3
echo "Kernel: $(uname -r)"
if lspci 2>/dev/null | grep -qi 'NVIDIA'; then
echo ""
echo "--- GPU ---"
nvidia-smi -L 2>&1 | head -4
echo "Persistence: $(nvidia-smi -q 2>/dev/null | grep 'Persistence Mode' | head -1 || echo N/A)"
echo "IMEX: $(ls /dev/nvidia-caps-imex-channels/ 2>&1)"
fi
echo ""
echo "--- Network ---"
ip -br link show type ether 2>/dev/null || true
echo "RDMA: $(rdma link show 2>/dev/null | awk '{print $2}' | tr '\n' ', ' || echo N/A)"
echo ""
echo "--- Disk ---"
df -h /usr/local /data /mnt/stateful_partition 2>/dev/null | grep -v Filesystem
echo ""
echo "--- SSH ---"
echo "Ports: $(grep '^Port' /etc/ssh/sshd_config | awk '{print $2}' | tr '\n' ', ')"
# Load networking modules for Calico (VXLAN / IPIP)
echo "=== Loading networking modules ==="
sudo modprobe vxlan || true
sudo modprobe ipip || true
echo "vxlan" | sudo tee /etc/modules-load.d/vxlan.conf
echo "ipip" | sudo tee /etc/modules-load.d/ipip.conf
touch "$INIT_STAMP"
echo ""
echo "================================================================"
echo " Startup complete at $(date)"
echo "================================================================"2.2 批量创建 Worker VM
> ⚠️ **重要提示 (Placement Policy 限制与 16/18 容灾 SLA)**:
> 1. **严禁绕过 Placement Policy**:根据 GCP 官方物理限制,GB200/GB300 必须以 1x72 拓扑(18个节点)为最小单元绑定 Placement Policy 进行资源调度。如果在同一个 sub-block(NVLink 域)内绕过 policy 零散地创建虚拟机,会导致整台 NVL72 机架发生物理碎片化锁定,后续将彻底无法再用于完整的 1x72 拓扑调度。
> 2. **为什么是 `--min-count=16`**:GCP 官方针对 A4X/A4X-Max 机架的 SLA 可用性承诺为 **16/18**。这意味着在由 18 个计算节点组成的 Domain 中,GCP 允许最多存在 2 个状态为 `degraded hosts`(因硬件故障排队等待维修更换)的节点。为了确保即使物理机架中存在故障节点,您的集群依然能成功拉起可用的机器,在使用 `bulk create` 批量申请时,**必须将参数设置为 `--count=18` 且 `--min-count=16`**。如果您强行设置 `--min-count=18`,一旦该机架内有 1 台机器报修,整个创建命令都会因无法满足约束而彻底被拒!这也是为什么主流模型框架都在适配 **64 + 8 Spare (容灾备用)** 队列机制的根本硬件原因。
> 3. **故障节点生命周期与恢复机制**:
> * **16/18 容灾补充创建**:如果在首次执行 `bulk create` 时由于有部分 `degraded hosts` 而只成功拉起了 16 台虚拟机,那么缺失的 2 台在云端是**不存在**的。当底层物理机修复后,您**必须重新执行一次 bulk create**(只传入缺失的 2 台 VM 名称,如 `count=2`),并且**必须指定与前 16 台完全相同的 Placement Policy 与 Subblock**,否则将引发 `PLACEMENT_POLICY_CONSTRAINTS_INFEASIBLE` 碎片化报错。
``` bash
# 按 Domain 分批创建, 每批绑定对应的 Placement Policy
for d in $(seq 1 $NUM_DOMAINS); do
echo "=== Creating Domain $d workers ==="
gcloud beta compute instances bulk create \
--name-pattern="${WORKER_PREFIX}-d${d}-w##" \
--count=$NODES_PER_DOMAIN \
--min-count=16 \
--project=$PROJECT --zone=$ZONE \
--machine-type=$MACHINE_TYPE \
--image=$IMAGE --image-project=$IMAGE_PROJECT \
--boot-disk-size=1000GB --boot-disk-type=hyperdisk-balanced \
--scopes=cloud-platform \
--reservation-affinity=specific --reservation=$RESERVATION \
--provisioning-model=RESERVATION_BOUND \
--instance-termination-action=STOP \
--maintenance-policy=TERMINATE \
--restart-on-failure \
--resource-policies=${PLACEMENT_PREFIX}-${d} \
--network-interface=nic-type=GVNIC,network=$GVNIC_NET,subnet=$GVNIC_SUB \
--network-interface=nic-type=GVNIC,network=$GVNIC_NET_1,subnet=$GVNIC_SUB_1,no-address \
--network-interface=nic-type=MRDMA,network=$RDMA_NET,subnet=${RDMA_NET}-sub-0,no-address \
--network-interface=nic-type=MRDMA,network=$RDMA_NET,subnet=${RDMA_NET}-sub-1,no-address \
--network-interface=nic-type=MRDMA,network=$RDMA_NET,subnet=${RDMA_NET}-sub-2,no-address \
--network-interface=nic-type=MRDMA,network=$RDMA_NET,subnet=${RDMA_NET}-sub-3,no-address \
--metadata-from-file=startup-script=tlinux4-customer-init.sh
echo "Domain $d: workers creation request submitted"
done注意: - A4X VM 不支持
--local-ssd,Local SSD 由 startup script 自动检测和配置 -RESERVATION_BOUND表示 VM 只使用预留容量
2.3 创建后验证
# 检查所有 VM 状态
gcloud compute instances list --project=$PROJECT \
--filter="name~${WORKER_PREFIX}-d" \
--format="table(name,zone,status,networkInterfaces[0].networkIP)"
# SSH 到任意 worker 检查初始化日志
gcloud compute ssh ${WORKER_PREFIX}-d1-w0 --zone=$ZONE --tunnel-through-iap \
--command="tail -30 /var/log/tlinux4-customer-init.log"
# 检查 IMEX channels (Phase 1 reboot 后)
gcloud compute ssh ${WORKER_PREFIX}-d1-w0 --zone=$ZONE --tunnel-through-iap \
--command="ls /dev/nvidia-caps-imex-channels/ && nvidia-smi -L | head -4"Part 3: 拓扑信息与节点标签
GCP 为 GB200/A4X VM 提供物理拓扑信息。同一 NVL72 domain 内的 VM 返回相同的拓扑标识,可用于确定节点的 domain 归属,进而为 k8s 节点打标签。
3.1 获取拓扑信息
两种获取方式:
| 方式 | 端点 | 权限要求 | 返回值 |
|---|---|---|---|
| Metadata Server(推荐) | http://metadata.google.internal/computeMetadata/v1/instance/attributes/physical_host_topology |
无(VM 内部直接访问) | 拓扑哈希字符串 |
| Compute API | gcloud compute instances describe --format='value(resourceStatus.physicalHostTopology.subblock)' |
需要 compute.instances.get IAM 权限 |
结构化的 subblock 标识 |
# 方式 1: 在 VM 内部通过 Metadata Server 查询(无需权限)
curl -sf -H "Metadata-Flavor: Google" \
"http://metadata.google.internal/computeMetadata/v1/instance/attributes/physical_host_topology"
# 输出示例: 一串哈希字符串,如 "a1b2c3d4e5f6..."
# 方式 2: 通过 Compute API 查询(需要 IAM 权限)
gcloud compute instances describe ${INSTANCE_NAME} \
--zone=${ZONE} \
--format='value(resourceStatus.physicalHostTopology.subblock)'
# 输出示例: 结构化的 subblock 标识,如 "/clusters/xxx/blocks/yyy/subBlocks/zzz"说明: - Metadata Server 返回的是哈希值(不可读但稳定),同一 NVL72 domain 内的所有 VM 保证返回相同值 - Compute API 返回可读的 subblock 名称,但需要额外配置 IAM 权限 - 客户可根据以上信息,按自身运维规范选择标签命名方式
3.2 手动标记(小规模 / 验证环境)
# 标记 NVL72 Domain
for d in $(seq 1 $NUM_DOMAINS); do
for i in $(seq 0 $((NODES_PER_DOMAIN - 1))); do
kubectl label node ${WORKER_PREFIX}-d${d}-w${i} \
nvl72-domain=domain-${d} --overwrite
done
done
# 验证
kubectl get nodes --show-labels | grep nvl72-domain3.3 自动标记(Startup Script 方式,推荐生产环境)
在 Worker 的 startup script 中,kubeadm join 之前,从 Metadata Server
获取拓扑哈希并自动传递 --node-labels 给 kubelet:
# 在 Worker startup script 中(kubeadm join 之前)添加:
# 1. 从 Metadata Server 获取物理拓扑哈希(同 domain 的 VM 返回相同值)
TOPO_HASH=$(curl -sf -H "Metadata-Flavor: Google" \
"http://metadata.google.internal/computeMetadata/v1/instance/attributes/physical_host_topology" \
| md5sum | cut -c1-8) # 截取前 8 位作为域标识符
# 2. 配置 kubelet 启动参数,自动附带拓扑标签
mkdir -p /etc/systemd/system/kubelet.service.d
cat > /etc/default/kubelet <<KUBELET_EOF
KUBELET_EXTRA_ARGS=--node-labels=nvl72-domain=domain-${TOPO_HASH}
KUBELET_EOF
systemctl daemon-reload
# 3. kubeadm join 时节点自动携带标签
kubeadm join ${CP_IP}:6443 \
--token "${JOIN_TOKEN}" \
--discovery-token-ca-cert-hash "sha256:${JOIN_HASH}" \
--node-name "${NODE_NAME}" \
--ignore-preflight-errors=Hostname替代方式:kubeadm JoinConfiguration(声明式配置)
# kubeadm-join-config.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
- name: "node-labels"
value: "nvl72-domain=domain-${TOPO_HASH}"
discovery:
bootstrapToken:
apiServerEndpoint: "${CP_IP}:6443"
token: "${JOIN_TOKEN}"
caCertHashes:
- "sha256:${JOIN_HASH}"
# 使用方式:kubeadm join --config kubeadm-join-config.yaml两种方式功能等价。
--node-labels在 k8s 1.34 中未被弃用,kubeadm 的kubeletExtraArgs本质上仍是传递 CLI 参数。配置文件方式更符合 IaC 管理风格,但技术上无差异。
3.4 客户自定义标签
以上示例使用 nvl72-domain=domain-${TOPO_HASH}
作为标签格式,客户可根据自身规范自定义:
- Label key 和 value 均可自定义,例如
topology.mycompany.io/nvl72-rack=rack-001 - 核心要求:同一 NVL72 domain 内的节点必须具有相同标签值,不同 domain 的标签值必须不同
- 后续依赖:Section 4.6 ComputeDomain、Section 4.10 Kueue TAS topology 等步骤中引用的 label key 需与此处保持一致
说明:以上 Part 1-3 为基础设施准备的必需步骤。以下 Part 4-5 为我们基于测试环境验证的推荐最佳实践,供参考。
Part 4: Kubernetes 部署建议(推荐)
以下是基于我们测试环境验证的 k8s 部署方案。
4.1 Control Plane 初始化
将以下脚本保存为 kubeadm-control-plane.sh 并在 CP
节点上执行。
#!/bin/bash
set -euo pipefail
exec > >(tee /var/log/kubeadm-setup.log) 2>&1
K8S_VERSION="1.34"
K8S_PATCH="1.34.9"
POD_CIDR="10.244.0.0/16"
CALICO_VERSION="3.29.3"
echo "=== [1/7] Disable swap and SELinux ==="
swapoff -a
sed -i '/swap/d' /etc/fstab
setenforce 0 || true
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
echo "=== [2/7] Kernel modules and sysctl ==="
cat <<EOF > /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
echo "=== [3/7] Install containerd ==="
dnf install -y yum-utils
dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
dnf install -y containerd.io
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
systemctl enable --now containerd
echo "=== [4/7] Install kubeadm, kubelet, kubectl ==="
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/rpm/repodata/repomd.xml.key
EOF
dnf install -y kubelet-${K8S_PATCH} kubeadm-${K8S_PATCH} kubectl-${K8S_PATCH}
systemctl enable kubelet
echo "=== [5/7] Initialize kubeadm ==="
INTERNAL_IP=$(curl -s -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip)
kubeadm init \
--pod-network-cidr=${POD_CIDR} \
--apiserver-advertise-address=${INTERNAL_IP} \
--control-plane-endpoint=${INTERNAL_IP}
echo "=== [6/7] Configure kubectl ==="
mkdir -p /root/.kube
cp /etc/kubernetes/admin.conf /root/.kube/config
echo "=== [7/7] Install Calico CNI ==="
curl -s -O https://raw.githubusercontent.com/projectcalico/calico/v${CALICO_VERSION}/manifests/calico.yaml
# 修改 Calico MTU 避免启动失败
sed -i 's/veth_mtu: "0"/veth_mtu: "1450"/' calico.yaml
kubectl --kubeconfig=/root/.kube/config apply -f calico.yaml
echo "=== Generate join command ==="
kubeadm token create --print-join-command > /root/kubeadm-join-command.txt
cat /root/kubeadm-join-command.txt
echo "=== Control plane setup complete ==="4.2 Calico 配置
CP 初始化后,配置 Calico VXLAN 模式和网卡检测。
重要: GCP 组织策略可能删除 IPIP 协议的防火墙规则,必须使用 VXLAN 模式。
# 1. 配置 Felix 使用 VXLAN 模式并关闭 IPIP
kubectl patch felixconfiguration default --type=merge -p '{"spec":{"vxlanEnabled":true,"ipipEnabled":false}}'
# 2. 配置默认 IPPool 使用 VXLAN 模式 (必选,防 VTEP IP 分配不一致)
kubectl patch ippool default-ipv4-ippool --type=merge -p '{"spec":{"vxlanMode":"Always","ipipMode":"Never"}}'
# 3. A4X Worker 网卡由 startup script 改名为 bond0, 需要更新 IP 自动检测
kubectl set env daemonset/calico-node -n kube-system \
IP_AUTODETECTION_METHOD="interface=eth.*|bond0"
# 重启 Calico 使配置生效
kubectl rollout restart daemonset/calico-node -n kube-system4.3 Worker 加入集群
将以下脚本保存为 tlinux4-k8s-worker.sh,在每个 worker
上执行(v5.3 镜像已内置 GPU/IMEX/GDRCopy 等组件)。
#!/bin/bash
# Worker k8s 组件安装 + 加入集群
# 前提: v5.3 镜像已内置 NVIDIA driver, IMEX, GDRCopy, nvidia-persistenced
set -euo pipefail
exec > >(tee /var/log/k8s-worker-setup.log) 2>&1
K8S_VERSION="1.34"
K8S_PATCH="1.34.9"
echo "=== [1/6] Disable swap and SELinux ==="
swapoff -a
sed -i '/swap/d' /etc/fstab
setenforce 0 || true
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config 2>/dev/null || true
echo "=== [2/6] Kernel modules ==="
cat > /etc/modules-load.d/k8s.conf <<EOF
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter
cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
echo "=== [3/6] Install containerd ==="
dnf install -y --enablerepo=AppStream container-selinux 2>/dev/null || true
DOCKER_ARCH=$(uname -m)
cat > /etc/yum.repos.d/docker-ce.repo <<REPOEOF
[docker-ce-stable]
name=Docker CE Stable - ${DOCKER_ARCH}
baseurl=https://download.docker.com/linux/rhel/9/${DOCKER_ARCH}/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/rhel/gpg
REPOEOF
dnf install -y containerd.io
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
mkdir -p /data/containerd
systemctl enable --now containerd
echo "=== [4/6] Install NVIDIA container toolkit ==="
curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
tee /etc/yum.repos.d/nvidia-container-toolkit.repo
dnf install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=containerd --set-as-default
# containerd config default uses single quotes; nvidia-ctk may rewrite with double quotes
sed -i "s|root = .*/var/lib/containerd.*|root = \"/data/containerd\"|" /etc/containerd/config.toml
systemctl restart containerd
echo "=== [5/6] Install kubeadm/kubelet/kubectl ==="
cat > /etc/yum.repos.d/kubernetes.repo <<REPOEOF
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/rpm/repodata/repomd.xml.key
REPOEOF
dnf install -y --enablerepo=AppStream kubelet-${K8S_PATCH} kubeadm-${K8S_PATCH} kubectl-${K8S_PATCH} \
conntrack-tools socat iptables ethtool
systemctl enable kubelet
echo "=== [6/6] kubeadm join ==="
# 方式 1: 从 GCE Metadata 获取 join 信息 (如果创建 VM 时传了 metadata)
CP_IP=$(curl -sf -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/attributes/cp-ip 2>/dev/null || true)
JOIN_TOKEN=$(curl -sf -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/attributes/join-token 2>/dev/null || true)
JOIN_HASH=$(curl -sf -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/attributes/join-hash 2>/dev/null || true)
NODE_NAME=$(curl -sf -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/name 2>/dev/null || hostname -s)
if [[ -n "$CP_IP" && -n "$JOIN_TOKEN" && -n "$JOIN_HASH" ]]; then
kubeadm join ${CP_IP}:6443 \
--token "${JOIN_TOKEN}" \
--discovery-token-ca-cert-hash "sha256:${JOIN_HASH}" \
--node-name "${NODE_NAME}" \
--ignore-preflight-errors=Hostname
else
echo "Missing CP metadata. Please run manually:"
echo " kubeadm join <CP_IP>:6443 --token <TOKEN> --discovery-token-ca-cert-hash sha256:<HASH> --node-name ${NODE_NAME}"
echo ""
echo "On CP, run: kubeadm token create --print-join-command"
fi
echo "=== Worker k8s setup complete at $(date) ==="重要: NVIDIA container toolkit 必须设为 containerd 的默认 runtime (
--set-as-default)。GPU 容器需要 nvidia-container-runtime 注入驱动库。
内部参考:一体化 Worker 脚本(v5.3 镜像 + k8s,支持人员/验证用)
以下脚本在 v5.3 镜像上完成文件系统初始化、sshd 22/56000、RDMA IB 设备改名,并安装 k8s worker 组件。适用于内部快速搭建验证环境。
注意: - v5.3 镜像不再需要重启。脚本一次运行即可完成所有配置,含 k8s 安装和 kubeadm join。 - TLinux 4 基础镜像的 sshd 默认端口为 36000,脚本会显式启用 Port 22 和 Port 56000。 - containerd 数据目录配置在
/data/containerd(50G 根分区空间不足)。
使用方式:
# 1. 在 CP 上生成 join 信息
kubeadm token create --print-join-command
# 输出中提取: CP_IP, TOKEN, HASH
# 2. 创建 Worker VM 时,将 join 信息和此脚本一起传入
gcloud compute instances create ${WORKER_NAME} \
... \
--metadata=cp-ip=${CP_IP},join-token=${TOKEN},join-hash=${HASH} \
--metadata-from-file=startup-script=tlinux4-internal-init.sh完整脚本(保存为
tlinux4-internal-init.sh):
#!/bin/bash
# TLinux 4 (TencentOS Server 4.0) — v5.3 Image Internal Test Environment
# Includes k8s worker setup (containerd + nvidia-ctk + kubeadm join)
#
# v5.3 镜像已内置: NVIDIA driver, GVE, IMEX, GDRCopy, nvidia-persistenced,
# SELinux/firewalld disabled, memlock unlimited, sudoers, SSH
#
# 本脚本负责: 文件系统初始化 + sshd 22/56000 端口 + 网卡改名 (bond0-bond5) + k8s worker
set -euo pipefail
exec > >(tee /var/log/tlinux4-internal-init.log) 2>&1
K8S_VERSION="1.34"
K8S_PATCH="1.34.9"
INIT_STAMP=/etc/.internal-init-done
###############################################################################
# Network interface rename (A4X only: bond0-bond5) — runs every boot
# GVNIC: enp0s3→bond0, enp192s2→bond1 (persisted via udev after first boot)
# RDMA: mlx5_0→bond2, mlx5_1→bond3, mlx5_2→bond4, mlx5_3→bond5
###############################################################################
if rdma link show 2>/dev/null | grep -q .; then
echo "=== Network interface rename (A4X) ==="
# GVNIC rename (Linux netdev: ip link set name)
for pair in "enp0s3:bond0" "enp192s2:bond1"; do
OLD="${pair%%:*}"
NEW="${pair##*:}"
if ip link show "$OLD" &>/dev/null; then
ip link set dev "$OLD" down
ip link set dev "$OLD" name "$NEW"
ip link set dev "$NEW" up
echo " $OLD → $NEW"
elif ip link show "$NEW" &>/dev/null; then
echo " $NEW already exists (skip)"
fi
done
# Update NM connection profile to match new name
NM_CONN="/etc/NetworkManager/system-connections/enp0s3-gvnic.nmconnection"
if [ -f "$NM_CONN" ] && grep -q "interface-name=enp0s3" "$NM_CONN"; then
sed -i 's/interface-name=enp0s3/interface-name=bond0/' "$NM_CONN"
sed -i 's/id=enp0s3-gvnic/id=bond0-gvnic/' "$NM_CONN"
nmcli connection reload 2>/dev/null || true
echo " Updated NM profile: enp0s3-gvnic → bond0-gvnic"
fi
# Create udev rule for persistent GVNIC naming across reboots (one-time)
UDEV_RULE="/etc/udev/rules.d/70-a4x-bond-rename.rules"
if [ ! -f "$UDEV_RULE" ]; then
PCI0=$(basename "$(readlink -f /sys/class/net/bond0/device)")
PCI1=$(basename "$(readlink -f /sys/class/net/bond1/device)")
cat > "$UDEV_RULE" << UDEVEOF
SUBSYSTEM=="net", ACTION=="add", KERNELS=="$PCI0", NAME="bond0"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="$PCI1", NAME="bond1"
UDEVEOF
echo " Created udev rule for persistent GVNIC naming"
fi
# RDMA IB device rename (rdma dev set — runtime only, needs every boot)
for i in 0 1 2 3; do
if rdma dev set mlx5_${i} name bond$((i+2)) 2>/dev/null; then
echo " mlx5_${i} → bond$((i+2))"
fi
done
echo " --- Result ---"
ip -br link show type ether 2>/dev/null || true
rdma link show 2>/dev/null || true
fi
# v5.3 image has stale stamp from build; remove if partitions not yet created
if [ -f "$INIT_STAMP" ] && [ ! -f "/etc/.partitions-created" ]; then
echo "Removing stale init stamp from image build..."
rm -f "$INIT_STAMP"
fi
if [ -f "$INIT_STAMP" ]; then
echo "Internal initialization already completed. Skipping."
exit 0
fi
echo "================================================================"
echo " TLinux 4 v5.3 — Internal Test Startup Script"
echo " $(date)"
echo "================================================================"
###############################################################################
# [1] Boot disk partitioning: /usr/local (20G) + /data (remaining)
###############################################################################
echo "=== [1] Boot disk partitioning ==="
BOOT_DEV=$(findmnt -n -o SOURCE / | sed 's/p[0-9]*$//')
PARTITION_STAMP=/etc/.partitions-created
if [ ! -f "$PARTITION_STAMP" ]; then
echo "--- Creating boot disk partitions (1TB disk) ---"
# Expand GPT to full disk (image GPT backup may be at original image size)
sgdisk -e "$BOOT_DEV" 2>/dev/null || true
sgdisk -n 3:0:+20G -t 3:8300 "$BOOT_DEV"
sgdisk -n 4:0:0 -t 4:8300 "$BOOT_DEV"
partprobe "$BOOT_DEV" 2>/dev/null || true
sleep 2
mkfs.xfs -f "${BOOT_DEV}p3"
mkfs.xfs -f "${BOOT_DEV}p4"
TMPMNT=$(mktemp -d)
mount "${BOOT_DEV}p3" "$TMPMNT"
cp -a /usr/local/. "$TMPMNT/" 2>/dev/null || true
umount "$TMPMNT"
rmdir "$TMPMNT"
mount "${BOOT_DEV}p3" /usr/local
mkdir -p /data
mount "${BOOT_DEV}p4" /data
P3_UUID=$(blkid -s UUID -o value "${BOOT_DEV}p3")
P4_UUID=$(blkid -s UUID -o value "${BOOT_DEV}p4")
if [ -n "$P3_UUID" ] && ! grep -q "$P3_UUID" /etc/fstab; then
echo "UUID=$P3_UUID /usr/local xfs defaults,nofail 0 2" >> /etc/fstab
fi
if [ -n "$P4_UUID" ] && ! grep -q "$P4_UUID" /etc/fstab; then
echo "UUID=$P4_UUID /data xfs defaults,nofail 0 2" >> /etc/fstab
fi
touch "$PARTITION_STAMP"
echo "Boot disk partitioned: p3=/usr/local(20G) p4=/data(remaining)"
fi
###############################################################################
# [2] Local SSD RAID0 → /mnt/stateful_partition
###############################################################################
echo "=== [2] Local SSD RAID0 setup ==="
SSD_DEVICES=()
for disk in /dev/nvme*n1; do
[ -b "$disk" ] || continue
PART_COUNT=$(lsblk -n -o NAME "$disk" 2>/dev/null | wc -l)
if [ "$PART_COUNT" -le 1 ]; then
SSD_DEVICES+=("$disk")
echo " Detected local SSD: $disk ($(lsblk -n -d -o SIZE "$disk"))"
fi
done
NUM_SSDS=${#SSD_DEVICES[@]}
echo " Found $NUM_SSDS local SSD device(s)"
if [ "$NUM_SSDS" -gt 0 ]; then
MOUNT_POINT="/mnt/stateful_partition"
mkdir -p "$MOUNT_POINT"
if [ "$NUM_SSDS" -eq 1 ]; then
TARGET_DEV="${SSD_DEVICES[0]}"
mkfs.xfs -f "$TARGET_DEV"
mount "$TARGET_DEV" "$MOUNT_POINT"
DEV_UUID=$(blkid -s UUID -o value "$TARGET_DEV")
if [ -n "$DEV_UUID" ] && ! grep -q "$DEV_UUID" /etc/fstab; then
echo "UUID=$DEV_UUID $MOUNT_POINT xfs defaults,nofail 0 2" >> /etc/fstab
fi
else
mdadm --stop /dev/md0 2>/dev/null || true
mdadm --create /dev/md0 --level=0 --raid-devices="$NUM_SSDS" "${SSD_DEVICES[@]}" --force --run
mkfs.xfs -f /dev/md0
mount /dev/md0 "$MOUNT_POINT"
mkdir -p /etc/mdadm
mdadm --detail --scan >> /etc/mdadm/mdadm.conf
DEV_UUID=$(blkid -s UUID -o value /dev/md0)
if [ -n "$DEV_UUID" ] && ! grep -q "$DEV_UUID" /etc/fstab; then
echo "UUID=$DEV_UUID $MOUNT_POINT xfs defaults,nofail 0 2" >> /etc/fstab
fi
fi
mkdir -p "$MOUNT_POINT"/{gcsfuse-cache,scratch-data}
echo "$MOUNT_POINT mounted (total: $(df -h "$MOUNT_POINT" | awk 'NR==2{print $2}'))"
else
echo "WARNING: No local SSD devices found"
mkdir -p /mnt/stateful_partition
fi
###############################################################################
# [3] sshd port 56000
###############################################################################
echo "=== [3] sshd port 22 + 56000 ==="
# TLinux 4 base image uses Port 36000 with Port 22 commented out.
# Ensure both Port 22 and Port 56000 are explicitly configured.
SSHD_CHANGED=0
if ! grep -q '^Port 22' /etc/ssh/sshd_config; then
sed -i '1i Port 22' /etc/ssh/sshd_config
SSHD_CHANGED=1
fi
if ! grep -q '^Port 56000' /etc/ssh/sshd_config; then
sed -i '/^Port 22/a Port 56000' /etc/ssh/sshd_config
SSHD_CHANGED=1
fi
if [ "$SSHD_CHANGED" -eq 1 ]; then
systemctl restart sshd
echo "sshd now listening on port 22 and 56000"
else
echo "sshd ports 22 and 56000 already configured"
fi
###############################################################################
# [4] k8s worker setup (containerd + nvidia-ctk + kubeadm join)
###############################################################################
echo "=== [4] k8s $K8S_VERSION worker setup ==="
# [4.1] Swap off
echo "--- [4.1] Disable swap ---"
swapoff -a
sed -i '/swap/d' /etc/fstab
# [4.2] Kernel modules for k8s
echo "--- [4.2] Kernel modules ---"
cat > /etc/modules-load.d/k8s.conf <<EOF
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter
cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
# [4.3] Install containerd
echo "--- [4.3] containerd ---"
dnf install -y --enablerepo=AppStream container-selinux 2>/dev/null || true
DOCKER_ARCH=$(uname -m)
cat > /etc/yum.repos.d/docker-ce.repo <<REPOEOF
[docker-ce-stable]
name=Docker CE Stable - ${DOCKER_ARCH}
baseurl=https://download.docker.com/linux/rhel/9/${DOCKER_ARCH}/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/rhel/gpg
REPOEOF
dnf install -y containerd.io
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
mkdir -p /data/containerd
systemctl enable --now containerd
# [4.4] NVIDIA container toolkit
echo "--- [4.4] NVIDIA container toolkit ---"
curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
tee /etc/yum.repos.d/nvidia-container-toolkit.repo
dnf install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=containerd --set-as-default
# containerd config default uses single quotes; nvidia-ctk may rewrite with double quotes
sed -i "s|root = .*/var/lib/containerd.*|root = \"/data/containerd\"|" /etc/containerd/config.toml
systemctl restart containerd
# [4.5] kubeadm / kubelet / kubectl
echo "--- [4.5] kubeadm/kubelet/kubectl ---"
cat > /etc/yum.repos.d/kubernetes.repo <<REPOEOF
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/rpm/repodata/repomd.xml.key
REPOEOF
dnf install -y --enablerepo=AppStream kubelet-${K8S_PATCH} kubeadm-${K8S_PATCH} kubectl-${K8S_PATCH} conntrack-tools socat iptables ethtool
systemctl enable kubelet
# [4.6] kubeadm join
echo "--- [4.6] kubeadm join ---"
CP_IP=$(curl -sf -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/attributes/cp-ip 2>/dev/null || true)
JOIN_TOKEN=$(curl -sf -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/attributes/join-token 2>/dev/null || true)
JOIN_HASH=$(curl -sf -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/attributes/join-hash 2>/dev/null || true)
NODE_NAME=$(cat /proc/sys/kernel/hostname 2>/dev/null || curl -sf -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/name || echo "unknown")
if [[ -n "$CP_IP" && -n "$JOIN_TOKEN" && -n "$JOIN_HASH" ]]; then
kubeadm join ${CP_IP}:6443 \
--token "${JOIN_TOKEN}" \
--discovery-token-ca-cert-hash "sha256:${JOIN_HASH}" \
--node-name "${NODE_NAME}" \
--ignore-preflight-errors=Hostname 2>&1 || echo "kubeadm join failed, manual join required"
else
echo "Missing CP metadata (cp-ip, join-token, join-hash). Manual join required."
echo "Run on CP: kubeadm token create --print-join-command"
fi
###############################################################################
# Verification
###############################################################################
echo ""
echo "================================================================"
echo " Verification"
echo "================================================================"
echo "--- OS ---"
cat /etc/os-release | head -3
echo "Kernel: $(uname -r)"
if lspci 2>/dev/null | grep -qi 'NVIDIA'; then
echo ""
echo "--- GPU ---"
nvidia-smi -L 2>&1 | head -4
echo "Persistence: $(nvidia-smi -q 2>/dev/null | grep 'Persistence Mode' | head -1 || echo N/A)"
echo "IMEX: $(ls /dev/nvidia-caps-imex-channels/ 2>&1)"
fi
echo ""
echo "--- Network ---"
ip -br link show type ether 2>/dev/null || true
echo "RDMA: $(rdma link show 2>/dev/null | awk '{print $2}' | tr '\n' ', ' || echo N/A)"
echo ""
echo "--- Services ---"
for svc in containerd kubelet; do
echo "$svc: $(systemctl is-active $svc 2>/dev/null || echo inactive)"
done
echo ""
echo "--- Disk ---"
df -h /usr/local /data /mnt/stateful_partition 2>/dev/null | grep -v Filesystem
echo ""
echo "--- SSH ---"
echo "Ports: $(grep '^Port' /etc/ssh/sshd_config | awk '{print $2}' | tr '\n' ', ')"
echo ""
echo "--- k8s ---"
kubeadm version 2>/dev/null || echo "kubeadm not installed"
touch "$INIT_STAMP"
echo ""
echo "================================================================"
echo " Internal initialization complete at $(date)"
echo "================================================================"4.4 GFD/NFD 与 DRA GPU Driver 安装
为了确保 DRA GPU Driver 驱动能够正常在节点上调度,必须先部署 Node Feature Discovery (NFD) 和 GPU Feature Discovery (GFD) 对 GPU 节点进行硬件属性标记。
# 1. 部署 GFD 并自动部署 NFD 依赖
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update
helm upgrade --install gpu-feature-discovery nvdp/gpu-feature-discovery \
--version 0.19.2 -n kube-system --set nfd.deploy=true
# 2. 安装 NVIDIA Device Plugin 用于常规的 nvidia.com/gpu 资源调度与分配
helm upgrade --install nvidia-device-plugin nvdp/nvidia-device-plugin \
--version 0.17.0 -n kube-system
# 等待 GFD/NFD 及 Device Plugin pods Ready 并验证节点 labels
kubectl get pods -n kube-system -l app.kubernetes.io/name=gpu-feature-discovery -w
kubectl get pods -n kube-system -l app.kubernetes.io/name=nvidia-device-plugin -w
kubectl get nodes --show-labels
# 3. 禁用 nvidia-imex.service (ComputeDomain 会管理 IMEX daemon lifecycle)
# 在所有 worker 上执行:
for d in $(seq 1 $NUM_DOMAINS); do
for i in $(seq 0 $((NODES_PER_DOMAIN - 1))); do
gcloud compute ssh ${WORKER_PREFIX}-d${d}-w${i} --zone=$ZONE --tunnel-through-iap \
--command="sudo systemctl disable --now nvidia-imex.service 2>/dev/null; sudo systemctl mask nvidia-imex.service" &
done
done
wait
# 3. 安装 DRA GPU Driver via Helm (CNCF 版本)
helm install dra-driver-nvidia-gpu \
oci://registry.k8s.io/dra-driver-nvidia/charts/dra-driver-nvidia-gpu \
--version v0.4.0 \
--namespace gpu-operator --create-namespace \
--set nvidiaDriverRoot=/ \
--set gpuResourcesEnabledOverride=true
# 等待 kubelet-plugin pods Ready (每个 worker 一个)
kubectl get pods -n gpu-operator -w替代方案:使用 NGC 版本 (v25.12.0)
NVIDIA DRA GPU Driver 有两条发布线,源码相同 (kubernetes-sigs/dra-driver-nvidia-gpu): - CNCF 版本 (v0.4.x):chart
dra-driver-nvidia-gpu,镜像registry.k8s.io/dra-driver-nvidia/dra-driver-nvidia-gpu- NGC 版本 (v25.x.x):chartnvidia-dra-driver-gpu,镜像nvcr.io/nvidia/k8s-dra-driver-gpuv25.12.0 (2025-02) 是 NGC 线最新 GA 版本,GPU allocation 功能正式 GA。
选择建议:CNCF 版本更新更快(v0.4.1 = 2025-06);NGC 版本适合已有 NVIDIA Helm 仓库的环境。
# 添加 NVIDIA Helm 仓库
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
# 安装 NGC 版本 DRA GPU Driver
helm install nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu \
--version="25.12.0" \
--create-namespace \
--namespace nvidia-dra-driver-gpu \
--set nvidiaDriverRoot=/
# 等待 kubelet-plugin pods Ready
kubectl get pods -n nvidia-dra-driver-gpu -w注意事项:
nvidiaDriverRoot=/用于宿主机直装 GPU 驱动(我们的场景);GPU Operator 环境改为/run/nvidia/driver- 两个版本的 chart 名称不同,不要混装
- 升级时 CRD 不会自动更新,需先手动 apply:
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/dra-driver-nvidia-gpu/v25.12.0/deployments/helm/nvidia-dra-driver-gpu/crds/resource.nvidia.com_computedomains.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/dra-driver-nvidia-gpu/v25.12.0/deployments/helm/nvidia-dra-driver-gpu/crds/resource.nvidia.com_computedomaincliques.yaml- NGC 镜像已确认支持 ARM64 (aarch64),在 Grace CPU 上可用
4.5 Scheduler DRA RBAC
关键: kubeadm 1.34.x 的默认 ClusterRole 缺少 DRA RBAC 权限。如果跳过此步骤,ComputeDomain daemon pods 会永久卡在 ContainerCreating 且无报错信息。
cat <<'EOF' | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:kube-scheduler:dra
rules:
- apiGroups: ["resource.k8s.io"]
resources: ["resourceclaims"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["resource.k8s.io"]
resources: ["resourceclaims/status"]
verbs: ["get", "update", "patch"]
- apiGroups: ["resource.k8s.io"]
resources: ["resourceslices"]
verbs: ["get", "list", "watch"]
- apiGroups: ["resource.k8s.io"]
resources: ["deviceclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:kube-scheduler:dra
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-scheduler:dra
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: system:kube-scheduler
EOF
# 重启 Scheduler 使 informer 重新同步
kubectl delete pod -n kube-system -l component=kube-scheduler
kubectl get pods -n kube-system -l component=kube-scheduler -w4.6 ComputeDomain
每个 NVL72 Domain 创建一个 ComputeDomain。
for d in $(seq 1 $NUM_DOMAINS); do
CD_NAME="domain-${d}-cd"
echo "=== Creating ComputeDomain: ${CD_NAME} ==="
cat <<EOF | kubectl apply -f -
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
name: ${CD_NAME}
spec:
numNodes: ${NODES_PER_DOMAIN}
channel:
allocationMode: Single
resourceClaimTemplate:
name: ${CD_NAME}-channel
EOF
# 获取 UID 并标记域内节点
CD_UID=$(kubectl get computedomain ${CD_NAME} -o jsonpath='{.metadata.uid}')
echo "ComputeDomain UID: $CD_UID"
for i in $(seq 0 $((NODES_PER_DOMAIN - 1))); do
kubectl label node ${WORKER_PREFIX}-d${d}-w${i} \
resource.nvidia.com/computeDomain=$CD_UID --overwrite
done
done
# 等待 daemon pods 启动 (~30-60s)
kubectl get pods -n gpu-operator -l app=computedomain-daemon -w
# 验证
kubectl get computedomain -o wide4.7 DRANET + RDMA DeviceClass
# 安装 DRANET
helm install dranet oci://registry.k8s.io/networking/charts/dranet \
--version v1.3.0 -n kube-system
# 创建 RDMA DeviceClass
cat <<'EOF' | kubectl apply -f -
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
name: rdma-devices
spec:
selectors:
- cel:
expression: |
device.driver == "dra.net" &&
has(device.attributes["dra.net"].rdma) && device.attributes["dra.net"].rdma == true
EOF
# 验证 RDMA 设备发现
kubectl get deviceclass
kubectl get resourceslice -o wide | head -204.8 MPI Operator
kubectl apply --server-side -f \
https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.8.0/deploy/v2beta1/mpi-operator.yaml
# 验证
kubectl get pods -n mpi-operator -w4.9 Kueue TAS + JobSet
# 安装 Kueue v0.18+
kubectl apply --server-side -f \
https://github.com/kubernetes-sigs/kueue/releases/download/v0.18.0/manifests.yaml
# 安装 JobSet v0.12+
kubectl apply --server-side -f \
https://github.com/kubernetes-sigs/jobset/releases/download/v0.12.0/manifests.yaml
# 等待就绪
kubectl get pods -n kueue-system -w
kubectl get pods -n jobset-system -w创建 Topology + Queue 资源:
cat <<'EOF' | kubectl apply -f -
apiVersion: kueue.x-k8s.io/v1beta2
kind: Topology
metadata:
name: gb200-nvl72-topology
spec:
levels:
- nodeLabel: "nvl72-domain"
- nodeLabel: "kubernetes.io/hostname"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
name: gb200-flavor
spec:
nodeLabels:
nvl72-domain: domain-1
topologyName: gb200-nvl72-topology
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
name: gpu-cluster-queue
spec:
namespaceSelector: {}
resourceGroups:
- coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
flavors:
- name: gb200-flavor
resources:
- name: "cpu"
nominalQuota: 50000
- name: "memory"
nominalQuota: 200000Gi
- name: "nvidia.com/gpu"
nominalQuota: 1296
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
name: default-queue
namespace: default
spec:
clusterQueue: gpu-cluster-queue
EOF注意: Kueue v0.18 使用
v1beta2API。ResourceFlavor设置topologyName时必须同时提供nodeLabels。ClusterQueue需要namespaceSelector: {}。
4.10 64+8 备用方案
使用 PriorityClass 实现 64 卡训练 + 8 卡备用的弹性容量管理。训练 Pod 抢占低优先级占位 Pod 实现秒级故障切换。
# PriorityClass 定义
cat <<'EOF' | kubectl apply -f -
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: gpu-training
value: 1000
preemptionPolicy: PreemptLowerPriority
globalDefault: false
description: "High priority for GPU training workloads. Preempts placeholder pods."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: gpu-placeholder
value: -1
preemptionPolicy: Never
globalDefault: false
description: "Low priority placeholder to hold spare GPU capacity."
EOF占位 Pod Deployment(按实际备用节点数调整 replicas):
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: gpu-placeholder
labels:
app: gpu-placeholder
spec:
replicas: 2
selector:
matchLabels:
app: gpu-placeholder
template:
metadata:
labels:
app: gpu-placeholder
spec:
priorityClassName: gpu-placeholder
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
- key: kubernetes.io/arch
operator: Equal
value: arm64
effect: NoSchedule
terminationGracePeriodSeconds: 0
containers:
- name: pause
image: registry.k8s.io/pause:3.9
resources:
limits:
nvidia.com/gpu: 4
EOF训练 Pod 使用 gpu-training PriorityClass
即可自动抢占占位 Pod:
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: training-preempt-test
spec:
priorityClassName: gpu-training
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
- key: kubernetes.io/arch
operator: Equal
value: arm64
effect: NoSchedule
restartPolicy: Never
containers:
- name: gpu-test
image: nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04
command: ["bash", "-c"]
args:
- |
echo "=== Training pod started ==="
nvidia-smi
sleep 300
resources:
limits:
nvidia.com/gpu: 4
EOFPart 5: NCCL 测试示例(推荐)
5.1 GIB 使用方式
NCCL-GIB 通过 init container 注入到工作容器中。关键环境变量:
| 环境变量 | 值 | 说明 |
|---|---|---|
LD_LIBRARY_PATH |
/usr/local/gib/lib64:... |
GIB NCCL 库路径 |
NCCL_ENV_PLUGIN |
gcp |
加载 GCP 环境插件, 自动设置 A4X 优化参数 |
NCCL_MNNVL_ENABLE |
2 |
启用 MNNVL (同域) / 0 (跨域) |
NCCL_CUMEM_ENABLE |
1 |
启用 CUDA Memory (同域) / 0 (跨域) |
GIB init container 模式:
initContainers:
- name: gib-installer
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
command: ["/bin/sh", "-c"]
args:
- |
set -e
/scripts/container_entry.sh install --install-nccl
cp -a /usr/local/gib/. /target/gib/
cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
mkdir -p /target/gib/lib64/libibverbs
cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
volumeMounts:
- name: gib
mountPath: /target/gib工作容器中初始化 GIB 环境:
source /usr/local/gib/scripts/set_nccl_env.sh # 设置 NCCL_ENV_PLUGIN=gcp
export LD_LIBRARY_PATH=/usr/local/gib/lib64:$LD_LIBRARY_PATH💡 架构设计说明:关于 GB200 容器内 Mellanox 网络路由配置的必要性
- 为什么本节的同域测试不需要配置静态路由脚本?
- 本文推荐的同域测试(
### 5.2 同域 2 节点 MNNVL NCCL 测试)运行在同一个物理机架的 NVLink Subblock(NVLink Domain)中。- 在该测试中,我们显式声明了
NCCL_MNNVL_ENABLE=2和NCCL_CUMEM_ENABLE=1,这使所有的 GPU 集合通信流量直接走物理 Multi-Node NVLink Switch 互联背板进行传输。- 此时,数据流量完全绕过了 Mellanox ConnectX-7 网卡。NCCL 网络插件初始化时如果扫描到副网卡处于 Down 状态(未配置 IP),会直接在日志中将其跳过而不报错中断。因此,本节测试无需容器内路由脚本即可满速通过。
- 什么时候必须引入容器内路由配置脚本?
- 如果您未来将本自建 K8s 集群扩展至 跨机架/跨域(Cross-Domain) 测试或生产环境,GPU 节点之间的通信就无法通过 NVLink 直连,必须降级走 Mellanox ConnectX-7 的三层 RoCE 网络。
- 在跨域自建场景下,由于开源社区版的
dranet无法自动完成云厂商特定的网络配置,您必须在 Pod 中声明并运行类似于 GB300 部署指南中的setup_rdma_ips.py脚本,从而在容器内动态分配网卡 IPv6 地址、启用网卡、并将明细 Rail/56路由指向 GCP Link-Local 网关fe80::1。- 若非 GKE 托管集群,缺失该步骤将导致跨机架通信错误地流向管理网卡
eth0导致丢包或性能断崖式下跌。
5.2 同域 2 节点 MNNVL NCCL 测试
使用 MPI Operator 运行同域 8 GPU (2 节点 × 4 GPU) all_reduce 测试。
首先创建 ResourceClaimTemplate:
cat <<'EOF' | kubectl apply -f -
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
name: rdma-nics-mpi-sd
spec:
spec:
devices:
requests:
- name: rdma-nics
exactly:
deviceClassName: rdma-devices
count: 4
EOF授权 Default ServiceAccount 读取 Pod 状态的权限 (Launcher 容器解析 Worker IP 所需):
cat <<'EOF' | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-reader
namespace: default
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: pod-reader
namespace: default
subjects:
- kind: ServiceAccount
name: default
namespace: default
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
EOF提交 MPIJob:
cat <<'YAML' | kubectl apply -f -
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
name: nccl-mpi-sd
spec:
slotsPerWorker: 4
runPolicy:
ttlSecondsAfterFinished: 600
cleanPodPolicy: Running
sshAuthMountPath: /root/.ssh
launcherCreationPolicy: WaitForWorkersReady
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
nodeSelector:
nvl72-domain: domain-1
initContainers:
- name: gib-installer
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
command: ["/bin/sh", "-c"]
args:
- |
set -e
/scripts/container_entry.sh install --install-nccl
cp -a /usr/local/gib/. /target/gib/
cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
mkdir -p /target/gib/lib64/libibverbs
cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
volumeMounts:
- name: gib
mountPath: /target/gib
containers:
- name: launcher
image: nvcr.io/nvidia/pytorch:26.05-py3
imagePullPolicy: IfNotPresent
env:
- name: LD_LIBRARY_PATH
value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
command:
- bash
- -c
- |
set -e
echo "Resolving worker IPs via k8s API..."
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
API="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"
for attempt in $(seq 1 60); do
W0_IP=$(curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" \
"$API/api/v1/namespaces/default/pods/nccl-mpi-sd-worker-0" 2>/dev/null \
| python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",{}).get("podIP",""))' 2>/dev/null || echo "")
W1_IP=$(curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" \
"$API/api/v1/namespaces/default/pods/nccl-mpi-sd-worker-1" 2>/dev/null \
| python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",{}).get("podIP",""))' 2>/dev/null || echo "")
if [ -n "$W0_IP" ] && [ -n "$W1_IP" ]; then
echo "Worker-0 IP: $W0_IP"
echo "Worker-1 IP: $W1_IP"
break
fi
echo "Waiting for worker IPs (attempt $attempt)..."
sleep 5
done
if [ -z "$W0_IP" ] || [ -z "$W1_IP" ]; then
echo "ERROR: Could not resolve worker IPs"
exit 1
fi
echo "$W0_IP nccl-mpi-sd-worker-0.nccl-mpi-sd.default.svc nccl-mpi-sd-worker-0" >> /etc/hosts
echo "$W1_IP nccl-mpi-sd-worker-1.nccl-mpi-sd.default.svc nccl-mpi-sd-worker-1" >> /etc/hosts
echo "Waiting 120s for workers to compile nccl-tests..."
sleep 120
echo "Starting mpirun..."
/usr/local/mpi/bin/mpirun --allow-run-as-root \
-np 8 -npernode 4 \
-x LD_LIBRARY_PATH=/usr/local/gib/lib64:/usr/local/nvidia/lib64 \
-x NCCL_NET_PLUGIN=/usr/local/gib/lib64/libnccl-net.so \
-x NCCL_ENV_PLUGIN=gcp \
-x NCCL_MNNVL_ENABLE=2 \
-x NCCL_CUMEM_ENABLE=1 \
--mca plm_rsh_args '-o StrictHostKeyChecking=no' \
/tmp/nccl-tests/build/all_reduce_perf -b 1M -e 16G -f 2 -g 1 -w 50 -n 100
volumeMounts:
- name: gib
mountPath: /usr/local/gib
resources:
requests:
cpu: "4"
memory: 16Gi
limits:
cpu: "4"
memory: 16Gi
volumes:
- name: gib
emptyDir: {}
Worker:
replicas: 2
template:
spec:
nodeSelector:
nvl72-domain: domain-1
dnsPolicy: None
dnsConfig:
nameservers:
- "10.96.0.10"
- "8.8.8.8"
searches:
- "default.svc.cluster.local"
- "svc.cluster.local"
- "cluster.local"
resourceClaims:
- name: compute-domain-channel
resourceClaimTemplateName: domain-1-cd-channel
- name: rdma-nics
resourceClaimTemplateName: rdma-nics-mpi-sd
initContainers:
- name: gib-installer
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
command: ["/bin/sh", "-c"]
args:
- |
set -e
/scripts/container_entry.sh install --install-nccl
cp -a /usr/local/gib/. /target/gib/
cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
mkdir -p /target/gib/lib64/libibverbs
cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
volumeMounts:
- name: gib
mountPath: /target/gib
containers:
- name: worker
image: nvcr.io/nvidia/pytorch:26.05-py3
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
env:
- name: LD_LIBRARY_PATH
value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
- name: NCCL_NET_PLUGIN
value: /usr/local/gib/lib64/libnccl-net.so
- name: NCCL_ENV_PLUGIN
value: gcp
command:
- bash
- -c
- |
apt-get update -qq && apt-get install -y -qq openssh-server > /dev/null 2>&1
mkdir -p /run/sshd
printf 'Port 22\nPermitRootLogin yes\nPubkeyAuthentication yes\nAuthorizedKeysFile /root/.ssh/authorized_keys\nStrictModes no\nPasswordAuthentication no\nUsePAM no\nPidFile /run/sshd/sshd.pid\n' > /etc/ssh/mpi_sshd_config
ssh-keygen -A
for i in $(seq 1 30); do
cd /tmp && rm -rf nccl-tests && git clone --depth 1 https://github.com/NVIDIA/nccl-tests.git && break
echo "git clone attempt $i failed, retrying in 10s..."
sleep 10
done
cd /tmp/nccl-tests && make -j$(nproc) MPI=1 \
MPI_HOME=/usr/local/mpi \
NCCL_HOME=/usr/local/gib \
CUDA_HOME=/usr/local/cuda
/usr/sbin/sshd -De -f /etc/ssh/mpi_sshd_config
volumeMounts:
- name: gib
mountPath: /usr/local/gib
- name: dshm
mountPath: /dev/shm
resources:
requests:
cpu: "96"
memory: 800Gi
limits:
cpu: "96"
memory: 800Gi
nvidia.com/gpu: 4
claims:
- name: compute-domain-channel
- name: rdma-nics
volumes:
- name: gib
emptyDir: {}
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 250Gi
YAML说明: -
resourceClaimTemplateName: domain-1-cd-channel是 ComputeDomain 自动创建的 ResourceClaimTemplate,名称格式为${CD_NAME}-channel- Worker 编译 nccl-tests 需要 MPI 支持(MPI=1),GIB 镜像自带的 nccl-tests 未链接 MPI - 同域测试设置NCCL_MNNVL_ENABLE=2和NCCL_CUMEM_ENABLE=1-NCCL_ENV_PLUGIN=gcp自动加载 A4X 优化参数(PXN_C2C、IB_ADAPTIVE_ROUTING 等)
5.3 查看测试结果
# 等待 Launcher 完成
kubectl get pods -l training.kubeflow.org/job-name=nccl-mpi-sd -w
# 查看结果
kubectl logs nccl-mpi-sd-launcher
# 预期输出 (同域 MNNVL, 8 GPU):
# all_reduce_perf @8G busbw ≈ 835 GB/s5.4 使用 NCCL_DEBUG 验证 GIB 初始化
添加 -x NCCL_DEBUG=INFO 到 mpirun
参数中,输出应包含:
NCCL INFO Successfully loaded external env plugin /usr/local/gib/lib64/libnccl-env-gcp.so
NCCL INFO NCCL version 2.30.4+cuda13.0
NCCL INFO NET/gIB : Initializing gIB v1.1.2
NCCL INFO NET/gIB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE
NCCL INFO Using network gIB
NCCL INFO comm ... MNNVL 1
实例调度与生命周期限制说明 (Important Restrictions)
Visibility: Confidential
本文档旨在澄清在 Google Cloud 平台上部署 A4X (GB200) 与 A4X-Max (GB300) 实例时,关于放置策略(Placement Policy)以及实例生命周期(删除与重建)的预期产品行为与官方限制的差异。
1. 放置策略(Placement Policy)与 Sub-block 锁定限制
适用范围: GB200 (A4X) 虚拟机与 GB300 (A4X-Max) 裸金属均受此限制约束。
现象描述 在尝试指定 Placement Policy(如 1x72
拓扑)创建 VM 或 GKE 节点池时,若对应的 sub-block(NVLink
Domain)中已经存在单台独立创建的 VM,系统会拒绝调度或发生映射失败,提示
PLACEMENT_POLICY_CONSTRAINTS_INFEASIBLE。删除重建 Placement
Policy 似乎能“绕过”此问题。
根本原因 这不是系统后端的缓存 Bug,而是 GCP 为了确保 NVLink 物理拓扑完整性而设计的强制锁定机制。因为 GB200 和 GB300 在底层均依赖 1x72 NVLink 拓扑,一个 NVLink Domain(sub-block)必须且只能被唯一的放置策略独占。一旦 Domain 中被创建了没有该策略的单机 VM,硬件即被视为“物理碎片化”,系统将拒绝任何需要独占整个 Domain 的 1x72 策略下发。
官方依据 根据 GCP 公开官方文档 Create an AI-optimized instance with A4X 中的重要警告说明: > Important: “When you create an A4X instance with a 1x72 topology, Compute Engine allocates an entire 18-instance sub-block (NVLink domain) to the placement policy used in the creation.”(当您使用 1x72 拓扑创建实例时,Compute Engine 会将整个包含 18 个实例的 sub-block 完全分配给创建时所使用的那个 Placement Policy。) > “If you create fewer than 18 instances, the unused instances in that sub-block remain idle and are unavailable for other requests that use a different placement policy.”(如果您创建了少于 18 个实例,该 sub-block 中未使用的实例将保持空闲状态,并且无法被使用了不同放置策略的其他请求所使用。)
最佳实践建议
为了最大化资源利用率并避免锁冲突,官方建议:强烈推荐一次性创建完整的
18 个实例。建议使用 instances.bulkInsert API 或
gcloud compute instances bulk create 命令,将 18 台 VM
作为一个整体一次性下发到空闲的 Domain 中,绝不要在同一个 Domain
内混搭创建独立 VM 和集群 VM。
2. 裸金属实例删除后的 1 小时“硬件锁定” (Transient Degraded Status)
适用范围: 仅限于 GB300 (A4X-Max) 裸金属 (Bare Metal) 实例。GB200 (A4X) 虚拟机 (VM) 由于只需回收虚拟化资源,不受此深度物理重置的限制。
现象描述 当删除一个 GB300 裸金属机器的 Domain (Subblock) 以释放配额时,GCP UI 会在长达约 1 小时的时间内将该 sub-block 标记为 “degraded hosts”,在此期间资源被锁定,无法将这些机器重新分配用于新的实例创建。
根本原因 这属于预期的产品核心安全行为,并非硬件故障或 UI 延迟。与 GB200 等普通虚拟机不同,GB300 裸金属 (Bare Metal) 实例在被删除后,必须经历一个称为 Attested Power Cycle(已验证的电源循环) 的深度安全重置流程。 为了保障多租户环境下的物理安全,系统必须移除前一个客户的任何潜在数据或状态。此过程包括 NVMe 磁盘的深度擦除、固件刷新、物理断电重启以及安全启动项的验证。该“消毒”过程需要花费大约 1 小时才能完成并自我恢复。在收到正确的密码学验证证明(Cryptographic proof)之前,平台会锁定硬件,禁止下一次分配。
最佳实践建议 对于 GB300 裸金属资源:在进行容量测试或日常运维编排时,请将这 1 小时的安全重置时间纳入工作流预期中。避免设计依赖于“秒级销毁并立即在原物理机上重建”的业务逻辑,这在 GCP Bare Metal 架构中是不被支持的。
附录
附录 A: RDMA 设备与 IB 设备改名
A.1 GCP A4X RDMA 设备概述
A4X 的 MRDMA 网卡是 SR-IOV VF,只暴露 RDMA 能力,没有 Linux 网络接口:
# IB 设备存在且状态正常
$ rdma link show
link mlx5_0/1 state ACTIVE physical_state LINK_UP
link mlx5_1/1 state ACTIVE physical_state LINK_UP
link mlx5_2/1 state ACTIVE physical_state LINK_UP
link mlx5_3/1 state ACTIVE physical_state LINK_UP
# 没有对应的 Linux 网络接口
$ ibdev2netdev
(空)
$ ls /sys/class/infiniband/mlx5_0/device/net/
(空)
# VM 上只有 2 个 GVNIC 网络接口 (startup script 改名后显示为 bond0/bond1)
$ ip -br link show
lo UNKNOWN ...
bond0 UP ... ← 主 GVNIC (原 enp0s3, MTU 8896)
bond1 UP ... ← 辅 GVNIC (原 enp192s2, MTU 8896)A.2 网卡改名
Startup script 会将所有 6 个网络接口统一改名为 bond0-bond5:
| 原名 | 新名 | 类型 | 方式 |
|---|---|---|---|
| enp0s3 | bond0 | GVNIC | ip link set name |
| enp192s2 | bond1 | GVNIC | ip link set name |
| mlx5_0 | bond2 | RDMA IB | rdma dev set |
| mlx5_1 | bond3 | RDMA IB | rdma dev set |
| mlx5_2 | bond4 | RDMA IB | rdma dev set |
| mlx5_3 | bond5 | RDMA IB | rdma dev set |
手动改名(如需临时测试):
# GVNIC
ip link set dev enp0s3 down
ip link set dev enp0s3 name bond0
ip link set dev bond0 up
# RDMA IB
rdma dev set mlx5_0 name bond2
rdma dev set mlx5_1 name bond3
rdma dev set mlx5_2 name bond4
rdma dev set mlx5_3 name bond5验证:
$ rdma link show
link bond2/1 state ACTIVE physical_state LINK_UP
link bond3/1 state ACTIVE physical_state LINK_UP
link bond4/1 state ACTIVE physical_state LINK_UP
link bond5/1 state ACTIVE physical_state LINK_UP
$ ls /sys/class/infiniband/
bond2 bond3 bond4 bond5A.3 持久化配置
Startup script 通过以下机制实现持久化: - GVNIC:
首次运行时创建 udev 规则
/etc/udev/rules.d/70-a4x-bond-rename.rules,后续重启由 udev
直接命名为 bond0/bond1 - RDMA IB:
rdma dev set 仅运行时生效,startup script 在 stamp check
之前执行,确保每次重启都会改名
说明: v5.3 及以上版本的镜像已将此步骤集成到 startup script 中,无需手动配置。
A.4 NCCL 验证
已实测:改名后在 2 节点 8 GPU 环境运行
all_reduce_perf(1M-16G),性能正常,0 errors。NCCL/GIB
通过 IB verbs API 访问 RDMA 设备,不依赖 IB 设备名称。k8s DRANET 通过
PCI 地址发现设备,同样不受改名影响。
附录 B: 预构建镜像软件版本信息
以下为 GCP 预构建的 TencentOS Server 4.0 ARM64 GCE 镜像中各核心组件的版本信息,供 GCP Support 评估环境兼容性和制定支持方案时参考。
B.1 基础环境
| 组件 | 版本 | 说明 |
|---|---|---|
| 操作系统 | TencentOS Server 4.0 (Build 4.6) | ARM64 (aarch64) |
| Linux 内核 | 6.6.119-49.101.nvidia.tl4.aarch64 | 客户定制 NVIDIA 内核 |
| 架构 | ARM64 / aarch64 | GB200 NVL72 平台 |
B.2 NVIDIA 软件栈
| 组件 | 版本 | 说明 |
|---|---|---|
| NVIDIA 驱动 | 580.159.04 | Open 类型 (–kernel-module-type=open), R580 分支, Dual MIT/GPL |
| CUDA | 13.0 | 通过 cuda-cudart-devel-13-0 安装 |
| nvidia-modprobe | 580.159.04 | IMEX 依赖 |
| IMEX Daemon | 580.159.04 | 跟随驱动版本, NVreg_CreateImexChannel0=1 |
| nvidia-persistenced | 580.159.04 | GPU-conditional systemd service, v5.3 新增 |
B.3 网络驱动
| 组件 | 版本 | 说明 |
|---|---|---|
| GVE (GVNIC) 驱动 | 1.4.10-oot | Out-of-tree 编译, 源码来自 Google compute-virtual-ethernet-linux |
| IDPF (Intel NIC) 驱动 | v1.0.11 | Out-of-tree 编译, 仅 GB300 需要, GB200 不安装 |
B.4 GPU 辅助组件
| 组件 | 版本 | 说明 |
|---|---|---|
| GDRCopy | v2.6 | GPU-conditional systemd service, 仅 GPU 节点启动 |
| Lustre Client | DDN 2.14 | DKMS 方式安装, 不自动挂载 |
B.5 GCE 集成
| 组件 | 说明 |
|---|---|
| google-compute-engine | GCE 集成包 (系统配置) |
| google-guest-agent | GCE Guest Agent (元数据/账号管理) |
| google-osconfig-agent | OS Config Agent (补丁管理) |
| boost-regex | 1.75 (从 Rocky 9 注入, GCE Guest Agent 依赖) |
| libicu | 67.1 (从 Rocky 9 注入, GCE Guest Agent 依赖) |
说明:镜像构建详细步骤请参考 TencentOS Server 4.0 ARM64 GCE Image Build Guide (V2)。
附录 C: 跨项目复制镜像
当镜像所在项目受到组织策略限制无法直接共享时,可通过以下步骤将镜像复制到目标项目。此操作无需 GPU VM,也无需启动任何虚拟机。
# 变量设置
SOURCE_PROJECT="source-project-id" # 镜像来源项目
TARGET_PROJECT="your-project-id" # 目标项目
ZONE="us-central1-f" # 临时磁盘所在区域
IMAGE_NAME="tlinux-server-4-gb200-v5dot3" # 镜像名称
# 1. 从来源镜像创建临时磁盘
gcloud compute disks create temp-tlinux-copy \
--project=$TARGET_PROJECT \
--zone=$ZONE \
--image=$IMAGE_NAME \
--image-project=$SOURCE_PROJECT \
--size=1000GB --type=pd-ssd
# 2. 从磁盘创建本地镜像
gcloud compute images create $IMAGE_NAME \
--project=$TARGET_PROJECT \
--source-disk=temp-tlinux-copy \
--source-disk-zone=$ZONE \
--guest-os-features="UEFI_COMPATIBLE,GVNIC,IDPF" \
--architecture="ARM64" \
--family="tlinux-server-4-gb200"
# 3. 删除临时磁盘
gcloud compute disks delete temp-tlinux-copy \
--project=$TARGET_PROJECT \
--zone=$ZONE --quiet
# 4. 验证
gcloud compute images describe $IMAGE_NAME \
--project=$TARGET_PROJECT \
--format="table(name,status,archiveSizeBytes,guestOsFeatures[].type)"注意: 镜像根盘为 1TB,临时磁盘
--size不能小于 1000GB。复制完成后务必删除临时磁盘以避免不必要的存储费用。
常见问题与故障排除 (Troubleshooting)
问题:mpi-operator 反复崩溃或 apiserver
响应超时 如果集群变得不稳定,或者 mpi-operator
因为租约超时(lease timeouts)反复重启,这通常是由于安装了冗余的 NFD
(Node Feature Discovery) 插件引起的(例如 gpu-operator
默认自带的 NFD 与集群已有的 NFD 冲突)。冲突会导致数百个 worker pod 陷入
CrashLoopBackOff,并向 apiserver 发送大量垃圾请求,导致 API Server
瘫痪。 解决方案:如果您使用了完整的
nvidia/gpu-operator Helm chart 并且集群中已存在其他 NFD
组件: > ⚠️ 警告:
本指南采用的是模块化部署(分离安装了 GFD 和 Device
Plugin),并未直接安装完整的 gpu-operator。若您自行使用了
nvidia/gpu-operator,请务必设置
--set nfd.enabled=false 以避免 NFD 导致的 Crash Loop
冲突。