GB200 A4X 部署指南

适用于 GCP A4X (GB200/GB300 NVL72) 自建 Kubernetes 环境。支持多集群、多区域部署。


全局变量

修改以下变量后,后续所有命令可直接复制粘贴执行。本指南支持在不同区域部署不同规模的集群,请根据当前实际交付的集群(GB200 或 GB300)取消对应区块的注释,并注释掉另一个区块。

# ======================== 项目 ========================
export PROJECT="tencent-gcp-taiji"

# ======================== 区域、规模与机器配置 ========================

# 【选项 A】: GB200 集群 (1296 GPU = 324 VM, us-east1)
export REGION="us-east1"
export ZONE="us-east1-d"
export TOTAL_GPUS=1296
export MACHINE_TYPE="a4x-highgpu-4g"
export CLUSTER_PREFIX="gb200-east"

# GVNIC 网段 (假设客户主 VPC 为 taiji-net-0)
export GVNIC_NET="taiji-net-0"
export GVNIC_SUB="${CLUSTER_PREFIX}-gvnic-sub"
export GVNIC_CIDR="10.0.0.0/22"            # /22 提供 1020 个可用 IP

export GVNIC_NET_1="${CLUSTER_PREFIX}-gvnic-net-1"
export GVNIC_SUB_1="${CLUSTER_PREFIX}-gvnic-sub-1"
export GVNIC_1_CIDR="10.0.4.0/22"          # /22 提供 1020 个可用 IP

# RDMA 网段 (独立的区域性 RoCE VPC)
export RDMA_NET="${CLUSTER_PREFIX}-rdma-net"
export RDMA_CIDRS=("10.0.16.0/22" "10.0.20.0/22" "10.0.24.0/22" "10.0.28.0/22")

# ======================== 集群 B: GB300 (us-central1) ========================
# 【选项 B】: GB300 集群 (1800 GPU = 450 VM, us-central1)
# export REGION="us-central1"
# export ZONE="us-central1-b"
# export TOTAL_GPUS=1800
# export MACHINE_TYPE="a4x-maxgpu-4g-metal"  # GB300 裸金属机型
# export CLUSTER_PREFIX="gb300-central"

# GVNIC 网段 (共享同一个客户主 VPC taiji-net-0,使用不重叠的 CIDR)
# export GVNIC_NET="taiji-net-0"
# export GVNIC_SUB="${CLUSTER_PREFIX}-gvnic-sub"
# export GVNIC_CIDR="10.0.32.0/22"         # /22 提供 1020 个可用 IP

# export GVNIC_NET_1="${CLUSTER_PREFIX}-gvnic-net-1"
# export GVNIC_SUB_1="${CLUSTER_PREFIX}-gvnic-sub-1"
# export GVNIC_1_CIDR="10.0.36.0/22"       # /22 提供 1020 个可用 IP

# RDMA 网段 (独立的区域性 RoCE VPC,使用不重叠的名称和 CIDR)
# export RDMA_NET="${CLUSTER_PREFIX}-rdma-net"
# export RDMA_CIDRS=("10.0.48.0/22" "10.0.52.0/22" "10.0.56.0/22" "10.0.60.0/22")

# ======================== 镜像与预留 ========================
export IMAGE="your-tlinux4-gb200-image"     # 客户提供的 OS 镜像名 (GB300 裸金属需确保为 ARM64 构建)
export IMAGE_PROJECT="your-image-project"   # 镜像所在项目
export RESERVATION="your-reservation-name"  # 预留实例名称

# ======================== 拓扑计算 ========================
export GPUS_PER_NODE=4
export NODES_PER_DOMAIN=18                  # NVL72 = 18 节点/域
export NUM_DOMAINS=$((TOTAL_GPUS / GPUS_PER_NODE / NODES_PER_DOMAIN))

export CP_NAME="${CLUSTER_PREFIX}-cp"
export WORKER_PREFIX="${CLUSTER_PREFIX}"     # Worker: ${WORKER_PREFIX}-d${DOMAIN}-w${INDEX}
export PLACEMENT_PREFIX="${CLUSTER_PREFIX}-nvl72-policy"
export SUPERNET="10.0.0.0/16"              # 防火墙规则用

# ======================== Kubernetes ========================
export K8S_VERSION="1.34"
export K8S_PATCH="1.34.9"                  # >= 1.34.5 (修复 kubelet DRA gRPC 死锁)
export POD_CIDR="10.244.0.0/16"
export CALICO_VERSION="3.29.3"

Part 1: 网络基础设施

1.1 创建 VPC

注意:两个集群如果共享主管理网络,第二次部署时已有的 GVNIC VPC 可能会提示已存在,此为正常现象,脚本中的 || true 会忽略该报错并继续执行。RDMA 网络在不同区域内是完全隔离的,每个集群均独立创建。

# 主 GVNIC 网络 — MTU 8896 (Jumbo Frames, 必须)
gcloud compute networks create $GVNIC_NET \
  --subnet-mode=custom --mtu=8896 --project=$PROJECT || true

# 辅 GVNIC 网络
gcloud compute networks create $GVNIC_NET_1 \
  --subnet-mode=custom --mtu=8896 --project=$PROJECT || true

# RDMA HPC 网络 (必须基于当前 Zone 的网络配置文件)
gcloud beta compute networks create $RDMA_NET \
  --network-profile=${ZONE}-vpc-roce \
  --subnet-mode=custom --project=$PROJECT

1.2 创建子网

# 主 GVNIC 子网
gcloud compute networks subnets create $GVNIC_SUB \
  --network=$GVNIC_NET --region=$REGION \
  --range=$GVNIC_CIDR --project=$PROJECT

# 辅 GVNIC 子网
gcloud compute networks subnets create $GVNIC_SUB_1 \
  --network=$GVNIC_NET_1 --region=$REGION \
  --range=$GVNIC_1_CIDR --project=$PROJECT

# RDMA 子网 (采用 Rail-optimized 架构,每张网卡接入一个独立的子网)
for n in 0 1 2 3; do
  gcloud compute networks subnets create ${RDMA_NET}-sub-${n} \
    --network=$RDMA_NET --region=$REGION \
    --range="${RDMA_CIDRS[$n]}" --project=$PROJECT
done

1.3 创建防火墙规则

注意: 为了防止不同集群在共享 VPC 时防火墙规则同名报错,以下规则名称中已加入 ${CLUSTER_PREFIX} 前缀。

# IAP SSH (用于 gcloud compute ssh --tunnel-through-iap)
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-iap-ssh \
  --project=$PROJECT --network=$GVNIC_NET \
  --direction=INGRESS --priority=900 --action=ALLOW \
  --rules=tcp:22,tcp:56000 --source-ranges=35.235.240.0/20

# 主 GVNIC 内部通信 (节点间 + Pod CIDR)
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-internal-gvnic \
  --project=$PROJECT --network=$GVNIC_NET \
  --direction=INGRESS --priority=1000 --action=ALLOW \
  --rules=tcp,udp,icmp --source-ranges="${GVNIC_CIDR},${POD_CIDR}"

# 辅 GVNIC 内部通信
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-internal-gvnic1 \
  --project=$PROJECT --network=$GVNIC_NET_1 \
  --direction=INGRESS --priority=1000 --action=ALLOW \
  --rules=tcp,udp,icmp --source-ranges="${GVNIC_1_CIDR}"

注意: 如果组织策略会自动删除防火墙规则,需在每次 SSH 前检查规则是否存在。RDMA 网络的防火墙由 Network Profile 管理,无需手动创建。

1.4 创建 Placement Policy

每个 NVL72 Domain 需要一个独立的 Placement Policy。

for d in $(seq 1 $NUM_DOMAINS); do
  gcloud beta compute resource-policies create group-placement \
    ${PLACEMENT_PREFIX}-${d} \
    --collocation=collocated \
    --gpu-topology=1x72 \
    --project=$PROJECT --region=$REGION
done

# 验证
gcloud compute resource-policies list --project=$PROJECT --filter="name~${PLACEMENT_PREFIX}"

Part 2: VM 创建

2.1 Startup Script

将以下脚本保存为 tlinux4-customer-init.sh。v5.3 镜像已内置 NVIDIA driver、GVE、IMEX、GDRCopy、nvidia-persistenced 等所有 GPU 相关组件,startup script 仅负责: 1. 文件系统初始化(Boot disk 分区 + Local SSD RAID0) 2. sshd port 22 + 56000 配置 3. RDMA IB 设备改名(mlx5_X → bondY)

注意: - v5.3 镜像不再需要重启。脚本一次运行即可完成所有配置。 - TLinux 4 基础镜像的 sshd 默认端口为 36000(Port 22 被注释掉),startup script 会显式启用 Port 22 和 Port 56000。

#!/bin/bash
# TLinux 4 (TencentOS Server 4.0) — v5.3 Image Startup Script
# Compatible with: A4X/GB200, GB300, H100, CPU-only instances
#
# v5.3 镜像已内置: NVIDIA driver, GVE, IMEX, GDRCopy, nvidia-persistenced,
#                   SELinux/firewalld disabled, memlock unlimited, sudoers, SSH
#
# 本脚本仅负责: 文件系统初始化 + sshd 22/56000 端口 + 网卡改名 (bond0-bond5)
set -euo pipefail
exec > >(tee /var/log/tlinux4-customer-init.log) 2>&1

INIT_STAMP=/etc/.customer-init-done

###############################################################################
# Network interface rename (A4X only: bond0-bond5) — runs every boot
# GVNIC: enp0s3→bond0, enp192s2→bond1 (persisted via udev after first boot)
# RDMA: mlx5_0→bond2, mlx5_1→bond3, mlx5_2→bond4, mlx5_3→bond5
###############################################################################
if rdma link show 2>/dev/null | grep -q .; then
  echo "=== Network interface rename (A4X) ==="

  # GVNIC rename (Linux netdev: ip link set name)
  for pair in "enp0s3:bond0" "enp192s2:bond1"; do
    OLD="${pair%%:*}"
    NEW="${pair##*:}"
    if ip link show "$OLD" &>/dev/null; then
      ip link set dev "$OLD" down
      ip link set dev "$OLD" name "$NEW"
      ip link set dev "$NEW" up
      echo "  $OLD → $NEW"
    elif ip link show "$NEW" &>/dev/null; then
      echo "  $NEW already exists (skip)"
    fi
  done

  # Update NM connection profile to match new name
  NM_CONN="/etc/NetworkManager/system-connections/enp0s3-gvnic.nmconnection"
  if [ -f "$NM_CONN" ] && grep -q "interface-name=enp0s3" "$NM_CONN"; then
    sed -i 's/interface-name=enp0s3/interface-name=bond0/' "$NM_CONN"
    sed -i 's/id=enp0s3-gvnic/id=bond0-gvnic/' "$NM_CONN"
    nmcli connection reload 2>/dev/null || true
    echo "  Updated NM profile: enp0s3-gvnic → bond0-gvnic"
  fi

  # Create udev rule for persistent GVNIC naming across reboots (one-time)
  UDEV_RULE="/etc/udev/rules.d/70-a4x-bond-rename.rules"
  if [ ! -f "$UDEV_RULE" ]; then
    PCI0=$(basename "$(readlink -f /sys/class/net/bond0/device)")
    PCI1=$(basename "$(readlink -f /sys/class/net/bond1/device)")
    cat > "$UDEV_RULE" << UDEVEOF
SUBSYSTEM=="net", ACTION=="add", KERNELS=="$PCI0", NAME="bond0"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="$PCI1", NAME="bond1"
UDEVEOF
    echo "  Created udev rule for persistent GVNIC naming"
  fi

  # RDMA IB device rename (rdma dev set — runtime only, needs every boot)
  for i in 0 1 2 3; do
    if rdma dev set mlx5_${i} name bond$((i+2)) 2>/dev/null; then
      echo "  mlx5_${i} → bond$((i+2))"
    fi
  done

  echo "  --- Result ---"
  ip -br link show type ether 2>/dev/null || true
  rdma link show 2>/dev/null || true
fi

if [ -f "$INIT_STAMP" ]; then
  echo "Customer initialization already completed. Skipping."
  exit 0
fi

echo "================================================================"
echo "  TLinux 4 v5.3 — Startup Script"
echo "  $(date)"
echo "================================================================"

###############################################################################
# [1] Boot disk partitioning: /usr/local (20G) + /data (remaining)
###############################################################################
echo "=== [1] Boot disk partitioning ==="

BOOT_DEV=$(findmnt -n -o SOURCE / | sed 's/p[0-9]*$//')
PARTITION_STAMP=/etc/.partitions-created

if [ ! -f "$PARTITION_STAMP" ]; then
  echo "--- Creating boot disk partitions (1TB disk) ---"

  # Expand GPT to full disk (image GPT backup may be at original image size)
  sgdisk -e "$BOOT_DEV" 2>/dev/null || true

  sgdisk -n 3:0:+20G -t 3:8300 "$BOOT_DEV"
  sgdisk -n 4:0:0 -t 4:8300 "$BOOT_DEV"
  partprobe "$BOOT_DEV" 2>/dev/null || true
  sleep 2

  mkfs.xfs -f "${BOOT_DEV}p3"
  mkfs.xfs -f "${BOOT_DEV}p4"

  TMPMNT=$(mktemp -d)
  mount "${BOOT_DEV}p3" "$TMPMNT"
  cp -a /usr/local/. "$TMPMNT/" 2>/dev/null || true
  umount "$TMPMNT"
  rmdir "$TMPMNT"

  mount "${BOOT_DEV}p3" /usr/local
  mkdir -p /data
  mount "${BOOT_DEV}p4" /data

  P3_UUID=$(blkid -s UUID -o value "${BOOT_DEV}p3")
  P4_UUID=$(blkid -s UUID -o value "${BOOT_DEV}p4")
  if [ -n "$P3_UUID" ] && ! grep -q "$P3_UUID" /etc/fstab; then
    echo "UUID=$P3_UUID /usr/local xfs defaults,nofail 0 2" >> /etc/fstab
  fi
  if [ -n "$P4_UUID" ] && ! grep -q "$P4_UUID" /etc/fstab; then
    echo "UUID=$P4_UUID /data xfs defaults,nofail 0 2" >> /etc/fstab
  fi

  touch "$PARTITION_STAMP"
  echo "Boot disk partitioned: p3=/usr/local(20G) p4=/data(remaining)"
fi

###############################################################################
# [2] Local SSD RAID0 → /mnt/stateful_partition
###############################################################################
echo "=== [2] Local SSD RAID0 setup ==="

SSD_DEVICES=()
for disk in /dev/nvme*n1; do
  [ -b "$disk" ] || continue
  PART_COUNT=$(lsblk -n -o NAME "$disk" 2>/dev/null | wc -l)
  if [ "$PART_COUNT" -le 1 ]; then
    SSD_DEVICES+=("$disk")
    echo "  Detected local SSD: $disk ($(lsblk -n -d -o SIZE "$disk"))"
  fi
done

NUM_SSDS=${#SSD_DEVICES[@]}
echo "  Found $NUM_SSDS local SSD device(s)"

if [ "$NUM_SSDS" -gt 0 ]; then
  MOUNT_POINT="/mnt/stateful_partition"
  mkdir -p "$MOUNT_POINT"

  if [ "$NUM_SSDS" -eq 1 ]; then
    TARGET_DEV="${SSD_DEVICES[0]}"
    mkfs.xfs -f "$TARGET_DEV"
    mount "$TARGET_DEV" "$MOUNT_POINT"
    DEV_UUID=$(blkid -s UUID -o value "$TARGET_DEV")
    if [ -n "$DEV_UUID" ] && ! grep -q "$DEV_UUID" /etc/fstab; then
      echo "UUID=$DEV_UUID $MOUNT_POINT xfs defaults,nofail 0 2" >> /etc/fstab
    fi
  else
    mdadm --stop /dev/md0 2>/dev/null || true
    mdadm --create /dev/md0 --level=0 --raid-devices="$NUM_SSDS" "${SSD_DEVICES[@]}" --force --run
    mkfs.xfs -f /dev/md0
    mount /dev/md0 "$MOUNT_POINT"
    mkdir -p /etc/mdadm
    mdadm --detail --scan >> /etc/mdadm/mdadm.conf
    DEV_UUID=$(blkid -s UUID -o value /dev/md0)
    if [ -n "$DEV_UUID" ] && ! grep -q "$DEV_UUID" /etc/fstab; then
      echo "UUID=$DEV_UUID $MOUNT_POINT xfs defaults,nofail 0 2" >> /etc/fstab
    fi
  fi

  mkdir -p "$MOUNT_POINT"/{gcsfuse-cache,scratch-data}
  echo "$MOUNT_POINT mounted (total: $(df -h "$MOUNT_POINT" | awk 'NR==2{print $2}'))"
else
  echo "WARNING: No local SSD devices found"
  mkdir -p /mnt/stateful_partition
fi

###############################################################################
# [3] sshd port 56000
###############################################################################
echo "=== [3] sshd port 22 + 56000 ==="

# TLinux 4 base image uses Port 36000 with Port 22 commented out.
# Ensure both Port 22 and Port 56000 are explicitly configured.
SSHD_CHANGED=0
if ! grep -q '^Port 22' /etc/ssh/sshd_config; then
  sed -i '1i Port 22' /etc/ssh/sshd_config
  SSHD_CHANGED=1
fi
if ! grep -q '^Port 56000' /etc/ssh/sshd_config; then
  sed -i '/^Port 22/a Port 56000' /etc/ssh/sshd_config
  SSHD_CHANGED=1
fi
if [ "$SSHD_CHANGED" -eq 1 ]; then
  systemctl restart sshd
  echo "sshd now listening on port 22 and 56000"
else
  echo "sshd ports 22 and 56000 already configured"
fi

###############################################################################
# Verification
###############################################################################
echo ""
echo "================================================================"
echo "  Verification"
echo "================================================================"

echo "--- OS ---"
cat /etc/os-release | head -3
echo "Kernel: $(uname -r)"

if lspci 2>/dev/null | grep -qi 'NVIDIA'; then
  echo ""
  echo "--- GPU ---"
  nvidia-smi -L 2>&1 | head -4
  echo "Persistence: $(nvidia-smi -q 2>/dev/null | grep 'Persistence Mode' | head -1 || echo N/A)"
  echo "IMEX: $(ls /dev/nvidia-caps-imex-channels/ 2>&1)"
fi

echo ""
echo "--- Network ---"
ip -br link show type ether 2>/dev/null || true
echo "RDMA: $(rdma link show 2>/dev/null | awk '{print $2}' | tr '\n' ', ' || echo N/A)"

echo ""
echo "--- Disk ---"
df -h /usr/local /data /mnt/stateful_partition 2>/dev/null | grep -v Filesystem

echo ""
echo "--- SSH ---"
echo "Ports: $(grep '^Port' /etc/ssh/sshd_config | awk '{print $2}' | tr '\n' ', ')"

# Load networking modules for Calico (VXLAN / IPIP)
echo "=== Loading networking modules ==="
sudo modprobe vxlan || true
sudo modprobe ipip || true
echo "vxlan" | sudo tee /etc/modules-load.d/vxlan.conf
echo "ipip" | sudo tee /etc/modules-load.d/ipip.conf

touch "$INIT_STAMP"
echo ""
echo "================================================================"
echo "  Startup complete at $(date)"
echo "================================================================"

2.2 批量创建 Worker VM

> ⚠️ **重要提示 (Placement Policy 限制与 16/18 容灾 SLA)**:
> 1. **严禁绕过 Placement Policy**:根据 GCP 官方物理限制,GB200/GB300 必须以 1x72 拓扑(18个节点)为最小单元绑定 Placement Policy 进行资源调度。如果在同一个 sub-block(NVLink 域)内绕过 policy 零散地创建虚拟机,会导致整台 NVL72 机架发生物理碎片化锁定,后续将彻底无法再用于完整的 1x72 拓扑调度。
> 2. **为什么是 `--min-count=16`**:GCP 官方针对 A4X/A4X-Max 机架的 SLA 可用性承诺为 **16/18**。这意味着在由 18 个计算节点组成的 Domain 中,GCP 允许最多存在 2 个状态为 `degraded hosts`(因硬件故障排队等待维修更换)的节点。为了确保即使物理机架中存在故障节点,您的集群依然能成功拉起可用的机器,在使用 `bulk create` 批量申请时,**必须将参数设置为 `--count=18` 且 `--min-count=16`**。如果您强行设置 `--min-count=18`,一旦该机架内有 1 台机器报修,整个创建命令都会因无法满足约束而彻底被拒!这也是为什么主流模型框架都在适配 **64 + 8 Spare (容灾备用)** 队列机制的根本硬件原因。
> 3. **故障节点生命周期与恢复机制**:
>    * **16/18 容灾补充创建**:如果在首次执行 `bulk create` 时由于有部分 `degraded hosts` 而只成功拉起了 16 台虚拟机,那么缺失的 2 台在云端是**不存在**的。当底层物理机修复后,您**必须重新执行一次 bulk create**(只传入缺失的 2 台 VM 名称,如 `count=2`),并且**必须指定与前 16 台完全相同的 Placement Policy 与 Subblock**,否则将引发 `PLACEMENT_POLICY_CONSTRAINTS_INFEASIBLE` 碎片化报错。

``` bash
# 按 Domain 分批创建, 每批绑定对应的 Placement Policy
for d in $(seq 1 $NUM_DOMAINS); do
  echo "=== Creating Domain $d workers ==="
  gcloud beta compute instances bulk create \
    --name-pattern="${WORKER_PREFIX}-d${d}-w##" \
    --count=$NODES_PER_DOMAIN \
    --min-count=16 \
    --project=$PROJECT --zone=$ZONE \
    --machine-type=$MACHINE_TYPE \
    --image=$IMAGE --image-project=$IMAGE_PROJECT \
    --boot-disk-size=1000GB --boot-disk-type=hyperdisk-balanced \
    --scopes=cloud-platform \
    --reservation-affinity=specific --reservation=$RESERVATION \
    --provisioning-model=RESERVATION_BOUND \
    --instance-termination-action=STOP \
    --maintenance-policy=TERMINATE \
    --restart-on-failure \
    --resource-policies=${PLACEMENT_PREFIX}-${d} \
    --network-interface=nic-type=GVNIC,network=$GVNIC_NET,subnet=$GVNIC_SUB \
    --network-interface=nic-type=GVNIC,network=$GVNIC_NET_1,subnet=$GVNIC_SUB_1,no-address \
    --network-interface=nic-type=MRDMA,network=$RDMA_NET,subnet=${RDMA_NET}-sub-0,no-address \
    --network-interface=nic-type=MRDMA,network=$RDMA_NET,subnet=${RDMA_NET}-sub-1,no-address \
    --network-interface=nic-type=MRDMA,network=$RDMA_NET,subnet=${RDMA_NET}-sub-2,no-address \
    --network-interface=nic-type=MRDMA,network=$RDMA_NET,subnet=${RDMA_NET}-sub-3,no-address \
    --metadata-from-file=startup-script=tlinux4-customer-init.sh
  echo "Domain $d: workers creation request submitted"
done

注意: - A4X VM 不支持 --local-ssd,Local SSD 由 startup script 自动检测和配置 - RESERVATION_BOUND 表示 VM 只使用预留容量

2.3 创建后验证

# 检查所有 VM 状态
gcloud compute instances list --project=$PROJECT \
  --filter="name~${WORKER_PREFIX}-d" \
  --format="table(name,zone,status,networkInterfaces[0].networkIP)"

# SSH 到任意 worker 检查初始化日志
gcloud compute ssh ${WORKER_PREFIX}-d1-w0 --zone=$ZONE --tunnel-through-iap \
  --command="tail -30 /var/log/tlinux4-customer-init.log"

# 检查 IMEX channels (Phase 1 reboot 后)
gcloud compute ssh ${WORKER_PREFIX}-d1-w0 --zone=$ZONE --tunnel-through-iap \
  --command="ls /dev/nvidia-caps-imex-channels/ && nvidia-smi -L | head -4"

Part 3: 拓扑信息与节点标签

GCP 为 GB200/A4X VM 提供物理拓扑信息。同一 NVL72 domain 内的 VM 返回相同的拓扑标识,可用于确定节点的 domain 归属,进而为 k8s 节点打标签。

3.1 获取拓扑信息

两种获取方式:

方式 端点 权限要求 返回值
Metadata Server(推荐) http://metadata.google.internal/computeMetadata/v1/instance/attributes/physical_host_topology 无(VM 内部直接访问) 拓扑哈希字符串
Compute API gcloud compute instances describe --format='value(resourceStatus.physicalHostTopology.subblock)' 需要 compute.instances.get IAM 权限 结构化的 subblock 标识
# 方式 1: 在 VM 内部通过 Metadata Server 查询(无需权限)
curl -sf -H "Metadata-Flavor: Google" \
  "http://metadata.google.internal/computeMetadata/v1/instance/attributes/physical_host_topology"
# 输出示例: 一串哈希字符串,如 "a1b2c3d4e5f6..."

# 方式 2: 通过 Compute API 查询(需要 IAM 权限)
gcloud compute instances describe ${INSTANCE_NAME} \
  --zone=${ZONE} \
  --format='value(resourceStatus.physicalHostTopology.subblock)'
# 输出示例: 结构化的 subblock 标识,如 "/clusters/xxx/blocks/yyy/subBlocks/zzz"

说明: - Metadata Server 返回的是哈希值(不可读但稳定),同一 NVL72 domain 内的所有 VM 保证返回相同值 - Compute API 返回可读的 subblock 名称,但需要额外配置 IAM 权限 - 客户可根据以上信息,按自身运维规范选择标签命名方式

3.2 手动标记(小规模 / 验证环境)

# 标记 NVL72 Domain
for d in $(seq 1 $NUM_DOMAINS); do
  for i in $(seq 0 $((NODES_PER_DOMAIN - 1))); do
    kubectl label node ${WORKER_PREFIX}-d${d}-w${i} \
      nvl72-domain=domain-${d} --overwrite
  done
done

# 验证
kubectl get nodes --show-labels | grep nvl72-domain

3.3 自动标记(Startup Script 方式,推荐生产环境)

在 Worker 的 startup script 中,kubeadm join 之前,从 Metadata Server 获取拓扑哈希并自动传递 --node-labels 给 kubelet:

# 在 Worker startup script 中(kubeadm join 之前)添加:

# 1. 从 Metadata Server 获取物理拓扑哈希(同 domain 的 VM 返回相同值)
TOPO_HASH=$(curl -sf -H "Metadata-Flavor: Google" \
  "http://metadata.google.internal/computeMetadata/v1/instance/attributes/physical_host_topology" \
  | md5sum | cut -c1-8)  # 截取前 8 位作为域标识符

# 2. 配置 kubelet 启动参数,自动附带拓扑标签
mkdir -p /etc/systemd/system/kubelet.service.d
cat > /etc/default/kubelet <<KUBELET_EOF
KUBELET_EXTRA_ARGS=--node-labels=nvl72-domain=domain-${TOPO_HASH}
KUBELET_EOF
systemctl daemon-reload

# 3. kubeadm join 时节点自动携带标签
kubeadm join ${CP_IP}:6443 \
  --token "${JOIN_TOKEN}" \
  --discovery-token-ca-cert-hash "sha256:${JOIN_HASH}" \
  --node-name "${NODE_NAME}" \
  --ignore-preflight-errors=Hostname

替代方式:kubeadm JoinConfiguration(声明式配置)

# kubeadm-join-config.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: JoinConfiguration
nodeRegistration:
  kubeletExtraArgs:
  - name: "node-labels"
    value: "nvl72-domain=domain-${TOPO_HASH}"
discovery:
  bootstrapToken:
    apiServerEndpoint: "${CP_IP}:6443"
    token: "${JOIN_TOKEN}"
    caCertHashes:
    - "sha256:${JOIN_HASH}"

# 使用方式:kubeadm join --config kubeadm-join-config.yaml

两种方式功能等价。--node-labels 在 k8s 1.34 中未被弃用,kubeadm 的 kubeletExtraArgs 本质上仍是传递 CLI 参数。配置文件方式更符合 IaC 管理风格,但技术上无差异。

3.4 客户自定义标签

以上示例使用 nvl72-domain=domain-${TOPO_HASH} 作为标签格式,客户可根据自身规范自定义:


说明:以上 Part 1-3 为基础设施准备的必需步骤。以下 Part 4-5 为我们基于测试环境验证的推荐最佳实践,供参考。


Part 4: Kubernetes 部署建议(推荐)

以下是基于我们测试环境验证的 k8s 部署方案。

4.1 Control Plane 初始化

将以下脚本保存为 kubeadm-control-plane.sh 并在 CP 节点上执行。

#!/bin/bash
set -euo pipefail
exec > >(tee /var/log/kubeadm-setup.log) 2>&1

K8S_VERSION="1.34"
K8S_PATCH="1.34.9"
POD_CIDR="10.244.0.0/16"
CALICO_VERSION="3.29.3"

echo "=== [1/7] Disable swap and SELinux ==="
swapoff -a
sed -i '/swap/d' /etc/fstab
setenforce 0 || true
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

echo "=== [2/7] Kernel modules and sysctl ==="
cat <<EOF > /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter

cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sysctl --system

echo "=== [3/7] Install containerd ==="
dnf install -y yum-utils
dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
dnf install -y containerd.io
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
systemctl enable --now containerd

echo "=== [4/7] Install kubeadm, kubelet, kubectl ==="
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/rpm/repodata/repomd.xml.key
EOF
dnf install -y kubelet-${K8S_PATCH} kubeadm-${K8S_PATCH} kubectl-${K8S_PATCH}
systemctl enable kubelet

echo "=== [5/7] Initialize kubeadm ==="
INTERNAL_IP=$(curl -s -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip)
kubeadm init \
  --pod-network-cidr=${POD_CIDR} \
  --apiserver-advertise-address=${INTERNAL_IP} \
  --control-plane-endpoint=${INTERNAL_IP}

echo "=== [6/7] Configure kubectl ==="
mkdir -p /root/.kube
cp /etc/kubernetes/admin.conf /root/.kube/config

echo "=== [7/7] Install Calico CNI ==="
curl -s -O https://raw.githubusercontent.com/projectcalico/calico/v${CALICO_VERSION}/manifests/calico.yaml
# 修改 Calico MTU 避免启动失败
sed -i 's/veth_mtu: "0"/veth_mtu: "1450"/' calico.yaml
kubectl --kubeconfig=/root/.kube/config apply -f calico.yaml

echo "=== Generate join command ==="
kubeadm token create --print-join-command > /root/kubeadm-join-command.txt
cat /root/kubeadm-join-command.txt

echo "=== Control plane setup complete ==="

4.2 Calico 配置

CP 初始化后,配置 Calico VXLAN 模式和网卡检测。

重要: GCP 组织策略可能删除 IPIP 协议的防火墙规则,必须使用 VXLAN 模式。

# 1. 配置 Felix 使用 VXLAN 模式并关闭 IPIP
kubectl patch felixconfiguration default --type=merge -p '{"spec":{"vxlanEnabled":true,"ipipEnabled":false}}'

# 2. 配置默认 IPPool 使用 VXLAN 模式 (必选,防 VTEP IP 分配不一致)
kubectl patch ippool default-ipv4-ippool --type=merge -p '{"spec":{"vxlanMode":"Always","ipipMode":"Never"}}'

# 3. A4X Worker 网卡由 startup script 改名为 bond0, 需要更新 IP 自动检测
kubectl set env daemonset/calico-node -n kube-system \
  IP_AUTODETECTION_METHOD="interface=eth.*|bond0"

# 重启 Calico 使配置生效
kubectl rollout restart daemonset/calico-node -n kube-system

4.3 Worker 加入集群

将以下脚本保存为 tlinux4-k8s-worker.sh,在每个 worker 上执行(v5.3 镜像已内置 GPU/IMEX/GDRCopy 等组件)。

#!/bin/bash
# Worker k8s 组件安装 + 加入集群
# 前提: v5.3 镜像已内置 NVIDIA driver, IMEX, GDRCopy, nvidia-persistenced
set -euo pipefail
exec > >(tee /var/log/k8s-worker-setup.log) 2>&1

K8S_VERSION="1.34"
K8S_PATCH="1.34.9"

echo "=== [1/6] Disable swap and SELinux ==="
swapoff -a
sed -i '/swap/d' /etc/fstab
setenforce 0 || true
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config 2>/dev/null || true

echo "=== [2/6] Kernel modules ==="
cat > /etc/modules-load.d/k8s.conf <<EOF
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter
cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sysctl --system

echo "=== [3/6] Install containerd ==="
dnf install -y --enablerepo=AppStream container-selinux 2>/dev/null || true
DOCKER_ARCH=$(uname -m)
cat > /etc/yum.repos.d/docker-ce.repo <<REPOEOF
[docker-ce-stable]
name=Docker CE Stable - ${DOCKER_ARCH}
baseurl=https://download.docker.com/linux/rhel/9/${DOCKER_ARCH}/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/rhel/gpg
REPOEOF
dnf install -y containerd.io
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
mkdir -p /data/containerd
systemctl enable --now containerd

echo "=== [4/6] Install NVIDIA container toolkit ==="
curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
  tee /etc/yum.repos.d/nvidia-container-toolkit.repo
dnf install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=containerd --set-as-default
# containerd config default uses single quotes; nvidia-ctk may rewrite with double quotes
sed -i "s|root = .*/var/lib/containerd.*|root = \"/data/containerd\"|" /etc/containerd/config.toml
systemctl restart containerd

echo "=== [5/6] Install kubeadm/kubelet/kubectl ==="
cat > /etc/yum.repos.d/kubernetes.repo <<REPOEOF
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/rpm/repodata/repomd.xml.key
REPOEOF
dnf install -y --enablerepo=AppStream kubelet-${K8S_PATCH} kubeadm-${K8S_PATCH} kubectl-${K8S_PATCH} \
  conntrack-tools socat iptables ethtool
systemctl enable kubelet

echo "=== [6/6] kubeadm join ==="
# 方式 1: 从 GCE Metadata 获取 join 信息 (如果创建 VM 时传了 metadata)
CP_IP=$(curl -sf -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/attributes/cp-ip 2>/dev/null || true)
JOIN_TOKEN=$(curl -sf -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/attributes/join-token 2>/dev/null || true)
JOIN_HASH=$(curl -sf -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/attributes/join-hash 2>/dev/null || true)

NODE_NAME=$(curl -sf -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/name 2>/dev/null || hostname -s)

if [[ -n "$CP_IP" && -n "$JOIN_TOKEN" && -n "$JOIN_HASH" ]]; then
  kubeadm join ${CP_IP}:6443 \
    --token "${JOIN_TOKEN}" \
    --discovery-token-ca-cert-hash "sha256:${JOIN_HASH}" \
    --node-name "${NODE_NAME}" \
    --ignore-preflight-errors=Hostname
else
  echo "Missing CP metadata. Please run manually:"
  echo "  kubeadm join <CP_IP>:6443 --token <TOKEN> --discovery-token-ca-cert-hash sha256:<HASH> --node-name ${NODE_NAME}"
  echo ""
  echo "On CP, run: kubeadm token create --print-join-command"
fi

echo "=== Worker k8s setup complete at $(date) ==="

重要: NVIDIA container toolkit 必须设为 containerd 的默认 runtime (--set-as-default)。GPU 容器需要 nvidia-container-runtime 注入驱动库。


内部参考:一体化 Worker 脚本(v5.3 镜像 + k8s,支持人员/验证用)

以下脚本在 v5.3 镜像上完成文件系统初始化、sshd 22/56000、RDMA IB 设备改名,并安装 k8s worker 组件。适用于内部快速搭建验证环境。

注意: - v5.3 镜像不再需要重启。脚本一次运行即可完成所有配置,含 k8s 安装和 kubeadm join。 - TLinux 4 基础镜像的 sshd 默认端口为 36000,脚本会显式启用 Port 22 和 Port 56000。 - containerd 数据目录配置在 /data/containerd(50G 根分区空间不足)。

使用方式:

# 1. 在 CP 上生成 join 信息
kubeadm token create --print-join-command
# 输出中提取: CP_IP, TOKEN, HASH

# 2. 创建 Worker VM 时,将 join 信息和此脚本一起传入
gcloud compute instances create ${WORKER_NAME} \
  ... \
  --metadata=cp-ip=${CP_IP},join-token=${TOKEN},join-hash=${HASH} \
  --metadata-from-file=startup-script=tlinux4-internal-init.sh

完整脚本(保存为 tlinux4-internal-init.sh):

#!/bin/bash
# TLinux 4 (TencentOS Server 4.0) — v5.3 Image Internal Test Environment
# Includes k8s worker setup (containerd + nvidia-ctk + kubeadm join)
#
# v5.3 镜像已内置: NVIDIA driver, GVE, IMEX, GDRCopy, nvidia-persistenced,
#                   SELinux/firewalld disabled, memlock unlimited, sudoers, SSH
#
# 本脚本负责: 文件系统初始化 + sshd 22/56000 端口 + 网卡改名 (bond0-bond5) + k8s worker
set -euo pipefail
exec > >(tee /var/log/tlinux4-internal-init.log) 2>&1

K8S_VERSION="1.34"
K8S_PATCH="1.34.9"
INIT_STAMP=/etc/.internal-init-done

###############################################################################
# Network interface rename (A4X only: bond0-bond5) — runs every boot
# GVNIC: enp0s3→bond0, enp192s2→bond1 (persisted via udev after first boot)
# RDMA: mlx5_0→bond2, mlx5_1→bond3, mlx5_2→bond4, mlx5_3→bond5
###############################################################################
if rdma link show 2>/dev/null | grep -q .; then
  echo "=== Network interface rename (A4X) ==="

  # GVNIC rename (Linux netdev: ip link set name)
  for pair in "enp0s3:bond0" "enp192s2:bond1"; do
    OLD="${pair%%:*}"
    NEW="${pair##*:}"
    if ip link show "$OLD" &>/dev/null; then
      ip link set dev "$OLD" down
      ip link set dev "$OLD" name "$NEW"
      ip link set dev "$NEW" up
      echo "  $OLD → $NEW"
    elif ip link show "$NEW" &>/dev/null; then
      echo "  $NEW already exists (skip)"
    fi
  done

  # Update NM connection profile to match new name
  NM_CONN="/etc/NetworkManager/system-connections/enp0s3-gvnic.nmconnection"
  if [ -f "$NM_CONN" ] && grep -q "interface-name=enp0s3" "$NM_CONN"; then
    sed -i 's/interface-name=enp0s3/interface-name=bond0/' "$NM_CONN"
    sed -i 's/id=enp0s3-gvnic/id=bond0-gvnic/' "$NM_CONN"
    nmcli connection reload 2>/dev/null || true
    echo "  Updated NM profile: enp0s3-gvnic → bond0-gvnic"
  fi

  # Create udev rule for persistent GVNIC naming across reboots (one-time)
  UDEV_RULE="/etc/udev/rules.d/70-a4x-bond-rename.rules"
  if [ ! -f "$UDEV_RULE" ]; then
    PCI0=$(basename "$(readlink -f /sys/class/net/bond0/device)")
    PCI1=$(basename "$(readlink -f /sys/class/net/bond1/device)")
    cat > "$UDEV_RULE" << UDEVEOF
SUBSYSTEM=="net", ACTION=="add", KERNELS=="$PCI0", NAME="bond0"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="$PCI1", NAME="bond1"
UDEVEOF
    echo "  Created udev rule for persistent GVNIC naming"
  fi

  # RDMA IB device rename (rdma dev set — runtime only, needs every boot)
  for i in 0 1 2 3; do
    if rdma dev set mlx5_${i} name bond$((i+2)) 2>/dev/null; then
      echo "  mlx5_${i} → bond$((i+2))"
    fi
  done

  echo "  --- Result ---"
  ip -br link show type ether 2>/dev/null || true
  rdma link show 2>/dev/null || true
fi

# v5.3 image has stale stamp from build; remove if partitions not yet created
if [ -f "$INIT_STAMP" ] && [ ! -f "/etc/.partitions-created" ]; then
  echo "Removing stale init stamp from image build..."
  rm -f "$INIT_STAMP"
fi

if [ -f "$INIT_STAMP" ]; then
  echo "Internal initialization already completed. Skipping."
  exit 0
fi

echo "================================================================"
echo "  TLinux 4 v5.3 — Internal Test Startup Script"
echo "  $(date)"
echo "================================================================"

###############################################################################
# [1] Boot disk partitioning: /usr/local (20G) + /data (remaining)
###############################################################################
echo "=== [1] Boot disk partitioning ==="

BOOT_DEV=$(findmnt -n -o SOURCE / | sed 's/p[0-9]*$//')
PARTITION_STAMP=/etc/.partitions-created

if [ ! -f "$PARTITION_STAMP" ]; then
  echo "--- Creating boot disk partitions (1TB disk) ---"

  # Expand GPT to full disk (image GPT backup may be at original image size)
  sgdisk -e "$BOOT_DEV" 2>/dev/null || true

  sgdisk -n 3:0:+20G -t 3:8300 "$BOOT_DEV"
  sgdisk -n 4:0:0 -t 4:8300 "$BOOT_DEV"
  partprobe "$BOOT_DEV" 2>/dev/null || true
  sleep 2

  mkfs.xfs -f "${BOOT_DEV}p3"
  mkfs.xfs -f "${BOOT_DEV}p4"

  TMPMNT=$(mktemp -d)
  mount "${BOOT_DEV}p3" "$TMPMNT"
  cp -a /usr/local/. "$TMPMNT/" 2>/dev/null || true
  umount "$TMPMNT"
  rmdir "$TMPMNT"

  mount "${BOOT_DEV}p3" /usr/local
  mkdir -p /data
  mount "${BOOT_DEV}p4" /data

  P3_UUID=$(blkid -s UUID -o value "${BOOT_DEV}p3")
  P4_UUID=$(blkid -s UUID -o value "${BOOT_DEV}p4")
  if [ -n "$P3_UUID" ] && ! grep -q "$P3_UUID" /etc/fstab; then
    echo "UUID=$P3_UUID /usr/local xfs defaults,nofail 0 2" >> /etc/fstab
  fi
  if [ -n "$P4_UUID" ] && ! grep -q "$P4_UUID" /etc/fstab; then
    echo "UUID=$P4_UUID /data xfs defaults,nofail 0 2" >> /etc/fstab
  fi

  touch "$PARTITION_STAMP"
  echo "Boot disk partitioned: p3=/usr/local(20G) p4=/data(remaining)"
fi

###############################################################################
# [2] Local SSD RAID0 → /mnt/stateful_partition
###############################################################################
echo "=== [2] Local SSD RAID0 setup ==="

SSD_DEVICES=()
for disk in /dev/nvme*n1; do
  [ -b "$disk" ] || continue
  PART_COUNT=$(lsblk -n -o NAME "$disk" 2>/dev/null | wc -l)
  if [ "$PART_COUNT" -le 1 ]; then
    SSD_DEVICES+=("$disk")
    echo "  Detected local SSD: $disk ($(lsblk -n -d -o SIZE "$disk"))"
  fi
done

NUM_SSDS=${#SSD_DEVICES[@]}
echo "  Found $NUM_SSDS local SSD device(s)"

if [ "$NUM_SSDS" -gt 0 ]; then
  MOUNT_POINT="/mnt/stateful_partition"
  mkdir -p "$MOUNT_POINT"

  if [ "$NUM_SSDS" -eq 1 ]; then
    TARGET_DEV="${SSD_DEVICES[0]}"
    mkfs.xfs -f "$TARGET_DEV"
    mount "$TARGET_DEV" "$MOUNT_POINT"
    DEV_UUID=$(blkid -s UUID -o value "$TARGET_DEV")
    if [ -n "$DEV_UUID" ] && ! grep -q "$DEV_UUID" /etc/fstab; then
      echo "UUID=$DEV_UUID $MOUNT_POINT xfs defaults,nofail 0 2" >> /etc/fstab
    fi
  else
    mdadm --stop /dev/md0 2>/dev/null || true
    mdadm --create /dev/md0 --level=0 --raid-devices="$NUM_SSDS" "${SSD_DEVICES[@]}" --force --run
    mkfs.xfs -f /dev/md0
    mount /dev/md0 "$MOUNT_POINT"
    mkdir -p /etc/mdadm
    mdadm --detail --scan >> /etc/mdadm/mdadm.conf
    DEV_UUID=$(blkid -s UUID -o value /dev/md0)
    if [ -n "$DEV_UUID" ] && ! grep -q "$DEV_UUID" /etc/fstab; then
      echo "UUID=$DEV_UUID $MOUNT_POINT xfs defaults,nofail 0 2" >> /etc/fstab
    fi
  fi

  mkdir -p "$MOUNT_POINT"/{gcsfuse-cache,scratch-data}
  echo "$MOUNT_POINT mounted (total: $(df -h "$MOUNT_POINT" | awk 'NR==2{print $2}'))"
else
  echo "WARNING: No local SSD devices found"
  mkdir -p /mnt/stateful_partition
fi

###############################################################################
# [3] sshd port 56000
###############################################################################
echo "=== [3] sshd port 22 + 56000 ==="

# TLinux 4 base image uses Port 36000 with Port 22 commented out.
# Ensure both Port 22 and Port 56000 are explicitly configured.
SSHD_CHANGED=0
if ! grep -q '^Port 22' /etc/ssh/sshd_config; then
  sed -i '1i Port 22' /etc/ssh/sshd_config
  SSHD_CHANGED=1
fi
if ! grep -q '^Port 56000' /etc/ssh/sshd_config; then
  sed -i '/^Port 22/a Port 56000' /etc/ssh/sshd_config
  SSHD_CHANGED=1
fi
if [ "$SSHD_CHANGED" -eq 1 ]; then
  systemctl restart sshd
  echo "sshd now listening on port 22 and 56000"
else
  echo "sshd ports 22 and 56000 already configured"
fi

###############################################################################
# [4] k8s worker setup (containerd + nvidia-ctk + kubeadm join)
###############################################################################
echo "=== [4] k8s $K8S_VERSION worker setup ==="

# [4.1] Swap off
echo "--- [4.1] Disable swap ---"
swapoff -a
sed -i '/swap/d' /etc/fstab

# [4.2] Kernel modules for k8s
echo "--- [4.2] Kernel modules ---"
cat > /etc/modules-load.d/k8s.conf <<EOF
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter
cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sysctl --system

# [4.3] Install containerd
echo "--- [4.3] containerd ---"
dnf install -y --enablerepo=AppStream container-selinux 2>/dev/null || true
DOCKER_ARCH=$(uname -m)
cat > /etc/yum.repos.d/docker-ce.repo <<REPOEOF
[docker-ce-stable]
name=Docker CE Stable - ${DOCKER_ARCH}
baseurl=https://download.docker.com/linux/rhel/9/${DOCKER_ARCH}/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/rhel/gpg
REPOEOF
dnf install -y containerd.io
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
mkdir -p /data/containerd
systemctl enable --now containerd

# [4.4] NVIDIA container toolkit
echo "--- [4.4] NVIDIA container toolkit ---"
curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
  tee /etc/yum.repos.d/nvidia-container-toolkit.repo
dnf install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=containerd --set-as-default
# containerd config default uses single quotes; nvidia-ctk may rewrite with double quotes
sed -i "s|root = .*/var/lib/containerd.*|root = \"/data/containerd\"|" /etc/containerd/config.toml
systemctl restart containerd

# [4.5] kubeadm / kubelet / kubectl
echo "--- [4.5] kubeadm/kubelet/kubectl ---"
cat > /etc/yum.repos.d/kubernetes.repo <<REPOEOF
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/rpm/repodata/repomd.xml.key
REPOEOF
dnf install -y --enablerepo=AppStream kubelet-${K8S_PATCH} kubeadm-${K8S_PATCH} kubectl-${K8S_PATCH} conntrack-tools socat iptables ethtool
systemctl enable kubelet

# [4.6] kubeadm join
echo "--- [4.6] kubeadm join ---"
CP_IP=$(curl -sf -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/attributes/cp-ip 2>/dev/null || true)
JOIN_TOKEN=$(curl -sf -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/attributes/join-token 2>/dev/null || true)
JOIN_HASH=$(curl -sf -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/attributes/join-hash 2>/dev/null || true)

NODE_NAME=$(cat /proc/sys/kernel/hostname 2>/dev/null || curl -sf -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/name || echo "unknown")
if [[ -n "$CP_IP" && -n "$JOIN_TOKEN" && -n "$JOIN_HASH" ]]; then
  kubeadm join ${CP_IP}:6443 \
    --token "${JOIN_TOKEN}" \
    --discovery-token-ca-cert-hash "sha256:${JOIN_HASH}" \
    --node-name "${NODE_NAME}" \
    --ignore-preflight-errors=Hostname 2>&1 || echo "kubeadm join failed, manual join required"
else
  echo "Missing CP metadata (cp-ip, join-token, join-hash). Manual join required."
  echo "Run on CP: kubeadm token create --print-join-command"
fi

###############################################################################
# Verification
###############################################################################
echo ""
echo "================================================================"
echo "  Verification"
echo "================================================================"

echo "--- OS ---"
cat /etc/os-release | head -3
echo "Kernel: $(uname -r)"

if lspci 2>/dev/null | grep -qi 'NVIDIA'; then
  echo ""
  echo "--- GPU ---"
  nvidia-smi -L 2>&1 | head -4
  echo "Persistence: $(nvidia-smi -q 2>/dev/null | grep 'Persistence Mode' | head -1 || echo N/A)"
  echo "IMEX: $(ls /dev/nvidia-caps-imex-channels/ 2>&1)"
fi

echo ""
echo "--- Network ---"
ip -br link show type ether 2>/dev/null || true
echo "RDMA: $(rdma link show 2>/dev/null | awk '{print $2}' | tr '\n' ', ' || echo N/A)"

echo ""
echo "--- Services ---"
for svc in containerd kubelet; do
  echo "$svc: $(systemctl is-active $svc 2>/dev/null || echo inactive)"
done

echo ""
echo "--- Disk ---"
df -h /usr/local /data /mnt/stateful_partition 2>/dev/null | grep -v Filesystem

echo ""
echo "--- SSH ---"
echo "Ports: $(grep '^Port' /etc/ssh/sshd_config | awk '{print $2}' | tr '\n' ', ')"

echo ""
echo "--- k8s ---"
kubeadm version 2>/dev/null || echo "kubeadm not installed"

touch "$INIT_STAMP"
echo ""
echo "================================================================"
echo "  Internal initialization complete at $(date)"
echo "================================================================"

4.4 GFD/NFD 与 DRA GPU Driver 安装

为了确保 DRA GPU Driver 驱动能够正常在节点上调度,必须先部署 Node Feature Discovery (NFD) 和 GPU Feature Discovery (GFD) 对 GPU 节点进行硬件属性标记。

# 1. 部署 GFD 并自动部署 NFD 依赖
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update
helm upgrade --install gpu-feature-discovery nvdp/gpu-feature-discovery \
  --version 0.19.2 -n kube-system --set nfd.deploy=true

# 2. 安装 NVIDIA Device Plugin 用于常规的 nvidia.com/gpu 资源调度与分配
helm upgrade --install nvidia-device-plugin nvdp/nvidia-device-plugin \
  --version 0.17.0 -n kube-system

# 等待 GFD/NFD 及 Device Plugin pods Ready 并验证节点 labels
kubectl get pods -n kube-system -l app.kubernetes.io/name=gpu-feature-discovery -w
kubectl get pods -n kube-system -l app.kubernetes.io/name=nvidia-device-plugin -w
kubectl get nodes --show-labels

# 3. 禁用 nvidia-imex.service (ComputeDomain 会管理 IMEX daemon lifecycle)
# 在所有 worker 上执行:
for d in $(seq 1 $NUM_DOMAINS); do
  for i in $(seq 0 $((NODES_PER_DOMAIN - 1))); do
    gcloud compute ssh ${WORKER_PREFIX}-d${d}-w${i} --zone=$ZONE --tunnel-through-iap \
      --command="sudo systemctl disable --now nvidia-imex.service 2>/dev/null; sudo systemctl mask nvidia-imex.service" &
  done
done
wait

# 3. 安装 DRA GPU Driver via Helm (CNCF 版本)
helm install dra-driver-nvidia-gpu \
  oci://registry.k8s.io/dra-driver-nvidia/charts/dra-driver-nvidia-gpu \
  --version v0.4.0 \
  --namespace gpu-operator --create-namespace \
  --set nvidiaDriverRoot=/ \
  --set gpuResourcesEnabledOverride=true

# 等待 kubelet-plugin pods Ready (每个 worker 一个)
kubectl get pods -n gpu-operator -w

替代方案:使用 NGC 版本 (v25.12.0)

NVIDIA DRA GPU Driver 有两条发布线,源码相同 (kubernetes-sigs/dra-driver-nvidia-gpu): - CNCF 版本 (v0.4.x):chart dra-driver-nvidia-gpu,镜像 registry.k8s.io/dra-driver-nvidia/dra-driver-nvidia-gpu - NGC 版本 (v25.x.x):chart nvidia-dra-driver-gpu,镜像 nvcr.io/nvidia/k8s-dra-driver-gpu

v25.12.0 (2025-02) 是 NGC 线最新 GA 版本,GPU allocation 功能正式 GA。

选择建议:CNCF 版本更新更快(v0.4.1 = 2025-06);NGC 版本适合已有 NVIDIA Helm 仓库的环境。

# 添加 NVIDIA Helm 仓库
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update

# 安装 NGC 版本 DRA GPU Driver
helm install nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu \
  --version="25.12.0" \
  --create-namespace \
  --namespace nvidia-dra-driver-gpu \
  --set nvidiaDriverRoot=/

# 等待 kubelet-plugin pods Ready
kubectl get pods -n nvidia-dra-driver-gpu -w

注意事项:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/dra-driver-nvidia-gpu/v25.12.0/deployments/helm/nvidia-dra-driver-gpu/crds/resource.nvidia.com_computedomains.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/dra-driver-nvidia-gpu/v25.12.0/deployments/helm/nvidia-dra-driver-gpu/crds/resource.nvidia.com_computedomaincliques.yaml

4.5 Scheduler DRA RBAC

关键: kubeadm 1.34.x 的默认 ClusterRole 缺少 DRA RBAC 权限。如果跳过此步骤,ComputeDomain daemon pods 会永久卡在 ContainerCreating 且无报错信息。

cat <<'EOF' | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:kube-scheduler:dra
rules:
- apiGroups: ["resource.k8s.io"]
  resources: ["resourceclaims"]
  verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["resource.k8s.io"]
  resources: ["resourceclaims/status"]
  verbs: ["get", "update", "patch"]
- apiGroups: ["resource.k8s.io"]
  resources: ["resourceslices"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["resource.k8s.io"]
  resources: ["deviceclasses"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-scheduler:dra
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-scheduler:dra
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:kube-scheduler
EOF

# 重启 Scheduler 使 informer 重新同步
kubectl delete pod -n kube-system -l component=kube-scheduler
kubectl get pods -n kube-system -l component=kube-scheduler -w

4.6 ComputeDomain

每个 NVL72 Domain 创建一个 ComputeDomain。

for d in $(seq 1 $NUM_DOMAINS); do
  CD_NAME="domain-${d}-cd"
  echo "=== Creating ComputeDomain: ${CD_NAME} ==="

  cat <<EOF | kubectl apply -f -
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: ${CD_NAME}
spec:
  numNodes: ${NODES_PER_DOMAIN}
  channel:
    allocationMode: Single
    resourceClaimTemplate:
      name: ${CD_NAME}-channel
EOF

  # 获取 UID 并标记域内节点
  CD_UID=$(kubectl get computedomain ${CD_NAME} -o jsonpath='{.metadata.uid}')
  echo "ComputeDomain UID: $CD_UID"

  for i in $(seq 0 $((NODES_PER_DOMAIN - 1))); do
    kubectl label node ${WORKER_PREFIX}-d${d}-w${i} \
      resource.nvidia.com/computeDomain=$CD_UID --overwrite
  done
done

# 等待 daemon pods 启动 (~30-60s)
kubectl get pods -n gpu-operator -l app=computedomain-daemon -w

# 验证
kubectl get computedomain -o wide

4.7 DRANET + RDMA DeviceClass

# 安装 DRANET
helm install dranet oci://registry.k8s.io/networking/charts/dranet \
  --version v1.3.0 -n kube-system

# 创建 RDMA DeviceClass
cat <<'EOF' | kubectl apply -f -
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: rdma-devices
spec:
  selectors:
  - cel:
      expression: |
        device.driver == "dra.net" &&
        has(device.attributes["dra.net"].rdma) && device.attributes["dra.net"].rdma == true
EOF

# 验证 RDMA 设备发现
kubectl get deviceclass
kubectl get resourceslice -o wide | head -20

4.8 MPI Operator

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.8.0/deploy/v2beta1/mpi-operator.yaml

# 验证
kubectl get pods -n mpi-operator -w

4.9 Kueue TAS + JobSet

# 安装 Kueue v0.18+
kubectl apply --server-side -f \
  https://github.com/kubernetes-sigs/kueue/releases/download/v0.18.0/manifests.yaml

# 安装 JobSet v0.12+
kubectl apply --server-side -f \
  https://github.com/kubernetes-sigs/jobset/releases/download/v0.12.0/manifests.yaml

# 等待就绪
kubectl get pods -n kueue-system -w
kubectl get pods -n jobset-system -w

创建 Topology + Queue 资源:

cat <<'EOF' | kubectl apply -f -
apiVersion: kueue.x-k8s.io/v1beta2
kind: Topology
metadata:
  name: gb200-nvl72-topology
spec:
  levels:
  - nodeLabel: "nvl72-domain"
  - nodeLabel: "kubernetes.io/hostname"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: gb200-flavor
spec:
  nodeLabels:
    nvl72-domain: domain-1
  topologyName: gb200-nvl72-topology
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: gpu-cluster-queue
spec:
  namespaceSelector: {}
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: gb200-flavor
      resources:
      - name: "cpu"
        nominalQuota: 50000
      - name: "memory"
        nominalQuota: 200000Gi
      - name: "nvidia.com/gpu"
        nominalQuota: 1296
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  name: default-queue
  namespace: default
spec:
  clusterQueue: gpu-cluster-queue
EOF

注意: Kueue v0.18 使用 v1beta2 API。ResourceFlavor 设置 topologyName 时必须同时提供 nodeLabels。ClusterQueue 需要 namespaceSelector: {}。

4.10 64+8 备用方案

使用 PriorityClass 实现 64 卡训练 + 8 卡备用的弹性容量管理。训练 Pod 抢占低优先级占位 Pod 实现秒级故障切换。

# PriorityClass 定义
cat <<'EOF' | kubectl apply -f -
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-training
value: 1000
preemptionPolicy: PreemptLowerPriority
globalDefault: false
description: "High priority for GPU training workloads. Preempts placeholder pods."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-placeholder
value: -1
preemptionPolicy: Never
globalDefault: false
description: "Low priority placeholder to hold spare GPU capacity."
EOF

占位 Pod Deployment(按实际备用节点数调整 replicas):

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpu-placeholder
  labels:
    app: gpu-placeholder
spec:
  replicas: 2
  selector:
    matchLabels:
      app: gpu-placeholder
  template:
    metadata:
      labels:
        app: gpu-placeholder
    spec:
      priorityClassName: gpu-placeholder
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      - key: kubernetes.io/arch
        operator: Equal
        value: arm64
        effect: NoSchedule
      terminationGracePeriodSeconds: 0
      containers:
      - name: pause
        image: registry.k8s.io/pause:3.9
        resources:
          limits:
            nvidia.com/gpu: 4
EOF

训练 Pod 使用 gpu-training PriorityClass 即可自动抢占占位 Pod:

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: training-preempt-test
spec:
  priorityClassName: gpu-training
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
  - key: kubernetes.io/arch
    operator: Equal
    value: arm64
    effect: NoSchedule
  restartPolicy: Never
  containers:
  - name: gpu-test
    image: nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04
    command: ["bash", "-c"]
    args:
    - |
      echo "=== Training pod started ==="
      nvidia-smi
      sleep 300
    resources:
      limits:
        nvidia.com/gpu: 4
EOF

Part 5: NCCL 测试示例(推荐)

5.1 GIB 使用方式

NCCL-GIB 通过 init container 注入到工作容器中。关键环境变量:

环境变量 值 说明
LD_LIBRARY_PATH /usr/local/gib/lib64:... GIB NCCL 库路径
NCCL_ENV_PLUGIN gcp 加载 GCP 环境插件, 自动设置 A4X 优化参数
NCCL_MNNVL_ENABLE 2 启用 MNNVL (同域) / 0 (跨域)
NCCL_CUMEM_ENABLE 1 启用 CUDA Memory (同域) / 0 (跨域)

GIB init container 模式:

initContainers:
- name: gib-installer
  image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
  command: ["/bin/sh", "-c"]
  args:
  - |
    set -e
    /scripts/container_entry.sh install --install-nccl
    cp -a /usr/local/gib/. /target/gib/
    cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
    cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
    cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
    cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
    mkdir -p /target/gib/lib64/libibverbs
    cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
  volumeMounts:
  - name: gib
    mountPath: /target/gib

工作容器中初始化 GIB 环境:

source /usr/local/gib/scripts/set_nccl_env.sh   # 设置 NCCL_ENV_PLUGIN=gcp
export LD_LIBRARY_PATH=/usr/local/gib/lib64:$LD_LIBRARY_PATH

💡 架构设计说明:关于 GB200 容器内 Mellanox 网络路由配置的必要性

  1. 为什么本节的同域测试不需要配置静态路由脚本?
    • 本文推荐的同域测试(### 5.2 同域 2 节点 MNNVL NCCL 测试)运行在同一个物理机架的 NVLink Subblock(NVLink Domain)中。
    • 在该测试中,我们显式声明了 NCCL_MNNVL_ENABLE=2 和 NCCL_CUMEM_ENABLE=1,这使所有的 GPU 集合通信流量直接走物理 Multi-Node NVLink Switch 互联背板进行传输。
    • 此时,数据流量完全绕过了 Mellanox ConnectX-7 网卡。NCCL 网络插件初始化时如果扫描到副网卡处于 Down 状态(未配置 IP),会直接在日志中将其跳过而不报错中断。因此,本节测试无需容器内路由脚本即可满速通过。
  2. 什么时候必须引入容器内路由配置脚本?
    • 如果您未来将本自建 K8s 集群扩展至 跨机架/跨域(Cross-Domain) 测试或生产环境,GPU 节点之间的通信就无法通过 NVLink 直连,必须降级走 Mellanox ConnectX-7 的三层 RoCE 网络。
    • 在跨域自建场景下,由于开源社区版的 dranet 无法自动完成云厂商特定的网络配置,您必须在 Pod 中声明并运行类似于 GB300 部署指南中的 setup_rdma_ips.py 脚本,从而在容器内动态分配网卡 IPv6 地址、启用网卡、并将明细 Rail /56 路由指向 GCP Link-Local 网关 fe80::1。
    • 若非 GKE 托管集群,缺失该步骤将导致跨机架通信错误地流向管理网卡 eth0 导致丢包或性能断崖式下跌。

5.2 同域 2 节点 MNNVL NCCL 测试

使用 MPI Operator 运行同域 8 GPU (2 节点 × 4 GPU) all_reduce 测试。

首先创建 ResourceClaimTemplate:

cat <<'EOF' | kubectl apply -f -
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: rdma-nics-mpi-sd
spec:
  spec:
    devices:
      requests:
      - name: rdma-nics
        exactly:
          deviceClassName: rdma-devices
          count: 4
EOF

授权 Default ServiceAccount 读取 Pod 状态的权限 (Launcher 容器解析 Worker IP 所需):

cat <<'EOF' | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-reader
  namespace: default
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io
EOF

提交 MPIJob:

cat <<'YAML' | kubectl apply -f -
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: nccl-mpi-sd
spec:
  slotsPerWorker: 4
  runPolicy:
    ttlSecondsAfterFinished: 600
    cleanPodPolicy: Running
  sshAuthMountPath: /root/.ssh
  launcherCreationPolicy: WaitForWorkersReady
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          nodeSelector:
            nvl72-domain: domain-1
          initContainers:
          - name: gib-installer
            image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
            command: ["/bin/sh", "-c"]
            args:
            - |
              set -e
              /scripts/container_entry.sh install --install-nccl
              cp -a /usr/local/gib/. /target/gib/
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
              mkdir -p /target/gib/lib64/libibverbs
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
            volumeMounts:
            - name: gib
              mountPath: /target/gib
          containers:
          - name: launcher
            image: nvcr.io/nvidia/pytorch:26.05-py3
            imagePullPolicy: IfNotPresent
            env:
            - name: LD_LIBRARY_PATH
              value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
            command:
            - bash
            - -c
            - |
              set -e
              echo "Resolving worker IPs via k8s API..."
              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              API="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"

              for attempt in $(seq 1 60); do
                W0_IP=$(curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" \
                  "$API/api/v1/namespaces/default/pods/nccl-mpi-sd-worker-0" 2>/dev/null \
                  | python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",{}).get("podIP",""))' 2>/dev/null || echo "")
                W1_IP=$(curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" \
                  "$API/api/v1/namespaces/default/pods/nccl-mpi-sd-worker-1" 2>/dev/null \
                  | python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",{}).get("podIP",""))' 2>/dev/null || echo "")
                if [ -n "$W0_IP" ] && [ -n "$W1_IP" ]; then
                  echo "Worker-0 IP: $W0_IP"
                  echo "Worker-1 IP: $W1_IP"
                  break
                fi
                echo "Waiting for worker IPs (attempt $attempt)..."
                sleep 5
              done

              if [ -z "$W0_IP" ] || [ -z "$W1_IP" ]; then
                echo "ERROR: Could not resolve worker IPs"
                exit 1
              fi

              echo "$W0_IP nccl-mpi-sd-worker-0.nccl-mpi-sd.default.svc nccl-mpi-sd-worker-0" >> /etc/hosts
              echo "$W1_IP nccl-mpi-sd-worker-1.nccl-mpi-sd.default.svc nccl-mpi-sd-worker-1" >> /etc/hosts

              echo "Waiting 120s for workers to compile nccl-tests..."
              sleep 120
              echo "Starting mpirun..."
              /usr/local/mpi/bin/mpirun --allow-run-as-root \
                -np 8 -npernode 4 \
                -x LD_LIBRARY_PATH=/usr/local/gib/lib64:/usr/local/nvidia/lib64 \
                -x NCCL_NET_PLUGIN=/usr/local/gib/lib64/libnccl-net.so \
                -x NCCL_ENV_PLUGIN=gcp \
                -x NCCL_MNNVL_ENABLE=2 \
                -x NCCL_CUMEM_ENABLE=1 \
                --mca plm_rsh_args '-o StrictHostKeyChecking=no' \
                /tmp/nccl-tests/build/all_reduce_perf -b 1M -e 16G -f 2 -g 1 -w 50 -n 100
            volumeMounts:
            - name: gib
              mountPath: /usr/local/gib
            resources:
              requests:
                cpu: "4"
                memory: 16Gi
              limits:
                cpu: "4"
                memory: 16Gi
          volumes:
          - name: gib
            emptyDir: {}
    Worker:
      replicas: 2
      template:
        spec:
          nodeSelector:
            nvl72-domain: domain-1
          dnsPolicy: None
          dnsConfig:
            nameservers:
            - "10.96.0.10"
            - "8.8.8.8"
            searches:
            - "default.svc.cluster.local"
            - "svc.cluster.local"
            - "cluster.local"
          resourceClaims:
          - name: compute-domain-channel
            resourceClaimTemplateName: domain-1-cd-channel
          - name: rdma-nics
            resourceClaimTemplateName: rdma-nics-mpi-sd
          initContainers:
          - name: gib-installer
            image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
            command: ["/bin/sh", "-c"]
            args:
            - |
              set -e
              /scripts/container_entry.sh install --install-nccl
              cp -a /usr/local/gib/. /target/gib/
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
              mkdir -p /target/gib/lib64/libibverbs
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
            volumeMounts:
            - name: gib
              mountPath: /target/gib
          containers:
          - name: worker
            image: nvcr.io/nvidia/pytorch:26.05-py3
            imagePullPolicy: IfNotPresent
            securityContext:
              privileged: true
            env:
            - name: LD_LIBRARY_PATH
              value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
            - name: NCCL_NET_PLUGIN
              value: /usr/local/gib/lib64/libnccl-net.so
            - name: NCCL_ENV_PLUGIN
              value: gcp
            command:
            - bash
            - -c
            - |
              apt-get update -qq && apt-get install -y -qq openssh-server > /dev/null 2>&1
              mkdir -p /run/sshd
              printf 'Port 22\nPermitRootLogin yes\nPubkeyAuthentication yes\nAuthorizedKeysFile /root/.ssh/authorized_keys\nStrictModes no\nPasswordAuthentication no\nUsePAM no\nPidFile /run/sshd/sshd.pid\n' > /etc/ssh/mpi_sshd_config
              ssh-keygen -A
              for i in $(seq 1 30); do
                cd /tmp && rm -rf nccl-tests && git clone --depth 1 https://github.com/NVIDIA/nccl-tests.git && break
                echo "git clone attempt $i failed, retrying in 10s..."
                sleep 10
              done
              cd /tmp/nccl-tests && make -j$(nproc) MPI=1 \
                MPI_HOME=/usr/local/mpi \
                NCCL_HOME=/usr/local/gib \
                CUDA_HOME=/usr/local/cuda
              /usr/sbin/sshd -De -f /etc/ssh/mpi_sshd_config
            volumeMounts:
            - name: gib
              mountPath: /usr/local/gib
            - name: dshm
              mountPath: /dev/shm
            resources:
              requests:
                cpu: "96"
                memory: 800Gi
              limits:
                cpu: "96"
                memory: 800Gi
                nvidia.com/gpu: 4
              claims:
              - name: compute-domain-channel
              - name: rdma-nics
          volumes:
          - name: gib
            emptyDir: {}
          - name: dshm
            emptyDir:
              medium: Memory
              sizeLimit: 250Gi
YAML

说明: - resourceClaimTemplateName: domain-1-cd-channel 是 ComputeDomain 自动创建的 ResourceClaimTemplate,名称格式为 ${CD_NAME}-channel - Worker 编译 nccl-tests 需要 MPI 支持(MPI=1),GIB 镜像自带的 nccl-tests 未链接 MPI - 同域测试设置 NCCL_MNNVL_ENABLE=2 和 NCCL_CUMEM_ENABLE=1 - NCCL_ENV_PLUGIN=gcp 自动加载 A4X 优化参数(PXN_C2C、IB_ADAPTIVE_ROUTING 等)

5.3 查看测试结果

# 等待 Launcher 完成
kubectl get pods -l training.kubeflow.org/job-name=nccl-mpi-sd -w

# 查看结果
kubectl logs nccl-mpi-sd-launcher

# 预期输出 (同域 MNNVL, 8 GPU):
# all_reduce_perf @8G busbw ≈ 835 GB/s

5.4 使用 NCCL_DEBUG 验证 GIB 初始化

添加 -x NCCL_DEBUG=INFO 到 mpirun 参数中,输出应包含:

NCCL INFO Successfully loaded external env plugin /usr/local/gib/lib64/libnccl-env-gcp.so
NCCL INFO NCCL version 2.30.4+cuda13.0
NCCL INFO NET/gIB : Initializing gIB v1.1.2
NCCL INFO NET/gIB : Using [0]mlx5_0:1/RoCE [1]mlx5_1:1/RoCE [2]mlx5_2:1/RoCE [3]mlx5_3:1/RoCE
NCCL INFO Using network gIB
NCCL INFO comm ... MNNVL 1


实例调度与生命周期限制说明 (Important Restrictions)

Visibility: Confidential

本文档旨在澄清在 Google Cloud 平台上部署 A4X (GB200) 与 A4X-Max (GB300) 实例时,关于放置策略(Placement Policy)以及实例生命周期(删除与重建)的预期产品行为与官方限制的差异。

1. 放置策略(Placement Policy)与 Sub-block 锁定限制

适用范围: GB200 (A4X) 虚拟机与 GB300 (A4X-Max) 裸金属均受此限制约束。

现象描述 在尝试指定 Placement Policy(如 1x72 拓扑)创建 VM 或 GKE 节点池时,若对应的 sub-block(NVLink Domain)中已经存在单台独立创建的 VM,系统会拒绝调度或发生映射失败,提示 PLACEMENT_POLICY_CONSTRAINTS_INFEASIBLE。删除重建 Placement Policy 似乎能“绕过”此问题。

根本原因 这不是系统后端的缓存 Bug,而是 GCP 为了确保 NVLink 物理拓扑完整性而设计的强制锁定机制。因为 GB200 和 GB300 在底层均依赖 1x72 NVLink 拓扑,一个 NVLink Domain(sub-block)必须且只能被唯一的放置策略独占。一旦 Domain 中被创建了没有该策略的单机 VM,硬件即被视为“物理碎片化”,系统将拒绝任何需要独占整个 Domain 的 1x72 策略下发。

官方依据 根据 GCP 公开官方文档 Create an AI-optimized instance with A4X 中的重要警告说明: > Important: “When you create an A4X instance with a 1x72 topology, Compute Engine allocates an entire 18-instance sub-block (NVLink domain) to the placement policy used in the creation.”(当您使用 1x72 拓扑创建实例时,Compute Engine 会将整个包含 18 个实例的 sub-block 完全分配给创建时所使用的那个 Placement Policy。) > “If you create fewer than 18 instances, the unused instances in that sub-block remain idle and are unavailable for other requests that use a different placement policy.”(如果您创建了少于 18 个实例,该 sub-block 中未使用的实例将保持空闲状态,并且无法被使用了不同放置策略的其他请求所使用。)

最佳实践建议 为了最大化资源利用率并避免锁冲突,官方建议:强烈推荐一次性创建完整的 18 个实例。建议使用 instances.bulkInsert API 或 gcloud compute instances bulk create 命令,将 18 台 VM 作为一个整体一次性下发到空闲的 Domain 中,绝不要在同一个 Domain 内混搭创建独立 VM 和集群 VM。

2. 裸金属实例删除后的 1 小时“硬件锁定” (Transient Degraded Status)

适用范围: 仅限于 GB300 (A4X-Max) 裸金属 (Bare Metal) 实例。GB200 (A4X) 虚拟机 (VM) 由于只需回收虚拟化资源,不受此深度物理重置的限制。

现象描述 当删除一个 GB300 裸金属机器的 Domain (Subblock) 以释放配额时,GCP UI 会在长达约 1 小时的时间内将该 sub-block 标记为 “degraded hosts”,在此期间资源被锁定,无法将这些机器重新分配用于新的实例创建。

根本原因 这属于预期的产品核心安全行为,并非硬件故障或 UI 延迟。与 GB200 等普通虚拟机不同,GB300 裸金属 (Bare Metal) 实例在被删除后,必须经历一个称为 Attested Power Cycle(已验证的电源循环) 的深度安全重置流程。 为了保障多租户环境下的物理安全,系统必须移除前一个客户的任何潜在数据或状态。此过程包括 NVMe 磁盘的深度擦除、固件刷新、物理断电重启以及安全启动项的验证。该“消毒”过程需要花费大约 1 小时才能完成并自我恢复。在收到正确的密码学验证证明(Cryptographic proof)之前,平台会锁定硬件,禁止下一次分配。

最佳实践建议 对于 GB300 裸金属资源:在进行容量测试或日常运维编排时,请将这 1 小时的安全重置时间纳入工作流预期中。避免设计依赖于“秒级销毁并立即在原物理机上重建”的业务逻辑,这在 GCP Bare Metal 架构中是不被支持的。

附录

附录 A: RDMA 设备与 IB 设备改名

A.1 GCP A4X RDMA 设备概述

A4X 的 MRDMA 网卡是 SR-IOV VF,只暴露 RDMA 能力,没有 Linux 网络接口:

# IB 设备存在且状态正常
$ rdma link show
link mlx5_0/1 state ACTIVE physical_state LINK_UP
link mlx5_1/1 state ACTIVE physical_state LINK_UP
link mlx5_2/1 state ACTIVE physical_state LINK_UP
link mlx5_3/1 state ACTIVE physical_state LINK_UP

# 没有对应的 Linux 网络接口
$ ibdev2netdev
(空)

$ ls /sys/class/infiniband/mlx5_0/device/net/
(空)

# VM 上只有 2 个 GVNIC 网络接口 (startup script 改名后显示为 bond0/bond1)
$ ip -br link show
lo               UNKNOWN  ...
bond0            UP       ...    ← 主 GVNIC (原 enp0s3, MTU 8896)
bond1            UP       ...    ← 辅 GVNIC (原 enp192s2, MTU 8896)

A.2 网卡改名

Startup script 会将所有 6 个网络接口统一改名为 bond0-bond5:

原名 新名 类型 方式
enp0s3 bond0 GVNIC ip link set name
enp192s2 bond1 GVNIC ip link set name
mlx5_0 bond2 RDMA IB rdma dev set
mlx5_1 bond3 RDMA IB rdma dev set
mlx5_2 bond4 RDMA IB rdma dev set
mlx5_3 bond5 RDMA IB rdma dev set

手动改名(如需临时测试):

# GVNIC
ip link set dev enp0s3 down
ip link set dev enp0s3 name bond0
ip link set dev bond0 up

# RDMA IB
rdma dev set mlx5_0 name bond2
rdma dev set mlx5_1 name bond3
rdma dev set mlx5_2 name bond4
rdma dev set mlx5_3 name bond5

验证:

$ rdma link show
link bond2/1 state ACTIVE physical_state LINK_UP
link bond3/1 state ACTIVE physical_state LINK_UP
link bond4/1 state ACTIVE physical_state LINK_UP
link bond5/1 state ACTIVE physical_state LINK_UP

$ ls /sys/class/infiniband/
bond2  bond3  bond4  bond5

A.3 持久化配置

Startup script 通过以下机制实现持久化: - GVNIC: 首次运行时创建 udev 规则 /etc/udev/rules.d/70-a4x-bond-rename.rules,后续重启由 udev 直接命名为 bond0/bond1 - RDMA IB: rdma dev set 仅运行时生效,startup script 在 stamp check 之前执行,确保每次重启都会改名

说明: v5.3 及以上版本的镜像已将此步骤集成到 startup script 中,无需手动配置。

A.4 NCCL 验证

已实测:改名后在 2 节点 8 GPU 环境运行 all_reduce_perf(1M-16G),性能正常,0 errors。NCCL/GIB 通过 IB verbs API 访问 RDMA 设备,不依赖 IB 设备名称。k8s DRANET 通过 PCI 地址发现设备,同样不受改名影响。

附录 B: 预构建镜像软件版本信息

以下为 GCP 预构建的 TencentOS Server 4.0 ARM64 GCE 镜像中各核心组件的版本信息,供 GCP Support 评估环境兼容性和制定支持方案时参考。

B.1 基础环境

组件 版本 说明
操作系统 TencentOS Server 4.0 (Build 4.6) ARM64 (aarch64)
Linux 内核 6.6.119-49.101.nvidia.tl4.aarch64 客户定制 NVIDIA 内核
架构 ARM64 / aarch64 GB200 NVL72 平台

B.2 NVIDIA 软件栈

组件 版本 说明
NVIDIA 驱动 580.159.04 Open 类型 (–kernel-module-type=open), R580 分支, Dual MIT/GPL
CUDA 13.0 通过 cuda-cudart-devel-13-0 安装
nvidia-modprobe 580.159.04 IMEX 依赖
IMEX Daemon 580.159.04 跟随驱动版本, NVreg_CreateImexChannel0=1
nvidia-persistenced 580.159.04 GPU-conditional systemd service, v5.3 新增

B.3 网络驱动

组件 版本 说明
GVE (GVNIC) 驱动 1.4.10-oot Out-of-tree 编译, 源码来自 Google compute-virtual-ethernet-linux
IDPF (Intel NIC) 驱动 v1.0.11 Out-of-tree 编译, 仅 GB300 需要, GB200 不安装

B.4 GPU 辅助组件

组件 版本 说明
GDRCopy v2.6 GPU-conditional systemd service, 仅 GPU 节点启动
Lustre Client DDN 2.14 DKMS 方式安装, 不自动挂载

B.5 GCE 集成

组件 说明
google-compute-engine GCE 集成包 (系统配置)
google-guest-agent GCE Guest Agent (元数据/账号管理)
google-osconfig-agent OS Config Agent (补丁管理)
boost-regex 1.75 (从 Rocky 9 注入, GCE Guest Agent 依赖)
libicu 67.1 (从 Rocky 9 注入, GCE Guest Agent 依赖)

说明:镜像构建详细步骤请参考 TencentOS Server 4.0 ARM64 GCE Image Build Guide (V2)。

附录 C: 跨项目复制镜像

当镜像所在项目受到组织策略限制无法直接共享时,可通过以下步骤将镜像复制到目标项目。此操作无需 GPU VM,也无需启动任何虚拟机。

# 变量设置
SOURCE_PROJECT="source-project-id"          # 镜像来源项目
TARGET_PROJECT="your-project-id"            # 目标项目
ZONE="us-central1-f"                        # 临时磁盘所在区域
IMAGE_NAME="tlinux-server-4-gb200-v5dot3"   # 镜像名称

# 1. 从来源镜像创建临时磁盘
gcloud compute disks create temp-tlinux-copy \
    --project=$TARGET_PROJECT \
    --zone=$ZONE \
    --image=$IMAGE_NAME \
    --image-project=$SOURCE_PROJECT \
    --size=1000GB --type=pd-ssd

# 2. 从磁盘创建本地镜像
gcloud compute images create $IMAGE_NAME \
    --project=$TARGET_PROJECT \
    --source-disk=temp-tlinux-copy \
    --source-disk-zone=$ZONE \
    --guest-os-features="UEFI_COMPATIBLE,GVNIC,IDPF" \
    --architecture="ARM64" \
    --family="tlinux-server-4-gb200"

# 3. 删除临时磁盘
gcloud compute disks delete temp-tlinux-copy \
    --project=$TARGET_PROJECT \
    --zone=$ZONE --quiet

# 4. 验证
gcloud compute images describe $IMAGE_NAME \
    --project=$TARGET_PROJECT \
    --format="table(name,status,archiveSizeBytes,guestOsFeatures[].type)"

注意: 镜像根盘为 1TB,临时磁盘 --size 不能小于 1000GB。复制完成后务必删除临时磁盘以避免不必要的存储费用。

常见问题与故障排除 (Troubleshooting)

问题:mpi-operator 反复崩溃或 apiserver 响应超时 如果集群变得不稳定,或者 mpi-operator 因为租约超时(lease timeouts)反复重启,这通常是由于安装了冗余的 NFD (Node Feature Discovery) 插件引起的(例如 gpu-operator 默认自带的 NFD 与集群已有的 NFD 冲突)。冲突会导致数百个 worker pod 陷入 CrashLoopBackOff,并向 apiserver 发送大量垃圾请求,导致 API Server 瘫痪。 解决方案:如果您使用了完整的 nvidia/gpu-operator Helm chart 并且集群中已存在其他 NFD 组件: > ⚠️ 警告: 本指南采用的是模块化部署(分离安装了 GFD 和 Device Plugin),并未直接安装完整的 gpu-operator。若您自行使用了 nvidia/gpu-operator,请务必设置 --set nfd.enabled=false 以避免 NFD 导致的 Crash Loop 冲突。