GB200 A4X 部署验证操作规程

本指南用于在 GCP 生产项目 tencent-gcp-taiji 中,验证 GB200 (A4X HighGPU) 自建 Kubernetes 集群的部署流程。 所有资源名称均采用标准生产命名,对应物理容量为 14 个 Domain(共 252 个工作节点,1008 张 GPU,完整占用共享物理预留)。


目录导航

  1. 全局变量配置
  2. Part 1: 网络与安全基础设施
  3. Part 2: 创建并配置 Lustre 存储实例
  4. Part 3: Control Plane (Master) 节点部署
  5. Part 4: Worker 节点部署
  6. Part 5: Kubernetes 拓扑与 GPU DRA 配置
  7. Part 6: NCCL 测试与验证
  8. Part 7: 清理资源

全局变量配置

请在您本地的 Linux 开发终端执行以下变量声明。后续所有命令均可直接复制粘贴执行:

# ======================== 项目与可用区 (生产环境) ========================
export PROJECT="tencent-gcp-taiji"
export REGION="us-east1"
export ZONE="us-east1-d"

# ======================== 部署规模与前缀 ========================
export CLUSTER_PREFIX="gb200-east"
export MACHINE_TYPE="a4x-highgpu-4g"
export TOTAL_GPUS=1008                         # 完整预留规模:252 节点 * 4 GPU = 1008 GPU
export GPUS_PER_NODE=4
export NODES_PER_DOMAIN=18                     # 物理 NVL72 架构:18 节点/域
export NUM_DOMAINS=$((TOTAL_GPUS / GPUS_PER_NODE / NODES_PER_DOMAIN)) # 计算得出 14 个 Domain

# ======================== 共享物理预留与拓扑前缀 ========================
export RESERVATION="nvidia-gb200-xvb8jh1huhy5y"
export PLACEMENT_PREFIX="${CLUSTER_PREFIX}-nvl72-policy"

export CP_NAME="${CLUSTER_PREFIX}-cp"
export WORKER_PREFIX="${CLUSTER_PREFIX}"

# ======================== 系统镜像 (生产定制版) ========================
export IMAGE="tlinux-server-4-gb200-v5dot3"
export IMAGE_PROJECT="tencent-gcp-taiji"

# ======================== 生产环境 GVNIC 网络 (已存在) ========================
export GVNIC_NET="gb200-east-gvnic-net-0"
export GVNIC_SUB="gb200-east-gvnic-sub-0"
export GVNIC_CIDR="22.142.40.0/22"

export GVNIC_NET_1="gb200-east-gvnic-net-1"
export GVNIC_SUB_1="gb200-east-gvnic-sub-1"
export GVNIC_1_CIDR="22.142.44.0/22"

# ======================== 生产环境 RDMA 网络 (已存在) ========================
export RDMA_NET="gb200-east-rdma-net"
# RDMA 子网 CIDR 列表 (生产环境对应网卡的子网)
export RDMA_CIDRS=("10.0.16.0/22" "10.0.20.0/22" "10.0.24.0/22" "10.0.28.0/22")

# ======================== Lustre 存储挂载 (创建后获取) ========================
# 已就位的 VPC Peering 服务段 (gcp.totorochina.com 已确认)
export LUSTRE_PEERING_CIDR="22.142.60.0/23"
# 示例: "10.142.40.100@tcp:/lustrefs" (如果不需要挂载,保留空值或不声明)
export LUSTRE_MGS_NID=""

Part 1: 网络与安全基础设施

生产环境的 VPC 网络 (gb200-east-gvnic-net-0 等) 和子网已经存在。我们需要验证并补充/修复必要的防火墙安全规则和置放策略。

1.1 修复与新建防火墙安全规则

🔍 生产环境现状自检: - IAP SSH 规则:生产项目中已存在 gb200-gvnic-net0-allow-iap-ssh 和 gb200-gvnic-net1-allow-iap-ssh,无需重复创建。 - GVNIC 0 内部通信规则:已存在 gb200-allow-internal-gvnic0,但其 Source Range 错误地配置为了 22.142.44.0/22(这会导致 GVNIC 0 平面的节点间无法互相通信)。我们需要将其更新修复为 22.142.40.0/22。 - GVNIC 1 内部通信规则:完全缺失,我们需要全新创建 gb200-allow-internal-gvnic1。

请在终端中执行以下防火墙修复和新建操作:

# 1. 修复 GVNIC 0 内部规则 (更新 Source Range)
gcloud compute firewall-rules update gb200-allow-internal-gvnic0 \
  --project=${PROJECT} \
  --source-ranges=${GVNIC_CIDR}

# 2. 全新创建 GVNIC 1 内部放行规则
gcloud compute firewall-rules create gb200-allow-internal-gvnic1 \
  --project=${PROJECT} \
  --network=${GVNIC_NET_1} \
  --direction=INGRESS --priority=1000 --action=ALLOW \
  --rules=tcp,udp,icmp --source-ranges=${GVNIC_1_CIDR}

1.2 批量创建 Placement Policy (置放策略)

GB200 物理机在没有硬件故障的情况下,必须使用 Compact 置放策略配合 --gpu-topology=1x72 参数进行创建,以使 GSC 调度器能正确绑定物理 NVLink。 由于我们有 14 个 Domain,需要为每个 Domain 分别创建一个置放策略:

# 循环创建 14 个 Domain 的置放策略
for d in $(seq 1 $NUM_DOMAINS); do
  # 补足四位对齐,如 0001, 0002...
  POLICY_ID=$(printf "%04d" $d)
  
  echo "Creating Placement Policy: ${PLACEMENT_PREFIX}-${POLICY_ID}"
  gcloud beta compute resource-policies create group-placement \
    ${PLACEMENT_PREFIX}-${POLICY_ID} \
    --collocation=collocated \
    --gpu-topology=1x72 \
    --project=${PROJECT} --region=${REGION}
done

# 验证置放策略状态
gcloud compute resource-policies list --project=${PROJECT} --region=${REGION} --filter="name~${PLACEMENT_PREFIX}"

Part 2: 创建并配置 Lustre 存储实例

GCP Managed Lustre (基于 DDN 2.14) 部署在辅管理网络(Jumbo Frame 平面) gb200-east-gvnic-net-1 上,以获得最高存储吞吐性能。

2.1 确认 VPC 专用服务连接 (Private Service Access)

🔍 生产环境现状自检: 经查询,生产项目在网络 gb200-east-gvnic-net-1 上已经建立了与 Google 服务的专用连接(VPC Peering servicenetworking-googleapis-com),并预留了名为 gb200-psa-service-range 的 IP 段(22.142.60.0/23)。

因此,无需重复申请 IP 地址段与连接服务。我们只需创建防火墙放行规则以允许 Lustre 流量访问该 IP 段即可。

# 1. 为 Lustre 对接端创建防火墙放行规则 (放行至 GVNIC Net 1,使用已有的服务段)
gcloud compute firewall-rules create gb200-allow-lustre-peering \
  --project=${PROJECT} \
  --network=${GVNIC_NET_1} \
  --allow=tcp:988,tcp:6988 \
  --source-ranges=${LUSTRE_PEERING_CIDR}

2.2 创建 Lustre 实例

本验证环境创建符合生产要求的 125 MB/s/TiB 性能 Tier,以及最小容量 72TB (72000 GiB):

gcloud lustre instances create ${CLUSTER_PREFIX}-lustre \
  --project=${PROJECT} \
  --location=${ZONE} \
  --network=projects/${PROJECT}/global/networks/${GVNIC_NET_1} \
  --capacity-gib=72000 \
  --per-unit-storage-throughput=125 \
  --filesystem=lustrefs

💡 获取 Lustre NID:创建可能需要 10-15 分钟。完成后运行以下命令: gcloud lustre instances describe ${CLUSTER_PREFIX}-lustre --location=${ZONE} --project=${PROJECT} --format="value(mountPoint)" 记下输出的 NID 地址(例如 10.142.60.X@tcp:/lustrefs),将其导出为全局变量 LUSTRE_MGS_NID。


Part 3: Control Plane (Master) 节点部署

Master 节点采用标准的 n2-standard-16 虚拟机,仅接入主管理网络即可。

3.1 创建 Master 节点

# 创建 Master VM,并注入 Master 专属初始化脚本
gcloud compute instances create ${CP_NAME} \
  --project=${PROJECT} \
  --zone=${ZONE} \
  --machine-type="n2-standard-16" \
  --image-family="rocky-linux-9" \
  --image-project="rocky-linux-cloud" \
  --boot-disk-type="pd-ssd" \
  --boot-disk-size="200GB" \
  --network-interface=network=${GVNIC_NET},subnet=${GVNIC_SUB} \
  --metadata-from-file=startup-script=scripts/gb200-east-master-init.sh

3.2 Master 节点初始化脚本原文

脚本将自动执行并输出日志至 /var/log/kubeadm-setup.log。其负责关闭 Swap、开启 K8s 转发、安装 containerd、kubeadm、并部署 Calico。

#!/bin/bash
# Master VM Startup Script for GB200 Cluster
# Installs containerd, K8s control plane components, and initializes the cluster.
set -euo pipefail
exec > >(tee /var/log/kubeadm-setup.log) 2>&1

K8S_VERSION="1.34"
K8S_PATCH="1.34.9"
POD_CIDR="10.244.0.0/16"

echo "=== [1/7] Disable swap and SELinux enforcement ==="
swapoff -a
sed -i '/swap/d' /etc/fstab
setenforce 0 || true
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

echo "=== [2/7] Kernel modules and sysctl ==="
cat <<EOF > /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter

cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sysctl --system

echo "=== [3/7] Install containerd ==="
dnf install -y yum-utils
dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
dnf install -y containerd.io
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
systemctl enable --now containerd

echo "=== [4/7] Install kubeadm, kubelet, kubectl ==="
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v\${K8S_VERSION}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v\${K8S_VERSION}/rpm/repodata/repomd.xml.key
EOF

dnf install -y kubelet-\${K8S_PATCH} kubeadm-\${K8S_PATCH} kubectl-\${K8S_PATCH} conntrack-tools socat iptables ethtool
systemctl enable kubelet

echo "=== [5/7] Initialize kubeadm ==="
INTERNAL_IP=\$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip)
kubeadm init \
  --pod-network-cidr=\${POD_CIDR} \
  --apiserver-advertise-address=\${INTERNAL_IP} \
  --control-plane-endpoint=\${INTERNAL_IP} \
  --ignore-preflight-errors=SystemVerification

echo "=== [6/7] Configure kubectl ==="
mkdir -p /root/.kube
cp /etc/kubernetes/admin.conf /root/.kube/config
chown root:root /root/.kube/config

echo "=== [7/7] Install Calico CNI (MTU 1450 patched) ==="
curl -L -s -O https://raw.githubusercontent.com/projectcalico/calico/v3.29.3/manifests/calico.yaml
# Patch the Calico MTU in the ConfigMap
sed -i 's/veth_mtu: "0"/veth_mtu: "1450"/' calico.yaml
kubectl --kubeconfig=/root/.kube/config apply -f calico.yaml

echo "=== Generate join command ==="
kubeadm token create --print-join-command > /root/kubeadm-join-command.txt
cat /root/kubeadm-join-command.txt

echo "=== Generate SSH key pair for worker access ==="
ssh-keygen -t rsa -b 4096 -f /root/.ssh/id_rsa -N '' -q
cat /root/.ssh/id_rsa.pub

echo "=== Control plane setup complete ==="

3.3 获取 Kubeadm 加入凭证

等待 Master 节点启动完成后,通过 SSH 登录 Master 获取 Worker 加入集群所需的 Token 和 Hash。

# 登录 Master 获取 K8s Join 命令
gcloud compute ssh ${CP_NAME} --zone=${ZONE} --project=${PROJECT} \
  --command="cat /root/kubeadm-join-command.txt"

提取输出命令中的 Token 与 Hash,例如: kubeadm join 22.142.40.X:6443 --token ll80fd.d752jd96pta6epju --discovery-token-ca-cert-hash sha256:f8c1e17b9db...


Part 4: Worker 节点部署

由于不同 Domain 的 Worker 必须绑定各自的置放策略和物理预留子模块,我们按 Domain 分批调用 bulk create 命令。

4.1 批量创建 Worker 节点

请将从 Master 节点获取到的 JOIN_TOKEN 和 JOIN_HASH 替换至下方命令,如有部署 Lustre,也请将获取到的 LUSTRE_MGS_NID 写入对应变量:

⚠️ 重要提示 (Placement Policy 限制与 16/18 容灾 SLA): 1. 严禁绕过 Placement Policy:根据 GCP 官方物理限制,GB200/GB300 必须以 1x72 拓扑(18个节点)为最小单元绑定 Placement Policy 进行资源调度。如果在同一个 sub-block(NVLink 域)内绕过 policy 零散地创建虚拟机,会导致整台 NVL72 机架发生物理碎片化锁定,后续将彻底无法再用于完整的 1x72 拓扑调度。 2. 为什么是 --min-count=16:GCP 官方针对 A4X/A4X-Max 机架的 SLA 可用性承诺为 16/18。这意味着在由 18 个计算节点组成的 Domain 中,GCP 允许最多存在 2 个状态为 degraded hosts(因硬件故障排队等待维修更换)的节点。为了确保即使物理机架中存在故障节点,您的集群依然能成功拉起可用的机器,在使用 bulk create 批量申请时,必须将参数设置为 --count=18 且 --min-count=16。如果您强行设置 --min-count=18,一旦该机架内有 1 台机器报修,整个创建命令都会因无法满足约束而彻底被拒!这也是为什么主流模型框架都在适配 64 + 8 Spare (容灾备用) 队列机制的根本硬件原因。

# 替换为您的实际 Token 和 Hash 串
export CP_IP="<MASTER_INTERNAL_IP>"   # 通常为 22.142.40.X
export JOIN_TOKEN="<YOUR_TOKEN>"
export JOIN_HASH="<YOUR_HASH>"

# 选填:如果部署了 Lustre 存储,请在此处填写 NID
export LUSTRE_MGS_NID="10.142.60.X@tcp:/lustrefs"

# 循环创建所有 14 个 Domain
for d in $(seq 1 $NUM_DOMAINS); do
  POLICY_ID=$(printf "%04d" $d)
  echo "=== Creating Domain $d workers (Subblock $POLICY_ID) ==="
  
  # 组合 Domain 内 18 个 VM 的名称列表 (使用 0 进制索引,w0 到 w17)
  NAMES=""
  for i in $(seq 0 $((NODES_PER_DOMAIN - 1))); do
    if [ -z "$NAMES" ]; then
      NAMES="${WORKER_PREFIX}-d${d}-w${i}"
    else
      NAMES="${NAMES},${WORKER_PREFIX}-d${d}-w${i}"
    fi
  done

  # 动态生成该 Domain 对应的物理子预留路径
  CURRENT_RESERVATION="projects/${PROJECT}/reservations/${RESERVATION}/reservationBlocks/${RESERVATION}-block-0001/reservationSubBlocks/${RESERVATION}-block-0001-subblock-${POLICY_ID}"

  # 拼接 metadata。如果 LUSTRE_MGS_NID 不为空则附加 lustre-mgs
  METADATA="serial-port-enable=1,cp-ip=${CP_IP},join-token=${JOIN_TOKEN},join-hash=${JOIN_HASH}"
  if [ -n "${LUSTRE_MGS_NID:-}" ]; then
    METADATA="${METADATA},lustre-mgs=${LUSTRE_MGS_NID}"
  fi

  # 执行 bulk 批量创建该域的 18 台虚拟机
  gcloud beta compute instances bulk create \
    --predefined-names=${NAMES} \
    --count=${NODES_PER_DOMAIN} \
    --min-count=$((NODES_PER_DOMAIN - 2)) \
    --project=${PROJECT} \
    --zone=${ZONE} \
    --machine-type=${MACHINE_TYPE} \
    --image=${IMAGE} \
    --image-project=${IMAGE_PROJECT} \
    --boot-disk-type=hyperdisk-balanced \
    --boot-disk-size=1000GB \
    --scopes=cloud-platform \
    --reservation-affinity=specific \
    --reservation=${CURRENT_RESERVATION} \
    --provisioning-model=RESERVATION_BOUND \
    --on-host-maintenance=TERMINATE \
    --restart-on-failure \
    --resource-policies=${PLACEMENT_PREFIX}-${POLICY_ID} \
    --metadata="${METADATA}" \
    --metadata-from-file=startup-script=scripts/gb200-east-worker-init.sh \
    --network-interface=subnet=${GVNIC_SUB},nic-type=GVNIC \
    --network-interface=subnet=${GVNIC_SUB_1},nic-type=GVNIC,no-address \
    --network-interface=subnet=${RDMA_NET}-sub-0,nic-type=MRDMA,no-address \
    --network-interface=subnet=${RDMA_NET}-sub-1,nic-type=MRDMA,no-address \
    --network-interface=subnet=${RDMA_NET}-sub-2,nic-type=MRDMA,no-address \
    --network-interface=subnet=${RDMA_NET}-sub-3,nic-type=MRDMA,no-address
done

4.2 Worker 节点一体化初始化脚本原文

脚本配置了 Kubelet 自动获取 physical_host_topology 并在加入集群时携带对应的拓扑标签(生产推荐做法)。其还负责网卡改名、磁盘分区、containerd 安装及 NVIDIA Container Toolkit 配置,并支持根据传入的 lustre-mgs 自动挂载存储。

#!/bin/bash
# Worker VM Startup Script for GB200 Cluster
# Configures:
#   1. Automatic network interface renaming (GVNIC as bond0-1, MRDMA as bond2-5)
#   2. Boot disk partitioning & Local SSD RAID0 setup
#   3. containerd, nvidia-container-toolkit, and K8s Worker components installation
#   4. Auto-labeling topology config & Auto-join to K8s Control Plane via GCE metadata
set -euo pipefail
exec > >(tee /var/log/tlinux4-internal-init.log) 2>&1

K8S_VERSION="1.34"
K8S_PATCH="1.34.9"
INIT_STAMP=/etc/.internal-init-done

# v5.3 image has stale stamp from build; remove if partitions not yet created
if [ -f "$INIT_STAMP" ] && [ ! -f "/etc/.partitions-created" ]; then
  echo "Removing stale init stamp from image build..."
  rm -f "$INIT_STAMP"
fi

if [ -f "$INIT_STAMP" ]; then
  echo "Internal initialization already completed. Skipping."
  exit 0
fi

echo "================================================================"
echo "  TLinux 4 v5.3 — GB200 Worker Startup Script"
echo "  \$(date)"
echo "================================================================"

###############################################################################
# [1] Network interface rename (auto-detect GVNIC & MRDMA)
###############################################################################
if rdma link show 2>/dev/null | grep -q .; then
  echo "=== Network interface rename (auto-detect) ==="
  BOND_IDX=0

  # Detect primary NIC (carries default route)
  PRIMARY_NIC=\$(ip route show default 2>/dev/null | awk '{print \$5}' | head -1)

  # Collect physical ethernet NICs needing rename
  MGMT_NICS=()
  for devpath in /sys/class/net/*/device; do
    [ -e "\$devpath" ] || continue
    nic=\$(basename "\$(dirname "\$devpath")")
    [[ "\$nic" == lo || "\$nic" == bond* ]] && continue
    [[ -d "/sys/class/net/\$nic/device/infiniband" ]] && continue
    MGMT_NICS+=("\$nic")
  done

  if [ \${#MGMT_NICS[@]} -gt 0 ]; then
    # ---- First boot: rename management NICs ----
    if [[ -n "\$PRIMARY_NIC" ]]; then
      ip link set dev "\$PRIMARY_NIC" down
      ip link set dev "\$PRIMARY_NIC" name bond0
      ip link set dev bond0 up
      echo "  \$PRIMARY_NIC → bond0 (primary)"
      BOND_IDX=1
    fi

    # Remaining NICs → bond1, bond2, ... sorted by PCI bus address
    SORTED=\$(for n in "\${MGMT_NICS[@]}"; do
      [[ "\$n" == "\$PRIMARY_NIC" ]] && continue
      pci=\$(basename "\$(readlink -f "/sys/class/net/\$n/device")" 2>/dev/null || echo "zzzz")
      echo "\$pci \$n"
    done | sort | awk '{print \$2}')

    for nic in \$SORTED; do
      if ip link show "\$nic" &>/dev/null; then
        ip link set dev "\$nic" down
        ip link set dev "\$nic" name "bond\${BOND_IDX}"
        ip link set dev "bond\${BOND_IDX}" up
        echo "  \$nic → bond\${BOND_IDX}"
        BOND_IDX=\$((\$BOND_IDX + 1))
      fi
    done
    MGMT_COUNT=\$BOND_IDX

    # Update NM connection profile for primary NIC
    if [[ -n "\$PRIMARY_NIC" ]]; then
      for nmf in /etc/NetworkManager/system-connections/*.nmconnection; do
        [ -f "\$nmf" ] || continue
        if grep -q "interface-name=\${PRIMARY_NIC}" "\$nmf" 2>/dev/null; then
          sed -i "s/interface-name=\${PRIMARY_NIC}/interface-name=bond0/" "\$nmf"
          sed -i "s/id=\${PRIMARY_NIC}/id=bond0/" "\$nmf"
          nmcli connection reload 2>/dev/null || true
          echo "  Updated NM profile: \${PRIMARY_NIC} → bond0"
          break
        fi
      done
    fi

    # Create udev rules for persistent naming across reboots (one-time)
    UDEV_RULE="/etc/udev/rules.d/70-a4x-bond-rename.rules"
    if [ ! -f "\$UDEV_RULE" ]; then
      : > "\$UDEV_RULE"
      for bidx in \$(seq 0 \$((\$MGMT_COUNT - 1))); do
        PCI=\$(basename "\$(readlink -f "/sys/class/net/bond\${bidx}/device")" 2>/dev/null || true)
        [ -n "\$PCI" ] && echo "SUBSYSTEM==\"net\", ACTION==\"add\", KERNELS==\"\$PCI\", NAME=\"bond\${bidx}\"" >> "\$UDEV_RULE"
      done
      echo "  Created udev rules for \${MGMT_COUNT} management NICs"
    fi
  else
    # ---- Reboot: management NICs already named by udev ----
    while ip link show "bond\${BOND_IDX}" &>/dev/null; do
      BOND_IDX=\$((\$BOND_IDX + 1))
    done
    [ \$BOND_IDX -gt 0 ] && echo "  Management NICs already renamed (bond0-bond\$((\$BOND_IDX - 1)))"
    MGMT_COUNT=\$BOND_IDX
  fi

  # ---- RDMA IB devices (runtime rename, needs every boot) ----
  for rdev in \$(rdma dev 2>/dev/null | awk '{print \$2}' | sed 's/:\$//' | sort); do
    [[ "\$rdev" == bond* ]] && continue
    if rdma dev set "\$rdev" name "bond\${BOND_IDX}" 2>/dev/null; then
      echo "  \$rdev → bond\${BOND_IDX}"
      BOND_IDX=\$((\$BOND_IDX + 1))
    fi
  done

  RDMA_COUNT=\$((\$BOND_IDX - \$MGMT_COUNT))
  echo "  --- Result: \${MGMT_COUNT} mgmt + \${RDMA_COUNT} RDMA = \${BOND_IDX} total ---"
  ip -br link show type ether 2>/dev/null || true
  rdma link show 2>/dev/null || true
fi

###############################################################################
# [2] Boot disk partitioning: /usr/local (20G) + /data (remaining)
###############################################################################
echo "=== [2] Boot disk partitioning ==="
BOOT_DEV=\$(findmnt -n -o SOURCE / | sed 's/p[0-9]*\$//')
PARTITION_STAMP=/etc/.partitions-created

if [ ! -f "\$PARTITION_STAMP" ]; then
  echo "--- Creating boot disk partitions (1TB disk) ---"
  sgdisk -e "\$BOOT_DEV" 2>/dev/null || true
  sgdisk -n 3:0:+20G -t 3:8300 "$BOOT_DEV"
  sgdisk -n 4:0:0 -t 4:8300 "$BOOT_DEV"
  partprobe "$BOOT_DEV" 2>/dev/null || true
  sleep 2

  mkfs.xfs -f "\${BOOT_DEV}p3"
  mkfs.xfs -f "\${BOOT_DEV}p4"

  TMPMNT=\$(mktemp -d)
  mount "\${BOOT_DEV}p3" "$TMPMNT"
  cp -a /usr/local/. "$TMPMNT/" 2>/dev/null || true
  umount "\$TMPMNT"
  rmdir "\$TMPMNT"

  mount "\${BOOT_DEV}p3" /usr/local
  mkdir -p /data
  mount "\${BOOT_DEV}p4" /data

  P3_UUID=\$(blkid -s UUID -o value "\${BOOT_DEV}p3")
  P4_UUID=\$(blkid -s UUID -o value "\${BOOT_DEV}p4")
  if [ -n "\$P3_UUID" ] && ! grep -q "\$P3_UUID" /etc/fstab; then
    echo "UUID=\$P3_UUID /usr/local xfs defaults,nofail 0 2" >> /etc/fstab
  fi
  if [ -n "\$P4_UUID" ] && ! grep -q "\$P4_UUID" /etc/fstab; then
    echo "UUID=\$P4_UUID /data xfs defaults,nofail 0 2" >> /etc/fstab
  fi
  touch "$PARTITION_STAMP"
  echo "Boot disk partitioned: p3=/usr/local(20G) p4=/data(remaining)"
fi

###############################################################################
# [3] Local SSD RAID0 setup (4x 3TB NVMe Local SSDs)
###############################################################################
echo "=== [3] Local SSD RAID0 setup ==="
SSD_DEVICES=()
for disk in /dev/nvme*n1; do
  [ -b "\$disk" ] || continue
  PART_COUNT=\$(lsblk -n -o NAME "\$disk" 2>/dev/null | wc -l)
  if [ "$PART_COUNT" -le 1 ]; then
    SSD_DEVICES+=("\$disk")
    echo "  Detected local SSD: \$disk (\$(lsblk -n -d -o SIZE "\$disk"))"
  fi
done

NUM_SSDS=\text{\${#SSD_DEVICES[@]}}
echo "  Found \$NUM_SSDS local SSD device(s)"

if [ "\$NUM_SSDS" -gt 0 ]; then
  MOUNT_POINT="/mnt/stateful_partition"
  mkdir -p "$MOUNT_POINT"

  if [ "\$NUM_SSDS" -eq 1 ]; then
    TARGET_DEV="\${SSD_DEVICES[0]}"
    mkfs.xfs -f "\$TARGET_DEV"
    mount "$TARGET_DEV" "$MOUNT_POINT"
    DEV_UUID=\$(blkid -s UUID -o value "$TARGET_DEV")
    if [ -n "$DEV_UUID" ] && ! grep -q "$DEV_UUID" /etc/fstab; then
      echo "UUID=\$DEV_UUID \$MOUNT_POINT xfs defaults,nofail 0 2" >> /etc/fstab
    fi
  else
    mdadm --stop /dev/md0 2>/dev/null || true
    mdadm --create /dev/md0 --level=0 --raid-devices="\$NUM_SSDS" "\${SSD_DEVICES[@]}" --force --run
    mkfs.xfs -f /dev/md0
    mount /dev/md0 "$MOUNT_POINT"
    mkdir -p /etc/mdadm
    mdadm --detail --scan >> /etc/mdadm/mdadm.conf
    DEV_UUID=\$(blkid -s UUID -o value /dev/md0)
    if [ -n "\$DEV_UUID" ] && ! grep -q "$DEV_UUID" /etc/fstab; then
      echo "UUID=\$DEV_UUID \$MOUNT_POINT xfs defaults,nofail 0 2" >> /etc/fstab
    fi
  fi
  mkdir -p "$MOUNT_POINT"/{gcsfuse-cache,scratch-data}
  echo "\$MOUNT_POINT mounted"
else
  echo "WARNING: No local SSD devices found"
  mkdir -p /mnt/stateful_partition
fi

###############################################################################
# [4] sshd ports 22 + 56000
###############################################################################
echo "=== [4] sshd port 22 + 56000 ==="
SSHD_CHANGED=0
if ! grep -q '^Port 22' /etc/ssh/sshd_config; then
  sed -i '1i Port 22' /etc/ssh/sshd_config
  SSHD_CHANGED=1
fi
if ! grep -q '^Port 56000' /etc/ssh/sshd_config; then
  sed -i '/^Port 22/a Port 56000' /etc/ssh/sshd_config
  SSHD_CHANGED=1
fi
if [ "\$SSHD_CHANGED" -eq 1 ]; then
  systemctl restart sshd
fi

###############################################################################
# [4.5] Lustre Storage network routing & mount (dynamic)
# Lustre is connected to GVNIC Net 1 (bond1, MTU 8896 Jumbo Frame)
###############################################################################
echo "=== [4.5] Lustre Storage routing & mount ==="
LUSTRE_MGS=\$(curl -sf -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/attributes/lustre-mgs || echo "")

if [ -n "\$LUSTRE_MGS" ]; then
  echo "Lustre MGS detected: \$LUSTRE_MGS"
  # Set modprobe networks option to tcp(bond1) (Lustre is on Jumbo Frame net)
  echo 'options lnet networks="tcp(bond1)"' > /etc/modprobe.d/lustre.conf

  # Ensure the modules are loaded correctly (must happen after creating lustre.conf if not already loaded)
  modprobe lnet || true
  modprobe lustre || true

  lnetctl lnet configure || true
  lnetctl net add --net tcp --if bond1 2>/dev/null || lnetctl net add --net tcp --if eth1 || true

  # Extract Lustre IP and create a static route via bond1's gateway
  LUSTRE_IP=$(echo "$LUSTRE_MGS" | cut -d'@' -f1)
  BOND1_GW=$(curl -sf -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/1/gateway)
  
  if [ -n "$BOND1_GW" ] && [ -n "$LUSTRE_IP" ]; then
    # Route the entire /16 containing the MGS to ensure OSTs are reachable
    LUSTRE_CIDR="$(echo "$LUSTRE_IP" | cut -d'.' -f1-2).0.0/16"
    echo "Adding static route for Lustre ($LUSTRE_CIDR) via bond1 gateway ($BOND1_GW)"
    ip route add "$LUSTRE_CIDR" via "$BOND1_GW" dev bond1 || true
  fi

  mkdir -p /gstore
  mount -t lustre "\$LUSTRE_MGS" /gstore || echo "Failed to mount /gstore"

  # Mount to fresh path to bypass Kubelet hostPath caching
  mkdir -p /lustre
  mount -t lustre "\$LUSTRE_MGS" /lustre || echo "Failed to mount /lustre"
else
  echo "No lustre-mgs metadata attribute found. Skipping Lustre mount."
fi


# Load networking modules for Calico (VXLAN / IPIP)
echo "=== Loading networking modules ==="
sudo modprobe vxlan || true
sudo modprobe ipip || true
echo "vxlan" | sudo tee /etc/modules-load.d/vxlan.conf
echo "ipip" | sudo tee /etc/modules-load.d/ipip.conf

###############################################################################
# [5] k8s worker setup (containerd + nvidia-ctk + kubeadm join)
###############################################################################
echo "=== [5] Configuring Kernel Modules for Kubernetes ==="
cat > /etc/modules-load.d/k8s.conf <<EOF
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter

cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sysctl --system

echo "=== [6] Installing containerd ==="
dnf install -y container-selinux 2>/dev/null || true
DOCKER_ARCH=\$(uname -m)
cat > /etc/yum.repos.d/docker-ce.repo <<REPOEOF
[docker-ce-stable]
name=Docker CE Stable - \text{\${DOCKER_ARCH}}
baseurl=https://download.docker.com/linux/rhel/9/\text{\${DOCKER_ARCH}}/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/rhel/gpg
REPOEOF

dnf install -y containerd.io
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
mkdir -p /data/containerd
sed -i "s|root = .*/var/lib/containerd.*|root = \"/data/containerd\"|" /etc/containerd/config.toml
systemctl enable --now containerd

echo "=== [7] Installing NVIDIA Container Toolkit ==="
curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
  tee /etc/yum.repos.d/nvidia-container-toolkit.repo
dnf install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=containerd --set-as-default
systemctl restart containerd

echo "=== [8] Installing K8s packages (kubeadm/kubelet/kubectl) ==="
cat > /etc/yum.repos.d/kubernetes.repo <<REPOEOF
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v\${K8S_VERSION}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v\${K8S_VERSION}/rpm/repodata/repomd.xml.key
EOF

dnf install -y kubelet-\${K8S_PATCH} kubeadm-\${K8S_PATCH} kubectl-\${K8S_PATCH} conntrack-tools socat iptables ethtool
systemctl enable kubelet

# [8.5] Configure auto-labeling for Kubelet (Production recommended)
echo "=== [8.5] Configure Kubelet auto-labeling ==="
TOPO_HASH=\$(curl -sf -H "Metadata-Flavor: Google" \
  "http://metadata.google.internal/computeMetadata/v1/instance/attributes/physical_host_topology" \
  | md5sum | cut -c1-8 || echo "unknown")

if [ "\$TOPO_HASH" != "unknown" ]; then
  mkdir -p /etc/systemd/system/kubelet.service.d
  cat > /etc/default/kubelet <<KUBELET_EOF
KUBELET_EXTRA_ARGS=--node-labels=nvl72-domain=domain-\${TOPO_HASH}
KUBELET_EOF
  systemctl daemon-reload
  echo "  Kubelet configured to auto-label with nvl72-domain=domain-\text{\${TOPO_HASH}}"
else
  echo "  WARNING: Failed to retrieve physical_host_topology from metadata"
fi

echo "=== [9] Kubeadm Join ==="
CP_IP=\$(curl -sf -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/attributes/cp-ip 2>/dev/null || true)
JOIN_TOKEN=\$(curl -sf -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/attributes/join-token 2>/dev/null || true)
JOIN_HASH=\$(curl -sf -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/attributes/join-hash 2>/dev/null || true)

NODE_NAME=\$(cat /proc/sys/kernel/hostname 2>/dev/null || curl -sf -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/name || echo "unknown")

if [[ -n "\$CP_IP" && -n "\$JOIN_TOKEN" && -n "\$JOIN_HASH" ]]; then
  echo "  Joining cluster at \text{\${CP_IP}}:6443..."
  kubeadm join \${CP_IP}:6443 \
    --token "\${JOIN_TOKEN}" \
    --discovery-token-ca-cert-hash "sha256:\${JOIN_HASH}" \
    --node-name "\text{\${NODE_NAME}}" \
    --ignore-preflight-errors=Hostname,SystemVerification 2>&1 || echo "kubeadm join failed, manual join required"
else
  echo "  Missing CP metadata parameters (cp-ip, join-token, join-hash). Skipping auto-join."
fi

touch "\$INIT_STAMP"
echo "=== Worker Initialization Complete ==="

4.3 验证 Worker 加入与拓扑标签状态

等待 5-8 分钟使 Worker VM 硬件开机自检及系统启动,之后在 Master 节点上验证节点是否就绪,且是否自动携带了 nvl72-domain 物理拓扑标签:

# 登录 Master 检查节点 and 标签
gcloud compute ssh ${CP_NAME} --zone=${ZONE} --project=${PROJECT} \
  --command="sudo kubectl get nodes --show-labels"

期待输出所有 Worker 节点处于 Ready 状态,并且均有类似于 nvl72-domain=domain-a1b2c3d4 的标签。


Part 5: Kubernetes 拓扑与 GPU DRA 配置

5.1 部署 Scheduler DRA RBAC

由于默认的 K8s Scheduler 缺少 DRA 调度的必要权限,需要手动部署 RBAC:

gcloud compute ssh ${CP_NAME} --zone=${ZONE} --project=${PROJECT} <<'EOF'
cat <<'YAML' | sudo kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:kube-scheduler:dra
rules:
- apiGroups: ["resource.k8s.io"]
  resources: ["resourceclaims"]
  verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["resource.k8s.io"]
  resources: ["resourceclaims/status"]
  verbs: ["get", "update", "patch"]
- apiGroups: ["resource.k8s.io"]
  resources: ["resourceslices"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["resource.k8s.io"]
  resources: ["deviceclasses"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-scheduler:dra
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-scheduler:dra
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:kube-scheduler
YAML

# 重启 Scheduler 载入权限
sudo kubectl delete pod -n kube-system -l component=kube-scheduler
sudo kubectl get pods -n kube-system -l component=kube-scheduler
EOF

5.2 部署 NFD & GFD & GPU DRA Driver

这是 NVIDIA 官方组件,用于使 K8s 检测 GPU 的物理属性:

# 登录 Master 部署 NVIDIA 相关 Operator / Helm Chart
gcloud compute ssh ${CP_NAME} --zone=${ZONE} --project=${PROJECT} <<'EOF'
# 1. 部署 Node Feature Discovery (NFD)
sudo helm repo add nfd https://kubernetes-sigs.github.io/node-feature-discovery/charts || true
sudo helm repo update
sudo helm install nfd nfd/node-feature-discovery \
  --namespace node-feature-discovery \
  --create-namespace \
  --set-string master.extraLabelNs={"nvidia.com"}

# 2. 部署 GPU DRA Driver & GFD
sudo helm repo add nvidia https://helm.ngc.nvidia.com/nvidia || true
sudo helm repo update
sudo helm install gpu-operator nvidia/gpu-operator \
  --set nfd.enabled=false \
  --namespace gpu-operator \
  --create-namespace \
  --set validator.repository=nvcr.io/nvidia \
  --set driver.enabled=false \
  --set toolkit.enabled=true \
  --set toolkit.version=v1.16.2-centos7 \
  --set toolkit.env[0].name=CONTAINERD_CONFIG \
  --set toolkit.env[0].value=/etc/containerd/config.toml \
  --set toolkit.env[1].name=CONTAINERD_SOCKET \
  --set toolkit.env[1].value=/run/containerd/containerd.sock

# 检查 Operator 与 NFD pods 状态
sudo kubectl get pods -A | grep -E "gpu-operator|nfd"
EOF

5.3 部署 ComputeDomain (MNNVL 通道)

我们需要为每个物理 Domain 部署一个 ComputeDomain 拓扑组。这里以 Domain 1 为例进行配置(生产部署时需对所有 14 个 Domain 重复此操作):

gcloud compute ssh ${CP_NAME} --zone=${ZONE} --project=${PROJECT} <<'EOF'
cat <<'YAML' | sudo kubectl apply -f -
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: domain-1-cd
spec:
  numNodes: 18                                 # 满规模 NVL72 域为 18 节点
  channel:
    allocationMode: Single
    resourceClaimTemplate:
      name: domain-1-cd-channel
YAML

# 获取 UID 并标记 Domain 1 内部的 18 个节点属性
CD_UID=$(sudo kubectl get computedomain domain-1-cd -o jsonpath='{.metadata.uid}')
for i in $(seq 0 17); do
  sudo kubectl label node ${WORKER_PREFIX}-d1-w${i} resource.nvidia.com/computeDomain=$CD_UID --overwrite
done

# 验证 ComputeDomain 状态
sudo kubectl get computedomain -o wide
EOF

5.4 部署 DRANET 与 DeviceClass

最后安装 DRANET 网卡管理框架 and DeviceClass 设备类:

gcloud compute ssh ${CP_NAME} --zone=${ZONE} --project=${PROJECT} <<'EOF'
# 1. Helm 部署 DRANET
sudo helm install dranet oci://registry.k8s.io/networking/charts/dranet \
  --version v1.3.0 -n kube-system

# 2. 创建 RDMA 设备选择类
cat <<'YAML' | sudo kubectl apply -f -
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: rdma-devices
spec:
  selectors:
  - cel:
      expression: |
        device.driver == "dra.net" &&
        has(device.attributes["dra.net"].rdma) && device.attributes["dra.net"].rdma == true
YAML

# 验证资源状态
sudo kubectl get deviceclass
sudo kubectl get resourceslice -o wide
EOF

Part 6: NCCL 测试与验证

本章节部署一个双节点的 NCCL 测试 JobSet,验证底层 MNNVL 网络是否连通,数据带宽是否达标。

6.1 部署 JobSet 控制器 与 Kueue

gcloud compute ssh ${CP_NAME} --zone=${ZONE} --project=${PROJECT} <<'EOF'
# 安装 Kueue 队列服务
sudo kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.18.0/manifests.yaml

# 安装 JobSet 容器编排
sudo kubectl apply --server-side -f https://github.com/kubernetes-sigs/jobset/releases/download/v0.12.0/manifests.yaml

# 等待 Controller Pods 处于 Running
sudo kubectl get pods -n kueue-system
sudo kubectl get pods -n jobset-system
EOF

6.2 提交双节点 NCCL JobSet 任务

这会拉起一个跨两个物理节点的测试任务(通常选择同一个 Domain 内部的两个 VM 以测出 MNNVL 互联最大性能):

gcloud compute ssh ${CP_NAME} --zone=${ZONE} --project=${PROJECT} <<'EOF'
cat <<'YAML' | sudo kubectl apply -f -
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: nccl-test-2nodes
spec:
  replicatedJobs:
  - name: workers
    replicas: 1
    template:
      spec:
        parallelism: 2
        completions: 2
        backoffLimit: 0
        template:
          spec:
            hostNetwork: true
            dnsPolicy: ClusterFirstWithHostNet
            tolerations:
            - key: nvidia.com/gpu
              operator: Exists
              effect: NoSchedule
            containers:
            - name: nccl-runner
              image: nvcr.io/nvidia/pytorch:24.01-py3
              command:
              - /bin/bash
              - -c
              - |
                echo "Running NCCL Test..."
                # 这里运行具体的 nccl-tests 命令,例如:
                # /workspace/nccl-tests/build/all_reduce_perf -b 8 -e 128M -f 2 -g 4
                sleep 3600
              resources:
                limits:
                  nvidia.com/gpu: 4
              securityContext:
                capabilities:
                  add: ["IPC_LOCK"]
YAML

# 观察任务 Pod 的拉起和运行日志
sudo kubectl get pods -o wide
EOF

💡 带宽评估:同域内跨机 MNNVL 互联理论带宽应在 400GB/s 以上。若测得数据偏低,请检查 GSC backend 中物理拓扑 mapping 状态。


Part 7: 清理资源

测试完成后,请务必删除 VM 实例与网络置放策略,避免产生额外的预留收费:

# 1. 循环删除所有 Domain 的 VM 节点
for d in $(seq 1 $NUM_DOMAINS); do
  echo "Deleting Domain $d workers..."
  gcloud compute instances delete \
    $(seq -f "${WORKER_PREFIX}-d${d}-w%g" 0 $((NODES_PER_DOMAIN - 1)) | tr '\n' ' ') \
    --project=${PROJECT} --zone=${ZONE} --quiet
done

# 2. 删除 Control Plane 节点
gcloud compute instances delete ${CP_NAME} \
  --project=${PROJECT} --zone=${ZONE} --quiet

# 3. 删除自定义防火墙规则
gcloud compute firewall-rules delete \
  ${CLUSTER_PREFIX}-allow-iap-ssh \
  ${CLUSTER_PREFIX}-allow-internal-gvnic0 \
  ${CLUSTER_PREFIX}-allow-internal-gvnic1 \
  --project=${PROJECT} --quiet

# 4. 批量删除置放策略
for d in $(seq 1 $NUM_DOMAINS); do
  POLICY_ID=$(printf "%04d" $d)
  gcloud compute resource-policies delete ${PLACEMENT_PREFIX}-${POLICY_ID} \
    --project=${PROJECT} --region=${REGION} --quiet
done

Part 8: 验证测试总结报告 (2026-07-16)

本验证环节旨在测试单 Domain(64 GPU)及双 Domain 跨机架(128 GPU)的 K8s 节点自动化拉起及 NCCL 集合通信基准测试。

核心结论与问题记录

  1. BeyondCorp SSH 限制与应对方案
    • 现象:在尝试通过 IAP 或原生 GCP SSH 自动登录 ivy-gb200-east-cp 控制平面以提取 Kubeadm Token 时,触发了 Google BeyondCorp 的 Location Restriction 拦截,导致自动化脚本受阻。
    • 解决方案:此限制属于预期内的安全策略。依赖具备对应位置权限的管理员手工通过内网跳板机(或有权限的终端)执行 kubeadm token create --print-join-command 并人工注入部署变量中。已规避通过重启 CP 节点利用 Startup Script 获取可能引发的集群业务中断风险。
  2. Lustre 资源限制与旁路方案
    • 现象:测试 Project 内无法开出 Lustre 实例,资源耗尽。
    • 解决方案:本测试完全剥离了原指南中的 Lustre 强依赖。执行创建 Worker 节点的 gcloud beta compute instances bulk create 命令时,去除了 lustre-mgs metadata 注入。事实证明对于纯粹的 NCCL MPIJob 并不影响带宽验证。
  3. 单/双域 NCCL MNNVL 拓扑调优 (关键结论)
    • 单 Domain 测试 (64卡, 16节点):Worker 全部落在 domain-0001 下,MPIJob 必须通过指定 NCCL_MNNVL_ENABLE=2 和 NCCL_CUMEM_ENABLE=1 来最大化利用域内 NVLink Switch 性能。
    • 双 Domain 测试 (128卡, 32节点):Worker 跨域分布(例如16台在 0001,16台在 0002)。当您使用原生的 nccl-tests 专门为了压测 RoCE 跨架物理带宽极限时,建议注入 NCCL_MNNVL_ENABLE=0 与 NCCL_CUMEM_ENABLE=0 强制禁用 NVLink。这样可以排除域内高速 NVLink 的干扰,得到纯粹的跨机架以太网打流数据。 > ⚠️ 生产框架排雷警告:禁用 MNNVL 仅仅是为了网络基准测试(Benchmark)的控制变量。在运行真实的业务大模型训练框架(例如 Megatron-LM)时,绝对不要注入 NCCL_MNNVL_ENABLE=0!现代大模型训练框架底层的 NCCL 具备自动拓扑感知能力,能够智能地将通信密集的算子路由到同域 NVLink,跨域流量路由到 RDMA。强行关闭 MNNVL 会导致模型完全无法享受 1.8TB/s 的域内 NVLink 红利,整体吞吐性能将发生断崖式下跌。

沉淀输出物

已提取并修改了两份脱离 Lustre 强依赖的基准测试 YAML 供研发后续参考: - 单域 64 GPU: nccl-mpi-64g-single-domain.yaml - 双域 128 GPU: nccl-mpi-128g-dual-domain.yaml

实际测试结果记录 (2026-07-16 实测)

  1. 单域 64 GPU (16 节点, Domain 1)
    • 配置:NCCL_MNNVL_ENABLE=2, NCCL_CUMEM_ENABLE=1 (启用域内 NVLink)
    • 实测带宽:766.6 GB/s (Avg bus bandwidth)
    • 结论:域内 NVLink 集合通信符合预期,性能优异。
  2. 双域 128 GPU (32 节点, Domain 1 + Domain 2)
    • 配置:NCCL_MNNVL_ENABLE=0, NCCL_CUMEM_ENABLE=0 (仅用于 Benchmark 压测,强制走 RoCE/GIB。生产训练请勿添加此环境变量)
    • 实测带宽:158.8 GB/s (Avg bus bandwidth)
    • 网络验证:通过 NCCL 调试日志确认成功加载 gIB 插件,并使用 bond2 - bond5 (RoCE) 进行跨域集合通信。
    • 结论:跨域 RoCE 通信带宽达到理论限制的约 80% (单节点 4x 400Gbps NICs 约 200 GB/s 双向吞吐),符合预期。

常见问题与故障排除 (Troubleshooting)

问题:mpi-operator 反复崩溃或 apiserver 响应超时 如果集群变得不稳定,或者 mpi-operator 因为租约超时(lease timeouts)反复重启,这通常是由于安装了冗余的 NFD (Node Feature Discovery) 插件引起的(例如 gpu-operator 默认自带的 NFD 与集群已有的 NFD 冲突)。冲突会导致数百个 worker pod 陷入 CrashLoopBackOff,并向 apiserver 发送大量垃圾请求,导致 API Server 瘫痪。 解决方案:请确保在部署 gpu-operator 时禁用 NFD:

helm upgrade gpu-operator nvidia/gpu-operator --namespace gpu-operator --reuse-values --set nfd.enabled=false