GB200 A4X 部署验证操作规程
本指南用于在 GCP 生产项目 tencent-gcp-taiji 中,验证
GB200 (A4X HighGPU) 自建 Kubernetes 集群的部署流程。
所有资源名称均采用标准生产命名,对应物理容量为 14 个 Domain(共
252 个工作节点,1008 张 GPU,完整占用共享物理预留)。
目录导航
- 全局变量配置
- Part 1: 网络与安全基础设施
- Part 2: 创建并配置 Lustre 存储实例
- Part 3: Control Plane (Master) 节点部署
- Part 4: Worker 节点部署
- Part 5: Kubernetes 拓扑与 GPU DRA 配置
- Part 6: NCCL 测试与验证
- Part 7: 清理资源
全局变量配置
请在您本地的 Linux 开发终端执行以下变量声明。后续所有命令均可直接复制粘贴执行:
# ======================== 项目与可用区 (生产环境) ========================
export PROJECT="tencent-gcp-taiji"
export REGION="us-east1"
export ZONE="us-east1-d"
# ======================== 部署规模与前缀 ========================
export CLUSTER_PREFIX="gb200-east"
export MACHINE_TYPE="a4x-highgpu-4g"
export TOTAL_GPUS=1008 # 完整预留规模:252 节点 * 4 GPU = 1008 GPU
export GPUS_PER_NODE=4
export NODES_PER_DOMAIN=18 # 物理 NVL72 架构:18 节点/域
export NUM_DOMAINS=$((TOTAL_GPUS / GPUS_PER_NODE / NODES_PER_DOMAIN)) # 计算得出 14 个 Domain
# ======================== 共享物理预留与拓扑前缀 ========================
export RESERVATION="nvidia-gb200-xvb8jh1huhy5y"
export PLACEMENT_PREFIX="${CLUSTER_PREFIX}-nvl72-policy"
export CP_NAME="${CLUSTER_PREFIX}-cp"
export WORKER_PREFIX="${CLUSTER_PREFIX}"
# ======================== 系统镜像 (生产定制版) ========================
export IMAGE="tlinux-server-4-gb200-v5dot3"
export IMAGE_PROJECT="tencent-gcp-taiji"
# ======================== 生产环境 GVNIC 网络 (已存在) ========================
export GVNIC_NET="gb200-east-gvnic-net-0"
export GVNIC_SUB="gb200-east-gvnic-sub-0"
export GVNIC_CIDR="22.142.40.0/22"
export GVNIC_NET_1="gb200-east-gvnic-net-1"
export GVNIC_SUB_1="gb200-east-gvnic-sub-1"
export GVNIC_1_CIDR="22.142.44.0/22"
# ======================== 生产环境 RDMA 网络 (已存在) ========================
export RDMA_NET="gb200-east-rdma-net"
# RDMA 子网 CIDR 列表 (生产环境对应网卡的子网)
export RDMA_CIDRS=("10.0.16.0/22" "10.0.20.0/22" "10.0.24.0/22" "10.0.28.0/22")
# ======================== Lustre 存储挂载 (创建后获取) ========================
# 已就位的 VPC Peering 服务段 (gcp.totorochina.com 已确认)
export LUSTRE_PEERING_CIDR="22.142.60.0/23"
# 示例: "10.142.40.100@tcp:/lustrefs" (如果不需要挂载,保留空值或不声明)
export LUSTRE_MGS_NID=""Part 1: 网络与安全基础设施
生产环境的 VPC 网络 (gb200-east-gvnic-net-0 等)
和子网已经存在。我们需要验证并补充/修复必要的防火墙安全规则和置放策略。
1.1 修复与新建防火墙安全规则
🔍 生产环境现状自检: - IAP SSH 规则:生产项目中已存在
gb200-gvnic-net0-allow-iap-ssh和gb200-gvnic-net1-allow-iap-ssh,无需重复创建。 - GVNIC 0 内部通信规则:已存在gb200-allow-internal-gvnic0,但其 Source Range 错误地配置为了 22.142.44.0/22(这会导致 GVNIC 0 平面的节点间无法互相通信)。我们需要将其更新修复为22.142.40.0/22。 - GVNIC 1 内部通信规则:完全缺失,我们需要全新创建gb200-allow-internal-gvnic1。
请在终端中执行以下防火墙修复和新建操作:
# 1. 修复 GVNIC 0 内部规则 (更新 Source Range)
gcloud compute firewall-rules update gb200-allow-internal-gvnic0 \
--project=${PROJECT} \
--source-ranges=${GVNIC_CIDR}
# 2. 全新创建 GVNIC 1 内部放行规则
gcloud compute firewall-rules create gb200-allow-internal-gvnic1 \
--project=${PROJECT} \
--network=${GVNIC_NET_1} \
--direction=INGRESS --priority=1000 --action=ALLOW \
--rules=tcp,udp,icmp --source-ranges=${GVNIC_1_CIDR}1.2 批量创建 Placement Policy (置放策略)
GB200 物理机在没有硬件故障的情况下,必须使用 Compact
置放策略配合 --gpu-topology=1x72 参数进行创建,以使 GSC
调度器能正确绑定物理 NVLink。 由于我们有 14 个 Domain,需要为每个 Domain
分别创建一个置放策略:
# 循环创建 14 个 Domain 的置放策略
for d in $(seq 1 $NUM_DOMAINS); do
# 补足四位对齐,如 0001, 0002...
POLICY_ID=$(printf "%04d" $d)
echo "Creating Placement Policy: ${PLACEMENT_PREFIX}-${POLICY_ID}"
gcloud beta compute resource-policies create group-placement \
${PLACEMENT_PREFIX}-${POLICY_ID} \
--collocation=collocated \
--gpu-topology=1x72 \
--project=${PROJECT} --region=${REGION}
done
# 验证置放策略状态
gcloud compute resource-policies list --project=${PROJECT} --region=${REGION} --filter="name~${PLACEMENT_PREFIX}"Part 2: 创建并配置 Lustre 存储实例
GCP Managed Lustre (基于 DDN 2.14) 部署在辅管理网络(Jumbo Frame
平面) gb200-east-gvnic-net-1
上,以获得最高存储吞吐性能。
2.1 确认 VPC 专用服务连接 (Private Service Access)
🔍 生产环境现状自检: 经查询,生产项目在网络
gb200-east-gvnic-net-1上已经建立了与 Google 服务的专用连接(VPC Peeringservicenetworking-googleapis-com),并预留了名为gb200-psa-service-range的 IP 段(22.142.60.0/23)。因此,无需重复申请 IP 地址段与连接服务。我们只需创建防火墙放行规则以允许 Lustre 流量访问该 IP 段即可。
# 1. 为 Lustre 对接端创建防火墙放行规则 (放行至 GVNIC Net 1,使用已有的服务段)
gcloud compute firewall-rules create gb200-allow-lustre-peering \
--project=${PROJECT} \
--network=${GVNIC_NET_1} \
--allow=tcp:988,tcp:6988 \
--source-ranges=${LUSTRE_PEERING_CIDR}2.2 创建 Lustre 实例
本验证环境创建符合生产要求的 125 MB/s/TiB 性能 Tier,以及最小容量 72TB (72000 GiB):
gcloud lustre instances create ${CLUSTER_PREFIX}-lustre \
--project=${PROJECT} \
--location=${ZONE} \
--network=projects/${PROJECT}/global/networks/${GVNIC_NET_1} \
--capacity-gib=72000 \
--per-unit-storage-throughput=125 \
--filesystem=lustrefs💡 获取 Lustre NID:创建可能需要 10-15 分钟。完成后运行以下命令:
gcloud lustre instances describe ${CLUSTER_PREFIX}-lustre --location=${ZONE} --project=${PROJECT} --format="value(mountPoint)"记下输出的 NID 地址(例如10.142.60.X@tcp:/lustrefs),将其导出为全局变量LUSTRE_MGS_NID。
Part 3: Control Plane (Master) 节点部署
Master 节点采用标准的 n2-standard-16
虚拟机,仅接入主管理网络即可。
3.1 创建 Master 节点
# 创建 Master VM,并注入 Master 专属初始化脚本
gcloud compute instances create ${CP_NAME} \
--project=${PROJECT} \
--zone=${ZONE} \
--machine-type="n2-standard-16" \
--image-family="rocky-linux-9" \
--image-project="rocky-linux-cloud" \
--boot-disk-type="pd-ssd" \
--boot-disk-size="200GB" \
--network-interface=network=${GVNIC_NET},subnet=${GVNIC_SUB} \
--metadata-from-file=startup-script=scripts/gb200-east-master-init.sh3.2 Master 节点初始化脚本原文
脚本将自动执行并输出日志至
/var/log/kubeadm-setup.log。其负责关闭 Swap、开启 K8s
转发、安装 containerd、kubeadm、并部署 Calico。
#!/bin/bash
# Master VM Startup Script for GB200 Cluster
# Installs containerd, K8s control plane components, and initializes the cluster.
set -euo pipefail
exec > >(tee /var/log/kubeadm-setup.log) 2>&1
K8S_VERSION="1.34"
K8S_PATCH="1.34.9"
POD_CIDR="10.244.0.0/16"
echo "=== [1/7] Disable swap and SELinux enforcement ==="
swapoff -a
sed -i '/swap/d' /etc/fstab
setenforce 0 || true
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
echo "=== [2/7] Kernel modules and sysctl ==="
cat <<EOF > /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
echo "=== [3/7] Install containerd ==="
dnf install -y yum-utils
dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
dnf install -y containerd.io
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
systemctl enable --now containerd
echo "=== [4/7] Install kubeadm, kubelet, kubectl ==="
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v\${K8S_VERSION}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v\${K8S_VERSION}/rpm/repodata/repomd.xml.key
EOF
dnf install -y kubelet-\${K8S_PATCH} kubeadm-\${K8S_PATCH} kubectl-\${K8S_PATCH} conntrack-tools socat iptables ethtool
systemctl enable kubelet
echo "=== [5/7] Initialize kubeadm ==="
INTERNAL_IP=\$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip)
kubeadm init \
--pod-network-cidr=\${POD_CIDR} \
--apiserver-advertise-address=\${INTERNAL_IP} \
--control-plane-endpoint=\${INTERNAL_IP} \
--ignore-preflight-errors=SystemVerification
echo "=== [6/7] Configure kubectl ==="
mkdir -p /root/.kube
cp /etc/kubernetes/admin.conf /root/.kube/config
chown root:root /root/.kube/config
echo "=== [7/7] Install Calico CNI (MTU 1450 patched) ==="
curl -L -s -O https://raw.githubusercontent.com/projectcalico/calico/v3.29.3/manifests/calico.yaml
# Patch the Calico MTU in the ConfigMap
sed -i 's/veth_mtu: "0"/veth_mtu: "1450"/' calico.yaml
kubectl --kubeconfig=/root/.kube/config apply -f calico.yaml
echo "=== Generate join command ==="
kubeadm token create --print-join-command > /root/kubeadm-join-command.txt
cat /root/kubeadm-join-command.txt
echo "=== Generate SSH key pair for worker access ==="
ssh-keygen -t rsa -b 4096 -f /root/.ssh/id_rsa -N '' -q
cat /root/.ssh/id_rsa.pub
echo "=== Control plane setup complete ==="3.3 获取 Kubeadm 加入凭证
等待 Master 节点启动完成后,通过 SSH 登录 Master 获取 Worker 加入集群所需的 Token 和 Hash。
# 登录 Master 获取 K8s Join 命令
gcloud compute ssh ${CP_NAME} --zone=${ZONE} --project=${PROJECT} \
--command="cat /root/kubeadm-join-command.txt"提取输出命令中的 Token 与 Hash,例如:
kubeadm join 22.142.40.X:6443 --token ll80fd.d752jd96pta6epju --discovery-token-ca-cert-hash sha256:f8c1e17b9db...
Part 4: Worker 节点部署
由于不同 Domain 的 Worker
必须绑定各自的置放策略和物理预留子模块,我们按 Domain
分批调用 bulk create 命令。
4.1 批量创建 Worker 节点
请将从 Master 节点获取到的 JOIN_TOKEN 和
JOIN_HASH 替换至下方命令,如有部署 Lustre,也请将获取到的
LUSTRE_MGS_NID 写入对应变量:
⚠️ 重要提示 (Placement Policy 限制与 16/18 容灾 SLA): 1. 严禁绕过 Placement Policy:根据 GCP 官方物理限制,GB200/GB300 必须以 1x72 拓扑(18个节点)为最小单元绑定 Placement Policy 进行资源调度。如果在同一个 sub-block(NVLink 域)内绕过 policy 零散地创建虚拟机,会导致整台 NVL72 机架发生物理碎片化锁定,后续将彻底无法再用于完整的 1x72 拓扑调度。 2. 为什么是
--min-count=16:GCP 官方针对 A4X/A4X-Max 机架的 SLA 可用性承诺为 16/18。这意味着在由 18 个计算节点组成的 Domain 中,GCP 允许最多存在 2 个状态为degraded hosts(因硬件故障排队等待维修更换)的节点。为了确保即使物理机架中存在故障节点,您的集群依然能成功拉起可用的机器,在使用bulk create批量申请时,必须将参数设置为--count=18且--min-count=16。如果您强行设置--min-count=18,一旦该机架内有 1 台机器报修,整个创建命令都会因无法满足约束而彻底被拒!这也是为什么主流模型框架都在适配 64 + 8 Spare (容灾备用) 队列机制的根本硬件原因。
# 替换为您的实际 Token 和 Hash 串
export CP_IP="<MASTER_INTERNAL_IP>" # 通常为 22.142.40.X
export JOIN_TOKEN="<YOUR_TOKEN>"
export JOIN_HASH="<YOUR_HASH>"
# 选填:如果部署了 Lustre 存储,请在此处填写 NID
export LUSTRE_MGS_NID="10.142.60.X@tcp:/lustrefs"
# 循环创建所有 14 个 Domain
for d in $(seq 1 $NUM_DOMAINS); do
POLICY_ID=$(printf "%04d" $d)
echo "=== Creating Domain $d workers (Subblock $POLICY_ID) ==="
# 组合 Domain 内 18 个 VM 的名称列表 (使用 0 进制索引,w0 到 w17)
NAMES=""
for i in $(seq 0 $((NODES_PER_DOMAIN - 1))); do
if [ -z "$NAMES" ]; then
NAMES="${WORKER_PREFIX}-d${d}-w${i}"
else
NAMES="${NAMES},${WORKER_PREFIX}-d${d}-w${i}"
fi
done
# 动态生成该 Domain 对应的物理子预留路径
CURRENT_RESERVATION="projects/${PROJECT}/reservations/${RESERVATION}/reservationBlocks/${RESERVATION}-block-0001/reservationSubBlocks/${RESERVATION}-block-0001-subblock-${POLICY_ID}"
# 拼接 metadata。如果 LUSTRE_MGS_NID 不为空则附加 lustre-mgs
METADATA="serial-port-enable=1,cp-ip=${CP_IP},join-token=${JOIN_TOKEN},join-hash=${JOIN_HASH}"
if [ -n "${LUSTRE_MGS_NID:-}" ]; then
METADATA="${METADATA},lustre-mgs=${LUSTRE_MGS_NID}"
fi
# 执行 bulk 批量创建该域的 18 台虚拟机
gcloud beta compute instances bulk create \
--predefined-names=${NAMES} \
--count=${NODES_PER_DOMAIN} \
--min-count=$((NODES_PER_DOMAIN - 2)) \
--project=${PROJECT} \
--zone=${ZONE} \
--machine-type=${MACHINE_TYPE} \
--image=${IMAGE} \
--image-project=${IMAGE_PROJECT} \
--boot-disk-type=hyperdisk-balanced \
--boot-disk-size=1000GB \
--scopes=cloud-platform \
--reservation-affinity=specific \
--reservation=${CURRENT_RESERVATION} \
--provisioning-model=RESERVATION_BOUND \
--on-host-maintenance=TERMINATE \
--restart-on-failure \
--resource-policies=${PLACEMENT_PREFIX}-${POLICY_ID} \
--metadata="${METADATA}" \
--metadata-from-file=startup-script=scripts/gb200-east-worker-init.sh \
--network-interface=subnet=${GVNIC_SUB},nic-type=GVNIC \
--network-interface=subnet=${GVNIC_SUB_1},nic-type=GVNIC,no-address \
--network-interface=subnet=${RDMA_NET}-sub-0,nic-type=MRDMA,no-address \
--network-interface=subnet=${RDMA_NET}-sub-1,nic-type=MRDMA,no-address \
--network-interface=subnet=${RDMA_NET}-sub-2,nic-type=MRDMA,no-address \
--network-interface=subnet=${RDMA_NET}-sub-3,nic-type=MRDMA,no-address
done4.2 Worker 节点一体化初始化脚本原文
脚本配置了 Kubelet 自动获取 physical_host_topology
并在加入集群时携带对应的拓扑标签(生产推荐做法)。其还负责网卡改名、磁盘分区、containerd
安装及 NVIDIA Container Toolkit 配置,并支持根据传入的
lustre-mgs 自动挂载存储。
#!/bin/bash
# Worker VM Startup Script for GB200 Cluster
# Configures:
# 1. Automatic network interface renaming (GVNIC as bond0-1, MRDMA as bond2-5)
# 2. Boot disk partitioning & Local SSD RAID0 setup
# 3. containerd, nvidia-container-toolkit, and K8s Worker components installation
# 4. Auto-labeling topology config & Auto-join to K8s Control Plane via GCE metadata
set -euo pipefail
exec > >(tee /var/log/tlinux4-internal-init.log) 2>&1
K8S_VERSION="1.34"
K8S_PATCH="1.34.9"
INIT_STAMP=/etc/.internal-init-done
# v5.3 image has stale stamp from build; remove if partitions not yet created
if [ -f "$INIT_STAMP" ] && [ ! -f "/etc/.partitions-created" ]; then
echo "Removing stale init stamp from image build..."
rm -f "$INIT_STAMP"
fi
if [ -f "$INIT_STAMP" ]; then
echo "Internal initialization already completed. Skipping."
exit 0
fi
echo "================================================================"
echo " TLinux 4 v5.3 — GB200 Worker Startup Script"
echo " \$(date)"
echo "================================================================"
###############################################################################
# [1] Network interface rename (auto-detect GVNIC & MRDMA)
###############################################################################
if rdma link show 2>/dev/null | grep -q .; then
echo "=== Network interface rename (auto-detect) ==="
BOND_IDX=0
# Detect primary NIC (carries default route)
PRIMARY_NIC=\$(ip route show default 2>/dev/null | awk '{print \$5}' | head -1)
# Collect physical ethernet NICs needing rename
MGMT_NICS=()
for devpath in /sys/class/net/*/device; do
[ -e "\$devpath" ] || continue
nic=\$(basename "\$(dirname "\$devpath")")
[[ "\$nic" == lo || "\$nic" == bond* ]] && continue
[[ -d "/sys/class/net/\$nic/device/infiniband" ]] && continue
MGMT_NICS+=("\$nic")
done
if [ \${#MGMT_NICS[@]} -gt 0 ]; then
# ---- First boot: rename management NICs ----
if [[ -n "\$PRIMARY_NIC" ]]; then
ip link set dev "\$PRIMARY_NIC" down
ip link set dev "\$PRIMARY_NIC" name bond0
ip link set dev bond0 up
echo " \$PRIMARY_NIC → bond0 (primary)"
BOND_IDX=1
fi
# Remaining NICs → bond1, bond2, ... sorted by PCI bus address
SORTED=\$(for n in "\${MGMT_NICS[@]}"; do
[[ "\$n" == "\$PRIMARY_NIC" ]] && continue
pci=\$(basename "\$(readlink -f "/sys/class/net/\$n/device")" 2>/dev/null || echo "zzzz")
echo "\$pci \$n"
done | sort | awk '{print \$2}')
for nic in \$SORTED; do
if ip link show "\$nic" &>/dev/null; then
ip link set dev "\$nic" down
ip link set dev "\$nic" name "bond\${BOND_IDX}"
ip link set dev "bond\${BOND_IDX}" up
echo " \$nic → bond\${BOND_IDX}"
BOND_IDX=\$((\$BOND_IDX + 1))
fi
done
MGMT_COUNT=\$BOND_IDX
# Update NM connection profile for primary NIC
if [[ -n "\$PRIMARY_NIC" ]]; then
for nmf in /etc/NetworkManager/system-connections/*.nmconnection; do
[ -f "\$nmf" ] || continue
if grep -q "interface-name=\${PRIMARY_NIC}" "\$nmf" 2>/dev/null; then
sed -i "s/interface-name=\${PRIMARY_NIC}/interface-name=bond0/" "\$nmf"
sed -i "s/id=\${PRIMARY_NIC}/id=bond0/" "\$nmf"
nmcli connection reload 2>/dev/null || true
echo " Updated NM profile: \${PRIMARY_NIC} → bond0"
break
fi
done
fi
# Create udev rules for persistent naming across reboots (one-time)
UDEV_RULE="/etc/udev/rules.d/70-a4x-bond-rename.rules"
if [ ! -f "\$UDEV_RULE" ]; then
: > "\$UDEV_RULE"
for bidx in \$(seq 0 \$((\$MGMT_COUNT - 1))); do
PCI=\$(basename "\$(readlink -f "/sys/class/net/bond\${bidx}/device")" 2>/dev/null || true)
[ -n "\$PCI" ] && echo "SUBSYSTEM==\"net\", ACTION==\"add\", KERNELS==\"\$PCI\", NAME=\"bond\${bidx}\"" >> "\$UDEV_RULE"
done
echo " Created udev rules for \${MGMT_COUNT} management NICs"
fi
else
# ---- Reboot: management NICs already named by udev ----
while ip link show "bond\${BOND_IDX}" &>/dev/null; do
BOND_IDX=\$((\$BOND_IDX + 1))
done
[ \$BOND_IDX -gt 0 ] && echo " Management NICs already renamed (bond0-bond\$((\$BOND_IDX - 1)))"
MGMT_COUNT=\$BOND_IDX
fi
# ---- RDMA IB devices (runtime rename, needs every boot) ----
for rdev in \$(rdma dev 2>/dev/null | awk '{print \$2}' | sed 's/:\$//' | sort); do
[[ "\$rdev" == bond* ]] && continue
if rdma dev set "\$rdev" name "bond\${BOND_IDX}" 2>/dev/null; then
echo " \$rdev → bond\${BOND_IDX}"
BOND_IDX=\$((\$BOND_IDX + 1))
fi
done
RDMA_COUNT=\$((\$BOND_IDX - \$MGMT_COUNT))
echo " --- Result: \${MGMT_COUNT} mgmt + \${RDMA_COUNT} RDMA = \${BOND_IDX} total ---"
ip -br link show type ether 2>/dev/null || true
rdma link show 2>/dev/null || true
fi
###############################################################################
# [2] Boot disk partitioning: /usr/local (20G) + /data (remaining)
###############################################################################
echo "=== [2] Boot disk partitioning ==="
BOOT_DEV=\$(findmnt -n -o SOURCE / | sed 's/p[0-9]*\$//')
PARTITION_STAMP=/etc/.partitions-created
if [ ! -f "\$PARTITION_STAMP" ]; then
echo "--- Creating boot disk partitions (1TB disk) ---"
sgdisk -e "\$BOOT_DEV" 2>/dev/null || true
sgdisk -n 3:0:+20G -t 3:8300 "$BOOT_DEV"
sgdisk -n 4:0:0 -t 4:8300 "$BOOT_DEV"
partprobe "$BOOT_DEV" 2>/dev/null || true
sleep 2
mkfs.xfs -f "\${BOOT_DEV}p3"
mkfs.xfs -f "\${BOOT_DEV}p4"
TMPMNT=\$(mktemp -d)
mount "\${BOOT_DEV}p3" "$TMPMNT"
cp -a /usr/local/. "$TMPMNT/" 2>/dev/null || true
umount "\$TMPMNT"
rmdir "\$TMPMNT"
mount "\${BOOT_DEV}p3" /usr/local
mkdir -p /data
mount "\${BOOT_DEV}p4" /data
P3_UUID=\$(blkid -s UUID -o value "\${BOOT_DEV}p3")
P4_UUID=\$(blkid -s UUID -o value "\${BOOT_DEV}p4")
if [ -n "\$P3_UUID" ] && ! grep -q "\$P3_UUID" /etc/fstab; then
echo "UUID=\$P3_UUID /usr/local xfs defaults,nofail 0 2" >> /etc/fstab
fi
if [ -n "\$P4_UUID" ] && ! grep -q "\$P4_UUID" /etc/fstab; then
echo "UUID=\$P4_UUID /data xfs defaults,nofail 0 2" >> /etc/fstab
fi
touch "$PARTITION_STAMP"
echo "Boot disk partitioned: p3=/usr/local(20G) p4=/data(remaining)"
fi
###############################################################################
# [3] Local SSD RAID0 setup (4x 3TB NVMe Local SSDs)
###############################################################################
echo "=== [3] Local SSD RAID0 setup ==="
SSD_DEVICES=()
for disk in /dev/nvme*n1; do
[ -b "\$disk" ] || continue
PART_COUNT=\$(lsblk -n -o NAME "\$disk" 2>/dev/null | wc -l)
if [ "$PART_COUNT" -le 1 ]; then
SSD_DEVICES+=("\$disk")
echo " Detected local SSD: \$disk (\$(lsblk -n -d -o SIZE "\$disk"))"
fi
done
NUM_SSDS=\text{\${#SSD_DEVICES[@]}}
echo " Found \$NUM_SSDS local SSD device(s)"
if [ "\$NUM_SSDS" -gt 0 ]; then
MOUNT_POINT="/mnt/stateful_partition"
mkdir -p "$MOUNT_POINT"
if [ "\$NUM_SSDS" -eq 1 ]; then
TARGET_DEV="\${SSD_DEVICES[0]}"
mkfs.xfs -f "\$TARGET_DEV"
mount "$TARGET_DEV" "$MOUNT_POINT"
DEV_UUID=\$(blkid -s UUID -o value "$TARGET_DEV")
if [ -n "$DEV_UUID" ] && ! grep -q "$DEV_UUID" /etc/fstab; then
echo "UUID=\$DEV_UUID \$MOUNT_POINT xfs defaults,nofail 0 2" >> /etc/fstab
fi
else
mdadm --stop /dev/md0 2>/dev/null || true
mdadm --create /dev/md0 --level=0 --raid-devices="\$NUM_SSDS" "\${SSD_DEVICES[@]}" --force --run
mkfs.xfs -f /dev/md0
mount /dev/md0 "$MOUNT_POINT"
mkdir -p /etc/mdadm
mdadm --detail --scan >> /etc/mdadm/mdadm.conf
DEV_UUID=\$(blkid -s UUID -o value /dev/md0)
if [ -n "\$DEV_UUID" ] && ! grep -q "$DEV_UUID" /etc/fstab; then
echo "UUID=\$DEV_UUID \$MOUNT_POINT xfs defaults,nofail 0 2" >> /etc/fstab
fi
fi
mkdir -p "$MOUNT_POINT"/{gcsfuse-cache,scratch-data}
echo "\$MOUNT_POINT mounted"
else
echo "WARNING: No local SSD devices found"
mkdir -p /mnt/stateful_partition
fi
###############################################################################
# [4] sshd ports 22 + 56000
###############################################################################
echo "=== [4] sshd port 22 + 56000 ==="
SSHD_CHANGED=0
if ! grep -q '^Port 22' /etc/ssh/sshd_config; then
sed -i '1i Port 22' /etc/ssh/sshd_config
SSHD_CHANGED=1
fi
if ! grep -q '^Port 56000' /etc/ssh/sshd_config; then
sed -i '/^Port 22/a Port 56000' /etc/ssh/sshd_config
SSHD_CHANGED=1
fi
if [ "\$SSHD_CHANGED" -eq 1 ]; then
systemctl restart sshd
fi
###############################################################################
# [4.5] Lustre Storage network routing & mount (dynamic)
# Lustre is connected to GVNIC Net 1 (bond1, MTU 8896 Jumbo Frame)
###############################################################################
echo "=== [4.5] Lustre Storage routing & mount ==="
LUSTRE_MGS=\$(curl -sf -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/attributes/lustre-mgs || echo "")
if [ -n "\$LUSTRE_MGS" ]; then
echo "Lustre MGS detected: \$LUSTRE_MGS"
# Set modprobe networks option to tcp(bond1) (Lustre is on Jumbo Frame net)
echo 'options lnet networks="tcp(bond1)"' > /etc/modprobe.d/lustre.conf
# Ensure the modules are loaded correctly (must happen after creating lustre.conf if not already loaded)
modprobe lnet || true
modprobe lustre || true
lnetctl lnet configure || true
lnetctl net add --net tcp --if bond1 2>/dev/null || lnetctl net add --net tcp --if eth1 || true
# Extract Lustre IP and create a static route via bond1's gateway
LUSTRE_IP=$(echo "$LUSTRE_MGS" | cut -d'@' -f1)
BOND1_GW=$(curl -sf -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/1/gateway)
if [ -n "$BOND1_GW" ] && [ -n "$LUSTRE_IP" ]; then
# Route the entire /16 containing the MGS to ensure OSTs are reachable
LUSTRE_CIDR="$(echo "$LUSTRE_IP" | cut -d'.' -f1-2).0.0/16"
echo "Adding static route for Lustre ($LUSTRE_CIDR) via bond1 gateway ($BOND1_GW)"
ip route add "$LUSTRE_CIDR" via "$BOND1_GW" dev bond1 || true
fi
mkdir -p /gstore
mount -t lustre "\$LUSTRE_MGS" /gstore || echo "Failed to mount /gstore"
# Mount to fresh path to bypass Kubelet hostPath caching
mkdir -p /lustre
mount -t lustre "\$LUSTRE_MGS" /lustre || echo "Failed to mount /lustre"
else
echo "No lustre-mgs metadata attribute found. Skipping Lustre mount."
fi
# Load networking modules for Calico (VXLAN / IPIP)
echo "=== Loading networking modules ==="
sudo modprobe vxlan || true
sudo modprobe ipip || true
echo "vxlan" | sudo tee /etc/modules-load.d/vxlan.conf
echo "ipip" | sudo tee /etc/modules-load.d/ipip.conf
###############################################################################
# [5] k8s worker setup (containerd + nvidia-ctk + kubeadm join)
###############################################################################
echo "=== [5] Configuring Kernel Modules for Kubernetes ==="
cat > /etc/modules-load.d/k8s.conf <<EOF
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter
cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
echo "=== [6] Installing containerd ==="
dnf install -y container-selinux 2>/dev/null || true
DOCKER_ARCH=\$(uname -m)
cat > /etc/yum.repos.d/docker-ce.repo <<REPOEOF
[docker-ce-stable]
name=Docker CE Stable - \text{\${DOCKER_ARCH}}
baseurl=https://download.docker.com/linux/rhel/9/\text{\${DOCKER_ARCH}}/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/rhel/gpg
REPOEOF
dnf install -y containerd.io
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
mkdir -p /data/containerd
sed -i "s|root = .*/var/lib/containerd.*|root = \"/data/containerd\"|" /etc/containerd/config.toml
systemctl enable --now containerd
echo "=== [7] Installing NVIDIA Container Toolkit ==="
curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
tee /etc/yum.repos.d/nvidia-container-toolkit.repo
dnf install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=containerd --set-as-default
systemctl restart containerd
echo "=== [8] Installing K8s packages (kubeadm/kubelet/kubectl) ==="
cat > /etc/yum.repos.d/kubernetes.repo <<REPOEOF
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v\${K8S_VERSION}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v\${K8S_VERSION}/rpm/repodata/repomd.xml.key
EOF
dnf install -y kubelet-\${K8S_PATCH} kubeadm-\${K8S_PATCH} kubectl-\${K8S_PATCH} conntrack-tools socat iptables ethtool
systemctl enable kubelet
# [8.5] Configure auto-labeling for Kubelet (Production recommended)
echo "=== [8.5] Configure Kubelet auto-labeling ==="
TOPO_HASH=\$(curl -sf -H "Metadata-Flavor: Google" \
"http://metadata.google.internal/computeMetadata/v1/instance/attributes/physical_host_topology" \
| md5sum | cut -c1-8 || echo "unknown")
if [ "\$TOPO_HASH" != "unknown" ]; then
mkdir -p /etc/systemd/system/kubelet.service.d
cat > /etc/default/kubelet <<KUBELET_EOF
KUBELET_EXTRA_ARGS=--node-labels=nvl72-domain=domain-\${TOPO_HASH}
KUBELET_EOF
systemctl daemon-reload
echo " Kubelet configured to auto-label with nvl72-domain=domain-\text{\${TOPO_HASH}}"
else
echo " WARNING: Failed to retrieve physical_host_topology from metadata"
fi
echo "=== [9] Kubeadm Join ==="
CP_IP=\$(curl -sf -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/attributes/cp-ip 2>/dev/null || true)
JOIN_TOKEN=\$(curl -sf -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/attributes/join-token 2>/dev/null || true)
JOIN_HASH=\$(curl -sf -H "Metadata-Flavor: Google" \
http://metadata.google.internal/computeMetadata/v1/instance/attributes/join-hash 2>/dev/null || true)
NODE_NAME=\$(cat /proc/sys/kernel/hostname 2>/dev/null || curl -sf -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/name || echo "unknown")
if [[ -n "\$CP_IP" && -n "\$JOIN_TOKEN" && -n "\$JOIN_HASH" ]]; then
echo " Joining cluster at \text{\${CP_IP}}:6443..."
kubeadm join \${CP_IP}:6443 \
--token "\${JOIN_TOKEN}" \
--discovery-token-ca-cert-hash "sha256:\${JOIN_HASH}" \
--node-name "\text{\${NODE_NAME}}" \
--ignore-preflight-errors=Hostname,SystemVerification 2>&1 || echo "kubeadm join failed, manual join required"
else
echo " Missing CP metadata parameters (cp-ip, join-token, join-hash). Skipping auto-join."
fi
touch "\$INIT_STAMP"
echo "=== Worker Initialization Complete ==="4.3 验证 Worker 加入与拓扑标签状态
等待 5-8 分钟使 Worker VM 硬件开机自检及系统启动,之后在 Master
节点上验证节点是否就绪,且是否自动携带了
nvl72-domain 物理拓扑标签:
# 登录 Master 检查节点 and 标签
gcloud compute ssh ${CP_NAME} --zone=${ZONE} --project=${PROJECT} \
--command="sudo kubectl get nodes --show-labels"期待输出所有 Worker 节点处于 Ready 状态,并且均有类似于
nvl72-domain=domain-a1b2c3d4 的标签。
Part 5: Kubernetes 拓扑与 GPU DRA 配置
5.1 部署 Scheduler DRA RBAC
由于默认的 K8s Scheduler 缺少 DRA 调度的必要权限,需要手动部署 RBAC:
gcloud compute ssh ${CP_NAME} --zone=${ZONE} --project=${PROJECT} <<'EOF'
cat <<'YAML' | sudo kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:kube-scheduler:dra
rules:
- apiGroups: ["resource.k8s.io"]
resources: ["resourceclaims"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["resource.k8s.io"]
resources: ["resourceclaims/status"]
verbs: ["get", "update", "patch"]
- apiGroups: ["resource.k8s.io"]
resources: ["resourceslices"]
verbs: ["get", "list", "watch"]
- apiGroups: ["resource.k8s.io"]
resources: ["deviceclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:kube-scheduler:dra
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-scheduler:dra
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: system:kube-scheduler
YAML
# 重启 Scheduler 载入权限
sudo kubectl delete pod -n kube-system -l component=kube-scheduler
sudo kubectl get pods -n kube-system -l component=kube-scheduler
EOF5.2 部署 NFD & GFD & GPU DRA Driver
这是 NVIDIA 官方组件,用于使 K8s 检测 GPU 的物理属性:
# 登录 Master 部署 NVIDIA 相关 Operator / Helm Chart
gcloud compute ssh ${CP_NAME} --zone=${ZONE} --project=${PROJECT} <<'EOF'
# 1. 部署 Node Feature Discovery (NFD)
sudo helm repo add nfd https://kubernetes-sigs.github.io/node-feature-discovery/charts || true
sudo helm repo update
sudo helm install nfd nfd/node-feature-discovery \
--namespace node-feature-discovery \
--create-namespace \
--set-string master.extraLabelNs={"nvidia.com"}
# 2. 部署 GPU DRA Driver & GFD
sudo helm repo add nvidia https://helm.ngc.nvidia.com/nvidia || true
sudo helm repo update
sudo helm install gpu-operator nvidia/gpu-operator \
--set nfd.enabled=false \
--namespace gpu-operator \
--create-namespace \
--set validator.repository=nvcr.io/nvidia \
--set driver.enabled=false \
--set toolkit.enabled=true \
--set toolkit.version=v1.16.2-centos7 \
--set toolkit.env[0].name=CONTAINERD_CONFIG \
--set toolkit.env[0].value=/etc/containerd/config.toml \
--set toolkit.env[1].name=CONTAINERD_SOCKET \
--set toolkit.env[1].value=/run/containerd/containerd.sock
# 检查 Operator 与 NFD pods 状态
sudo kubectl get pods -A | grep -E "gpu-operator|nfd"
EOF5.3 部署 ComputeDomain (MNNVL 通道)
我们需要为每个物理 Domain 部署一个 ComputeDomain
拓扑组。这里以 Domain 1 为例进行配置(生产部署时需对所有 14 个 Domain
重复此操作):
gcloud compute ssh ${CP_NAME} --zone=${ZONE} --project=${PROJECT} <<'EOF'
cat <<'YAML' | sudo kubectl apply -f -
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
name: domain-1-cd
spec:
numNodes: 18 # 满规模 NVL72 域为 18 节点
channel:
allocationMode: Single
resourceClaimTemplate:
name: domain-1-cd-channel
YAML
# 获取 UID 并标记 Domain 1 内部的 18 个节点属性
CD_UID=$(sudo kubectl get computedomain domain-1-cd -o jsonpath='{.metadata.uid}')
for i in $(seq 0 17); do
sudo kubectl label node ${WORKER_PREFIX}-d1-w${i} resource.nvidia.com/computeDomain=$CD_UID --overwrite
done
# 验证 ComputeDomain 状态
sudo kubectl get computedomain -o wide
EOF5.4 部署 DRANET 与 DeviceClass
最后安装 DRANET 网卡管理框架 and DeviceClass 设备类:
gcloud compute ssh ${CP_NAME} --zone=${ZONE} --project=${PROJECT} <<'EOF'
# 1. Helm 部署 DRANET
sudo helm install dranet oci://registry.k8s.io/networking/charts/dranet \
--version v1.3.0 -n kube-system
# 2. 创建 RDMA 设备选择类
cat <<'YAML' | sudo kubectl apply -f -
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
name: rdma-devices
spec:
selectors:
- cel:
expression: |
device.driver == "dra.net" &&
has(device.attributes["dra.net"].rdma) && device.attributes["dra.net"].rdma == true
YAML
# 验证资源状态
sudo kubectl get deviceclass
sudo kubectl get resourceslice -o wide
EOFPart 6: NCCL 测试与验证
本章节部署一个双节点的 NCCL 测试 JobSet,验证底层 MNNVL 网络是否连通,数据带宽是否达标。
6.1 部署 JobSet 控制器 与 Kueue
gcloud compute ssh ${CP_NAME} --zone=${ZONE} --project=${PROJECT} <<'EOF'
# 安装 Kueue 队列服务
sudo kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.18.0/manifests.yaml
# 安装 JobSet 容器编排
sudo kubectl apply --server-side -f https://github.com/kubernetes-sigs/jobset/releases/download/v0.12.0/manifests.yaml
# 等待 Controller Pods 处于 Running
sudo kubectl get pods -n kueue-system
sudo kubectl get pods -n jobset-system
EOF6.2 提交双节点 NCCL JobSet 任务
这会拉起一个跨两个物理节点的测试任务(通常选择同一个 Domain 内部的两个 VM 以测出 MNNVL 互联最大性能):
gcloud compute ssh ${CP_NAME} --zone=${ZONE} --project=${PROJECT} <<'EOF'
cat <<'YAML' | sudo kubectl apply -f -
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
name: nccl-test-2nodes
spec:
replicatedJobs:
- name: workers
replicas: 1
template:
spec:
parallelism: 2
completions: 2
backoffLimit: 0
template:
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
containers:
- name: nccl-runner
image: nvcr.io/nvidia/pytorch:24.01-py3
command:
- /bin/bash
- -c
- |
echo "Running NCCL Test..."
# 这里运行具体的 nccl-tests 命令,例如:
# /workspace/nccl-tests/build/all_reduce_perf -b 8 -e 128M -f 2 -g 4
sleep 3600
resources:
limits:
nvidia.com/gpu: 4
securityContext:
capabilities:
add: ["IPC_LOCK"]
YAML
# 观察任务 Pod 的拉起和运行日志
sudo kubectl get pods -o wide
EOF💡 带宽评估:同域内跨机 MNNVL 互联理论带宽应在 400GB/s 以上。若测得数据偏低,请检查 GSC backend 中物理拓扑 mapping 状态。
Part 7: 清理资源
测试完成后,请务必删除 VM 实例与网络置放策略,避免产生额外的预留收费:
# 1. 循环删除所有 Domain 的 VM 节点
for d in $(seq 1 $NUM_DOMAINS); do
echo "Deleting Domain $d workers..."
gcloud compute instances delete \
$(seq -f "${WORKER_PREFIX}-d${d}-w%g" 0 $((NODES_PER_DOMAIN - 1)) | tr '\n' ' ') \
--project=${PROJECT} --zone=${ZONE} --quiet
done
# 2. 删除 Control Plane 节点
gcloud compute instances delete ${CP_NAME} \
--project=${PROJECT} --zone=${ZONE} --quiet
# 3. 删除自定义防火墙规则
gcloud compute firewall-rules delete \
${CLUSTER_PREFIX}-allow-iap-ssh \
${CLUSTER_PREFIX}-allow-internal-gvnic0 \
${CLUSTER_PREFIX}-allow-internal-gvnic1 \
--project=${PROJECT} --quiet
# 4. 批量删除置放策略
for d in $(seq 1 $NUM_DOMAINS); do
POLICY_ID=$(printf "%04d" $d)
gcloud compute resource-policies delete ${PLACEMENT_PREFIX}-${POLICY_ID} \
--project=${PROJECT} --region=${REGION} --quiet
donePart 8: 验证测试总结报告 (2026-07-16)
本验证环节旨在测试单 Domain(64 GPU)及双 Domain 跨机架(128 GPU)的 K8s 节点自动化拉起及 NCCL 集合通信基准测试。
核心结论与问题记录
- BeyondCorp SSH 限制与应对方案
- 现象:在尝试通过 IAP 或原生 GCP SSH 自动登录
ivy-gb200-east-cp控制平面以提取 Kubeadm Token 时,触发了 Google BeyondCorp 的 Location Restriction 拦截,导致自动化脚本受阻。 - 解决方案:此限制属于预期内的安全策略。依赖具备对应位置权限的管理员手工通过内网跳板机(或有权限的终端)执行
kubeadm token create --print-join-command并人工注入部署变量中。已规避通过重启 CP 节点利用 Startup Script 获取可能引发的集群业务中断风险。
- 现象:在尝试通过 IAP 或原生 GCP SSH 自动登录
- Lustre 资源限制与旁路方案
- 现象:测试 Project 内无法开出 Lustre 实例,资源耗尽。
- 解决方案:本测试完全剥离了原指南中的 Lustre
强依赖。执行创建 Worker 节点的
gcloud beta compute instances bulk create命令时,去除了lustre-mgsmetadata 注入。事实证明对于纯粹的 NCCL MPIJob 并不影响带宽验证。
- 单/双域 NCCL MNNVL 拓扑调优 (关键结论)
- 单 Domain 测试 (64卡, 16节点):Worker 全部落在
domain-0001下,MPIJob 必须通过指定NCCL_MNNVL_ENABLE=2和NCCL_CUMEM_ENABLE=1来最大化利用域内 NVLink Switch 性能。 - 双 Domain 测试 (128卡, 32节点):Worker
跨域分布(例如16台在 0001,16台在 0002)。当您使用原生的
nccl-tests专门为了压测 RoCE 跨架物理带宽极限时,建议注入NCCL_MNNVL_ENABLE=0与NCCL_CUMEM_ENABLE=0强制禁用 NVLink。这样可以排除域内高速 NVLink 的干扰,得到纯粹的跨机架以太网打流数据。 > ⚠️ 生产框架排雷警告:禁用 MNNVL 仅仅是为了网络基准测试(Benchmark)的控制变量。在运行真实的业务大模型训练框架(例如 Megatron-LM)时,绝对不要注入NCCL_MNNVL_ENABLE=0!现代大模型训练框架底层的 NCCL 具备自动拓扑感知能力,能够智能地将通信密集的算子路由到同域 NVLink,跨域流量路由到 RDMA。强行关闭 MNNVL 会导致模型完全无法享受 1.8TB/s 的域内 NVLink 红利,整体吞吐性能将发生断崖式下跌。
- 单 Domain 测试 (64卡, 16节点):Worker 全部落在
沉淀输出物
已提取并修改了两份脱离 Lustre 强依赖的基准测试 YAML 供研发后续参考:
- 单域 64 GPU: nccl-mpi-64g-single-domain.yaml - 双域 128
GPU: nccl-mpi-128g-dual-domain.yaml
实际测试结果记录 (2026-07-16 实测)
- 单域 64 GPU (16 节点, Domain 1)
- 配置:
NCCL_MNNVL_ENABLE=2,NCCL_CUMEM_ENABLE=1(启用域内 NVLink) - 实测带宽:766.6 GB/s (Avg bus bandwidth)
- 结论:域内 NVLink 集合通信符合预期,性能优异。
- 配置:
- 双域 128 GPU (32 节点, Domain 1 + Domain 2)
- 配置:
NCCL_MNNVL_ENABLE=0,NCCL_CUMEM_ENABLE=0(仅用于 Benchmark 压测,强制走 RoCE/GIB。生产训练请勿添加此环境变量) - 实测带宽:158.8 GB/s (Avg bus bandwidth)
- 网络验证:通过 NCCL 调试日志确认成功加载
gIB插件,并使用bond2-bond5(RoCE) 进行跨域集合通信。 - 结论:跨域 RoCE 通信带宽达到理论限制的约 80% (单节点 4x 400Gbps NICs 约 200 GB/s 双向吞吐),符合预期。
- 配置:
常见问题与故障排除 (Troubleshooting)
问题:mpi-operator 反复崩溃或 apiserver
响应超时 如果集群变得不稳定,或者 mpi-operator
因为租约超时(lease timeouts)反复重启,这通常是由于安装了冗余的 NFD
(Node Feature Discovery) 插件引起的(例如 gpu-operator
默认自带的 NFD 与集群已有的 NFD 冲突)。冲突会导致数百个 worker pod 陷入
CrashLoopBackOff,并向 apiserver 发送大量垃圾请求,导致 API Server
瘫痪。 解决方案:请确保在部署 gpu-operator
时禁用 NFD:
helm upgrade gpu-operator nvidia/gpu-operator --namespace gpu-operator --reuse-values --set nfd.enabled=false