GB300 A4X-Max 裸金属手动验证操作规程
本指南用于在 GCP 测试项目 tencent-gcp-taiji 中,手动验证
GB300 (A4X-Max Bare Metal) 自建 Kubernetes 集群部署流程。
所有资源名称均已加上 hzchen-mval- 前缀以防冲突,默认规模为
18 个工作节点(共 72 张 GPU,使用 subblock-0001
物理子预留),以规避物理机架故障并节省 Quota。
目录导航
全局变量配置
请在您本地的 Linux 开发终端执行以下变量声明。后续所有命令均可直接复制粘贴执行:
# ======================== 项目与可用区 ========================
export PROJECT="tencent-gcp-taiji"
export REGION="us-central1"
export ZONE="us-central1-b"
# ======================== 验证规模与前缀 ========================
export CLUSTER_PREFIX="gb300-central"
export MACHINE_TYPE="a4x-maxgpu-4g-metal"
export TOTAL_GPUS=72
export GPUS_PER_NODE=4
export NODES_PER_DOMAIN=18 # 测试完整 18 节点 NVLink 域
export NUM_DOMAINS=1
# ======================== 共享物理预留与拓扑策略 ========================
# 强制绑定到空闲且健康的 subblock-0003 以防冲突(生产环境部署)
export CURRENT_RESERVATION="projects/${PROJECT}/reservations/nvidia-gb300-dxkhoz4ypk4mh/reservationBlocks/nvidia-gb300-dxkhoz4ypk4mh-block-0001/reservationSubBlocks/nvidia-gb300-dxkhoz4ypk4mh-block-0001-subblock-0001"
export PLACEMENT_POLICY="${CLUSTER_PREFIX}-nvl72-policy-0001"
export CP_NAME="${CLUSTER_PREFIX}-cp"
export WORKER_PREFIX="${CLUSTER_PREFIX}-worker"
# ======================== 系统镜像 ========================
export IMAGE="tlinux-server-4-gb300-v5dot4-ipv6"
export IMAGE_PROJECT="tencent-gcp-taiji"
# ======================== 管理网 VPC (NIC 0) ========================
export MGMT_NET="${CLUSTER_PREFIX}-mgmt-net"
export MGMT_SUB="${CLUSTER_PREFIX}-mgmt-sub"
export MGMT_CIDR="10.205.0.0/22"
# ======================== Lustre 存储网 VPC (NIC 1) ========================
export LUSTRE_NET="${CLUSTER_PREFIX}-lustre-net"
export LUSTRE_SUB="${CLUSTER_PREFIX}-lustre-sub"
export LUSTRE_CIDR="10.206.0.0/22"
# ======================== RDMA 计算网 VPC (NIC 2-9) ========================
export RDMA_NET="${CLUSTER_PREFIX}-rdma-net"
export RDMA_PROFILE="us-central1-b-vpc-roce-metal"Part 1: 网络基础设施部署 (必须)
1.1 创建 VPC 拓扑
创建三个独立的网络以满足不同的 MTU 与硬件底层 Profile:
# 1. 创建控制管理网络 (MTU 1500)
gcloud compute --project=${PROJECT} \
networks create ${MGMT_NET} \
--subnet-mode=custom --mtu=1500
# 2. 创建 Lustre 存储网络 (MTU 8896)
gcloud compute --project=${PROJECT} \
networks create ${LUSTRE_NET} \
--subnet-mode=custom --mtu=8896
# 3. 创建 RDMA 计算网络 (关联 vpc-roce 硬件 Profile)
gcloud beta compute --project=${PROJECT} \
networks create ${RDMA_NET} \
--network-profile=${RDMA_PROFILE} \
--subnet-mode=custom1.2 创建子网
# 创建控制管理网子网 (IPv4)
gcloud compute --project=${PROJECT} \
networks subnets create ${MGMT_SUB} \
--network=${MGMT_NET} \
--region=${REGION} \
--range=${MGMT_CIDR}
# 创建 Lustre 存储网子网 (IPv4)
gcloud compute --project=${PROJECT} \
networks subnets create ${LUSTRE_SUB} \
--network=${LUSTRE_NET} \
--region=${REGION} \
--range=${LUSTRE_CIDR}💡 RDMA 子网说明:RDMA 计算网底层关联了
vpc-roceProfile,GCP 会自动在us-central1-b下创建名为default-subnet-1-${RDMA_NET}的全局 IPv6-Only 子网,无需人工执行子网划分命令。您可以运行以下命令进行验证:gcloud compute networks subnets list --network=${RDMA_NET} --project=${PROJECT}
1.3 创建防火墙安全规则
# 允许 Google IAP 隧道登录 (必须)
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-iap-ssh \
--project=${PROJECT} --network=${MGMT_NET} \
--direction=INGRESS --priority=900 --action=ALLOW \
--rules=tcp:22,tcp:56000 --source-ranges=35.235.240.0/20
# 允许管理网段内部全端口全协议互通
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-internal-mgmt \
--project=${PROJECT} --network=${MGMT_NET} \
--direction=INGRESS --priority=1000 --action=ALLOW \
--rules=all --source-ranges=${MGMT_CIDR}
# 允许 Lustre 网段内部全端口全协议互通
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-internal-lustre \
--project=${PROJECT} --network=${LUSTRE_NET} \
--direction=INGRESS --priority=1000 --action=ALLOW \
--rules=all --source-ranges=${LUSTRE_CIDR}1.4 创建 Placement Policy (物理亲和性策略)
为我们的验证 Domain 创建一个拓扑组:
gcloud beta compute resource-policies create group-placement \
${PLACEMENT_POLICY} \
--collocation=collocated \
--gpu-topology=1x72 \
--project=${PROJECT} --region=${REGION}Part 2: 创建并配置 Lustre 存储实例 (可选)
如果您需要在此轮验证中挂载 Lustre,请执行本章节;如只需测试 NCCL,可跳过此章节。
2.1 建立 VPC 专用服务连接 (Private Service Access)
# 1. 申请用于 VPC Peering 的 IP 地址段
gcloud compute addresses create ${CLUSTER_PREFIX}-lustre-ip-range \
--global \
--purpose=VPC_PEERING \
--prefix-length=20 \
--description="Managed Lustre VPC Peering" \
--network=${LUSTRE_NET} \
--project=${PROJECT}
# 2. 建立私有服务对接
gcloud services vpc-peerings connect \
--network=${LUSTRE_NET} \
--ranges=${CLUSTER_PREFIX}-lustre-ip-range \
--service=servicenetworking.googleapis.com \
--project=${PROJECT}
# 3. 为 Lustre 对接端创建防火墙放行规则
PEERING_CIDR=$(gcloud compute addresses describe ${CLUSTER_PREFIX}-lustre-ip-range --global --project=${PROJECT} --format="value[separator=/](address, prefixLength)")
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-lustre-peering \
--project=${PROJECT} \
--network=${LUSTRE_NET} \
--allow=tcp:988,tcp:6988 \
--source-ranges=$PEERING_CIDR2.2 创建 Lustre 实例
gcloud lustre instances create ${CLUSTER_PREFIX}-lustre \
--project=${PROJECT} \
--location=${ZONE} \
--network=projects/${PROJECT}/global/networks/${LUSTRE_NET} \
--capacity-gib=72000 \
--per-unit-storage-throughput=125 \
--filesystem=lustrefs💡 获取 Lustre NID:创建可能需要 10-15 分钟。完成后运行:
gcloud lustre instances describe ${CLUSTER_PREFIX}-lustre --location=${ZONE} --project=${PROJECT} --format="value(mountPoint)"记下输出的 NID 地址(格式为10.X.X.X@tcp:/lustrefs),后续需要作为 metadata 传入 Worker。
Part 3: Control Plane (Master) 节点部署 (必须)
3.1 创建 Master VM
Master VM 采用标准的 n2-standard-16
虚拟机,无需绑定物理预留或 Placement Policy:
gcloud compute instances create ${CP_NAME} \
--project=${PROJECT} \
--zone=${ZONE} \
--machine-type="n2-standard-16" \
--image-family="rocky-linux-9" \
--image-project="rocky-linux-cloud" \
--boot-disk-type="pd-ssd" \
--boot-disk-size="200GB" \
--network-interface=network=${MGMT_NET},subnet=${MGMT_SUB} \
--network-interface=network=${LUSTRE_NET},subnet=${LUSTRE_SUB} \
--metadata-from-file=startup-script=scripts/hzchen-mval-master-init.sh3.2 提取 K8s Control Plane 配置参数
创建后,VM 会通过 startup-script 自动安装 K8s 控制面并预加载 Calico。稍等 2-3 分钟,通过 SSH 登录 Master 节点,获取 K8s Join 命令及凭证。
⚠️ 关于 IAP Tunnel 4033 授权报错与直连 SSH 绕行方案: 如果您在使用
gcloud compute ssh --tunnel-through-iap登录时遇到如下报错:ConnectionCreationError: Error while connecting [4033: 'not authorized']这是因为您的 Google 账号在tencent-gcp-taiji项目中缺少roles/iap.tunnelResourceAccessor角色。绕行解决方案: 1. 在本地终端放行您当前的 Cloudtop 公网 IP 入站:
bash gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-ssh-direct \ --project=${PROJECT} --network=${MGMT_NET} \ --direction=INGRESS --priority=950 --action=ALLOW \ --rules=tcp:22 --source-ranges=104.199.159.39,35.201.140.1612. 获取 Master 的公网 IP:bash export CP_EXTERNAL_IP=$(gcloud compute instances describe ${CP_NAME} --zone=${ZONE} --project=${PROJECT} --format="value(networkInterfaces[0].accessConfigs[0].natIP)") echo "Master Public IP: ${CP_EXTERNAL_IP}"3. 使用原生 SSH 直连公网 IP,并显式禁用 ProxyCommand 绕过 BeyondCorp 拦截:bash ssh -o ProxyCommand=none -i ~/.ssh/google_compute_engine -o StrictHostKeyChecking=no hzchen@${CP_EXTERNAL_IP}
登录后,在 Master 上查看自动生成的 kubeadm 令牌:
cat /root/kubeadm-join-command.txt
# 预期输出格式: kubeadm join 10.205.0.2:6443 --token <TOKEN> --discovery-token-ca-cert-hash sha256:<HASH>请记录 Master 的内网 IP(即
10.205.0.2)、<TOKEN> 与
<HASH> 串,填入后续 Worker 创建的变量中。
Part 4: Worker 节点部署 (必须)
4.1 声明 Join 参数与 Lustre 变量
请在您的本地终端声明您在 Step 3.2 中获取的值:
# 修改为您在 Master 节点上获取的实际值
export CP_IP="10.205.0.2"
export JOIN_TOKEN="xxxxxx.xxxxxxxxxxxxxxxx"
export JOIN_HASH="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
# 如果 Part 2 创建了 Lustre,请修改为 Lustre Mount NID 串,否则留空:
export LUSTRE_MGS_NID=""
# 💡 拓扑策略绑定:为了保留完整的物理亲和性验证(推荐),
# 我们需要使用项目中已经与 `subblock-0003` 绑定的 Placement Policy,而不是新建 policy。
# 底层预留 Subblock 与 Policy 的映射关系为 1对1 对应(Subblock-0003 对应 policy-0003)。
# 如果您在创建时遇到故障节点需要临时绕过拓扑审查,可以将其设为空 `""`。
export PLACEMENT_POLICY="${CLUSTER_PREFIX}-nvl72-policy-0001"4.2 创建 Worker 虚拟机
⚠️ 重要提示 (Placement Policy 限制与 16/18 容灾 SLA): 1. 严禁绕过 Placement Policy:根据 GCP 官方物理限制,GB200/GB300 必须以 1x72 拓扑(18个节点)为最小单元绑定 Placement Policy 进行资源调度。如果在同一个 sub-block(NVLink 域)内绕过 policy 零散地创建虚拟机,会导致整台 NVL72 机架发生物理碎片化锁定,后续将彻底无法再用于完整的 1x72 拓扑调度。 2. 为什么是
--min-count=16:GCP 官方针对 A4X/A4X-Max 机架的 SLA 可用性承诺为 16/18。这意味着在由 18 个计算节点组成的 Domain 中,GCP 允许最多存在 2 个状态为degraded hosts(因硬件故障排队等待维修更换)的节点。为了确保即使物理机架中存在故障节点,您的集群依然能成功拉起可用的机器,在使用bulk create批量申请时,必须将参数设置为--count=18且--min-count=16。如果您强行设置--min-count=18,一旦该机架内有 1 台机器报修,整个创建命令都会因无法满足约束而彻底被拒!这也是为什么主流模型框架都在适配 64 + 8 Spare (容灾备用) 队列机制的根本硬件原因。 3. 故障节点生命周期与恢复机制: * 16/18 容灾补充创建:如果在首次执行bulk create时由于有部分degraded hosts而只成功拉起了 16 台虚拟机,那么缺失的 2 台在云端是不存在的。当底层物理机修复后,您必须重新执行一次 bulk create(只传入缺失的 2 台 VM 名称,如count=2),并且必须指定与前 16 台完全相同的 Placement Policy 与 Subblock,否则将引发PLACEMENT_POLICY_CONSTRAINTS_INFEASIBLE碎片化报错。 * 运行中节点突发硬件故障:如果您的 18 台虚拟机已经RUNNING,后来有 2 台发生硬件故障变为degraded hosts,由于 GPU/裸金属不支持热迁移(Live Migration),因此这两台虚拟机会被强制终止 (TERMINATED)(由命令中的--on-host-maintenance=TERMINATE决定)。但是,在命令默认开启的--restart-on-failure保护下,一旦底层硬件被 GCP 修复完毕,控制平面会自动 (Automatically) 将这两台机器重新拉起并恢复为RUNNING状态,您不需要额外干预或执行任何命令。
批量创建整组 18 台虚拟机
必须使用以下命令一次性创建整组 (18台) 虚拟机,并严格启用 Resource Policy。
# 生成 18 台 VM 的名称列表
NAMES=""
for i in {0..17}; do
if [ -z "$NAMES" ]; then
NAMES="${WORKER_PREFIX}-w${i}"
else
NAMES="${NAMES},${WORKER_PREFIX}-w${i}"
fi
done
# 使用 bulk create 一次性原子化申请 18 台虚拟机
gcloud beta compute instances bulk create \
--predefined-names=${NAMES} \
--count=18 \
--min-count=16 \
--project=${PROJECT} \
--zone=${ZONE} \
--machine-type=${MACHINE_TYPE} \
--image=${IMAGE} \
--image-project=${IMAGE_PROJECT} \
--boot-disk-type=hyperdisk-balanced \
--boot-disk-size=1000GB \
--scopes=cloud-platform \
--reservation-affinity=specific \
--reservation=${CURRENT_RESERVATION} \
--provisioning-model=RESERVATION_BOUND \
--on-host-maintenance=TERMINATE \
--restart-on-failure \
--resource-policies=${PLACEMENT_POLICY} \
--metadata=serial-port-enable=1,cp-ip=${CP_IP},join-token=${JOIN_TOKEN},join-hash=${JOIN_HASH} \
--metadata-from-file=startup-script=scripts/hzchen-mval-worker-init.sh \
--network-interface=subnet=${MGMT_SUB},nic-type=IDPF \
--network-interface=subnet=${LUSTRE_SUB},nic-type=IDPF,no-address \
--network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
--network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
--network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
--network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
--network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
--network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
--network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
--network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA4.3 验证 Worker 初始化与 K8s 入群状态
# 检查 VM 开机状态与内网 IP
gcloud compute instances list --project=${PROJECT} --filter="name~${WORKER_PREFIX}-d"
# SSH 登录 Master 查看 K8s 节点加入情况
gcloud compute ssh ${CP_NAME} --zone=${ZONE} --tunnel-through-iap --command="kubectl get nodes -o wide"💡 节点就绪可能需要 3-4 分钟。如果状态为
Ready且 K8s 识别到正确的 OS 与 IP,则说明加入成功。
Part 5: Kubernetes 基础资源与硬件驱动部署 (必须)
请通过 SSH 登录到 Master 节点上执行后续所有 Kubernetes 控制面操作。
gcloud compute ssh ${CP_NAME} --zone=${ZONE} --tunnel-through-iap5.1 对 Worker 节点打标
# 手动将节点归类到同一 NVLink Domain 1 下,并标记为裸金属类型
for i in {0..17}; do
kubectl label node hzchen-mval-worker-d1-w${i} nvl72-domain=domain-1 --overwrite
kubectl label node hzchen-mval-worker-d1-w${i} node-type=a4x-max --overwrite
done5.2 配置 Scheduler DRA RBAC 权限
cat <<'EOF' | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:kube-scheduler:dra
rules:
- apiGroups: ["resource.k8s.io"]
resources: ["resourceclaims"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["resource.k8s.io"]
resources: ["resourceclaims/status"]
verbs: ["get", "update", "patch"]
- apiGroups: ["resource.k8s.io"]
resources: ["resourceslices"]
verbs: ["get", "list", "watch"]
- apiGroups: ["resource.k8s.io"]
resources: ["deviceclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:kube-scheduler:dra
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-scheduler:dra
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: system:kube-scheduler
EOF
# 重启 Scheduler 强制使权限生效
kubectl delete pod -n kube-system -l component=kube-scheduler5.3 部署 asapd-lite
(裸金属网络管理)
参考信息:GB300 裸金属实例依赖
asapd-liteDaemonSet 来接管管理网卡之外的高速网络设备(CX8 物理网卡)。官方维护的 COS 版本 yaml 模板请参考: asapd-lite-installer-a4x-max-bm-cos.yaml
# 部署 TLinux/Rocky 适配版 asapd-lite 安装文件
kubectl apply -f https://gcp.totorochina.com/asapd-lite-tlinux.yaml
# 验证 Pod 处于 Running 状态
kubectl get ds -n kube-system asapd-lite💡 注:为了下载 asapd-lite-tlinux.yaml,我们已在 Totoro 网站上传了该 YAML 的静态映射版本。
5.4 部署 NVIDIA GPU 驱动与 DRA 算力控制面
# 1. 部署 NVIDIA Device Plugin 限制 nodeSelector 为我们打过标的节点
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update
helm install nvidia-device-plugin nvdp/nvidia-device-plugin \
--namespace gpu-operator \
--create-namespace \
--set-string nodeSelector."nvidia\.com/gpu\.present"=true \
--wait --timeout 300s
# 2. 部署 K8s DRA Driver (版本对齐 v25.12.0)
helm repo add nvidia-dra https://nvidia.github.io/k8s-dra-driver
helm repo update
helm install nvidia-dra-driver-gpu nvidia-dra/nvidia-dra-driver-gpu \
--namespace gpu-operator \
--version 25.12.0 \
--set gpuResourcesEnabledOverride=true \
--wait --timeout 300s5.5 部署 DRANET 网络分配器
helm install dranet oci://registry.k8s.io/networking/charts/dranet \
--version v1.3.0 \
-n kube-system
# 声明并注册 DeviceClass 资源,允许 Pod 挂载 MRDMA 设备文件
cat << 'EOF' | kubectl apply -f -
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
name: rdma-devices
spec:
selectors:
- cel:
expression: |
device.driver == "dra.net" &&
has(device.attributes["dra.net"].rdma) && device.attributes["dra.net"].rdma == true
EOF5.6 注册 ComputeDomain 物理互联域
向控制器注册机架内 1x72 NVLink Clique,建立 NVLink 通道并提取 K8s UID 写入节点:
# 1. 创建 ComputeDomain
cat <<EOF | kubectl apply -f -
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
name: domain-1-cd
spec:
numNodes: 0
channel:
allocationMode: Single
resourceClaimTemplate:
name: domain-1-cd-channel
EOF
# 2. 提取 UID
CD_UID=$(kubectl get computedomain domain-1-cd -o jsonpath='{.metadata.uid}')
echo "ComputeDomain UID: $CD_UID"
# 3. 将 UID 打标至两台节点,激活多机 GPU 组播拓扑
for i in {0..17}; do
kubectl label node hzchen-mval-worker-d1-w${i} \
resource.nvidia.com/computeDomain=$CD_UID --overwrite
done
# 验证守护进程状态 (应显示 2 个 healthy 实例)
kubectl get pods -n gpu-operator -l app=computedomain-daemon5.7 安装 Kubeflow MPI Operator
kubectl apply --server-side -f \
https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.8.0/deploy/v2beta1/mpi-operator.yamlPart 6: 双机 MPIJob NCCL AllReduce 测试 (可选)
6.1 创建 ResourceClaimTemplate
cat <<'EOF' | kubectl apply -f -
# 1. GPU 申请模板
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
name: gpu-devices-mpi-sd
spec:
spec:
devices:
requests:
- name: gpus
exactly:
deviceClassName: gpu.nvidia.com
count: 4
---
# 2. MRDMA 8 网卡申请模板
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
name: rdma-nics-mpi-sd
spec:
spec:
devices:
requests:
- name: rdma-nics
exactly:
deviceClassName: rdma-devices
count: 8
EOF6.2 注册 MRDMA 静态路由 ConfigMap
由于 CNI 无法在三层环境中直接寻址,该脚本负责通过 metadata 服务将网卡关联的 IPv6 路由直接写入容器网关中:
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
name: setup-rdma-ips-script
namespace: default
data:
setup_rdma_ips.py: |
import json
import urllib.request
import subprocess
import os
import glob
import sys
def get_metadata():
req = urllib.request.Request("http://169.254.169.254/computeMetadata/v1/instance/network-interfaces/?recursive=true")
req.add_header("Metadata-Flavor", "Google")
try:
with urllib.request.urlopen(req) as response:
return json.loads(response.read().decode())
except Exception as e:
print(f"Failed to query metadata: {e}")
return None
def get_local_interfaces():
mac_map = {}
for path in glob.glob("/sys/class/net/gpu*/address"):
ifname = path.split("/")[-2]
try:
with open(path, "r") as f:
mac = f.read().strip().lower()
mac_map[mac] = ifname
except Exception as e:
print(f"Failed to read MAC for {ifname}: {e}")
return mac_map
def parse_ipv6_prefix_56(ip_str):
parts = ip_str.split(":")
if "" in parts:
idx = parts.index("")
missing = 8 - len(parts) + 1
parts = parts[:idx] + ["0"] * missing + parts[idx+1:]
block4 = parts[3]
block4_int = int(block4, 16)
block4_56 = (block4_int & 0xFF00)
parts[3] = f"{block4_56:x}"
return ":".join(parts[:4]) + "::/56"
def main():
metadata = get_metadata()
if not metadata:
sys.exit(1)
local_interfaces = get_local_interfaces()
gateway_v6 = "fe80::1"
for nic in metadata:
if nic.get("nicType") != "MRDMA":
continue
mac = nic.get("mac", "").lower()
ipv6_list = nic.get("ipv6s", [])
if not ipv6_list:
ipv6_list = nic.get("ipv6", [])
if not ipv6_list:
print(f"No IPv6 allocated for MRDMA interface with MAC {mac}")
continue
ipv6_addr = ipv6_list[0]
if "/" not in ipv6_addr:
ipv6_with_prefix = f"{ipv6_addr}/128"
else:
ipv6_with_prefix = ipv6_addr
ipv6_addr = ipv6_addr.split("/")[0]
route_target = parse_ipv6_prefix_56(ipv6_addr)
ifname = local_interfaces.get(mac)
if not ifname:
print(f"Could not find local interface for MAC {mac}")
continue
print(f"Configuring container NIC {ifname} ({mac}) with IP {ipv6_with_prefix}, route {route_target} via {gateway_v6}")
try:
subprocess.run(["ip", "addr", "flush", "dev", ifname], check=True)
subprocess.run(["ip", "addr", "add", ipv6_with_prefix, "dev", ifname], check=True)
subprocess.run(["ip", "link", "set", "dev", ifname, "up"], check=True)
subprocess.run(["ip", "-6", "route", "replace", route_target, "via", gateway_v6, "dev", ifname], check=True)
print(f"Successfully configured {ifname}")
except Exception as e:
print(f"Failed to configure interface {ifname}: {e}")
if __name__ == "__main__":
main()
EOF6.3 运行 MPIJob 压力测试 Pod
该 YAML 将向 K8s 申领 2 台节点上共计 8 张 GPU,通过 Containerd 动态分配物理设备。
cat <<'EOF' | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-reader
namespace: default
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-pods
namespace: default
subjects:
- kind: ServiceAccount
name: default
namespace: default
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
---
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
name: nccl-mpi-sd
namespace: default
spec:
slotsPerWorker: 4
runPolicy:
cleanPodPolicy: Running
sshAuthMountPath: /root/.ssh
launcherCreationPolicy: WaitForWorkersReady
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
nodeSelector:
nvl72-domain: domain-1
initContainers:
- name: gib-installer
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
command: ["/bin/sh", "-c"]
args:
- |
set -e
/scripts/container_entry.sh install --install-nccl
cp -a /usr/local/gib/. /target/gib/
cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
mkdir -p /target/gib/lib64/libibverbs
cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
volumeMounts:
- name: gib
mountPath: /target/gib
containers:
- name: launcher
image: nvcr.io/nvidia/pytorch:26.05-py3
imagePullPolicy: IfNotPresent
env:
- name: LD_LIBRARY_PATH
value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
command:
- bash
- -c
- |
set -e
echo "Resolving worker IPs via K8s ServiceAccount API..."
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
API="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"
for i in $(seq 0 $((REPLICAS-1))); do
WORKER_NAME="nccl-mpi-sd-worker-${i}"
for attempt in $(seq 1 60); do
WORKER_IP=$(curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" \
"$API/api/v1/namespaces/default/pods/$WORKER_NAME" 2>/dev/null \
| python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",{}).get("podIP",""))' 2>/dev/null || echo "")
if [ -n "$WORKER_IP" ]; then
echo "$WORKER_NAME IP: $WORKER_IP"
# 注册短名和完整 FQDN 以解决 DNS 解析边界问题
echo "$WORKER_IP $WORKER_NAME $WORKER_NAME.default.svc.cluster.local" >> /etc/hosts
break
fi
echo "Waiting for $WORKER_NAME IP (attempt $attempt)..."
sleep 5
done
if [ -z "$WORKER_IP" ]; then
echo "ERROR: Could not resolve $WORKER_NAME IP"
exit 1
fi
done
apt-get update -qq && apt-get install -y -qq netcat-openbsd > /dev/null 2>&1
echo "Waiting for workers SSH port to open..."
for i in $(seq 0 $((REPLICAS-1))); do
WORKER_NAME="nccl-mpi-sd-worker-${i}"
until nc -z $WORKER_NAME 22; do
echo "Waiting for SSH on $WORKER_NAME..."
sleep 3
done
echo "SSH on $WORKER_NAME is active!"
done
echo "Starting MPI AllReduce benchmark (8 Ranks total, 4 per node)..."
/usr/local/mpi/bin/mpirun --allow-run-as-root \
-np 8 -npernode 4 \
--mca pml ob1 \
--mca btl tcp,self \
--mca btl_tcp_if_include eth0 \
--mca coll_ucc_enable 0 \
-x LD_LIBRARY_PATH=/usr/local/gib/lib64:/usr/local/nvidia/lib64 \
-x NCCL_NET_PLUGIN=/usr/local/gib/lib64/libnccl-net.so \
-x NCCL_ENV_PLUGIN=gcp \
-x NCCL_MNNVL_ENABLE=2 \
-x NCCL_CUMEM_ENABLE=1 \
-x NCCL_SOCKET_IFNAME=eth0 \
-x UCX_NET_DEVICES=eth0 \
-x NCCL_DEBUG=INFO \
--mca plm_rsh_args '-o StrictHostKeyChecking=no' \
/tmp/nccl-tests/build/all_reduce_perf -b 512M -e 8G -f 2 -g 1
volumeMounts:
- name: gib
mountPath: /usr/local/gib
resources:
requests:
cpu: "4"
memory: 16Gi
limits:
cpu: "4"
memory: 16Gi
volumes:
- name: gib
emptyDir: {}
Worker:
replicas: 2
template:
spec:
nodeSelector:
nvl72-domain: domain-1
resourceClaims:
- name: compute-domain-channel
resourceClaimTemplateName: domain-1-cd-channel
- name: rdma-nics
resourceClaimTemplateName: rdma-nics-mpi-sd
- name: gpu-devices
resourceClaimTemplateName: gpu-devices-mpi-sd
initContainers:
- name: gib-installer
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
command: ["/bin/sh", "-c"]
args:
- |
set -e
/scripts/container_entry.sh install --install-nccl
cp -a /usr/local/gib/. /target/gib/
cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
mkdir -p /target/gib/lib64/libibverbs
cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
volumeMounts:
- name: gib
mountPath: /target/gib
containers:
- name: worker
image: nvcr.io/nvidia/pytorch:26.05-py3
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
env:
- name: LD_LIBRARY_PATH
value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
- name: NCCL_NET_PLUGIN
value: /usr/local/gib/lib64/libnccl-net.so
- name: NCCL_ENV_PLUGIN
value: gcp
command:
- bash
- -c
- |
apt-get update -qq && apt-get install -y -qq openssh-server iproute2 > /dev/null 2>&1
mkdir -p /run/sshd
ssh-keygen -A
printf 'Port 22\nPermitRootLogin yes\nPubkeyAuthentication yes\nAuthorizedKeysFile /root/.ssh/authorized_keys\nStrictModes no\nPasswordAuthentication no\nUsePAM no\nPidFile /run/sshd/sshd.pid\n' > /etc/ssh/mpi_sshd_config
python3 /scripts/setup_rdma_ips.py
# 带重试机制的 git clone,防止由于 Github 网络波动导致 Worker 拉起失败
cd /tmp
for i in $(seq 1 30); do
git clone --depth 1 https://github.com/NVIDIA/nccl-tests.git && break || sleep 5
done
cd /tmp/nccl-tests && make -j$(nproc) MPI=1 MPI_HOME=/usr/local/mpi NCCL_HOME=/usr/local/gib CUDA_HOME=/usr/local/cuda CXXFLAGS="-g" CFLAGS="-g"
/usr/sbin/sshd -De -f /etc/ssh/mpi_sshd_config
volumeMounts:
- name: gib
mountPath: /usr/local/gib
- name: dshm
mountPath: /dev/shm
- name: setup-script
mountPath: /scripts
resources:
requests:
cpu: "96"
memory: 800Gi
limits:
cpu: "96"
memory: 800Gi
claims:
- name: gpu-devices
- name: rdma-nics
- name: compute-domain-channel
volumes:
- name: gib
emptyDir: {}
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 250Gi
- name: setup-script
configMap:
name: setup-rdma-ips-script
EOF6.4 查看测试结果
# 实时跟踪 Launcher 日志
kubectl logs -f -l training.kubeflow.org/job-role=launcher预期输出指标: 同域(Same-Domain,走机架内多节点
NVLink,即 MNNVL 协议)双机 all_reduce 带宽指标: - Avg bus
bandwidth 应该在 295+ GB/s
以上。
Part 7: 实验后清理
清理测试所占资源,释放 CPU / GPU Quotas:
# 1. 删除虚拟机
gcloud compute instances delete ${NAMES} \
--project=${PROJECT} --zone=${ZONE} --quiet
gcloud compute instances delete ${CP_NAME} \
--project=${PROJECT} --zone=${ZONE} --quiet
# 2. 删除 Placement Policy
gcloud compute resource-policies delete ${PLACEMENT_POLICY} \
--project=${PROJECT} --region=${REGION} --quiet
# 3. 删除专用网络 (自下而上)
# 删除防火墙规则
gcloud compute firewall-rules delete \
${CLUSTER_PREFIX}-allow-iap-ssh \
${CLUSTER_PREFIX}-allow-internal-mgmt \
${CLUSTER_PREFIX}-allow-internal-lustre \
--project=${PROJECT} --quiet
# 删除子网
gcloud compute networks subnets delete ${MGMT_SUB} --project=${PROJECT} --region=${REGION} --quiet
gcloud compute networks subnets delete ${LUSTRE_SUB} --project=${PROJECT} --region=${REGION} --quiet
# 删除 VPC 网络
gcloud compute networks delete ${MGMT_NET} --project=${PROJECT} --quiet
gcloud compute networks delete ${LUSTRE_NET} --project=${PROJECT} --quiet
gcloud compute networks delete ${RDMA_NET} --project=${PROJECT} --quiet
# 4. 删除 Lustre 实例 (如果创建了)
gcloud lustre instances delete ${CLUSTER_PREFIX}-lustre \
--location=${ZONE} --project=${PROJECT} --quiet常见问题与故障排除 (Troubleshooting)
问题:mpi-operator 反复崩溃或 apiserver
响应超时 如果集群变得不稳定,或者 mpi-operator
因为租约超时(lease timeouts)反复重启,这通常是由于安装了冗余的 NFD
(Node Feature Discovery) 插件引起的(例如 gpu-operator
默认自带的 NFD 与集群已有的 NFD 冲突)。冲突会导致数百个 worker pod 陷入
CrashLoopBackOff,并向 apiserver 发送大量垃圾请求,导致 API Server
瘫痪。 解决方案:请确保在部署 gpu-operator
时禁用 NFD:
helm upgrade gpu-operator nvidia/gpu-operator --namespace gpu-operator --reuse-values --set nfd.enabled=false