GB300 A4X-Max 裸金属手动验证操作规程

本指南用于在 GCP 测试项目 tencent-gcp-taiji 中,手动验证 GB300 (A4X-Max Bare Metal) 自建 Kubernetes 集群部署流程。 所有资源名称均已加上 hzchen-mval- 前缀以防冲突,默认规模为 18 个工作节点(共 72 张 GPU,使用 subblock-0001 物理子预留),以规避物理机架故障并节省 Quota。


目录导航


全局变量配置

请在您本地的 Linux 开发终端执行以下变量声明。后续所有命令均可直接复制粘贴执行:

# ======================== 项目与可用区 ========================
export PROJECT="tencent-gcp-taiji"
export REGION="us-central1"
export ZONE="us-central1-b"

# ======================== 验证规模与前缀 ========================
export CLUSTER_PREFIX="gb300-central"
export MACHINE_TYPE="a4x-maxgpu-4g-metal"
export TOTAL_GPUS=72
export GPUS_PER_NODE=4
export NODES_PER_DOMAIN=18                     # 测试完整 18 节点 NVLink 域
export NUM_DOMAINS=1

# ======================== 共享物理预留与拓扑策略 ========================
# 强制绑定到空闲且健康的 subblock-0003 以防冲突(生产环境部署)
export CURRENT_RESERVATION="projects/${PROJECT}/reservations/nvidia-gb300-dxkhoz4ypk4mh/reservationBlocks/nvidia-gb300-dxkhoz4ypk4mh-block-0001/reservationSubBlocks/nvidia-gb300-dxkhoz4ypk4mh-block-0001-subblock-0001"
export PLACEMENT_POLICY="${CLUSTER_PREFIX}-nvl72-policy-0001"

export CP_NAME="${CLUSTER_PREFIX}-cp"
export WORKER_PREFIX="${CLUSTER_PREFIX}-worker"

# ======================== 系统镜像 ========================
export IMAGE="tlinux-server-4-gb300-v5dot4-ipv6"
export IMAGE_PROJECT="tencent-gcp-taiji"

# ======================== 管理网 VPC (NIC 0) ========================
export MGMT_NET="${CLUSTER_PREFIX}-mgmt-net"
export MGMT_SUB="${CLUSTER_PREFIX}-mgmt-sub"
export MGMT_CIDR="10.205.0.0/22"

# ======================== Lustre 存储网 VPC (NIC 1) ========================
export LUSTRE_NET="${CLUSTER_PREFIX}-lustre-net"
export LUSTRE_SUB="${CLUSTER_PREFIX}-lustre-sub"
export LUSTRE_CIDR="10.206.0.0/22"

# ======================== RDMA 计算网 VPC (NIC 2-9) ========================
export RDMA_NET="${CLUSTER_PREFIX}-rdma-net"
export RDMA_PROFILE="us-central1-b-vpc-roce-metal"

Part 1: 网络基础设施部署 (必须)

1.1 创建 VPC 拓扑

创建三个独立的网络以满足不同的 MTU 与硬件底层 Profile:

# 1. 创建控制管理网络 (MTU 1500)
gcloud compute --project=${PROJECT} \
  networks create ${MGMT_NET} \
  --subnet-mode=custom --mtu=1500

# 2. 创建 Lustre 存储网络 (MTU 8896)
gcloud compute --project=${PROJECT} \
  networks create ${LUSTRE_NET} \
  --subnet-mode=custom --mtu=8896

# 3. 创建 RDMA 计算网络 (关联 vpc-roce 硬件 Profile)
gcloud beta compute --project=${PROJECT} \
  networks create ${RDMA_NET} \
  --network-profile=${RDMA_PROFILE} \
  --subnet-mode=custom

1.2 创建子网

# 创建控制管理网子网 (IPv4)
gcloud compute --project=${PROJECT} \
  networks subnets create ${MGMT_SUB} \
  --network=${MGMT_NET} \
  --region=${REGION} \
  --range=${MGMT_CIDR}

# 创建 Lustre 存储网子网 (IPv4)
gcloud compute --project=${PROJECT} \
  networks subnets create ${LUSTRE_SUB} \
  --network=${LUSTRE_NET} \
  --region=${REGION} \
  --range=${LUSTRE_CIDR}

💡 RDMA 子网说明:RDMA 计算网底层关联了 vpc-roce Profile,GCP 会自动在 us-central1-b 下创建名为 default-subnet-1-${RDMA_NET} 的全局 IPv6-Only 子网,无需人工执行子网划分命令。您可以运行以下命令进行验证: gcloud compute networks subnets list --network=${RDMA_NET} --project=${PROJECT}

1.3 创建防火墙安全规则

# 允许 Google IAP 隧道登录 (必须)
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-iap-ssh \
  --project=${PROJECT} --network=${MGMT_NET} \
  --direction=INGRESS --priority=900 --action=ALLOW \
  --rules=tcp:22,tcp:56000 --source-ranges=35.235.240.0/20

# 允许管理网段内部全端口全协议互通
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-internal-mgmt \
  --project=${PROJECT} --network=${MGMT_NET} \
  --direction=INGRESS --priority=1000 --action=ALLOW \
  --rules=all --source-ranges=${MGMT_CIDR}

# 允许 Lustre 网段内部全端口全协议互通
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-internal-lustre \
  --project=${PROJECT} --network=${LUSTRE_NET} \
  --direction=INGRESS --priority=1000 --action=ALLOW \
  --rules=all --source-ranges=${LUSTRE_CIDR}

1.4 创建 Placement Policy (物理亲和性策略)

为我们的验证 Domain 创建一个拓扑组:

gcloud beta compute resource-policies create group-placement \
  ${PLACEMENT_POLICY} \
  --collocation=collocated \
  --gpu-topology=1x72 \
  --project=${PROJECT} --region=${REGION}

Part 2: 创建并配置 Lustre 存储实例 (可选)

如果您需要在此轮验证中挂载 Lustre,请执行本章节;如只需测试 NCCL,可跳过此章节。

2.1 建立 VPC 专用服务连接 (Private Service Access)

# 1. 申请用于 VPC Peering 的 IP 地址段
gcloud compute addresses create ${CLUSTER_PREFIX}-lustre-ip-range \
  --global \
  --purpose=VPC_PEERING \
  --prefix-length=20 \
  --description="Managed Lustre VPC Peering" \
  --network=${LUSTRE_NET} \
  --project=${PROJECT}

# 2. 建立私有服务对接
gcloud services vpc-peerings connect \
  --network=${LUSTRE_NET} \
  --ranges=${CLUSTER_PREFIX}-lustre-ip-range \
  --service=servicenetworking.googleapis.com \
  --project=${PROJECT}

# 3. 为 Lustre 对接端创建防火墙放行规则
PEERING_CIDR=$(gcloud compute addresses describe ${CLUSTER_PREFIX}-lustre-ip-range --global --project=${PROJECT} --format="value[separator=/](address, prefixLength)")
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-lustre-peering \
  --project=${PROJECT} \
  --network=${LUSTRE_NET} \
  --allow=tcp:988,tcp:6988 \
  --source-ranges=$PEERING_CIDR

2.2 创建 Lustre 实例

gcloud lustre instances create ${CLUSTER_PREFIX}-lustre \
  --project=${PROJECT} \
  --location=${ZONE} \
  --network=projects/${PROJECT}/global/networks/${LUSTRE_NET} \
  --capacity-gib=72000 \
  --per-unit-storage-throughput=125 \
  --filesystem=lustrefs

💡 获取 Lustre NID:创建可能需要 10-15 分钟。完成后运行: gcloud lustre instances describe ${CLUSTER_PREFIX}-lustre --location=${ZONE} --project=${PROJECT} --format="value(mountPoint)" 记下输出的 NID 地址(格式为 10.X.X.X@tcp:/lustrefs),后续需要作为 metadata 传入 Worker。


Part 3: Control Plane (Master) 节点部署 (必须)

3.1 创建 Master VM

Master VM 采用标准的 n2-standard-16 虚拟机,无需绑定物理预留或 Placement Policy:

gcloud compute instances create ${CP_NAME} \
  --project=${PROJECT} \
  --zone=${ZONE} \
  --machine-type="n2-standard-16" \
  --image-family="rocky-linux-9" \
  --image-project="rocky-linux-cloud" \
  --boot-disk-type="pd-ssd" \
  --boot-disk-size="200GB" \
  --network-interface=network=${MGMT_NET},subnet=${MGMT_SUB} \
  --network-interface=network=${LUSTRE_NET},subnet=${LUSTRE_SUB} \
  --metadata-from-file=startup-script=scripts/hzchen-mval-master-init.sh

3.2 提取 K8s Control Plane 配置参数

创建后,VM 会通过 startup-script 自动安装 K8s 控制面并预加载 Calico。稍等 2-3 分钟,通过 SSH 登录 Master 节点,获取 K8s Join 命令及凭证。

⚠️ 关于 IAP Tunnel 4033 授权报错与直连 SSH 绕行方案: 如果您在使用 gcloud compute ssh --tunnel-through-iap 登录时遇到如下报错: ConnectionCreationError: Error while connecting [4033: 'not authorized'] 这是因为您的 Google 账号在 tencent-gcp-taiji 项目中缺少 roles/iap.tunnelResourceAccessor 角色。

绕行解决方案: 1. 在本地终端放行您当前的 Cloudtop 公网 IP 入站: bash gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-ssh-direct \ --project=${PROJECT} --network=${MGMT_NET} \ --direction=INGRESS --priority=950 --action=ALLOW \ --rules=tcp:22 --source-ranges=104.199.159.39,35.201.140.161 2. 获取 Master 的公网 IP: bash export CP_EXTERNAL_IP=$(gcloud compute instances describe ${CP_NAME} --zone=${ZONE} --project=${PROJECT} --format="value(networkInterfaces[0].accessConfigs[0].natIP)") echo "Master Public IP: ${CP_EXTERNAL_IP}" 3. 使用原生 SSH 直连公网 IP,并显式禁用 ProxyCommand 绕过 BeyondCorp 拦截: bash ssh -o ProxyCommand=none -i ~/.ssh/google_compute_engine -o StrictHostKeyChecking=no hzchen@${CP_EXTERNAL_IP}

登录后,在 Master 上查看自动生成的 kubeadm 令牌:

cat /root/kubeadm-join-command.txt
# 预期输出格式: kubeadm join 10.205.0.2:6443 --token <TOKEN> --discovery-token-ca-cert-hash sha256:<HASH>

请记录 Master 的内网 IP(即 10.205.0.2)、<TOKEN> 与 <HASH> 串,填入后续 Worker 创建的变量中。


Part 4: Worker 节点部署 (必须)

4.1 声明 Join 参数与 Lustre 变量

请在您的本地终端声明您在 Step 3.2 中获取的值:

# 修改为您在 Master 节点上获取的实际值
export CP_IP="10.205.0.2"
export JOIN_TOKEN="xxxxxx.xxxxxxxxxxxxxxxx"
export JOIN_HASH="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

# 如果 Part 2 创建了 Lustre,请修改为 Lustre Mount NID 串,否则留空:
export LUSTRE_MGS_NID="" 

# 💡 拓扑策略绑定:为了保留完整的物理亲和性验证(推荐),
# 我们需要使用项目中已经与 `subblock-0003` 绑定的 Placement Policy,而不是新建 policy。
# 底层预留 Subblock 与 Policy 的映射关系为 1对1 对应(Subblock-0003 对应 policy-0003)。
# 如果您在创建时遇到故障节点需要临时绕过拓扑审查,可以将其设为空 `""`。
export PLACEMENT_POLICY="${CLUSTER_PREFIX}-nvl72-policy-0001"

4.2 创建 Worker 虚拟机

⚠️ 重要提示 (Placement Policy 限制与 16/18 容灾 SLA): 1. 严禁绕过 Placement Policy:根据 GCP 官方物理限制,GB200/GB300 必须以 1x72 拓扑(18个节点)为最小单元绑定 Placement Policy 进行资源调度。如果在同一个 sub-block(NVLink 域)内绕过 policy 零散地创建虚拟机,会导致整台 NVL72 机架发生物理碎片化锁定,后续将彻底无法再用于完整的 1x72 拓扑调度。 2. 为什么是 --min-count=16:GCP 官方针对 A4X/A4X-Max 机架的 SLA 可用性承诺为 16/18。这意味着在由 18 个计算节点组成的 Domain 中,GCP 允许最多存在 2 个状态为 degraded hosts(因硬件故障排队等待维修更换)的节点。为了确保即使物理机架中存在故障节点,您的集群依然能成功拉起可用的机器,在使用 bulk create 批量申请时,必须将参数设置为 --count=18 且 --min-count=16。如果您强行设置 --min-count=18,一旦该机架内有 1 台机器报修,整个创建命令都会因无法满足约束而彻底被拒!这也是为什么主流模型框架都在适配 64 + 8 Spare (容灾备用) 队列机制的根本硬件原因。 3. 故障节点生命周期与恢复机制: * 16/18 容灾补充创建:如果在首次执行 bulk create 时由于有部分 degraded hosts 而只成功拉起了 16 台虚拟机,那么缺失的 2 台在云端是不存在的。当底层物理机修复后,您必须重新执行一次 bulk create(只传入缺失的 2 台 VM 名称,如 count=2),并且必须指定与前 16 台完全相同的 Placement Policy 与 Subblock,否则将引发 PLACEMENT_POLICY_CONSTRAINTS_INFEASIBLE 碎片化报错。 * 运行中节点突发硬件故障:如果您的 18 台虚拟机已经 RUNNING,后来有 2 台发生硬件故障变为 degraded hosts,由于 GPU/裸金属不支持热迁移(Live Migration),因此这两台虚拟机会被强制终止 (TERMINATED)(由命令中的 --on-host-maintenance=TERMINATE 决定)。但是,在命令默认开启的 --restart-on-failure 保护下,一旦底层硬件被 GCP 修复完毕,控制平面会自动 (Automatically) 将这两台机器重新拉起并恢复为 RUNNING 状态,您不需要额外干预或执行任何命令。

批量创建整组 18 台虚拟机

必须使用以下命令一次性创建整组 (18台) 虚拟机,并严格启用 Resource Policy。

# 生成 18 台 VM 的名称列表
NAMES=""
for i in {0..17}; do
  if [ -z "$NAMES" ]; then
    NAMES="${WORKER_PREFIX}-w${i}"
  else
    NAMES="${NAMES},${WORKER_PREFIX}-w${i}"
  fi
done

# 使用 bulk create 一次性原子化申请 18 台虚拟机
gcloud beta compute instances bulk create \
  --predefined-names=${NAMES} \
  --count=18 \
  --min-count=16 \
  --project=${PROJECT} \
  --zone=${ZONE} \
  --machine-type=${MACHINE_TYPE} \
  --image=${IMAGE} \
  --image-project=${IMAGE_PROJECT} \
  --boot-disk-type=hyperdisk-balanced \
  --boot-disk-size=1000GB \
  --scopes=cloud-platform \
  --reservation-affinity=specific \
  --reservation=${CURRENT_RESERVATION} \
  --provisioning-model=RESERVATION_BOUND \
  --on-host-maintenance=TERMINATE \
  --restart-on-failure \
  --resource-policies=${PLACEMENT_POLICY} \
  --metadata=serial-port-enable=1,cp-ip=${CP_IP},join-token=${JOIN_TOKEN},join-hash=${JOIN_HASH} \
  --metadata-from-file=startup-script=scripts/hzchen-mval-worker-init.sh \
  --network-interface=subnet=${MGMT_SUB},nic-type=IDPF \
  --network-interface=subnet=${LUSTRE_SUB},nic-type=IDPF,no-address \
  --network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
  --network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
  --network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
  --network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
  --network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
  --network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
  --network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
  --network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA

4.3 验证 Worker 初始化与 K8s 入群状态

# 检查 VM 开机状态与内网 IP
gcloud compute instances list --project=${PROJECT} --filter="name~${WORKER_PREFIX}-d"

# SSH 登录 Master 查看 K8s 节点加入情况
gcloud compute ssh ${CP_NAME} --zone=${ZONE} --tunnel-through-iap --command="kubectl get nodes -o wide"

💡 节点就绪可能需要 3-4 分钟。如果状态为 Ready 且 K8s 识别到正确的 OS 与 IP,则说明加入成功。


Part 5: Kubernetes 基础资源与硬件驱动部署 (必须)

请通过 SSH 登录到 Master 节点上执行后续所有 Kubernetes 控制面操作。

gcloud compute ssh ${CP_NAME} --zone=${ZONE} --tunnel-through-iap

5.1 对 Worker 节点打标

# 手动将节点归类到同一 NVLink Domain 1 下,并标记为裸金属类型
for i in {0..17}; do
  kubectl label node hzchen-mval-worker-d1-w${i} nvl72-domain=domain-1 --overwrite
  kubectl label node hzchen-mval-worker-d1-w${i} node-type=a4x-max --overwrite
done

5.2 配置 Scheduler DRA RBAC 权限

cat <<'EOF' | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:kube-scheduler:dra
rules:
- apiGroups: ["resource.k8s.io"]
  resources: ["resourceclaims"]
  verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["resource.k8s.io"]
  resources: ["resourceclaims/status"]
  verbs: ["get", "update", "patch"]
- apiGroups: ["resource.k8s.io"]
  resources: ["resourceslices"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["resource.k8s.io"]
  resources: ["deviceclasses"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-scheduler:dra
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-scheduler:dra
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:kube-scheduler
EOF

# 重启 Scheduler 强制使权限生效
kubectl delete pod -n kube-system -l component=kube-scheduler

5.3 部署 asapd-lite (裸金属网络管理)

参考信息:GB300 裸金属实例依赖 asapd-lite DaemonSet 来接管管理网卡之外的高速网络设备(CX8 物理网卡)。官方维护的 COS 版本 yaml 模板请参考: asapd-lite-installer-a4x-max-bm-cos.yaml

# 部署 TLinux/Rocky 适配版 asapd-lite 安装文件
kubectl apply -f https://gcp.totorochina.com/asapd-lite-tlinux.yaml

# 验证 Pod 处于 Running 状态
kubectl get ds -n kube-system asapd-lite

💡 注:为了下载 asapd-lite-tlinux.yaml,我们已在 Totoro 网站上传了该 YAML 的静态映射版本。

5.4 部署 NVIDIA GPU 驱动与 DRA 算力控制面

# 1. 部署 NVIDIA Device Plugin 限制 nodeSelector 为我们打过标的节点
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update
helm install nvidia-device-plugin nvdp/nvidia-device-plugin \
  --namespace gpu-operator \
  --create-namespace \
  --set-string nodeSelector."nvidia\.com/gpu\.present"=true \
  --wait --timeout 300s

# 2. 部署 K8s DRA Driver (版本对齐 v25.12.0)
helm repo add nvidia-dra https://nvidia.github.io/k8s-dra-driver
helm repo update
helm install nvidia-dra-driver-gpu nvidia-dra/nvidia-dra-driver-gpu \
  --namespace gpu-operator \
  --version 25.12.0 \
  --set gpuResourcesEnabledOverride=true \
  --wait --timeout 300s

5.5 部署 DRANET 网络分配器

helm install dranet oci://registry.k8s.io/networking/charts/dranet \
  --version v1.3.0 \
  -n kube-system

# 声明并注册 DeviceClass 资源,允许 Pod 挂载 MRDMA 设备文件
cat << 'EOF' | kubectl apply -f -
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: rdma-devices
spec:
  selectors:
  - cel:
      expression: |
        device.driver == "dra.net" &&
        has(device.attributes["dra.net"].rdma) && device.attributes["dra.net"].rdma == true
EOF

5.6 注册 ComputeDomain 物理互联域

向控制器注册机架内 1x72 NVLink Clique,建立 NVLink 通道并提取 K8s UID 写入节点:

# 1. 创建 ComputeDomain
cat <<EOF | kubectl apply -f -
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: domain-1-cd
spec:
  numNodes: 0
  channel:
    allocationMode: Single
    resourceClaimTemplate:
      name: domain-1-cd-channel
EOF

# 2. 提取 UID
CD_UID=$(kubectl get computedomain domain-1-cd -o jsonpath='{.metadata.uid}')
echo "ComputeDomain UID: $CD_UID"

# 3. 将 UID 打标至两台节点,激活多机 GPU 组播拓扑
for i in {0..17}; do
  kubectl label node hzchen-mval-worker-d1-w${i} \
    resource.nvidia.com/computeDomain=$CD_UID --overwrite
done

# 验证守护进程状态 (应显示 2 个 healthy 实例)
kubectl get pods -n gpu-operator -l app=computedomain-daemon

5.7 安装 Kubeflow MPI Operator

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.8.0/deploy/v2beta1/mpi-operator.yaml

Part 6: 双机 MPIJob NCCL AllReduce 测试 (可选)

6.1 创建 ResourceClaimTemplate

cat <<'EOF' | kubectl apply -f -
# 1. GPU 申请模板
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: gpu-devices-mpi-sd
spec:
  spec:
    devices:
      requests:
      - name: gpus
        exactly:
          deviceClassName: gpu.nvidia.com
          count: 4
---
# 2. MRDMA 8 网卡申请模板
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: rdma-nics-mpi-sd
spec:
  spec:
    devices:
      requests:
      - name: rdma-nics
        exactly:
          deviceClassName: rdma-devices
          count: 8
EOF

6.2 注册 MRDMA 静态路由 ConfigMap

由于 CNI 无法在三层环境中直接寻址,该脚本负责通过 metadata 服务将网卡关联的 IPv6 路由直接写入容器网关中:

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: setup-rdma-ips-script
  namespace: default
data:
  setup_rdma_ips.py: |
    import json
    import urllib.request
    import subprocess
    import os
    import glob
    import sys

    def get_metadata():
        req = urllib.request.Request("http://169.254.169.254/computeMetadata/v1/instance/network-interfaces/?recursive=true")
        req.add_header("Metadata-Flavor", "Google")
        try:
            with urllib.request.urlopen(req) as response:
                return json.loads(response.read().decode())
        except Exception as e:
            print(f"Failed to query metadata: {e}")
            return None

    def get_local_interfaces():
        mac_map = {}
        for path in glob.glob("/sys/class/net/gpu*/address"):
            ifname = path.split("/")[-2]
            try:
                with open(path, "r") as f:
                    mac = f.read().strip().lower()
                    mac_map[mac] = ifname
            except Exception as e:
                print(f"Failed to read MAC for {ifname}: {e}")
        return mac_map

    def parse_ipv6_prefix_56(ip_str):
        parts = ip_str.split(":")
        if "" in parts:
            idx = parts.index("")
            missing = 8 - len(parts) + 1
            parts = parts[:idx] + ["0"] * missing + parts[idx+1:]
        block4 = parts[3]
        block4_int = int(block4, 16)
        block4_56 = (block4_int & 0xFF00)
        parts[3] = f"{block4_56:x}"
        return ":".join(parts[:4]) + "::/56"

    def main():
        metadata = get_metadata()
        if not metadata:
            sys.exit(1)
        
        local_interfaces = get_local_interfaces()
        gateway_v6 = "fe80::1"
        
        for nic in metadata:
            if nic.get("nicType") != "MRDMA":
                continue
            
            mac = nic.get("mac", "").lower()
            ipv6_list = nic.get("ipv6s", [])
            if not ipv6_list:
                ipv6_list = nic.get("ipv6", [])
                
            if not ipv6_list:
                print(f"No IPv6 allocated for MRDMA interface with MAC {mac}")
                continue
                
            ipv6_addr = ipv6_list[0]
            if "/" not in ipv6_addr:
                ipv6_with_prefix = f"{ipv6_addr}/128"
            else:
                ipv6_with_prefix = ipv6_addr
                ipv6_addr = ipv6_addr.split("/")[0]
                
            route_target = parse_ipv6_prefix_56(ipv6_addr)
            
            ifname = local_interfaces.get(mac)
            if not ifname:
                print(f"Could not find local interface for MAC {mac}")
                continue
                
            print(f"Configuring container NIC {ifname} ({mac}) with IP {ipv6_with_prefix}, route {route_target} via {gateway_v6}")
            
            try:
                subprocess.run(["ip", "addr", "flush", "dev", ifname], check=True)
                subprocess.run(["ip", "addr", "add", ipv6_with_prefix, "dev", ifname], check=True)
                subprocess.run(["ip", "link", "set", "dev", ifname, "up"], check=True)
                subprocess.run(["ip", "-6", "route", "replace", route_target, "via", gateway_v6, "dev", ifname], check=True)
                print(f"Successfully configured {ifname}")
            except Exception as e:
                print(f"Failed to configure interface {ifname}: {e}")

    if __name__ == "__main__":
        main()
EOF

6.3 运行 MPIJob 压力测试 Pod

该 YAML 将向 K8s 申领 2 台节点上共计 8 张 GPU,通过 Containerd 动态分配物理设备。

cat <<'EOF' | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: nccl-mpi-sd
  namespace: default
spec:
  slotsPerWorker: 4
  runPolicy:
    cleanPodPolicy: Running
  sshAuthMountPath: /root/.ssh
  launcherCreationPolicy: WaitForWorkersReady
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          nodeSelector:
            nvl72-domain: domain-1
          initContainers:
          - name: gib-installer
            image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
            command: ["/bin/sh", "-c"]
            args:
            - |
              set -e
              /scripts/container_entry.sh install --install-nccl
              cp -a /usr/local/gib/. /target/gib/
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
              mkdir -p /target/gib/lib64/libibverbs
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
            volumeMounts:
            - name: gib
              mountPath: /target/gib
          containers:
          - name: launcher
            image: nvcr.io/nvidia/pytorch:26.05-py3
            imagePullPolicy: IfNotPresent
            env:
            - name: LD_LIBRARY_PATH
              value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
            command:
            - bash
            - -c
            - |
              set -e
              echo "Resolving worker IPs via K8s ServiceAccount API..."
              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              API="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"

              for i in $(seq 0 $((REPLICAS-1))); do
                WORKER_NAME="nccl-mpi-sd-worker-${i}"
                for attempt in $(seq 1 60); do
                  WORKER_IP=$(curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" \
                    "$API/api/v1/namespaces/default/pods/$WORKER_NAME" 2>/dev/null \
                    | python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",{}).get("podIP",""))' 2>/dev/null || echo "")
                  if [ -n "$WORKER_IP" ]; then
                    echo "$WORKER_NAME IP: $WORKER_IP"
                    # 注册短名和完整 FQDN 以解决 DNS 解析边界问题
                    echo "$WORKER_IP $WORKER_NAME $WORKER_NAME.default.svc.cluster.local" >> /etc/hosts
                    break
                  fi
                  echo "Waiting for $WORKER_NAME IP (attempt $attempt)..."
                  sleep 5
                done
                if [ -z "$WORKER_IP" ]; then
                  echo "ERROR: Could not resolve $WORKER_NAME IP"
                  exit 1
                fi
              done

              apt-get update -qq && apt-get install -y -qq netcat-openbsd > /dev/null 2>&1
              echo "Waiting for workers SSH port to open..."
              for i in $(seq 0 $((REPLICAS-1))); do
                WORKER_NAME="nccl-mpi-sd-worker-${i}"
                until nc -z $WORKER_NAME 22; do
                  echo "Waiting for SSH on $WORKER_NAME..."
                  sleep 3
                done
                echo "SSH on $WORKER_NAME is active!"
              done

              echo "Starting MPI AllReduce benchmark (8 Ranks total, 4 per node)..."
              /usr/local/mpi/bin/mpirun --allow-run-as-root \
                -np 8 -npernode 4 \
                --mca pml ob1 \
                --mca btl tcp,self \
                --mca btl_tcp_if_include eth0 \
                --mca coll_ucc_enable 0 \
                -x LD_LIBRARY_PATH=/usr/local/gib/lib64:/usr/local/nvidia/lib64 \
                -x NCCL_NET_PLUGIN=/usr/local/gib/lib64/libnccl-net.so \
                -x NCCL_ENV_PLUGIN=gcp \
                -x NCCL_MNNVL_ENABLE=2 \
                -x NCCL_CUMEM_ENABLE=1 \
                -x NCCL_SOCKET_IFNAME=eth0 \
                -x UCX_NET_DEVICES=eth0 \
                -x NCCL_DEBUG=INFO \
                --mca plm_rsh_args '-o StrictHostKeyChecking=no' \
                /tmp/nccl-tests/build/all_reduce_perf -b 512M -e 8G -f 2 -g 1
            volumeMounts:
            - name: gib
              mountPath: /usr/local/gib
            resources:
              requests:
                cpu: "4"
                memory: 16Gi
              limits:
                cpu: "4"
                memory: 16Gi
          volumes:
          - name: gib
            emptyDir: {}
    Worker:
      replicas: 2
      template:
        spec:
          nodeSelector:
            nvl72-domain: domain-1
          resourceClaims:
          - name: compute-domain-channel
            resourceClaimTemplateName: domain-1-cd-channel
          - name: rdma-nics
            resourceClaimTemplateName: rdma-nics-mpi-sd
          - name: gpu-devices
            resourceClaimTemplateName: gpu-devices-mpi-sd
          initContainers:
          - name: gib-installer
            image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
            command: ["/bin/sh", "-c"]
            args:
            - |
              set -e
              /scripts/container_entry.sh install --install-nccl
              cp -a /usr/local/gib/. /target/gib/
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
              mkdir -p /target/gib/lib64/libibverbs
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
            volumeMounts:
            - name: gib
              mountPath: /target/gib
          containers:
          - name: worker
            image: nvcr.io/nvidia/pytorch:26.05-py3
            imagePullPolicy: IfNotPresent
            securityContext:
              privileged: true
            env:
            - name: LD_LIBRARY_PATH
              value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
            - name: NCCL_NET_PLUGIN
              value: /usr/local/gib/lib64/libnccl-net.so
            - name: NCCL_ENV_PLUGIN
              value: gcp
            command:
            - bash
            - -c
            - |
              apt-get update -qq && apt-get install -y -qq openssh-server iproute2 > /dev/null 2>&1
              mkdir -p /run/sshd
              ssh-keygen -A
              
              printf 'Port 22\nPermitRootLogin yes\nPubkeyAuthentication yes\nAuthorizedKeysFile /root/.ssh/authorized_keys\nStrictModes no\nPasswordAuthentication no\nUsePAM no\nPidFile /run/sshd/sshd.pid\n' > /etc/ssh/mpi_sshd_config
              
              python3 /scripts/setup_rdma_ips.py
              # 带重试机制的 git clone,防止由于 Github 网络波动导致 Worker 拉起失败
              cd /tmp
              for i in $(seq 1 30); do
                git clone --depth 1 https://github.com/NVIDIA/nccl-tests.git && break || sleep 5
              done
              
              cd /tmp/nccl-tests && make -j$(nproc) MPI=1 MPI_HOME=/usr/local/mpi NCCL_HOME=/usr/local/gib CUDA_HOME=/usr/local/cuda CXXFLAGS="-g" CFLAGS="-g"
              
              /usr/sbin/sshd -De -f /etc/ssh/mpi_sshd_config
            volumeMounts:
            - name: gib
              mountPath: /usr/local/gib
            - name: dshm
              mountPath: /dev/shm
            - name: setup-script
              mountPath: /scripts
            resources:
              requests:
                cpu: "96"
                memory: 800Gi
              limits:
                cpu: "96"
                memory: 800Gi
              claims:
              - name: gpu-devices
              - name: rdma-nics
              - name: compute-domain-channel
          volumes:
          - name: gib
            emptyDir: {}
          - name: dshm
            emptyDir:
              medium: Memory
              sizeLimit: 250Gi
          - name: setup-script
            configMap:
              name: setup-rdma-ips-script
EOF

6.4 查看测试结果

# 实时跟踪 Launcher 日志
kubectl logs -f -l training.kubeflow.org/job-role=launcher

预期输出指标: 同域(Same-Domain,走机架内多节点 NVLink,即 MNNVL 协议)双机 all_reduce 带宽指标: - Avg bus bandwidth 应该在 295+ GB/s 以上。


Part 7: 实验后清理

清理测试所占资源,释放 CPU / GPU Quotas:

# 1. 删除虚拟机
gcloud compute instances delete ${NAMES} \
  --project=${PROJECT} --zone=${ZONE} --quiet

gcloud compute instances delete ${CP_NAME} \
  --project=${PROJECT} --zone=${ZONE} --quiet

# 2. 删除 Placement Policy
gcloud compute resource-policies delete ${PLACEMENT_POLICY} \
  --project=${PROJECT} --region=${REGION} --quiet

# 3. 删除专用网络 (自下而上)
# 删除防火墙规则
gcloud compute firewall-rules delete \
  ${CLUSTER_PREFIX}-allow-iap-ssh \
  ${CLUSTER_PREFIX}-allow-internal-mgmt \
  ${CLUSTER_PREFIX}-allow-internal-lustre \
  --project=${PROJECT} --quiet

# 删除子网
gcloud compute networks subnets delete ${MGMT_SUB} --project=${PROJECT} --region=${REGION} --quiet
gcloud compute networks subnets delete ${LUSTRE_SUB} --project=${PROJECT} --region=${REGION} --quiet

# 删除 VPC 网络
gcloud compute networks delete ${MGMT_NET} --project=${PROJECT} --quiet
gcloud compute networks delete ${LUSTRE_NET} --project=${PROJECT} --quiet
gcloud compute networks delete ${RDMA_NET} --project=${PROJECT} --quiet

# 4. 删除 Lustre 实例 (如果创建了)
gcloud lustre instances delete ${CLUSTER_PREFIX}-lustre \
  --location=${ZONE} --project=${PROJECT} --quiet

常见问题与故障排除 (Troubleshooting)

问题:mpi-operator 反复崩溃或 apiserver 响应超时 如果集群变得不稳定,或者 mpi-operator 因为租约超时(lease timeouts)反复重启,这通常是由于安装了冗余的 NFD (Node Feature Discovery) 插件引起的(例如 gpu-operator 默认自带的 NFD 与集群已有的 NFD 冲突)。冲突会导致数百个 worker pod 陷入 CrashLoopBackOff,并向 apiserver 发送大量垃圾请求,导致 API Server 瘫痪。 解决方案:请确保在部署 gpu-operator 时禁用 NFD:

helm upgrade gpu-operator nvidia/gpu-operator --namespace gpu-operator --reuse-values --set nfd.enabled=false