GB200 集群验证执行操作规程

本指南提供了在 GB200 集群上手动运行 NCCL 验证测试的步骤。测试参数已根据用户的验证场景进行了调整(-b 1M -e 16G -f 2 -w 50 -n 100)。

更新后的 YAML 文件可以在以下位置找到: - 本地工作空间:nccl-mpi-64g-single-domain.yaml 和 nccl-mpi-128g-dual-domain.yaml - 主节点(Master Node):/tmp/nccl-mpi-64g-single-domain.yaml 和 /tmp/nccl-mpi-128g-dual-domain.yaml


1. 登录主节点 (Control Plane)

主节点的公网 IP 为 35.231.106.252 (内网 IP: 10.0.0.24, 实例名称: hzchen-test-cp)。

选项 A:通过 IAP 直接 SSH 登录(推荐)

如果您的本地网络环境允许(未被拦截),可以直接 SSH 登录到主节点:

gcloud compute ssh hzchen-test-cp \
  --project=tencent-gcp-taiji-poc \
  --zone=us-east1-d \
  --tunnel-through-iap

选项 B:通过跳板机登录(备用)

如果直接访问受到 BeyondCorp 策略的拦截限制,请通过跳板机中转: 1. SSH 登录到跳板机: bash gcloud compute ssh hzchen-build \ --project=gpu-launchpad-playground \ --zone=us-central1-a \ --tunnel-through-iap 2. 从跳板机 SSH 连接到主节点: bash ssh -i ~/.ssh/id_ed25519_worker jetski@35.231.106.252


2. 执行验证测试

登录主节点后,请按照以下步骤运行验证测试。

本实验用于验证单个 NVL72 机柜内部的通信带宽。

  1. 提交任务:

    sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl apply -f /tmp/nccl-mpi-64g-single-domain.yaml
  2. 监控状态: 等待相关 Pod 创建并进入 Running 状态:

    sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl get pods

    找到启动器(launcher)Pod(例如:nccl-mpi-64g-single-domain-launcher-xxxxx)。

  3. 查看日志:

    sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl logs -f [LAUNCHER_POD_NAME]

    注意:使用最新参数后,测试将覆盖 1M 到 16G 的数据块大小,包含 50 次 warmup 和 100 次 iterations。完成测试需要几分钟的时间,请耐心等待。

  4. 清理环境:

    sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl delete -f /tmp/nccl-mpi-64g-single-domain.yaml

测试实验 2:双域 (Dual Domain, 128 GPU, 跨柜 RoCE)

本实验用于验证通过 GPUDirect-TCPX (gIB) 跨越两个 NVL72 机柜的通信带宽。

  1. 提交任务:

    sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl apply -f /tmp/nccl-mpi-128g-dual-domain.yaml
  2. Monitor Status:

    sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl get pods

    找到启动器(launcher)Pod(例如:nccl-mpi-128g-dual-domain-launcher-xxxxx)。

  3. 查看日志:

    sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl logs -f [LAUNCHER_POD_NAME]

    注意:实验已开启 NCCL 调试日志 (NCCL_DEBUG=INFO)。您将在日志中看到类似 NCCL INFO NET/gIB : Using ... 的输出,这表明 gIB RDMA 网络处于激活状态。

  4. 清理环境:

    sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl delete -f /tmp/nccl-mpi-128g-dual-domain.yaml

3. 故障排除 (Troubleshooting)

mpi-operator 反复崩溃或 apiserver 响应超时

如果集群变得不稳定,或者 mpi-operator 因为租约超时(lease timeouts)反复重启,这通常是由于 gpu-operator 默认安装了冗余的 NFD (Node Feature Discovery) 插件引起的。该插件会导致大约 70 个 worker pod 陷入 CrashLoopBackOff 循环,并向 apiserver 发送大量垃圾请求。 要解决此问题,请在更新 gpu-operator 时禁用 NFD:

sudo KUBECONFIG=/etc/kubernetes/admin.conf helm upgrade gpu-operator nvidia/gpu-operator --namespace gpu-operator --reuse-values --set nfd.enabled=false


4. YAML 配置说明与原理

本节将解释在验证 YAML 文件中所使用的各项配置参数。整个验证过程基于 MPI Operator 和 NVIDIA NCCL Tests 构建。

4.1 YAML 执行流程与架构

mpi-operator 会负责调度一个 Launcher 节点以及指定数量的 Worker 节点。 * Worker Pods (工作节点):启动一个基础的 SSH 服务(sshd),并负责拉取和编译 nccl-tests 代码库。 * Launcher Pod (启动节点):负责等待所有的工作节点就绪。它会在启动脚本中循环查询 Kubernetes API,获取所有 Worker Pod 的 IP 地址,并将其解析写入自身的 /etc/hosts 文件中。 * 等待时间 (Wait Time 120s):在执行真正的 mpirun 前,Launcher 的脚本中硬编码了一个 sleep 120 步骤。这段等待时间非常关键,它的目的是确保所有 Worker 都有充足的时间来执行 apt-get install、克隆 nccl-tests 仓库并完成 make -j MPI=1 编译工作。 * 参考资料: MPI Operator 官方文档

4.2 同域与跨域的 NCCL 环境变量差异

在单纯的 nccl-tests 基准测试中,我们通过显式控制环境变量来强制隔离和测算不同硬件链路的极限带宽。但在实际的生产训练中,情况会有所不同。

4.3 节点数量 (Node Count) 修改指南

如果需要增减测试所用的节点数,请确保同时修改以下两处配置: 1. Worker 副本数:在 YAML 结构中的 mpiReplicaSpecs.Worker 下,将 replicas: X 更改为目标节点数。 2. Launcher 启动参数:在 mpiReplicaSpecs.Launcher 的 entrypoint 脚本中: * IP 解析循环:修改循环参数 seq 0 Y,这里的 Y 必须等于 replicas - 1(因为索引从 0 开始),以确保所有的 Worker IP 都能被正确解析。 * MPI 进程数:修改 /usr/local/mpi/bin/mpirun 命令中的 -np Z 参数。np 的值必须等于 Worker 节点数 * 每节点 GPU 数(对于 GB200 而言,每节点有 4 张 GPU)。例如:如果测试 2 个节点,则应该设置为 -np 8。

附录:YAML 配置清单

nccl-mpi-64g-single-domain.yaml
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: nccl-mpi-64g-single-domain
spec:
  slotsPerWorker: 4
  runPolicy:
    cleanPodPolicy: Running
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          nodeSelector:
            nvl72-domain: domain-0001
          initContainers:
          - name: gib-installer
            image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
            command: ["/bin/sh", "-c"]
            args:
            - |
              set -e
              /scripts/container_entry.sh install --install-nccl
              cp -a /usr/local/gib/. /target/gib/
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
              mkdir -p /target/gib/lib64/libibverbs
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
            volumeMounts:
            - name: gib
              mountPath: /target/gib
          containers:
          - name: launcher
            image: nvcr.io/nvidia/pytorch:26.05-py3
            imagePullPolicy: IfNotPresent
            env:
            - name: LD_LIBRARY_PATH
              value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
            command:
            - bash
            - -c
            - |
              set -e
              echo "Resolving worker IPs via k8s API..."
              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              API="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"

              for attempt in $(seq 1 60); do
                ALL_READY=true
                for i in $(seq 0 15); do
                  W_IP=$(curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" \
                    "$API/api/v1/namespaces/default/pods/nccl-mpi-64g-single-domain-worker-${i}" 2>/dev/null \
                    | python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",{}).get("podIP",""))' 2>/dev/null || echo "")
                  if [ -z "$W_IP" ]; then
                    ALL_READY=false
                    break
                  fi
                  echo "$W_IP nccl-mpi-64g-single-domain-worker-${i}.nccl-mpi-64g-single-domain.default.svc nccl-mpi-64g-single-domain-worker-${i}" >> /tmp/hosts_new
                done
                
                if [ "$ALL_READY" = true ]; then
                  cat /tmp/hosts_new >> /etc/hosts
                  echo "All worker IPs resolved."
                  break
                fi
                rm -f /tmp/hosts_new
                echo "Waiting for worker IPs (attempt $attempt)..."
                sleep 5
              done

              if [ "$ALL_READY" = false ]; then
                echo "ERROR: Could not resolve worker IPs"
                exit 1
              fi

              echo "Waiting 120s for workers to compile nccl-tests..."
              sleep 120
              echo "Starting mpirun..."
              /usr/local/mpi/bin/mpirun --allow-run-as-root \
                -np 64 -npernode 4 \
                -x LD_LIBRARY_PATH=/usr/local/gib/lib64:/usr/local/nvidia/lib64 \
                -x NCCL_NET_PLUGIN=/usr/local/gib/lib64/libnccl-net.so \
                -x NCCL_ENV_PLUGIN=gcp \
                -x NCCL_DEBUG=INFO \
                -x NCCL_MNNVL_ENABLE=2 \
                -x NCCL_CUMEM_ENABLE=1 \
                --mca plm_rsh_args '-o StrictHostKeyChecking=no' \
                /tmp/nccl-tests/build/all_reduce_perf -b 1M -e 16G -f 2 -g 1 -w 50 -n 100
            volumeMounts:
            - name: gib
              mountPath: /usr/local/gib
          volumes:
          - name: gib
            emptyDir: {}
    Worker:
      replicas: 16
      template:
        spec:
          nodeSelector:
            nvl72-domain: domain-0001
          dnsPolicy: None
          dnsConfig:
            nameservers:
            - "10.96.0.10"
            - "8.8.8.8"
            searches:
            - "default.svc.cluster.local"
            - "svc.cluster.local"
            - "cluster.local"
          resourceClaims:
          - name: compute-domain-channel
            resourceClaimTemplateName: domain-1-cd-channel
          - name: rdma-nics
            resourceClaimTemplateName: rdma-nics-mpi-sd
          initContainers:
          - name: gib-installer
            image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
            command: ["/bin/sh", "-c"]
            args:
            - |
              set -e
              /scripts/container_entry.sh install --install-nccl
              cp -a /usr/local/gib/. /target/gib/
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
              mkdir -p /target/gib/lib64/libibverbs
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
            volumeMounts:
            - name: gib
              mountPath: /target/gib
          containers:
          - name: worker
            image: nvcr.io/nvidia/pytorch:26.05-py3
            imagePullPolicy: IfNotPresent
            securityContext:
              privileged: true
            env:
            - name: LD_LIBRARY_PATH
              value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
            - name: NCCL_NET_PLUGIN
              value: /usr/local/gib/lib64/libnccl-net.so
            - name: NCCL_ENV_PLUGIN
              value: gcp
            command:
            - bash
            - -c
            - |
              apt-get update -qq && apt-get install -y -qq openssh-server > /dev/null 2>&1
              mkdir -p /run/sshd
              printf 'Port 22\nPermitRootLogin yes\nPubkeyAuthentication yes\nAuthorizedKeysFile /root/.ssh/authorized_keys\nStrictModes no\nPasswordAuthentication no\nUsePAM no\nPidFile /run/sshd/sshd.pid\n' > /etc/ssh/mpi_sshd_config
              ssh-keygen -A
              for i in $(seq 1 30); do
                cd /tmp && rm -rf nccl-tests && git clone --depth 1 https://github.com/NVIDIA/nccl-tests.git && break
                echo "git clone attempt $i failed, retrying in 10s..."
                sleep 10
              done
              cd /tmp/nccl-tests && make -j$(nproc) MPI=1 \
                MPI_HOME=/usr/local/mpi \
                NCCL_HOME=/usr/local/gib \
                CUDA_HOME=/usr/local/cuda
              /usr/sbin/sshd -De -f /etc/ssh/mpi_sshd_config
            volumeMounts:
            - name: gib
              mountPath: /usr/local/gib
            - name: dshm
              mountPath: /dev/shm
            resources:
              requests:
                cpu: "96"
                memory: 800Gi
              limits:
                cpu: "96"
                memory: 800Gi
                nvidia.com/gpu: 4
              claims:
              - name: compute-domain-channel
              - name: rdma-nics
          volumes:
          - name: gib
            emptyDir: {}
          - name: dshm
            emptyDir:
              medium: Memory
              sizeLimit: 250Gi
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: rdma-nics-mpi-sd
spec:
  spec:
    devices:
      requests:
      - name: nic-0
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu0rdma0"
      - name: nic-1
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu1rdma0"
      - name: nic-2
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu2rdma0"
      - name: nic-3
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu3rdma0"
nccl-mpi-128g-dual-domain.yaml
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: nccl-mpi-128g-dual-domain
spec:
  slotsPerWorker: 4
  runPolicy:
    cleanPodPolicy: Running
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          initContainers:
          - name: gib-installer
            image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
            command: ["/bin/sh", "-c"]
            args:
            - |
              set -e
              /scripts/container_entry.sh install --install-nccl
              cp -a /usr/local/gib/. /target/gib/
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
              mkdir -p /target/gib/lib64/libibverbs
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
            volumeMounts:
            - name: gib
              mountPath: /target/gib
          containers:
          - name: launcher
            image: nvcr.io/nvidia/pytorch:26.05-py3
            imagePullPolicy: IfNotPresent
            env:
            - name: LD_LIBRARY_PATH
              value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
            command:
            - bash
            - -c
            - |
              set -e
              echo "Resolving worker IPs via k8s API..."
              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              API="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"

              for attempt in $(seq 1 60); do
                ALL_READY=true
                for i in $(seq 0 31); do
                  W_IP=$(curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" \
                    "$API/api/v1/namespaces/default/pods/nccl-mpi-128g-dual-domain-worker-${i}" 2>/dev/null \
                    | python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",{}).get("podIP",""))' 2>/dev/null || echo "")
                  if [ -z "$W_IP" ]; then
                    ALL_READY=false
                    break
                  fi
                  echo "$W_IP nccl-mpi-128g-dual-domain-worker-${i}.nccl-mpi-128g-dual-domain.default.svc nccl-mpi-128g-dual-domain-worker-${i}" >> /tmp/hosts_new
                done
                
                if [ "$ALL_READY" = true ]; then
                  cat /tmp/hosts_new >> /etc/hosts
                  echo "All worker IPs resolved."
                  break
                fi
                rm -f /tmp/hosts_new
                echo "Waiting for worker IPs (attempt $attempt)..."
                sleep 5
              done

              if [ "$ALL_READY" = false ]; then
                echo "ERROR: Could not resolve worker IPs"
                exit 1
              fi

              echo "Waiting 120s for workers to compile nccl-tests..."
              sleep 120
              echo "Starting mpirun..."
              # MNNVL 和 CUMEM 设置为 0,启用跨域 RoCE
              /usr/local/mpi/bin/mpirun --allow-run-as-root \
                -np 128 -npernode 4 \
                -x LD_LIBRARY_PATH=/usr/local/gib/lib64:/usr/local/nvidia/lib64 \
                -x NCCL_NET_PLUGIN=/usr/local/gib/lib64/libnccl-net.so \
                -x NCCL_ENV_PLUGIN=gcp \
                -x NCCL_DEBUG=INFO \
                -x NCCL_MNNVL_ENABLE=0 \
                -x NCCL_CUMEM_ENABLE=0 \
                --mca plm_rsh_args '-o StrictHostKeyChecking=no' \
                /tmp/nccl-tests/build/all_reduce_perf -b 1M -e 16G -f 2 -g 1 -w 50 -n 100
            volumeMounts:
            - name: gib
              mountPath: /usr/local/gib
          volumes:
          - name: gib
            emptyDir: {}
    Worker:
      replicas: 32
      template:
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: nvl72-domain
                    operator: In
                    values: ["domain-0001", "domain-0002"]
          topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: nvl72-domain
            whenUnsatisfiable: DoNotSchedule
            labelSelector:
              matchLabels:
                training.kubeflow.org/job-name: nccl-mpi-128g-dual-domain
          dnsPolicy: None
          dnsConfig:
            nameservers:
            - "10.96.0.10"
            - "8.8.8.8"
            searches:
            - "default.svc.cluster.local"
            - "svc.cluster.local"
            - "cluster.local"
          resourceClaims:
          - name: rdma-nics
            resourceClaimTemplateName: rdma-nics-mpi-sd
          initContainers:
          - name: gib-installer
            image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
            command: ["/bin/sh", "-c"]
            args:
            - |
              set -e
              /scripts/container_entry.sh install --install-nccl
              cp -a /usr/local/gib/. /target/gib/
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
              mkdir -p /target/gib/lib64/libibverbs
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
            volumeMounts:
            - name: gib
              mountPath: /target/gib
          containers:
          - name: worker
            image: nvcr.io/nvidia/pytorch:26.05-py3
            imagePullPolicy: IfNotPresent
            securityContext:
              privileged: true
            env:
            - name: LD_LIBRARY_PATH
              value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
            - name: NCCL_NET_PLUGIN
              value: /usr/local/gib/lib64/libnccl-net.so
            - name: NCCL_ENV_PLUGIN
              value: gcp
            command:
            - bash
            - -c
            - |
              # 跨域路由和 IP 自动发现脚本 (简化示意,需根据实际网络配置实现 setup_rdma_ips.py)
              # python3 /usr/local/bin/setup_rdma_ips.py 
              
              apt-get update -qq && apt-get install -y -qq openssh-server > /dev/null 2>&1
              mkdir -p /run/sshd
              printf 'Port 22\nPermitRootLogin yes\nPubkeyAuthentication yes\nAuthorizedKeysFile /root/.ssh/authorized_keys\nStrictModes no\nPasswordAuthentication no\nUsePAM no\nPidFile /run/sshd/sshd.pid\n' > /etc/ssh/mpi_sshd_config
              ssh-keygen -A
              for i in $(seq 1 30); do
                cd /tmp && rm -rf nccl-tests && git clone --depth 1 https://github.com/NVIDIA/nccl-tests.git && break
                echo "git clone attempt $i failed, retrying in 10s..."
                sleep 10
              done
              cd /tmp/nccl-tests && make -j$(nproc) MPI=1 \
                MPI_HOME=/usr/local/mpi \
                NCCL_HOME=/usr/local/gib \
                CUDA_HOME=/usr/local/cuda
              /usr/sbin/sshd -De -f /etc/ssh/mpi_sshd_config
            volumeMounts:
            - name: gib
              mountPath: /usr/local/gib
            - name: dshm
              mountPath: /dev/shm
            resources:
              requests:
                cpu: "96"
                memory: 800Gi
              limits:
                cpu: "96"
                memory: 800Gi
                nvidia.com/gpu: 4
              claims:
              - name: rdma-nics
          volumes:
          - name: gib
            emptyDir: {}
          - name: dshm
            emptyDir:
              medium: Memory
              sizeLimit: 250Gi
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: rdma-nics-mpi-sd
spec:
  spec:
    devices:
      requests:
      - name: nic-0
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu0rdma0"
      - name: nic-1
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu1rdma0"
      - name: nic-2
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu2rdma0"
      - name: nic-3
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu3rdma0"
nccl-mpi-8g-single-domain-2nodes.yaml
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: nccl-mpi-8g-single-domain-2nodes
spec:
  slotsPerWorker: 4
  runPolicy:
    cleanPodPolicy: Running
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          nodeSelector:
            nvl72-domain: domain-0001
          initContainers:
          - name: gib-installer
            image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
            command: ["/bin/sh", "-c"]
            args:
            - |
              set -e
              /scripts/container_entry.sh install --install-nccl
              cp -a /usr/local/gib/. /target/gib/
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
              mkdir -p /target/gib/lib64/libibverbs
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
            volumeMounts:
            - name: gib
              mountPath: /target/gib
          containers:
          - name: launcher
            image: nvcr.io/nvidia/pytorch:26.05-py3
            imagePullPolicy: IfNotPresent
            env:
            - name: LD_LIBRARY_PATH
              value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
            command:
            - bash
            - -c
            - |
              set -e
              echo "Resolving worker IPs via k8s API..."
              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              API="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"

              for attempt in $(seq 1 60); do
                ALL_READY=true
                for i in $(seq 0 1); do
                  W_IP=$(curl -s --cacert $CA -H "Authorization: Bearer $TOKEN"                     "$API/api/v1/namespaces/default/pods/nccl-mpi-8g-single-domain-2nodes-worker-${i}" 2>/dev/null                     | python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",{}).get("podIP",""))' 2>/dev/null || echo "")
                  if [ -z "$W_IP" ]; then
                    ALL_READY=false
                    break
                  fi
                  echo "$W_IP nccl-mpi-8g-single-domain-2nodes-worker-${i}.nccl-mpi-8g-single-domain-2nodes.default.svc nccl-mpi-8g-single-domain-2nodes-worker-${i}" >> /tmp/hosts_new
                done
                
                if [ "$ALL_READY" = true ]; then
                  cat /tmp/hosts_new >> /etc/hosts
                  echo "All worker IPs resolved."
                  break
                fi
                rm -f /tmp/hosts_new
                echo "Waiting for worker IPs (attempt $attempt)..."
                sleep 5
              done

              if [ "$ALL_READY" = false ]; then
                echo "ERROR: Could not resolve worker IPs"
                exit 1
              fi

              echo "Waiting 120s for workers to compile nccl-tests..."
              sleep 120
              echo "Starting mpirun..."
              /usr/local/mpi/bin/mpirun --allow-run-as-root                 -np 8 -npernode 4                 -x LD_LIBRARY_PATH=/usr/local/gib/lib64:/usr/local/nvidia/lib64                 -x NCCL_NET_PLUGIN=/usr/local/gib/lib64/libnccl-net.so                 -x NCCL_ENV_PLUGIN=gcp                 -x NCCL_DEBUG=INFO                 -x NCCL_MNNVL_ENABLE=2                 -x NCCL_CUMEM_ENABLE=1                 --mca plm_rsh_args '-o StrictHostKeyChecking=no'                 /tmp/nccl-tests/build/all_reduce_perf -b 1M -e 16G -f 2 -g 1 -w 50 -n 100
            volumeMounts:
            - name: gib
              mountPath: /usr/local/gib
          volumes:
          - name: gib
            emptyDir: {}
    Worker:
      replicas: 2
      template:
        spec:
          nodeSelector:
            nvl72-domain: domain-0001
          dnsPolicy: None
          dnsConfig:
            nameservers:
            - "10.96.0.10"
            - "8.8.8.8"
            searches:
            - "default.svc.cluster.local"
            - "svc.cluster.local"
            - "cluster.local"
          resourceClaims:
          - name: compute-domain-channel
            resourceClaimTemplateName: domain-1-cd-channel
          - name: rdma-nics
            resourceClaimTemplateName: rdma-nics-mpi-sd
          initContainers:
          - name: gib-installer
            image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
            command: ["/bin/sh", "-c"]
            args:
            - |
              set -e
              /scripts/container_entry.sh install --install-nccl
              cp -a /usr/local/gib/. /target/gib/
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
              mkdir -p /target/gib/lib64/libibverbs
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
            volumeMounts:
            - name: gib
              mountPath: /target/gib
          containers:
          - name: worker
            image: nvcr.io/nvidia/pytorch:26.05-py3
            imagePullPolicy: IfNotPresent
            securityContext:
              privileged: true
            env:
            - name: LD_LIBRARY_PATH
              value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
            - name: NCCL_NET_PLUGIN
              value: /usr/local/gib/lib64/libnccl-net.so
            - name: NCCL_ENV_PLUGIN
              value: gcp
            command:
            - bash
            - -c
            - |
              apt-get update -qq && apt-get install -y -qq openssh-server > /dev/null 2>&1
              mkdir -p /run/sshd
              printf 'Port 22\nPermitRootLogin yes\nPubkeyAuthentication yes\nAuthorizedKeysFile /root/.ssh/authorized_keys\nStrictModes no\nPasswordAuthentication no\nUsePAM no\nPidFile /run/sshd/sshd.pid\n' > /etc/ssh/mpi_sshd_config
              ssh-keygen -A
              for i in $(seq 1 30); do
                cd /tmp && rm -rf nccl-tests && git clone --depth 1 https://github.com/NVIDIA/nccl-tests.git && break
                echo "git clone attempt $i failed, retrying in 10s..."
                sleep 10
              done
              cd /tmp/nccl-tests && make -j$(nproc) MPI=1                 MPI_HOME=/usr/local/mpi                 NCCL_HOME=/usr/local/gib                 CUDA_HOME=/usr/local/cuda
              /usr/sbin/sshd -De -f /etc/ssh/mpi_sshd_config
            volumeMounts:
            - name: gib
              mountPath: /usr/local/gib
            - name: dshm
              mountPath: /dev/shm
            resources:
              requests:
                cpu: "96"
                memory: 800Gi
              limits:
                cpu: "96"
                memory: 800Gi
                nvidia.com/gpu: 4
              claims:
              - name: compute-domain-channel
              - name: rdma-nics
          volumes:
          - name: gib
            emptyDir: {}
          - name: dshm
            emptyDir:
              medium: Memory
              sizeLimit: 250Gi
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: rdma-nics-mpi-sd
spec:
  spec:
    devices:
      requests:
      - name: nic-0
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu0rdma0"
      - name: nic-1
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu1rdma0"
      - name: nic-2
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu2rdma0"
      - name: nic-3
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu3rdma0"
nccl-mpi-8g-cross-domain-2nodes.yaml
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: nccl-mpi-8g-cross-domain-2nodes
spec:
  slotsPerWorker: 4
  runPolicy:
    cleanPodPolicy: Running
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          initContainers:
          - name: gib-installer
            image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
            command: ["/bin/sh", "-c"]
            args:
            - |
              set -e
              /scripts/container_entry.sh install --install-nccl
              cp -a /usr/local/gib/. /target/gib/
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
              mkdir -p /target/gib/lib64/libibverbs
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
            volumeMounts:
            - name: gib
              mountPath: /target/gib
          containers:
          - name: launcher
            image: nvcr.io/nvidia/pytorch:26.05-py3
            imagePullPolicy: IfNotPresent
            env:
            - name: LD_LIBRARY_PATH
              value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
            command:
            - bash
            - -c
            - |
              set -e
              echo "Resolving worker IPs via k8s API..."
              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              API="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"

              for attempt in $(seq 1 60); do
                ALL_READY=true
                for i in $(seq 0 1); do
                  W_IP=$(curl -s --cacert $CA -H "Authorization: Bearer $TOKEN"                     "$API/api/v1/namespaces/default/pods/nccl-mpi-8g-cross-domain-2nodes-worker-${i}" 2>/dev/null                     | python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",{}).get("podIP",""))' 2>/dev/null || echo "")
                  if [ -z "$W_IP" ]; then
                    ALL_READY=false
                    break
                  fi
                  echo "$W_IP nccl-mpi-8g-cross-domain-2nodes-worker-${i}.nccl-mpi-8g-cross-domain-2nodes.default.svc nccl-mpi-8g-cross-domain-2nodes-worker-${i}" >> /tmp/hosts_new
                done
                
                if [ "$ALL_READY" = true ]; then
                  cat /tmp/hosts_new >> /etc/hosts
                  echo "All worker IPs resolved."
                  break
                fi
                rm -f /tmp/hosts_new
                echo "Waiting for worker IPs (attempt $attempt)..."
                sleep 5
              done

              if [ "$ALL_READY" = false ]; then
                echo "ERROR: Could not resolve worker IPs"
                exit 1
              fi

              echo "Waiting 120s for workers to compile nccl-tests..."
              sleep 120
              echo "Starting mpirun..."
              /usr/local/mpi/bin/mpirun --allow-run-as-root                 -np 8 -npernode 4                 -x LD_LIBRARY_PATH=/usr/local/gib/lib64:/usr/local/nvidia/lib64                 -x NCCL_NET_PLUGIN=/usr/local/gib/lib64/libnccl-net.so                 -x NCCL_ENV_PLUGIN=gcp                 -x NCCL_DEBUG=INFO                 -x NCCL_MNNVL_ENABLE=0                 -x NCCL_CUMEM_ENABLE=0                 --mca plm_rsh_args '-o StrictHostKeyChecking=no'                 /tmp/nccl-tests/build/all_reduce_perf -b 1M -e 16G -f 2 -g 1 -w 50 -n 100
            volumeMounts:
            - name: gib
              mountPath: /usr/local/gib
          volumes:
          - name: gib
            emptyDir: {}
    Worker:
      replicas: 2
      template:
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: nvl72-domain
                    operator: In
                    values: ["domain-0001", "domain-0002"]
          topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: nvl72-domain
            whenUnsatisfiable: DoNotSchedule
            labelSelector:
              matchLabels:
                training.kubeflow.org/job-name: nccl-mpi-8g-cross-domain-2nodes
          dnsPolicy: None
          dnsConfig:
            nameservers:
            - "10.96.0.10"
            - "8.8.8.8"
            searches:
            - "default.svc.cluster.local"
            - "svc.cluster.local"
            - "cluster.local"
          resourceClaims:
          - name: rdma-nics
            resourceClaimTemplateName: rdma-nics-mpi-sd
          initContainers:
          - name: gib-installer
            image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
            command: ["/bin/sh", "-c"]
            args:
            - |
              set -e
              /scripts/container_entry.sh install --install-nccl
              cp -a /usr/local/gib/. /target/gib/
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
              mkdir -p /target/gib/lib64/libibverbs
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
            volumeMounts:
            - name: gib
              mountPath: /target/gib
          containers:
          - name: worker
            image: nvcr.io/nvidia/pytorch:26.05-py3
            imagePullPolicy: IfNotPresent
            securityContext:
              privileged: true
            env:
            - name: LD_LIBRARY_PATH
              value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
            - name: NCCL_NET_PLUGIN
              value: /usr/local/gib/lib64/libnccl-net.so
            - name: NCCL_ENV_PLUGIN
              value: gcp
            command:
            - bash
            - -c
            - |
              apt-get update -qq && apt-get install -y -qq openssh-server > /dev/null 2>&1
              mkdir -p /run/sshd
              printf 'Port 22\nPermitRootLogin yes\nPubkeyAuthentication yes\nAuthorizedKeysFile /root/.ssh/authorized_keys\nStrictModes no\nPasswordAuthentication no\nUsePAM no\nPidFile /run/sshd/sshd.pid\n' > /etc/ssh/mpi_sshd_config
              ssh-keygen -A
              for i in $(seq 1 30); do
                cd /tmp && rm -rf nccl-tests && git clone --depth 1 https://github.com/NVIDIA/nccl-tests.git && break
                echo "git clone attempt $i failed, retrying in 10s..."
                sleep 10
              done
              cd /tmp/nccl-tests && make -j$(nproc) MPI=1                 MPI_HOME=/usr/local/mpi                 NCCL_HOME=/usr/local/gib                 CUDA_HOME=/usr/local/cuda
              /usr/sbin/sshd -De -f /etc/ssh/mpi_sshd_config
            volumeMounts:
            - name: gib
              mountPath: /usr/local/gib
            - name: dshm
              mountPath: /dev/shm
            resources:
              requests:
                cpu: "96"
                memory: 800Gi
              limits:
                cpu: "96"
                memory: 800Gi
                nvidia.com/gpu: 4
              claims:
              - name: rdma-nics
          volumes:
          - name: gib
            emptyDir: {}
          - name: dshm
            emptyDir:
              medium: Memory
              sizeLimit: 250Gi
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: rdma-nics-mpi-sd
spec:
  spec:
    devices:
      requests:
      - name: nic-0
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu0rdma0"
      - name: nic-1
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu1rdma0"
      - name: nic-2
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu2rdma0"
      - name: nic-3
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu3rdma0"