GB200 集群验证执行操作规程
本指南提供了在 GB200 集群上手动运行 NCCL
验证测试的步骤。测试参数已根据用户的验证场景进行了调整(-b 1M -e 16G -f 2 -w 50 -n 100)。
更新后的 YAML 文件可以在以下位置找到: -
本地工作空间:nccl-mpi-64g-single-domain.yaml 和
nccl-mpi-128g-dual-domain.yaml - 主节点(Master
Node):/tmp/nccl-mpi-64g-single-domain.yaml 和
/tmp/nccl-mpi-128g-dual-domain.yaml
1. 登录主节点 (Control Plane)
主节点的公网 IP 为 35.231.106.252 (内网 IP:
10.0.0.24, 实例名称: hzchen-test-cp)。
选项 A:通过 IAP 直接 SSH 登录(推荐)
如果您的本地网络环境允许(未被拦截),可以直接 SSH 登录到主节点:
gcloud compute ssh hzchen-test-cp \
--project=tencent-gcp-taiji-poc \
--zone=us-east1-d \
--tunnel-through-iap选项 B:通过跳板机登录(备用)
如果直接访问受到 BeyondCorp 策略的拦截限制,请通过跳板机中转: 1.
SSH 登录到跳板机:
bash gcloud compute ssh hzchen-build \ --project=gpu-launchpad-playground \ --zone=us-central1-a \ --tunnel-through-iap
2. 从跳板机 SSH 连接到主节点:
bash ssh -i ~/.ssh/id_ed25519_worker jetski@35.231.106.252
2. 执行验证测试
登录主节点后,请按照以下步骤运行验证测试。
测试实验 1:单域 (Single Domain, 64 GPU, 柜内 NVLink)
本实验用于验证单个 NVL72 机柜内部的通信带宽。
提交任务:
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl apply -f /tmp/nccl-mpi-64g-single-domain.yaml监控状态: 等待相关 Pod 创建并进入 Running 状态:
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl get pods找到启动器(launcher)Pod(例如:
nccl-mpi-64g-single-domain-launcher-xxxxx)。查看日志:
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl logs -f [LAUNCHER_POD_NAME]注意:使用最新参数后,测试将覆盖 1M 到 16G 的数据块大小,包含 50 次 warmup 和 100 次 iterations。完成测试需要几分钟的时间,请耐心等待。
清理环境:
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl delete -f /tmp/nccl-mpi-64g-single-domain.yaml
测试实验 2:双域 (Dual Domain, 128 GPU, 跨柜 RoCE)
本实验用于验证通过 GPUDirect-TCPX (gIB) 跨越两个 NVL72 机柜的通信带宽。
提交任务:
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl apply -f /tmp/nccl-mpi-128g-dual-domain.yamlMonitor Status:
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl get pods找到启动器(launcher)Pod(例如:
nccl-mpi-128g-dual-domain-launcher-xxxxx)。查看日志:
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl logs -f [LAUNCHER_POD_NAME]注意:实验已开启 NCCL 调试日志 (
NCCL_DEBUG=INFO)。您将在日志中看到类似NCCL INFO NET/gIB : Using ...的输出,这表明 gIB RDMA 网络处于激活状态。清理环境:
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl delete -f /tmp/nccl-mpi-128g-dual-domain.yaml
3. 故障排除 (Troubleshooting)
mpi-operator 反复崩溃或 apiserver 响应超时
如果集群变得不稳定,或者 mpi-operator
因为租约超时(lease timeouts)反复重启,这通常是由于
gpu-operator 默认安装了冗余的 NFD (Node Feature Discovery)
插件引起的。该插件会导致大约 70 个 worker pod 陷入 CrashLoopBackOff
循环,并向 apiserver 发送大量垃圾请求。 要解决此问题,请在更新
gpu-operator 时禁用 NFD:
sudo KUBECONFIG=/etc/kubernetes/admin.conf helm upgrade gpu-operator nvidia/gpu-operator --namespace gpu-operator --reuse-values --set nfd.enabled=false4. YAML 配置说明与原理
本节将解释在验证 YAML 文件中所使用的各项配置参数。整个验证过程基于 MPI Operator 和 NVIDIA NCCL Tests 构建。
4.1 YAML 执行流程与架构
mpi-operator 会负责调度一个 Launcher
节点以及指定数量的 Worker 节点。 * Worker Pods
(工作节点):启动一个基础的 SSH
服务(sshd),并负责拉取和编译 nccl-tests
代码库。 * Launcher Pod
(启动节点):负责等待所有的工作节点就绪。它会在启动脚本中循环查询
Kubernetes API,获取所有 Worker Pod 的 IP 地址,并将其解析写入自身的
/etc/hosts 文件中。 * 等待时间 (Wait Time
120s):在执行真正的 mpirun 前,Launcher
的脚本中硬编码了一个 sleep 120
步骤。这段等待时间非常关键,它的目的是确保所有 Worker
都有充足的时间来执行 apt-get install、克隆 nccl-tests 仓库并完成
make -j MPI=1 编译工作。 * 参考资料: MPI Operator
官方文档
4.2 同域与跨域的 NCCL 环境变量差异
在单纯的 nccl-tests
基准测试中,我们通过显式控制环境变量来强制隔离和测算不同硬件链路的极限带宽。但在实际的生产训练中,情况会有所不同。
- 同域 Same Domain 测试 (验证 NVLink 极限):
- 关键参数:
NCCL_MNNVL_ENABLE=2和NCCL_CUMEM_ENABLE=1 - 作用:强制利用同 Domain 内部的 NVLink 互联。流量将高效地通过 NVSwitch 后端进行路由,而不是走外部网络。
- 关键参数:
- 跨域 Cross Domain 测试 (验证 RDMA 网络极限):
- 关键参数:
NCCL_MNNVL_ENABLE=0和NCCL_CUMEM_ENABLE=0 - 作用:在测试中禁用 MNNVL,是为了强制剥离 NVLink
的影响,让跨柜的
nccl-tests流量 100% 压在 RDMA CX-7 网卡上,从而准确测量 RoCE 网络的真实物理带宽(排除分层通信带来的带宽放大错觉)。
- 关键参数:
- 💡 生产训练场景(如 Megatron-LM)中的真实用法:
- 不要在生产中禁用 MNNVL:在真实的多 Domain
训练任务中,您不需要也不应该设置
NCCL_MNNVL_ENABLE=0。 - NCCL 的自动拓扑感知:生产环境中保持默认开启即可。NCCL 极其聪明,它是拓扑感知 (Topology-Aware) 的。当一个通信组(Communicator)跨越多个机柜时,NCCL 会自动探测两张卡之间的硬件链路:如果是同柜的 GPU,NCCL 会自动走 NVLink;如果是跨柜的 GPU,NCCL 会自动走 RDMA 网卡。它会将这两种网络无缝结合(例如通过分层通信 Hierarchical Rings/Trees)。
- 训练框架的并行策略:现代大模型训练框架(如 Megatron-LM)通过 3D/4D 并行策略来配合硬件拓扑。它们会将对带宽极度饥渴的张量并行 (TP - Tensor Parallel) 和专家并行 (EP) 严格调度在同一个 NVL72 Domain 内(享受 1.8TB/s 高速 NVLink);而将对带宽要求较低的流水线并行 (PP) 和数据并行 (DP) 调度到跨 Domain 的链路上(使用 RoCE/RDMA)。
- 不要在生产中禁用 MNNVL:在真实的多 Domain
训练任务中,您不需要也不应该设置
- 参考资料: NVIDIA NCCL 环境变量官方指南
4.3 节点数量 (Node Count) 修改指南
如果需要增减测试所用的节点数,请确保同时修改以下两处配置: 1.
Worker 副本数:在 YAML 结构中的
mpiReplicaSpecs.Worker 下,将 replicas: X
更改为目标节点数。 2. Launcher 启动参数:在
mpiReplicaSpecs.Launcher 的 entrypoint 脚本中: *
IP 解析循环:修改循环参数 seq 0 Y,这里的
Y 必须等于 replicas - 1(因为索引从 0
开始),以确保所有的 Worker IP 都能被正确解析。 * MPI
进程数:修改 /usr/local/mpi/bin/mpirun 命令中的
-np Z 参数。np 的值必须等于
Worker 节点数 * 每节点 GPU 数(对于 GB200 而言,每节点有 4
张 GPU)。例如:如果测试 2 个节点,则应该设置为 -np 8。
附录:YAML 配置清单
nccl-mpi-64g-single-domain.yaml
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
name: nccl-mpi-64g-single-domain
spec:
slotsPerWorker: 4
runPolicy:
cleanPodPolicy: Running
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
nodeSelector:
nvl72-domain: domain-0001
initContainers:
- name: gib-installer
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
command: ["/bin/sh", "-c"]
args:
- |
set -e
/scripts/container_entry.sh install --install-nccl
cp -a /usr/local/gib/. /target/gib/
cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
mkdir -p /target/gib/lib64/libibverbs
cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
volumeMounts:
- name: gib
mountPath: /target/gib
containers:
- name: launcher
image: nvcr.io/nvidia/pytorch:26.05-py3
imagePullPolicy: IfNotPresent
env:
- name: LD_LIBRARY_PATH
value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
command:
- bash
- -c
- |
set -e
echo "Resolving worker IPs via k8s API..."
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
API="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"
for attempt in $(seq 1 60); do
ALL_READY=true
for i in $(seq 0 15); do
W_IP=$(curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" \
"$API/api/v1/namespaces/default/pods/nccl-mpi-64g-single-domain-worker-${i}" 2>/dev/null \
| python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",{}).get("podIP",""))' 2>/dev/null || echo "")
if [ -z "$W_IP" ]; then
ALL_READY=false
break
fi
echo "$W_IP nccl-mpi-64g-single-domain-worker-${i}.nccl-mpi-64g-single-domain.default.svc nccl-mpi-64g-single-domain-worker-${i}" >> /tmp/hosts_new
done
if [ "$ALL_READY" = true ]; then
cat /tmp/hosts_new >> /etc/hosts
echo "All worker IPs resolved."
break
fi
rm -f /tmp/hosts_new
echo "Waiting for worker IPs (attempt $attempt)..."
sleep 5
done
if [ "$ALL_READY" = false ]; then
echo "ERROR: Could not resolve worker IPs"
exit 1
fi
echo "Waiting 120s for workers to compile nccl-tests..."
sleep 120
echo "Starting mpirun..."
/usr/local/mpi/bin/mpirun --allow-run-as-root \
-np 64 -npernode 4 \
-x LD_LIBRARY_PATH=/usr/local/gib/lib64:/usr/local/nvidia/lib64 \
-x NCCL_NET_PLUGIN=/usr/local/gib/lib64/libnccl-net.so \
-x NCCL_ENV_PLUGIN=gcp \
-x NCCL_DEBUG=INFO \
-x NCCL_MNNVL_ENABLE=2 \
-x NCCL_CUMEM_ENABLE=1 \
--mca plm_rsh_args '-o StrictHostKeyChecking=no' \
/tmp/nccl-tests/build/all_reduce_perf -b 1M -e 16G -f 2 -g 1 -w 50 -n 100
volumeMounts:
- name: gib
mountPath: /usr/local/gib
volumes:
- name: gib
emptyDir: {}
Worker:
replicas: 16
template:
spec:
nodeSelector:
nvl72-domain: domain-0001
dnsPolicy: None
dnsConfig:
nameservers:
- "10.96.0.10"
- "8.8.8.8"
searches:
- "default.svc.cluster.local"
- "svc.cluster.local"
- "cluster.local"
resourceClaims:
- name: compute-domain-channel
resourceClaimTemplateName: domain-1-cd-channel
- name: rdma-nics
resourceClaimTemplateName: rdma-nics-mpi-sd
initContainers:
- name: gib-installer
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
command: ["/bin/sh", "-c"]
args:
- |
set -e
/scripts/container_entry.sh install --install-nccl
cp -a /usr/local/gib/. /target/gib/
cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
mkdir -p /target/gib/lib64/libibverbs
cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
volumeMounts:
- name: gib
mountPath: /target/gib
containers:
- name: worker
image: nvcr.io/nvidia/pytorch:26.05-py3
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
env:
- name: LD_LIBRARY_PATH
value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
- name: NCCL_NET_PLUGIN
value: /usr/local/gib/lib64/libnccl-net.so
- name: NCCL_ENV_PLUGIN
value: gcp
command:
- bash
- -c
- |
apt-get update -qq && apt-get install -y -qq openssh-server > /dev/null 2>&1
mkdir -p /run/sshd
printf 'Port 22\nPermitRootLogin yes\nPubkeyAuthentication yes\nAuthorizedKeysFile /root/.ssh/authorized_keys\nStrictModes no\nPasswordAuthentication no\nUsePAM no\nPidFile /run/sshd/sshd.pid\n' > /etc/ssh/mpi_sshd_config
ssh-keygen -A
for i in $(seq 1 30); do
cd /tmp && rm -rf nccl-tests && git clone --depth 1 https://github.com/NVIDIA/nccl-tests.git && break
echo "git clone attempt $i failed, retrying in 10s..."
sleep 10
done
cd /tmp/nccl-tests && make -j$(nproc) MPI=1 \
MPI_HOME=/usr/local/mpi \
NCCL_HOME=/usr/local/gib \
CUDA_HOME=/usr/local/cuda
/usr/sbin/sshd -De -f /etc/ssh/mpi_sshd_config
volumeMounts:
- name: gib
mountPath: /usr/local/gib
- name: dshm
mountPath: /dev/shm
resources:
requests:
cpu: "96"
memory: 800Gi
limits:
cpu: "96"
memory: 800Gi
nvidia.com/gpu: 4
claims:
- name: compute-domain-channel
- name: rdma-nics
volumes:
- name: gib
emptyDir: {}
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 250Gi
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
name: rdma-nics-mpi-sd
spec:
spec:
devices:
requests:
- name: nic-0
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu0rdma0"
- name: nic-1
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu1rdma0"
- name: nic-2
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu2rdma0"
- name: nic-3
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu3rdma0"nccl-mpi-128g-dual-domain.yaml
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
name: nccl-mpi-128g-dual-domain
spec:
slotsPerWorker: 4
runPolicy:
cleanPodPolicy: Running
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
initContainers:
- name: gib-installer
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
command: ["/bin/sh", "-c"]
args:
- |
set -e
/scripts/container_entry.sh install --install-nccl
cp -a /usr/local/gib/. /target/gib/
cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
mkdir -p /target/gib/lib64/libibverbs
cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
volumeMounts:
- name: gib
mountPath: /target/gib
containers:
- name: launcher
image: nvcr.io/nvidia/pytorch:26.05-py3
imagePullPolicy: IfNotPresent
env:
- name: LD_LIBRARY_PATH
value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
command:
- bash
- -c
- |
set -e
echo "Resolving worker IPs via k8s API..."
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
API="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"
for attempt in $(seq 1 60); do
ALL_READY=true
for i in $(seq 0 31); do
W_IP=$(curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" \
"$API/api/v1/namespaces/default/pods/nccl-mpi-128g-dual-domain-worker-${i}" 2>/dev/null \
| python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",{}).get("podIP",""))' 2>/dev/null || echo "")
if [ -z "$W_IP" ]; then
ALL_READY=false
break
fi
echo "$W_IP nccl-mpi-128g-dual-domain-worker-${i}.nccl-mpi-128g-dual-domain.default.svc nccl-mpi-128g-dual-domain-worker-${i}" >> /tmp/hosts_new
done
if [ "$ALL_READY" = true ]; then
cat /tmp/hosts_new >> /etc/hosts
echo "All worker IPs resolved."
break
fi
rm -f /tmp/hosts_new
echo "Waiting for worker IPs (attempt $attempt)..."
sleep 5
done
if [ "$ALL_READY" = false ]; then
echo "ERROR: Could not resolve worker IPs"
exit 1
fi
echo "Waiting 120s for workers to compile nccl-tests..."
sleep 120
echo "Starting mpirun..."
# MNNVL 和 CUMEM 设置为 0,启用跨域 RoCE
/usr/local/mpi/bin/mpirun --allow-run-as-root \
-np 128 -npernode 4 \
-x LD_LIBRARY_PATH=/usr/local/gib/lib64:/usr/local/nvidia/lib64 \
-x NCCL_NET_PLUGIN=/usr/local/gib/lib64/libnccl-net.so \
-x NCCL_ENV_PLUGIN=gcp \
-x NCCL_DEBUG=INFO \
-x NCCL_MNNVL_ENABLE=0 \
-x NCCL_CUMEM_ENABLE=0 \
--mca plm_rsh_args '-o StrictHostKeyChecking=no' \
/tmp/nccl-tests/build/all_reduce_perf -b 1M -e 16G -f 2 -g 1 -w 50 -n 100
volumeMounts:
- name: gib
mountPath: /usr/local/gib
volumes:
- name: gib
emptyDir: {}
Worker:
replicas: 32
template:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: nvl72-domain
operator: In
values: ["domain-0001", "domain-0002"]
topologySpreadConstraints:
- maxSkew: 1
topologyKey: nvl72-domain
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
training.kubeflow.org/job-name: nccl-mpi-128g-dual-domain
dnsPolicy: None
dnsConfig:
nameservers:
- "10.96.0.10"
- "8.8.8.8"
searches:
- "default.svc.cluster.local"
- "svc.cluster.local"
- "cluster.local"
resourceClaims:
- name: rdma-nics
resourceClaimTemplateName: rdma-nics-mpi-sd
initContainers:
- name: gib-installer
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
command: ["/bin/sh", "-c"]
args:
- |
set -e
/scripts/container_entry.sh install --install-nccl
cp -a /usr/local/gib/. /target/gib/
cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
mkdir -p /target/gib/lib64/libibverbs
cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
volumeMounts:
- name: gib
mountPath: /target/gib
containers:
- name: worker
image: nvcr.io/nvidia/pytorch:26.05-py3
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
env:
- name: LD_LIBRARY_PATH
value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
- name: NCCL_NET_PLUGIN
value: /usr/local/gib/lib64/libnccl-net.so
- name: NCCL_ENV_PLUGIN
value: gcp
command:
- bash
- -c
- |
# 跨域路由和 IP 自动发现脚本 (简化示意,需根据实际网络配置实现 setup_rdma_ips.py)
# python3 /usr/local/bin/setup_rdma_ips.py
apt-get update -qq && apt-get install -y -qq openssh-server > /dev/null 2>&1
mkdir -p /run/sshd
printf 'Port 22\nPermitRootLogin yes\nPubkeyAuthentication yes\nAuthorizedKeysFile /root/.ssh/authorized_keys\nStrictModes no\nPasswordAuthentication no\nUsePAM no\nPidFile /run/sshd/sshd.pid\n' > /etc/ssh/mpi_sshd_config
ssh-keygen -A
for i in $(seq 1 30); do
cd /tmp && rm -rf nccl-tests && git clone --depth 1 https://github.com/NVIDIA/nccl-tests.git && break
echo "git clone attempt $i failed, retrying in 10s..."
sleep 10
done
cd /tmp/nccl-tests && make -j$(nproc) MPI=1 \
MPI_HOME=/usr/local/mpi \
NCCL_HOME=/usr/local/gib \
CUDA_HOME=/usr/local/cuda
/usr/sbin/sshd -De -f /etc/ssh/mpi_sshd_config
volumeMounts:
- name: gib
mountPath: /usr/local/gib
- name: dshm
mountPath: /dev/shm
resources:
requests:
cpu: "96"
memory: 800Gi
limits:
cpu: "96"
memory: 800Gi
nvidia.com/gpu: 4
claims:
- name: rdma-nics
volumes:
- name: gib
emptyDir: {}
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 250Gi
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
name: rdma-nics-mpi-sd
spec:
spec:
devices:
requests:
- name: nic-0
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu0rdma0"
- name: nic-1
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu1rdma0"
- name: nic-2
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu2rdma0"
- name: nic-3
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu3rdma0"nccl-mpi-8g-single-domain-2nodes.yaml
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
name: nccl-mpi-8g-single-domain-2nodes
spec:
slotsPerWorker: 4
runPolicy:
cleanPodPolicy: Running
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
nodeSelector:
nvl72-domain: domain-0001
initContainers:
- name: gib-installer
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
command: ["/bin/sh", "-c"]
args:
- |
set -e
/scripts/container_entry.sh install --install-nccl
cp -a /usr/local/gib/. /target/gib/
cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
mkdir -p /target/gib/lib64/libibverbs
cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
volumeMounts:
- name: gib
mountPath: /target/gib
containers:
- name: launcher
image: nvcr.io/nvidia/pytorch:26.05-py3
imagePullPolicy: IfNotPresent
env:
- name: LD_LIBRARY_PATH
value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
command:
- bash
- -c
- |
set -e
echo "Resolving worker IPs via k8s API..."
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
API="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"
for attempt in $(seq 1 60); do
ALL_READY=true
for i in $(seq 0 1); do
W_IP=$(curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" "$API/api/v1/namespaces/default/pods/nccl-mpi-8g-single-domain-2nodes-worker-${i}" 2>/dev/null | python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",{}).get("podIP",""))' 2>/dev/null || echo "")
if [ -z "$W_IP" ]; then
ALL_READY=false
break
fi
echo "$W_IP nccl-mpi-8g-single-domain-2nodes-worker-${i}.nccl-mpi-8g-single-domain-2nodes.default.svc nccl-mpi-8g-single-domain-2nodes-worker-${i}" >> /tmp/hosts_new
done
if [ "$ALL_READY" = true ]; then
cat /tmp/hosts_new >> /etc/hosts
echo "All worker IPs resolved."
break
fi
rm -f /tmp/hosts_new
echo "Waiting for worker IPs (attempt $attempt)..."
sleep 5
done
if [ "$ALL_READY" = false ]; then
echo "ERROR: Could not resolve worker IPs"
exit 1
fi
echo "Waiting 120s for workers to compile nccl-tests..."
sleep 120
echo "Starting mpirun..."
/usr/local/mpi/bin/mpirun --allow-run-as-root -np 8 -npernode 4 -x LD_LIBRARY_PATH=/usr/local/gib/lib64:/usr/local/nvidia/lib64 -x NCCL_NET_PLUGIN=/usr/local/gib/lib64/libnccl-net.so -x NCCL_ENV_PLUGIN=gcp -x NCCL_DEBUG=INFO -x NCCL_MNNVL_ENABLE=2 -x NCCL_CUMEM_ENABLE=1 --mca plm_rsh_args '-o StrictHostKeyChecking=no' /tmp/nccl-tests/build/all_reduce_perf -b 1M -e 16G -f 2 -g 1 -w 50 -n 100
volumeMounts:
- name: gib
mountPath: /usr/local/gib
volumes:
- name: gib
emptyDir: {}
Worker:
replicas: 2
template:
spec:
nodeSelector:
nvl72-domain: domain-0001
dnsPolicy: None
dnsConfig:
nameservers:
- "10.96.0.10"
- "8.8.8.8"
searches:
- "default.svc.cluster.local"
- "svc.cluster.local"
- "cluster.local"
resourceClaims:
- name: compute-domain-channel
resourceClaimTemplateName: domain-1-cd-channel
- name: rdma-nics
resourceClaimTemplateName: rdma-nics-mpi-sd
initContainers:
- name: gib-installer
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
command: ["/bin/sh", "-c"]
args:
- |
set -e
/scripts/container_entry.sh install --install-nccl
cp -a /usr/local/gib/. /target/gib/
cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
mkdir -p /target/gib/lib64/libibverbs
cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
volumeMounts:
- name: gib
mountPath: /target/gib
containers:
- name: worker
image: nvcr.io/nvidia/pytorch:26.05-py3
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
env:
- name: LD_LIBRARY_PATH
value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
- name: NCCL_NET_PLUGIN
value: /usr/local/gib/lib64/libnccl-net.so
- name: NCCL_ENV_PLUGIN
value: gcp
command:
- bash
- -c
- |
apt-get update -qq && apt-get install -y -qq openssh-server > /dev/null 2>&1
mkdir -p /run/sshd
printf 'Port 22\nPermitRootLogin yes\nPubkeyAuthentication yes\nAuthorizedKeysFile /root/.ssh/authorized_keys\nStrictModes no\nPasswordAuthentication no\nUsePAM no\nPidFile /run/sshd/sshd.pid\n' > /etc/ssh/mpi_sshd_config
ssh-keygen -A
for i in $(seq 1 30); do
cd /tmp && rm -rf nccl-tests && git clone --depth 1 https://github.com/NVIDIA/nccl-tests.git && break
echo "git clone attempt $i failed, retrying in 10s..."
sleep 10
done
cd /tmp/nccl-tests && make -j$(nproc) MPI=1 MPI_HOME=/usr/local/mpi NCCL_HOME=/usr/local/gib CUDA_HOME=/usr/local/cuda
/usr/sbin/sshd -De -f /etc/ssh/mpi_sshd_config
volumeMounts:
- name: gib
mountPath: /usr/local/gib
- name: dshm
mountPath: /dev/shm
resources:
requests:
cpu: "96"
memory: 800Gi
limits:
cpu: "96"
memory: 800Gi
nvidia.com/gpu: 4
claims:
- name: compute-domain-channel
- name: rdma-nics
volumes:
- name: gib
emptyDir: {}
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 250Gi
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
name: rdma-nics-mpi-sd
spec:
spec:
devices:
requests:
- name: nic-0
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu0rdma0"
- name: nic-1
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu1rdma0"
- name: nic-2
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu2rdma0"
- name: nic-3
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu3rdma0"nccl-mpi-8g-cross-domain-2nodes.yaml
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
name: nccl-mpi-8g-cross-domain-2nodes
spec:
slotsPerWorker: 4
runPolicy:
cleanPodPolicy: Running
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
initContainers:
- name: gib-installer
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
command: ["/bin/sh", "-c"]
args:
- |
set -e
/scripts/container_entry.sh install --install-nccl
cp -a /usr/local/gib/. /target/gib/
cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
mkdir -p /target/gib/lib64/libibverbs
cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
volumeMounts:
- name: gib
mountPath: /target/gib
containers:
- name: launcher
image: nvcr.io/nvidia/pytorch:26.05-py3
imagePullPolicy: IfNotPresent
env:
- name: LD_LIBRARY_PATH
value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
command:
- bash
- -c
- |
set -e
echo "Resolving worker IPs via k8s API..."
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
API="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"
for attempt in $(seq 1 60); do
ALL_READY=true
for i in $(seq 0 1); do
W_IP=$(curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" "$API/api/v1/namespaces/default/pods/nccl-mpi-8g-cross-domain-2nodes-worker-${i}" 2>/dev/null | python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",{}).get("podIP",""))' 2>/dev/null || echo "")
if [ -z "$W_IP" ]; then
ALL_READY=false
break
fi
echo "$W_IP nccl-mpi-8g-cross-domain-2nodes-worker-${i}.nccl-mpi-8g-cross-domain-2nodes.default.svc nccl-mpi-8g-cross-domain-2nodes-worker-${i}" >> /tmp/hosts_new
done
if [ "$ALL_READY" = true ]; then
cat /tmp/hosts_new >> /etc/hosts
echo "All worker IPs resolved."
break
fi
rm -f /tmp/hosts_new
echo "Waiting for worker IPs (attempt $attempt)..."
sleep 5
done
if [ "$ALL_READY" = false ]; then
echo "ERROR: Could not resolve worker IPs"
exit 1
fi
echo "Waiting 120s for workers to compile nccl-tests..."
sleep 120
echo "Starting mpirun..."
/usr/local/mpi/bin/mpirun --allow-run-as-root -np 8 -npernode 4 -x LD_LIBRARY_PATH=/usr/local/gib/lib64:/usr/local/nvidia/lib64 -x NCCL_NET_PLUGIN=/usr/local/gib/lib64/libnccl-net.so -x NCCL_ENV_PLUGIN=gcp -x NCCL_DEBUG=INFO -x NCCL_MNNVL_ENABLE=0 -x NCCL_CUMEM_ENABLE=0 --mca plm_rsh_args '-o StrictHostKeyChecking=no' /tmp/nccl-tests/build/all_reduce_perf -b 1M -e 16G -f 2 -g 1 -w 50 -n 100
volumeMounts:
- name: gib
mountPath: /usr/local/gib
volumes:
- name: gib
emptyDir: {}
Worker:
replicas: 2
template:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: nvl72-domain
operator: In
values: ["domain-0001", "domain-0002"]
topologySpreadConstraints:
- maxSkew: 1
topologyKey: nvl72-domain
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
training.kubeflow.org/job-name: nccl-mpi-8g-cross-domain-2nodes
dnsPolicy: None
dnsConfig:
nameservers:
- "10.96.0.10"
- "8.8.8.8"
searches:
- "default.svc.cluster.local"
- "svc.cluster.local"
- "cluster.local"
resourceClaims:
- name: rdma-nics
resourceClaimTemplateName: rdma-nics-mpi-sd
initContainers:
- name: gib-installer
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
command: ["/bin/sh", "-c"]
args:
- |
set -e
/scripts/container_entry.sh install --install-nccl
cp -a /usr/local/gib/. /target/gib/
cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
mkdir -p /target/gib/lib64/libibverbs
cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
volumeMounts:
- name: gib
mountPath: /target/gib
containers:
- name: worker
image: nvcr.io/nvidia/pytorch:26.05-py3
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
env:
- name: LD_LIBRARY_PATH
value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
- name: NCCL_NET_PLUGIN
value: /usr/local/gib/lib64/libnccl-net.so
- name: NCCL_ENV_PLUGIN
value: gcp
command:
- bash
- -c
- |
apt-get update -qq && apt-get install -y -qq openssh-server > /dev/null 2>&1
mkdir -p /run/sshd
printf 'Port 22\nPermitRootLogin yes\nPubkeyAuthentication yes\nAuthorizedKeysFile /root/.ssh/authorized_keys\nStrictModes no\nPasswordAuthentication no\nUsePAM no\nPidFile /run/sshd/sshd.pid\n' > /etc/ssh/mpi_sshd_config
ssh-keygen -A
for i in $(seq 1 30); do
cd /tmp && rm -rf nccl-tests && git clone --depth 1 https://github.com/NVIDIA/nccl-tests.git && break
echo "git clone attempt $i failed, retrying in 10s..."
sleep 10
done
cd /tmp/nccl-tests && make -j$(nproc) MPI=1 MPI_HOME=/usr/local/mpi NCCL_HOME=/usr/local/gib CUDA_HOME=/usr/local/cuda
/usr/sbin/sshd -De -f /etc/ssh/mpi_sshd_config
volumeMounts:
- name: gib
mountPath: /usr/local/gib
- name: dshm
mountPath: /dev/shm
resources:
requests:
cpu: "96"
memory: 800Gi
limits:
cpu: "96"
memory: 800Gi
nvidia.com/gpu: 4
claims:
- name: rdma-nics
volumes:
- name: gib
emptyDir: {}
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 250Gi
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
name: rdma-nics-mpi-sd
spec:
spec:
devices:
requests:
- name: nic-0
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu0rdma0"
- name: nic-1
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu1rdma0"
- name: nic-2
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu2rdma0"
- name: nic-3
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu3rdma0"