Google Cloud NCCL / gIB 官方说明

面向:在 Google Cloud GB200(A4X)/ GB300(A4X Max)等 RDMA GPU 机型上跑 NCCL / 分布式训练的用户。 本文以 Google Cloud 官方文档 + 官方 GitHub 为依据整理,可直接转发。 结尾「附录」为我方在统一镜像 v18dot2 上的实测交付细节,供对照。


1. 这是做什么的

gIB(Google InfiniBand)是 Google Cloud 为其 RoCE / RDMA GPU 机型深度优化的一套 NCCL 插件, 让 NCCL 走 GPUDirect RDMA 而非普通 TCP。官方定性:

*"NCCL/gIB is often more performant than upstream NCCL on Google infrastructure."*

四个组件(一套装,协同工作):

组件 作用
Network Plugin(gIB 本体,libnccl-net.so)NCCL 的 RoCE RDMA 网络后端,提供 guest-level 多路径、高吞吐低延迟
Environment Plugin(libnccl-env-gcp.so)运行时自动探测机型(GB200/GB300/H200…),套用该机型最优参数
Tuner Plugin(libnccl-tuner.so)按 GCP 硬件拓扑动态优化集合通信
Profiler Plugin(CoMMA,libnccl-profiler.so)性能遥测与诊断(此组件开源,见 §3)

适用机型:A4X Max = GB300(裸金属)、A4X = GB200(VM)、A4 = B200、A3 Ultra = H200。

官方性能数据:AllReduce 特定消息尺寸最高 ~12×;AllGather 拥塞下大消息 ~50% 提升 (对比未优化的 upstream NCCL)。

开源状况(如实告知):


2. 如何使用

统一安装落位:/usr/local/gib/(内含 lib64/ 库、configs/ 各机型配置、scripts/set_nccl_env.sh)。

2.1 Compute Engine VM / 裸机 / Slurm —— 用系统包管理器装

# RHEL / Rocky / CentOS / TencentOS(aarch64 有包)
sudo tee /etc/yum.repos.d/nccl-gib.repo <<'EOL'
[gpudirect-gib-rpm]
name=NCCL/gIB
baseurl=https://packages.cloud.google.com/yum/repos/gpudirect-gib-rpm
enabled=1
repo_gpgcheck=0
gpgcheck=0
EOL
sudo dnf install nccl-gib

# Debian / Ubuntu 则用 apt 源 gpudirect-gib-apt,包名同为 nccl-gib

包区别:nccl-gib = NCCL 本体 + gIB 插件(一站式);nccl-gib-plugins(v1.1.2+)= 仅插件,需另装 NCCL。

2.2 容器(Docker / Kubernetes)—— 官方要求把 gIB 放进容器,两种方式

2.3 NVIDIA NGC 容器(2026-07 起)

gIB 已内置,无需 initContainer / 手动安装,插件自动探测 GCP 平台并启用。

2.4 运行时环境变量(官方写法)

export NCCL_NET=gIB
source /usr/local/gib/scripts/set_nccl_env.sh          # 自动套本机机型最优参数
export LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/gib/lib64:$LD_LIBRARY_PATH
export NCCL_DEBUG=INFO                                   # 验证时用

自 v1.1.2 起,只要装了 nccl-gib,gIB 一般会自动启用。set_nccl_env.sh 会按机型加载 nccl.a4xmax.conf(GB300)/ nccl.a4x.conf(GB200)。

口径注(如实标注):该脚本内部即 export NCCL_ENV_PLUGIN=gcp。NCCL_ENV_PLUGIN 这个变量名 官方文档未单独列出——对外说明请以官方 source set_nccl_env.sh 写法为准;二者等价。

2.5 docker run 单机冒烟样例

docker run --rm -it --network host --ipc host --gpus all --cap-add IPC_LOCK \
  --device /dev/infiniband -v /usr/local/gib:/usr/local/gib:ro \
  -e NCCL_NET=gIB -e LD_LIBRARY_PATH=/usr/local/gib/lib64 -e NCCL_DEBUG=INFO \
  <nccl-tests 镜像> \
  bash -lc 'source /usr/local/gib/scripts/set_nccl_env.sh && all_reduce_perf -b 1M -e 8G -f 2 -g 4'

2.6 验证成功标志(NCCL_DEBUG=INFO 日志)

2.7 官方警告(务必转达)

  1. 不要在这些机型上装旧插件(FastSocket / TCPX / TCPXO)。
  2. A4X Max(GB300)需额外装 DOCA-OFED 更新用户态 RDMA 库。
  3. 容器场景插件必须集成进镜像或挂载,不能只装宿主机就指望容器内可用。

3. 官方文档 / GitHub 链接清单

官方文档(AI Hypercomputer,建议直接发给客户)

主题 链接
总览与架构 / 性能https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/overview
安装 gIBhttps://docs.cloud.google.com/ai-hypercomputer/docs/nccl/install-nccl
NGC 容器内置 gIBhttps://docs.cloud.google.com/ai-hypercomputer/docs/nccl/ngc-containers
日志 / 验证 / 排障https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/collect-and-understand
VM 上跑 NCCLhttps://docs.cloud.google.com/ai-hypercomputer/docs/nccl/test-vms
GKE A4X(GB200)https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/test-gke-custom-a4x
GKE A4X Max(GB300)https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/test-gke-custom-a4x-max
A4X Max Slurm 集群https://docs.cloud.google.com/cluster-toolkit/docs/deploy/slurm/create-a4x-max-cluster
版本说明(ARM64,GB200/GB300)https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/release-notes-arm64
版本说明(x86)https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/release-notes-x86
CoMMA 遥测配置https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/configure-comma
GPU 机型规格https://docs.cloud.google.com/ai-hypercomputer/docs/gpu

GitHub(开源配套)

仓库 说明
https://github.com/GoogleCloudPlatform/cluster-toolkit部署蓝图 / 示例
https://github.com/GoogleCloudPlatform/container-engine-acceleratorsGKE GPU 安装器(含 A4X gIB DaemonSet)
https://github.com/AI-Hypercomputer/xpkNCCL 测试样例(examples/nccl/nccl.md)
https://github.com/google/CoMMAProfiler 组件,Apache-2.0(唯一开源的插件本体)
https://github.com/google/nccl-plugin-gpudirecttcpx(非 gIB,A3 TCP 插件,供对照——GB200/GB300 勿用)

容器镜像 / 包源


附录 A:平台配置实际取值(机型差异)

取自 nccl-gib-1.1.2 的 configs/(set_nccl_env.sh / NCCL_ENV_PLUGIN=gcp 会按机型自动套用)。

nccl.a4xmax.conf —— GB300 / A4X Max

变量 值
NCCL_NETgIB
NCCL_PXN_C2C1
NCCL_IB_MERGE_VFS1
NCCL_IB_ADAPTIVE_ROUTING1
NCCL_IB_ADDR_FAMILYAF_INET6
NCCL_IB_TC52
NCCL_IB_FIFO_TC84
NCCL_IB_QPS_PER_CONNECTION4
NCCL_NCHANNELS_PER_NET_PEER1
NCCL_P2P_NET_CHUNKSIZE262144
NCCL_TUNER_PLUGINnone

nccl.a4x.conf —— GB200 / A4X

变量 值
NCCL_NETgIB
NCCL_PXN_C2C1
NCCL_IB_ADAPTIVE_ROUTING1
NCCL_IB_QPS_PER_CONNECTION4
NCCL_IB_TC52
NCCL_IB_FIFO_TC84
NCCL_TUNER_PLUGINnone

GB300 用 IPv6(AF_INET6)做 IB 地址族,GB200 用默认 IPv4——这是两机型 gIB 配置的主要差异之一。


附录 B:我方实测交付细节(v18dot2 统一镜像)

以下为我方在 tlinux 统一镜像上的落地情况,供对照;转发外部时可按需删减。

bin/       ncclras, ncclparam
configs/   nccl.conf, nccl.a4x.conf(GB200), nccl.a4xmax.conf(GB300), nccl.a3u/a4/a5x.conf
lib64/     libnccl-net.so(gIB 网络插件)  libnccl-env-gcp.so(env 插件)
           libnccl-tuner.so  libnccl.so.2.30.4(捆绑 NCCL)  libGPUViz.so  libnccl-profiler.so
scripts/   set_nccl_env.sh, run_nccl_tests.sh, gen_hostfiles.sh, init_ssh.sh, show_gids.sh

附录 C:实测验证过的 K8s 样例(GB200)

以下为我方在 GB200(A4X,NVL72,4 GPU/节点,aarch64) 上内部实测通过的 NCCL 测试清单, 采用 Kubeflow MPIJob v2beta1。共 4 个变体(单域 / 跨域 × 大规模 / 2 节点),下面给出 1 份 完整 canonical 清单 + 变体差异表,可据此拼出其余 3 个。

⚠️ 两处务必注意:

  1. 这些清单用 initContainer(gib-installer)把 gIB 拷进 emptyDir,适用于节点镜像未内置 gIB 的情况。 若使用我方 v18dot2 镜像(已把 gIB 烤进宿主机 /usr/local/gib),可删掉整个 gib-installer initContainer, 改为 hostPath 挂载 /usr/local/gib(见 §2.2 / 附录 B),两者等效且更简。
  2. 这里 env 用的是 NCCL_NET_PLUGIN=/usr/local/gib/lib64/libnccl-net.so(显式指库路径)+ NCCL_ENV_PLUGIN=gcp, 未用 NCCL_NET=gIB——三者都是有效写法(见 §2.4 口径注)。
  3. 本组仅覆盖 GB200,未含 GB300;GB300 机型由 env 插件自动套 nccl.a4xmax.conf,但 RDMA 网卡名/数量不同 (GB300 约 8 张 RDMA),ResourceClaimTemplate 的 ifName 选择器需按实际 rdma dev 调整。

C.1 完整 canonical 清单 —— nccl-mpi-64g-single-domain.yaml(单 NVL72 域,64 GPU / 16 worker)

apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: nccl-mpi-64g-single-domain
spec:
  slotsPerWorker: 4
  runPolicy:
    cleanPodPolicy: Running
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          nodeSelector:
            nvl72-domain: domain-0001
          initContainers:
          - name: gib-installer
            image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
            command: ["/bin/sh", "-c"]
            args:
            - |
              set -e
              /scripts/container_entry.sh install --install-nccl
              cp -a /usr/local/gib/. /target/gib/
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
              mkdir -p /target/gib/lib64/libibverbs
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
            volumeMounts:
            - name: gib
              mountPath: /target/gib
          containers:
          - name: launcher
            image: nvcr.io/nvidia/pytorch:26.05-py3
            imagePullPolicy: IfNotPresent
            env:
            - name: LD_LIBRARY_PATH
              value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
            command:
            - bash
            - -c
            - |
              set -e
              echo "Resolving worker IPs via k8s API..."
              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              API="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"

              for attempt in $(seq 1 60); do
                ALL_READY=true
                for i in $(seq 0 15); do
                  W_IP=$(curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" \
                    "$API/api/v1/namespaces/default/pods/nccl-mpi-64g-single-domain-worker-${i}" 2>/dev/null \
                    | python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",{}).get("podIP",""))' 2>/dev/null || echo "")
                  if [ -z "$W_IP" ]; then
                    ALL_READY=false
                    break
                  fi
                  echo "$W_IP nccl-mpi-64g-single-domain-worker-${i}.nccl-mpi-64g-single-domain.default.svc nccl-mpi-64g-single-domain-worker-${i}" >> /tmp/hosts_new
                done

                if [ "$ALL_READY" = true ]; then
                  cat /tmp/hosts_new >> /etc/hosts
                  echo "All worker IPs resolved."
                  break
                fi
                rm -f /tmp/hosts_new
                echo "Waiting for worker IPs (attempt $attempt)..."
                sleep 5
              done

              if [ "$ALL_READY" = false ]; then
                echo "ERROR: Could not resolve worker IPs"
                exit 1
              fi

              echo "Waiting 120s for workers to compile nccl-tests..."
              sleep 120
              echo "Starting mpirun..."
              /usr/local/mpi/bin/mpirun --allow-run-as-root \
                -np 64 -npernode 4 \
                -x LD_LIBRARY_PATH=/usr/local/gib/lib64:/usr/local/nvidia/lib64 \
                -x NCCL_NET_PLUGIN=/usr/local/gib/lib64/libnccl-net.so \
                -x NCCL_ENV_PLUGIN=gcp \
                -x NCCL_DEBUG=INFO \
                -x NCCL_MNNVL_ENABLE=2 \
                -x NCCL_CUMEM_ENABLE=1 \
                --mca plm_rsh_args '-o StrictHostKeyChecking=no' \
                /tmp/nccl-tests/build/all_reduce_perf -b 1M -e 16G -f 2 -g 1 -w 50 -n 100
            volumeMounts:
            - name: gib
              mountPath: /usr/local/gib
          volumes:
          - name: gib
            emptyDir: {}
    Worker:
      replicas: 16
      template:
        spec:
          nodeSelector:
            nvl72-domain: domain-0001
          dnsPolicy: None
          dnsConfig:
            nameservers:
            - "10.96.0.10"
            - "8.8.8.8"
            searches:
            - "default.svc.cluster.local"
            - "svc.cluster.local"
            - "cluster.local"
          resourceClaims:
          - name: compute-domain-channel
            resourceClaimTemplateName: domain-1-cd-channel
          - name: rdma-nics
            resourceClaimTemplateName: rdma-nics-mpi-sd
          initContainers:
          - name: gib-installer
            image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
            command: ["/bin/sh", "-c"]
            args:
            - |
              set -e
              /scripts/container_entry.sh install --install-nccl
              cp -a /usr/local/gib/. /target/gib/
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
              mkdir -p /target/gib/lib64/libibverbs
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
            volumeMounts:
            - name: gib
              mountPath: /target/gib
          containers:
          - name: worker
            image: nvcr.io/nvidia/pytorch:26.05-py3
            imagePullPolicy: IfNotPresent
            securityContext:
              privileged: true
            env:
            - name: LD_LIBRARY_PATH
              value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
            - name: NCCL_NET_PLUGIN
              value: /usr/local/gib/lib64/libnccl-net.so
            - name: NCCL_ENV_PLUGIN
              value: gcp
            command:
            - bash
            - -c
            - |
              apt-get update -qq && apt-get install -y -qq openssh-server > /dev/null 2>&1
              mkdir -p /run/sshd
              printf 'Port 22\nPermitRootLogin yes\nPubkeyAuthentication yes\nAuthorizedKeysFile /root/.ssh/authorized_keys\nStrictModes no\nPasswordAuthentication no\nUsePAM no\nPidFile /run/sshd/sshd.pid\n' > /etc/ssh/mpi_sshd_config
              ssh-keygen -A
              for i in $(seq 1 30); do
                cd /tmp && rm -rf nccl-tests && git clone --depth 1 https://github.com/NVIDIA/nccl-tests.git && break
                echo "git clone attempt $i failed, retrying in 10s..."
                sleep 10
              done
              cd /tmp/nccl-tests && make -j$(nproc) MPI=1 \
                MPI_HOME=/usr/local/mpi \
                NCCL_HOME=/usr/local/gib \
                CUDA_HOME=/usr/local/cuda
              /usr/sbin/sshd -De -f /etc/ssh/mpi_sshd_config
            volumeMounts:
            - name: gib
              mountPath: /usr/local/gib
            - name: dshm
              mountPath: /dev/shm
            resources:
              requests:
                cpu: "96"
                memory: 800Gi
              limits:
                cpu: "96"
                memory: 800Gi
                nvidia.com/gpu: 4
              claims:
              - name: compute-domain-channel
              - name: rdma-nics
          volumes:
          - name: gib
            emptyDir: {}
          - name: dshm
            emptyDir:
              medium: Memory
              sizeLimit: 250Gi
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: rdma-nics-mpi-sd
spec:
  spec:
    devices:
      requests:
      - name: nic-0
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu0rdma0"
      - name: nic-1
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu1rdma0"
      - name: nic-2
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu2rdma0"
      - name: nic-3
        exactly:
          deviceClassName: rdma-devices
          count: 1
          selectors:
          - cel:
              expression: device.attributes["dra.net"].ifName == "gpu3rdma0"

C.2 变体差异表(其余 3 个清单 = C.1 改这几处)

all_reduce_perf 参数四个变体完全一致:-b 1M -e 16G -f 2 -g 1 -w 50 -n 100;差异只在规模与「域」两类:

变体 Worker replicas IP 循环 seq 0 N -np Worker 调度 Worker resourceClaims NCCL_MNNVL_ENABLE / NCCL_CUMEM_ENABLE
64G 单域(C.1)16seq 0 1564nodeSelector: nvl72-domain=domain-0001compute-domain-channel + rdma-nics2 / 1
128G 跨域32seq 0 31128nodeAffinity(domain-0001/0002)+ topologySpreadConstraints仅 rdma-nics0 / 0
8G 单域(2 节点)2seq 0 18nodeSelector: nvl72-domain=domain-0001compute-domain-channel + rdma-nics2 / 1
8G 跨域(2 节点)2seq 0 18nodeAffinity(domain-0001/0002)+ topologySpreadConstraints仅 rdma-nics0 / 0

「域」语义(关键):

跨域 Worker 调度片段(替换 C.1 里 Worker 的 nodeSelector 那两行):

          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: nvl72-domain
                    operator: In
                    values: ["domain-0001", "domain-0002"]
          topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: nvl72-domain
            whenUnsatisfiable: DoNotSchedule
            labelSelector:
              matchLabels:
                training.kubeflow.org/job-name: <MPIJob 的 metadata.name>

(跨域时 Worker resourceClaims 只保留 rdma-nics,删掉 compute-domain-channel;Launcher 删掉 nodeSelector。)

C.3 nccl-tests 编译命令(Worker 容器内,四个清单一致)

cd /tmp/nccl-tests && make -j$(nproc) MPI=1 \
  MPI_HOME=/usr/local/mpi \
  NCCL_HOME=/usr/local/gib \
  CUDA_HOME=/usr/local/cuda

C.4 提交与查看

sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl apply -f nccl-mpi-64g-single-domain.yaml
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl get pods
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl logs -f <LAUNCHER_POD_NAME>
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl delete -f nccl-mpi-64g-single-domain.yaml

C.5 成功标志

本组清单实测时,Launcher 日志出现下面这行即表示 gIB RDMA 网络已激活(与 §2.6 的 Using network gIB 等义):

NCCL INFO NET/gIB : Using ...

注:内部记录未保存带实际带宽数字的完整成功日志;此处为激活标志行。