Google Cloud NCCL / gIB 官方说明
面向:在 Google Cloud GB200(A4X)/ GB300(A4X Max)等 RDMA GPU 机型上跑 NCCL / 分布式训练的用户。 本文以 Google Cloud 官方文档 + 官方 GitHub 为依据整理,可直接转发。 结尾「附录」为我方在统一镜像 v18dot2 上的实测交付细节,供对照。
1. 这是做什么的
gIB(Google InfiniBand)是 Google Cloud 为其 RoCE / RDMA GPU 机型深度优化的一套 NCCL 插件, 让 NCCL 走 GPUDirect RDMA 而非普通 TCP。官方定性:
*"NCCL/gIB is often more performant than upstream NCCL on Google infrastructure."*
四个组件(一套装,协同工作):
| 组件 | 作用 |
|---|---|
Network Plugin(gIB 本体,libnccl-net.so) | NCCL 的 RoCE RDMA 网络后端,提供 guest-level 多路径、高吞吐低延迟 |
Environment Plugin(libnccl-env-gcp.so) | 运行时自动探测机型(GB200/GB300/H200…),套用该机型最优参数 |
Tuner Plugin(libnccl-tuner.so) | 按 GCP 硬件拓扑动态优化集合通信 |
Profiler Plugin(CoMMA,libnccl-profiler.so) | 性能遥测与诊断(此组件开源,见 §3) |
适用机型:A4X Max = GB300(裸金属)、A4X = GB200(VM)、A4 = B200、A3 Ultra = H200。
官方性能数据:AllReduce 特定消息尺寸最高 ~12×;AllGather 拥塞下大消息 ~50% 提升 (对比未优化的 upstream NCCL)。
开源状况(如实告知):
- gIB 网络插件本体闭源——
libnccl-net.so、libnccl-env-gcp.so仅以预编译包 + 容器镜像分发。 - 开源的是配套件:部署蓝图/示例(cluster-toolkit)、GKE 安装器(container-engine-accelerators)、 测试样例(xpk)、遥测库(CoMMA)。详见 §3。
2. 如何使用
统一安装落位:/usr/local/gib/(内含 lib64/ 库、configs/ 各机型配置、scripts/set_nccl_env.sh)。
2.1 Compute Engine VM / 裸机 / Slurm —— 用系统包管理器装
# RHEL / Rocky / CentOS / TencentOS(aarch64 有包)
sudo tee /etc/yum.repos.d/nccl-gib.repo <<'EOL'
[gpudirect-gib-rpm]
name=NCCL/gIB
baseurl=https://packages.cloud.google.com/yum/repos/gpudirect-gib-rpm
enabled=1
repo_gpgcheck=0
gpgcheck=0
EOL
sudo dnf install nccl-gib
# Debian / Ubuntu 则用 apt 源 gpudirect-gib-apt,包名同为 nccl-gib
包区别:
nccl-gib= NCCL 本体 + gIB 插件(一站式);nccl-gib-plugins(v1.1.2+)= 仅插件,需另装 NCCL。
2.2 容器(Docker / Kubernetes)—— 官方要求把 gIB 放进容器,两种方式
- ① 镜像内安装:Dockerfile 里
dnf/apt install nccl-gib;或 - ② hostPath 挂载:把宿主机
/usr/local/gib挂进容器(GKE 即此模式——节点上装在/home/kubernetes/bin/gib,Pod 挂到/usr/local/gib)。
2.3 NVIDIA NGC 容器(2026-07 起)
gIB 已内置,无需 initContainer / 手动安装,插件自动探测 GCP 平台并启用。
2.4 运行时环境变量(官方写法)
export NCCL_NET=gIB
source /usr/local/gib/scripts/set_nccl_env.sh # 自动套本机机型最优参数
export LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/gib/lib64:$LD_LIBRARY_PATH
export NCCL_DEBUG=INFO # 验证时用
自 v1.1.2 起,只要装了
nccl-gib,gIB 一般会自动启用。set_nccl_env.sh会按机型加载nccl.a4xmax.conf(GB300)/nccl.a4x.conf(GB200)。口径注(如实标注):该脚本内部即
export NCCL_ENV_PLUGIN=gcp。NCCL_ENV_PLUGIN这个变量名 官方文档未单独列出——对外说明请以官方source set_nccl_env.sh写法为准;二者等价。
2.5 docker run 单机冒烟样例
docker run --rm -it --network host --ipc host --gpus all --cap-add IPC_LOCK \
--device /dev/infiniband -v /usr/local/gib:/usr/local/gib:ro \
-e NCCL_NET=gIB -e LD_LIBRARY_PATH=/usr/local/gib/lib64 -e NCCL_DEBUG=INFO \
<nccl-tests 镜像> \
bash -lc 'source /usr/local/gib/scripts/set_nccl_env.sh && all_reduce_perf -b 1M -e 8G -f 2 -g 4'
2.6 验证成功标志(NCCL_DEBUG=INFO 日志)
- ✅
NCCL INFO Using network gIB+NCCL INFO NET/IB : Using [0]mlx5_0 … - ❌
Using network Socket/NET/Socket= 退回 TCP(查LD_LIBRARY_PATH、RDMA 设备权限、NCCL_NET是否被别处覆盖) - ❌
Using network IB(而非gIB)= 用了 NCCL 自带 IB 后端,而非 gIB 插件
2.7 官方警告(务必转达)
- 不要在这些机型上装旧插件(FastSocket / TCPX / TCPXO)。
- A4X Max(GB300)需额外装 DOCA-OFED 更新用户态 RDMA 库。
- 容器场景插件必须集成进镜像或挂载,不能只装宿主机就指望容器内可用。
3. 官方文档 / GitHub 链接清单
官方文档(AI Hypercomputer,建议直接发给客户)
| 主题 | 链接 |
|---|---|
| 总览与架构 / 性能 | https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/overview |
| 安装 gIB | https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/install-nccl |
| NGC 容器内置 gIB | https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/ngc-containers |
| 日志 / 验证 / 排障 | https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/collect-and-understand |
| VM 上跑 NCCL | https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/test-vms |
| GKE A4X(GB200) | https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/test-gke-custom-a4x |
| GKE A4X Max(GB300) | https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/test-gke-custom-a4x-max |
| A4X Max Slurm 集群 | https://docs.cloud.google.com/cluster-toolkit/docs/deploy/slurm/create-a4x-max-cluster |
| 版本说明(ARM64,GB200/GB300) | https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/release-notes-arm64 |
| 版本说明(x86) | https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/release-notes-x86 |
| CoMMA 遥测配置 | https://docs.cloud.google.com/ai-hypercomputer/docs/nccl/configure-comma |
| GPU 机型规格 | https://docs.cloud.google.com/ai-hypercomputer/docs/gpu |
GitHub(开源配套)
| 仓库 | 说明 |
|---|---|
| https://github.com/GoogleCloudPlatform/cluster-toolkit | 部署蓝图 / 示例 |
| https://github.com/GoogleCloudPlatform/container-engine-accelerators | GKE GPU 安装器(含 A4X gIB DaemonSet) |
| https://github.com/AI-Hypercomputer/xpk | NCCL 测试样例(examples/nccl/nccl.md) |
| https://github.com/google/CoMMA | Profiler 组件,Apache-2.0(唯一开源的插件本体) |
| https://github.com/google/nccl-plugin-gpudirecttcpx | (非 gIB,A3 TCP 插件,供对照——GB200/GB300 勿用) |
容器镜像 / 包源
- 安装 / 诊断镜像(GB200/GB300 用 ARM64):
us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-arm64:v1.1.2us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
- YUM 源:
https://packages.cloud.google.com/yum/repos/gpudirect-gib-rpm - APT 源:
gpudirect-gib-apt
附录 A:平台配置实际取值(机型差异)
取自 nccl-gib-1.1.2 的 configs/(set_nccl_env.sh / NCCL_ENV_PLUGIN=gcp 会按机型自动套用)。
nccl.a4xmax.conf —— GB300 / A4X Max
| 变量 | 值 |
|---|---|
NCCL_NET | gIB |
NCCL_PXN_C2C | 1 |
NCCL_IB_MERGE_VFS | 1 |
NCCL_IB_ADAPTIVE_ROUTING | 1 |
NCCL_IB_ADDR_FAMILY | AF_INET6 |
NCCL_IB_TC | 52 |
NCCL_IB_FIFO_TC | 84 |
NCCL_IB_QPS_PER_CONNECTION | 4 |
NCCL_NCHANNELS_PER_NET_PEER | 1 |
NCCL_P2P_NET_CHUNKSIZE | 262144 |
NCCL_TUNER_PLUGIN | none |
nccl.a4x.conf —— GB200 / A4X
| 变量 | 值 |
|---|---|
NCCL_NET | gIB |
NCCL_PXN_C2C | 1 |
NCCL_IB_ADAPTIVE_ROUTING | 1 |
NCCL_IB_QPS_PER_CONNECTION | 4 |
NCCL_IB_TC | 52 |
NCCL_IB_FIFO_TC | 84 |
NCCL_TUNER_PLUGIN | none |
GB300 用 IPv6(
AF_INET6)做 IB 地址族,GB200 用默认 IPv4——这是两机型 gIB 配置的主要差异之一。
附录 B:我方实测交付细节(v18dot2 统一镜像)
以下为我方在 tlinux 统一镜像上的落地情况,供对照;转发外部时可按需删减。
- 镜像:
tlinux-gb-unified-ipv6-dual-50gb-v18dot2(ARM64,familytlinux-gb-unified) = Lustre Client DDN2.14 + NCCL/gIB 1.1.2(RPMnccl-gib,落位/usr/local/gib)。 - 收益:用户在容器里跑 NCCL 测试不再需要 initContainer 拉 nccl-gib—— Pod hostPath 挂
/usr/local/gib+ 设LD_LIBRARY_PATH/NCCL_ENV_PLUGIN=gcp即用宿主机 gIB。 - 配套交付 README:
deploy/nccl-gib-usage-guide.md(Pod / Job /docker run完整样例 + 配置表)。 - 镜像内
/usr/local/gib结构(RPMnccl-gib-1.1.2-1.aarch64):
bin/ ncclras, ncclparam
configs/ nccl.conf, nccl.a4x.conf(GB200), nccl.a4xmax.conf(GB300), nccl.a3u/a4/a5x.conf
lib64/ libnccl-net.so(gIB 网络插件) libnccl-env-gcp.so(env 插件)
libnccl-tuner.so libnccl.so.2.30.4(捆绑 NCCL) libGPUViz.so libnccl-profiler.so
scripts/ set_nccl_env.sh, run_nccl_tests.sh, gen_hostfiles.sh, init_ssh.sh, show_gids.sh
附录 C:实测验证过的 K8s 样例(GB200)
以下为我方在 GB200(A4X,NVL72,4 GPU/节点,aarch64) 上内部实测通过的 NCCL 测试清单, 采用 Kubeflow MPIJob v2beta1。共 4 个变体(单域 / 跨域 × 大规模 / 2 节点),下面给出 1 份 完整 canonical 清单 + 变体差异表,可据此拼出其余 3 个。
⚠️ 两处务必注意:
- 这些清单用
initContainer(gib-installer)把 gIB 拷进emptyDir,适用于节点镜像未内置 gIB 的情况。 若使用我方 v18dot2 镜像(已把 gIB 烤进宿主机/usr/local/gib),可删掉整个gib-installerinitContainer, 改为 hostPath 挂载/usr/local/gib(见 §2.2 / 附录 B),两者等效且更简。- 这里 env 用的是
NCCL_NET_PLUGIN=/usr/local/gib/lib64/libnccl-net.so(显式指库路径)+NCCL_ENV_PLUGIN=gcp, 未用NCCL_NET=gIB——三者都是有效写法(见 §2.4 口径注)。- 本组仅覆盖 GB200,未含 GB300;GB300 机型由 env 插件自动套
nccl.a4xmax.conf,但 RDMA 网卡名/数量不同 (GB300 约 8 张 RDMA),ResourceClaimTemplate的ifName选择器需按实际rdma dev调整。
C.1 完整 canonical 清单 —— nccl-mpi-64g-single-domain.yaml(单 NVL72 域,64 GPU / 16 worker)
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
name: nccl-mpi-64g-single-domain
spec:
slotsPerWorker: 4
runPolicy:
cleanPodPolicy: Running
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
nodeSelector:
nvl72-domain: domain-0001
initContainers:
- name: gib-installer
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
command: ["/bin/sh", "-c"]
args:
- |
set -e
/scripts/container_entry.sh install --install-nccl
cp -a /usr/local/gib/. /target/gib/
cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
mkdir -p /target/gib/lib64/libibverbs
cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
volumeMounts:
- name: gib
mountPath: /target/gib
containers:
- name: launcher
image: nvcr.io/nvidia/pytorch:26.05-py3
imagePullPolicy: IfNotPresent
env:
- name: LD_LIBRARY_PATH
value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
command:
- bash
- -c
- |
set -e
echo "Resolving worker IPs via k8s API..."
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
API="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"
for attempt in $(seq 1 60); do
ALL_READY=true
for i in $(seq 0 15); do
W_IP=$(curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" \
"$API/api/v1/namespaces/default/pods/nccl-mpi-64g-single-domain-worker-${i}" 2>/dev/null \
| python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",{}).get("podIP",""))' 2>/dev/null || echo "")
if [ -z "$W_IP" ]; then
ALL_READY=false
break
fi
echo "$W_IP nccl-mpi-64g-single-domain-worker-${i}.nccl-mpi-64g-single-domain.default.svc nccl-mpi-64g-single-domain-worker-${i}" >> /tmp/hosts_new
done
if [ "$ALL_READY" = true ]; then
cat /tmp/hosts_new >> /etc/hosts
echo "All worker IPs resolved."
break
fi
rm -f /tmp/hosts_new
echo "Waiting for worker IPs (attempt $attempt)..."
sleep 5
done
if [ "$ALL_READY" = false ]; then
echo "ERROR: Could not resolve worker IPs"
exit 1
fi
echo "Waiting 120s for workers to compile nccl-tests..."
sleep 120
echo "Starting mpirun..."
/usr/local/mpi/bin/mpirun --allow-run-as-root \
-np 64 -npernode 4 \
-x LD_LIBRARY_PATH=/usr/local/gib/lib64:/usr/local/nvidia/lib64 \
-x NCCL_NET_PLUGIN=/usr/local/gib/lib64/libnccl-net.so \
-x NCCL_ENV_PLUGIN=gcp \
-x NCCL_DEBUG=INFO \
-x NCCL_MNNVL_ENABLE=2 \
-x NCCL_CUMEM_ENABLE=1 \
--mca plm_rsh_args '-o StrictHostKeyChecking=no' \
/tmp/nccl-tests/build/all_reduce_perf -b 1M -e 16G -f 2 -g 1 -w 50 -n 100
volumeMounts:
- name: gib
mountPath: /usr/local/gib
volumes:
- name: gib
emptyDir: {}
Worker:
replicas: 16
template:
spec:
nodeSelector:
nvl72-domain: domain-0001
dnsPolicy: None
dnsConfig:
nameservers:
- "10.96.0.10"
- "8.8.8.8"
searches:
- "default.svc.cluster.local"
- "svc.cluster.local"
- "cluster.local"
resourceClaims:
- name: compute-domain-channel
resourceClaimTemplateName: domain-1-cd-channel
- name: rdma-nics
resourceClaimTemplateName: rdma-nics-mpi-sd
initContainers:
- name: gib-installer
image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
command: ["/bin/sh", "-c"]
args:
- |
set -e
/scripts/container_entry.sh install --install-nccl
cp -a /usr/local/gib/. /target/gib/
cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
mkdir -p /target/gib/lib64/libibverbs
cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
volumeMounts:
- name: gib
mountPath: /target/gib
containers:
- name: worker
image: nvcr.io/nvidia/pytorch:26.05-py3
imagePullPolicy: IfNotPresent
securityContext:
privileged: true
env:
- name: LD_LIBRARY_PATH
value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
- name: NCCL_NET_PLUGIN
value: /usr/local/gib/lib64/libnccl-net.so
- name: NCCL_ENV_PLUGIN
value: gcp
command:
- bash
- -c
- |
apt-get update -qq && apt-get install -y -qq openssh-server > /dev/null 2>&1
mkdir -p /run/sshd
printf 'Port 22\nPermitRootLogin yes\nPubkeyAuthentication yes\nAuthorizedKeysFile /root/.ssh/authorized_keys\nStrictModes no\nPasswordAuthentication no\nUsePAM no\nPidFile /run/sshd/sshd.pid\n' > /etc/ssh/mpi_sshd_config
ssh-keygen -A
for i in $(seq 1 30); do
cd /tmp && rm -rf nccl-tests && git clone --depth 1 https://github.com/NVIDIA/nccl-tests.git && break
echo "git clone attempt $i failed, retrying in 10s..."
sleep 10
done
cd /tmp/nccl-tests && make -j$(nproc) MPI=1 \
MPI_HOME=/usr/local/mpi \
NCCL_HOME=/usr/local/gib \
CUDA_HOME=/usr/local/cuda
/usr/sbin/sshd -De -f /etc/ssh/mpi_sshd_config
volumeMounts:
- name: gib
mountPath: /usr/local/gib
- name: dshm
mountPath: /dev/shm
resources:
requests:
cpu: "96"
memory: 800Gi
limits:
cpu: "96"
memory: 800Gi
nvidia.com/gpu: 4
claims:
- name: compute-domain-channel
- name: rdma-nics
volumes:
- name: gib
emptyDir: {}
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 250Gi
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
name: rdma-nics-mpi-sd
spec:
spec:
devices:
requests:
- name: nic-0
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu0rdma0"
- name: nic-1
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu1rdma0"
- name: nic-2
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu2rdma0"
- name: nic-3
exactly:
deviceClassName: rdma-devices
count: 1
selectors:
- cel:
expression: device.attributes["dra.net"].ifName == "gpu3rdma0"
C.2 变体差异表(其余 3 个清单 = C.1 改这几处)
all_reduce_perf 参数四个变体完全一致:-b 1M -e 16G -f 2 -g 1 -w 50 -n 100;差异只在规模与「域」两类:
| 变体 | Worker replicas |
IP 循环 seq 0 N |
-np |
Worker 调度 | Worker resourceClaims |
NCCL_MNNVL_ENABLE / NCCL_CUMEM_ENABLE |
|---|---|---|---|---|---|---|
| 64G 单域(C.1) | 16 | seq 0 15 | 64 | nodeSelector: nvl72-domain=domain-0001 | compute-domain-channel + rdma-nics | 2 / 1 |
| 128G 跨域 | 32 | seq 0 31 | 128 | nodeAffinity(domain-0001/0002)+ topologySpreadConstraints | 仅 rdma-nics | 0 / 0 |
| 8G 单域(2 节点) | 2 | seq 0 1 | 8 | nodeSelector: nvl72-domain=domain-0001 | compute-domain-channel + rdma-nics | 2 / 1 |
| 8G 跨域(2 节点) | 2 | seq 0 1 | 8 | nodeAffinity(domain-0001/0002)+ topologySpreadConstraints | 仅 rdma-nics | 0 / 0 |
「域」语义(关键):
- 同域 / 单域(NVLink 域内,走 NVLink):
NCCL_MNNVL_ENABLE=2、NCCL_CUMEM_ENABLE=1。 - 跨域 / 双域(强制走 RoCE/RDMA):
NCCL_MNNVL_ENABLE=0、NCCL_CUMEM_ENABLE=0,且 Launcher 去掉nodeSelector, Worker 改用nodeAffinity(横跨两个 domain)+topologySpreadConstraints(两域均摊)。 - 通用规则:
-np= Worker 节点数 × 4(GB200 每节点 4 GPU);IP 解析循环seq 0 N里N = replicas − 1。
跨域 Worker 调度片段(替换 C.1 里 Worker 的 nodeSelector 那两行):
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: nvl72-domain
operator: In
values: ["domain-0001", "domain-0002"]
topologySpreadConstraints:
- maxSkew: 1
topologyKey: nvl72-domain
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
training.kubeflow.org/job-name: <MPIJob 的 metadata.name>
(跨域时 Worker resourceClaims 只保留 rdma-nics,删掉 compute-domain-channel;Launcher 删掉 nodeSelector。)
C.3 nccl-tests 编译命令(Worker 容器内,四个清单一致)
cd /tmp/nccl-tests && make -j$(nproc) MPI=1 \
MPI_HOME=/usr/local/mpi \
NCCL_HOME=/usr/local/gib \
CUDA_HOME=/usr/local/cuda
C.4 提交与查看
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl apply -f nccl-mpi-64g-single-domain.yaml
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl get pods
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl logs -f <LAUNCHER_POD_NAME>
sudo KUBECONFIG=/etc/kubernetes/admin.conf kubectl delete -f nccl-mpi-64g-single-domain.yaml
C.5 成功标志
本组清单实测时,Launcher 日志出现下面这行即表示 gIB RDMA 网络已激活(与 §2.6 的 Using network gIB 等义):
NCCL INFO NET/gIB : Using ...
注:内部记录未保存带实际带宽数字的完整成功日志;此处为激活标志行。