☰ 目录

目录导航

GB300 A4X-Max 裸金属部署指南

适用于 GCP A4X-Max (GB300 Grace Blackwell NVL72) 裸金属架构上的自管理 Kubernetes 环境。本指南完全对齐客户交付标准,并集成了双网卡命名兼容、多 NUMA 巨页分配优化及底层硬件网卡管控组件的配置。


全局变量

修改以下变量后,后续所有命令可直接复制粘贴执行。本指南统一使用物理裸金属机型 a4x-maxgpu-4g-metal,包含 2 个 Intel IDPF 管理网卡及 8 个 MRDMA 高速 RDMA 物理网卡接口。

# ======================== 项目 ========================
export PROJECT="tencent-gcp-taiji-poc"

# ======================== 区域、规模与机器配置 ========================
export REGION="us-central1"
export ZONE="us-central1-b"

# 【生产规模配置例】:1800 GPU = 450 VM (25 个 NVL72 domains)
# 若仅测试单 Domain 72 GPU,可修改为 72
export TOTAL_GPUS=1800                           
export MACHINE_TYPE="a4x-maxgpu-4g-metal"      # GB300 裸金属机型
export CLUSTER_PREFIX="gb300-central"

# 共享物理预留前缀 (格式: subblock-0001 至 subblock-0025)
# 在创建 VM 的循环中,脚本将自动根据 domain 索引动态追加类似 "-0001" 到 "-0025" 的子网预留后缀,
# 以实现多 domain 分离调度及物理拓扑隔离。
export RESERVATION_PREFIX="projects/tencent-gcp-taiji/reservations/nvidia-gb300-dxkhoz4ypk4mh/reservationBlocks/nvidia-gb300-dxkhoz4ypk4mh-block-0001/reservationSubBlocks/nvidia-gb300-dxkhoz4ypk4mh-block-0001-subblock"

# ======================== 拓扑计算 ========================
export GPUS_PER_NODE=4
export NODES_PER_DOMAIN=18                      # GB300 NVL72 = 18 节点/域
export NUM_DOMAINS=$((TOTAL_GPUS / GPUS_PER_NODE / NODES_PER_DOMAIN))

export CP_NAME="${CLUSTER_PREFIX}-cp"
export WORKER_PREFIX="${CLUSTER_PREFIX}-worker"  # VM 命名格式: ${WORKER_PREFIX}-d${DOMAIN}-w${INDEX}
export PLACEMENT_PREFIX="${CLUSTER_PREFIX}-nvl72-policy"

# ======================== 系统镜像 (集成 UEFI 引导修复) ========================
export IMAGE="tlinux-server-4-gb300-v5dot4-ipv6"
export IMAGE_PROJECT="tencent-gcp-taiji-poc"

# ======================== 网卡 1 (主管理网, MTU 1500) ========================
# 【重要重要】MGMT_CIDR 的网段必须由客户提供,因为该网卡需要通过 Interconnect 对接到客户的内部骨干网络中。
export MGMT_NET="${CLUSTER_PREFIX}-mgmt-net"
export MGMT_SUB="${CLUSTER_PREFIX}-mgmt-sub"
export MGMT_CIDR="10.200.0.0/22"               # /22 提供 1020 个可用 IP (请根据客户实际分配的段进行替换)

# ======================== 网卡 2 (Lustre 数据存储网, MTU 8896) ========================
# LUSTRE_CIDR 仅用于集群内部与 Lustre 存储客户端的数据通信,不与客户内网直接对接,因此可以由交付团队自行指定。
export LUSTRE_NET="${CLUSTER_PREFIX}-lustre-net"
export LUSTRE_SUB="${CLUSTER_PREFIX}-lustre-sub"
export LUSTRE_CIDR="10.201.0.0/22"
export LUSTRE_FS_CIDR="172.16.0.0/24"          # 客户 Lustre 存储段地址 (若需挂载外部 Lustre 请同步对齐)

# ======================== RDMA 计算网络 (8-Way 独立子网) ========================
export RDMA_NET="${CLUSTER_PREFIX}-rdma-net"
# 【重要重要】GB300 Bare Metal (a4x-maxgpu-4g-metal) 必须关联 us-central1-b-vpc-roce-metal 底层 Profile
export RDMA_PROFILE="us-central1-b-vpc-roce-metal"

# ======================== Kubernetes ========================
export K8S_VERSION="1.34"
export K8S_PATCH="1.34.9"                      # >= 1.34.5 (包含 kubelet DRA gRPC 修复)
export POD_CIDR="10.244.0.0/16"
export CALICO_VERSION="3.28.0"
export CALICO_MTU=8846

Part 1: 网络基础设施

GB300 裸金属节点包含 10 个物理网口。为配合客户对管理网专线 MTU 1500 的硬性规范,同时保障 Lustre 客户端存储通信的巨型帧吞吐 (MTU 8896),我们设计了双管理/数据 VPC 网络架构。

+---------------------------------------------------------------------------------+
|                                  GB300 Node                                     |
|                                                                                 |
|  [IDPF eth0 (bond0)]  MTU 1500  ======>  MGMT VPC (MGMT_NET)                     |
|  [IDPF eth1 (bond1)]  MTU 8896  ======>  Lustre VPC (LUSTRE_NET) ===> Lustre FS   |
|                                                                                 |
|  [8x MRDMA (bond2-9)] IPv6-Only ======>  RDMA VPC (RDMA_NET)     ===> RoCE Fabric |
+---------------------------------------------------------------------------------+

1.1 创建 VPC 拓扑

创建三个独立的网络以满足不同的 MTU 与硬件底层 Profile:

# 1. 创建控制管理网络 (MTU 1500)
gcloud compute --project=${PROJECT} \
  networks create ${MGMT_NET} \
  --subnet-mode=custom --mtu=1500

# 2. 创建 Lustre 存储网络 (MTU 8896)
gcloud compute --project=${PROJECT} \
  networks create ${LUSTRE_NET} \
  --subnet-mode=custom --mtu=8896

# 3. 创建 RDMA 计算网络 (关联 vpc-roce 硬件 Profile)
gcloud beta compute --project=${PROJECT} \
  networks create ${RDMA_NET} \
  --network-profile=${RDMA_PROFILE} \
  --subnet-mode=custom

1.2 创建子网

为各个 VPC 划分对应的子网网段。高速 RDMA 网卡采用 IPv6-Only 声明:

# 创建控制管理网子网 (IPv4)
gcloud compute --project=${PROJECT} \
  networks subnets create ${MGMT_SUB} \
  --network=${MGMT_NET} \
  --region=${REGION} \
  --range=${MGMT_CIDR}

# 创建 Lustre 存储网子网 (IPv4)
gcloud compute --project=${PROJECT} \
  networks subnets create ${LUSTRE_SUB} \
  --network=${LUSTRE_NET} \
  --region=${REGION} \
  --range=${LUSTRE_CIDR}

# 【注意】对于使用 beta Profile 创建的 RDMA 计算网,底层策略是不允许手动划分自定义子网的:
# allowSubnetworkCreation: SUBNETWORK_CREATION_BLOCKED。
# 底层会自动在对应的 Zone 下为该 RoCE 裸金属网络自动创建名为 "default-subnet-1-${RDMA_NET}"
# 覆盖 fd36::/48 的全局 IPv6 唯一的子网段,所有 ConnectX-8 网卡直接挂载并从中分配。
#
# 交付人员只需要对此处的 default 物理子网进行核实,无需执行自定义网络划分命令:
gcloud compute networks subnets list --network=${RDMA_NET} --project=${PROJECT}

1.3 创建防火墙

# 允许 Google IAP 隧道进行控制登录
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-iap-ssh \
  --project=${PROJECT} --network=${MGMT_NET} \
  --direction=INGRESS --priority=900 --action=ALLOW \
  --rules=tcp:22,tcp:56000 --source-ranges=35.235.240.0/20

# 允许控制管理网段内部全协议互通
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-internal-mgmt \
  --project=${PROJECT} --network=${MGMT_NET} \
  --direction=INGRESS --priority=1000 --action=ALLOW \
  --rules=all --source-ranges=${MGMT_CIDR}

# 允许 Lustre 存储网段内部全协议互通
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-internal-lustre \
  --project=${PROJECT} --network=${LUSTRE_NET} \
  --direction=INGRESS --priority=1000 --action=ALLOW \
  --rules=all --source-ranges=${LUSTRE_CIDR}

# 允许来自本地 VPC Peering / Service Networking 的 Lustre 托管存储段入站全协议互通 (必须)
gcloud compute firewall-rules create ${CLUSTER_PREFIX}-allow-lustre-ingress-local \
  --project=${PROJECT} --network=${LUSTRE_NET} \
  --direction=INGRESS --priority=800 --action=ALLOW \
  --rules=all --source-ranges=10.75.0.0/16

1.4 创建 Placement Policy (物理拓扑亲和策略)

每个 NVL72 Domain (18 个 Worker 节点) 需要一个独立的 Placement Policy,用以强制调度器将节点分配到物理上紧邻的机架内。

# 遍历所有的 Domain,为每个 NVL72 物理域创建独立的 Placement Policy
for d in $(seq 1 ${NUM_DOMAINS}); do
  gcloud beta compute resource-policies create group-placement \
    ${PLACEMENT_PREFIX}-${d} \
    --collocation=collocated \
    --gpu-topology=1x72 \
    --project=${PROJECT} --region=${REGION}
done

# 验证已创建的策略列表
gcloud compute resource-policies list \
  --project=${PROJECT} --filter="name~${PLACEMENT_PREFIX}"

Part 2: Worker 节点创建

GCP 裸金属网卡 Bonding 架构说明

在传统的自建数据中心(On-Premises)中,客户常通过操作系统级的 bonding 驱动将两块管理网卡绑定为 bond0 (Active-Backup / LACP 模式),以及将 RDMA 网卡进行 MLAG/LAG 链路聚合以提供高可用与带宽扩容。而在 GCP A4X-Max 裸金属环境 中:

  1. 管理网络高可用已由底层托管:系统的主控制管理网(通过 IDPF 驱动暴露的 eth0/eth1)在宿主机层面连接到 Google 的 Andromeda SDN 网络。Andromeda 自动在硬件交换层提供完全透明的故障切换与路径冗余,操作系统层无法也无需再配置传统的 LACP 绑定(GCP 底层交换网络不支持 LACP 协议协商)。
  2. RDMA 网卡强依赖 Rail-Aligned (轨道对齐) 物理映射:GB300 的 8 个 MRDMA 网络接口(CX-8 双端口卡)是直连 GPU 的高性能通信通道。NCCL 等并行计算框架通过 “8-way rail-aligned” 拓扑进行跨节点集体通信。每个 GPU 必须物理绑定其对应的独立 MRDMA 网络接口(即 Rank 0 走 Rail 0,Rank 1 走 Rail 1 依次对应),如果强行在操作系统中将 8 个网口捆绑为一个 Logical Bond,将会破坏物理上的 Rail 对齐拓扑,导致流量在宿主机 PCIe 内部产生严重冲突与延迟抖动,甚至破坏 GPUDirect RDMA。

综上,GCP A4X-Max 裸金属物理网卡不支持、也不应该配置任何操作系统的物理双网卡 Bonding。
为了完全满足客户已有监控和软件模板对网卡命名规范(bond0 ~ bond9)的兼容,我们采用虚拟重命名方案:将 eth0/eth1 命名为 bond0/bond1,将 Mellanox 的 8 个 MRDMA 网口虚拟更名为 bond2 到 bond9。这仅仅是对接口设备名(netdev 和 ibdev)的别名封装,完全保留了底层的独立物理通道和 Rail-Aligned 特性。本步骤在定制初始化启动脚本中自动实现。

硬件故障容错与 Placement Policy 避障说明

  1. 标准部署流程:在正常的物理机房交付中,为了保证 18 台虚拟机完全分布在同一个 NVLink 物理 Subblock(同机架/同 Domain)中以启用高速 1.8 TB/s NVSwitch 组播转发,批量创建实例的命令中必须包含 --resource-policies=${RESOURCE_POLICY} 标志(其中 ${RESOURCE_POLICY} 对应 Part 1.4 中创建的 Placement Policy)。
  2. 硬件故障阻塞风险:如果交付分配的物理机架(Reservation Block)中存在故障硬件节点(例如:某个物理节点出现 GPU 损坏、IMEX 链路断开、或 UEFI 引导挂死),由于该节点无法成功开机或通过 GCE 底层的健康检查,GCE 调度器在满足紧凑性约束(COLLOCATED)时,会因为无法满足全部 18 台机器同时在同一物理拓扑内拉起的硬性限制,导致整组实例创建命令报错、挂起或超时失败。
  3. Triage 绕行预案:如果在测试或排查阶段遇到物理插槽中的故障节点导致整体调度挂死,您可以暂时从创建命令中移除 --resource-policies 标志(仅指定 --reservation 以保证占用的物理槽位)。这能让虚拟机绕开紧凑拓扑的强一致性审查,正常启动其余健康的节点进行单节点或局部多节点的 NCCL/存储测试。

官方文档与架构设计参考: - 关于网卡不支持 LACP Bonding:请参阅 Google Cloud 官方 VPC 设计说明 Virtual Private Cloud (VPC) network - Link aggregation,明确指出 GCP 虚拟网卡(包括 IDPF)由底层软件定义网络(Andromeda)提供冗余保障,操作系统层面不支持 Link Aggregation 协议协商。 - 关于 Rail-Aligned 多网卡设计:详见 Google Cloud 官方超级计算机架构白皮书 GPU network bandwidth and architecture - A4X-Max,说明了其内置 8 路 MRDMA 接口直连 Grace-Blackwell CPU 与 GPU 复合体的物理拓扑(8-way rail-aligned),阐明了保持网卡独立物理寻址对避免内部 PCIe 链路竞争的重要性。 - 关于 Placement Policy 级联锁死与故障退避:请参阅 Google Cloud 官方虚拟机调度规范 Compact placement policies。该指南详细说明了当使用 group-placement 且拓扑约束强度较高时,若可用节点中存在单点损坏或物理槽位资源不足,由于紧凑性条件无法满足,整组 VM 启动命令将强一致性报错,印证了故障时需要退避解除 --resource-policies 绑定的必要性。

2.1 批量创建 Worker 裸金属实例

使用 a4x-maxgpu-4g-metal 机型创建裸金属节点。创建命令中需包含 10 个网卡描述(2 个 IDPF 网口 + 8 个 MRDMA 接口)。启动脚本完整源码请参见 附录 E(内部自建 K8s 测试版请参见 附录 F):

# 按 Domain 分批并行创建,每批绑定各自的 Placement Policy 并分配对应的物理子预留
for d in $(seq 1 ${NUM_DOMAINS}); do
  echo "=== Creating Domain $d workers ==="
  
  # 计算当前 Domain 对应的 4 位格式化子预留名称 (例如 d=1 -> 0001)
  export CURRENT_RESERVATION="${RESERVATION_PREFIX}-$(printf "%04d" $d)"
  
  for i in $(seq 0 $((NODES_PER_DOMAIN - 1))); do
    gcloud compute instances create ${WORKER_PREFIX}-d${d}-w${i} \
      --project=${PROJECT} \
      --zone=${ZONE} \
      --machine-type=${MACHINE_TYPE} \
      --image=${IMAGE} \
      --image-project=${IMAGE_PROJECT} \
      --boot-disk-type=hyperdisk-balanced \
      --boot-disk-size=1000GB \
      --scopes=cloud-platform \
      --reservation-affinity=specific \
      --reservation=${CURRENT_RESERVATION} \
      --provisioning-model=RESERVATION_BOUND \
      --instance-termination-action=STOP \
      --maintenance-policy=TERMINATE \
      --restart-on-failure \
      --resource-policies=${PLACEMENT_PREFIX}-${d} \
      --metadata=serial-port-enable=1 \
      --metadata-from-file=startup-script=scripts/tlinux4-customer-init-gb300.sh \
      --network-interface=subnet=${MGMT_SUB},nic-type=IDPF \
      --network-interface=subnet=${LUSTRE_SUB},nic-type=IDPF,no-address \
      --network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
      --network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
      --network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
      --network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
      --network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
      --network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
      --network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA \
      --network-interface=subnet=default-subnet-1-${RDMA_NET},stack-type=IPV6_ONLY,nic-type=MRDMA &
  done
  wait
  echo "Domain $d: $NODES_PER_DOMAIN workers created."
done

2.2 创建后验证

# 检查所有 VM 状态与 IP
gcloud compute instances list --project=${PROJECT} \
  --filter="name~${WORKER_PREFIX}-d" \
  --format="table(name,zone,status,networkInterfaces[0].networkIP)"

# SSH 到任意 worker 检查初始化日志 (以 d1-w0 为例)
gcloud compute ssh ${WORKER_PREFIX}-d1-w0 --zone=${ZONE} --tunnel-through-iap \
  --command="tail -30 /var/log/tlinux4-customer-init.log"

# 检查 IMEX channels 状态
gcloud compute ssh ${WORKER_PREFIX}-d1-w0 --zone=${ZONE} --tunnel-through-iap \
  --command="ls /dev/nvidia-caps-imex-channels/ && nvidia-smi -L | head -4"

Part 3: 拓扑信息与节点标签

GCP 裸金属平台为 GB300 (A4X-Max) 节点提供高精度的物理拓扑标识。处于同一个机架即同一个 NVLink Clique 域 (IMEX Domain) 内的所有 VM,都将返回完全相同的物理拓扑标识。通过该标识将节点分入不同的 K8s 亲和性组,能够避免分布式计算跨越物理域导致 NCCL Fallback 产生重大性能损耗。

3.1 获取物理拓扑信息

我们推荐使用 Metadata Server 获取哈希,或使用 Compute API 查询:

3.2 手动标记 (小规模 / 验证环境)

通过 kubectl 对创建的多个 Domain 的节点进行分区标记:

# 遍历所有 Domain,对节点标记 domain-1, domain-2...
for d in $(seq 1 ${NUM_DOMAINS}); do
  for i in $(seq 0 $((NODES_PER_DOMAIN - 1))); do
    # 标记 IMEX Domain 归属与机器类型
    kubectl label node ${WORKER_PREFIX}-d${d}-w${i} nvl72-domain=domain-${d} --overwrite
    kubectl label node ${WORKER_PREFIX}-d${d}-w${i} node-type=a4x-max --overwrite
  done
done

# 验证标记结果
kubectl get nodes --show-labels | grep nvl72-domain

3.3 自动标记 (生产环境,推荐自动上报)

若要在节点入群时由 kubelet 自动汇报拓扑,可将以下逻辑注入 Worker 节点的 startup-script 中(在执行 kubeadm join 之前):

# 1. 抓取物理拓扑 Hash 并截取前 8 位
TOPO_HASH=$(curl -sf -H "Metadata-Flavor: Google" \
  "http://metadata.google.internal/computeMetadata/v1/instance/attributes/physical_host_topology" \
  | md5sum | cut -c1-8 || echo "unknown-domain")

# 2. 修改 kubelet 默认环境变量,加入 node-labels 参数
mkdir -p /etc/systemd/system/kubelet.service.d
cat > /etc/default/kubelet <<KUBELET_EOF
KUBELET_EXTRA_ARGS=--node-labels=nvl72-domain=domain-${TOPO_HASH},node-type=a4x-max
KUBELET_EOF

# 3. 重启 kubelet 生效
systemctl daemon-reload
systemctl restart kubelet

Part 4: Kubernetes 硬件接口管控与自建 K8s 部署

本章节分为两部分:必须项 (Required Steps)(客户自建 K8s 集集群接入 GB300 时必须在集群内应用的底层硬件与 CNI 配置)以及 建议/可选项 (Recommended / Optional Sandbox Setup)(用于测试环境自建 K8s 控制面的参考步骤)。

4.1 【必须项】部署 asapd-lite (裸金属硬件管控必备组件)

asapd-lite 组件部署说明

在 GB200 虚拟化架构中,底层网络管理程序运行在 Google 的 Hypervisor 主机空间内。但在 GB300 裸金属 (Bare Metal) 架构中无 Hypervisor 层,客户操作系统直接读写物理 CX-8 PCIe 接口。

因此,操作系统中必须在集群初始化阶段部署 asapd-lite DaemonSet。它负责:

  1. 利用 DPDK 驱动在用户空间接管 CX-8 物理设备以处理流量拥塞控制 (PCC/DOCA 3.1)。
  2. 为 8 个物理 MRDMA 接口自动分配及修复 DAD (Duplicate Address Detection) 异常。
  3. 提供网卡性能指标采集及 hairpin 回环诊断。

注:asapd-lite 的运行强依赖 16GB 的 NUMA 2MB 巨页分区(已在 Part 3.2 启动脚本中完成)。

设计根源与技术出处

asapd-lite 裸金属硬件管控方案源于 GCP 官方与开源社区核心实现:

  1. GCP 官方设计规范:详见 GCP A4X-Max GKE 部署指引。
  2. 上游容器服务开源仓库:该组件的底层容器镜像构建与安装资源配置源于 Google 的开源网络加速器仓库 container-engine-accelerators。
  3. 底层功能探测:对镜像 us-docker.pkg.dev/gce-ai-infra/asapd-lite/asapd-lite 的实际内容深度探测表明,其内部封装了:
    • asapd-lite-controller(位于 /asapd-lite/controller/):核心主控守护进程,动态轮询 CX-8 PF 设备 PCI 信息。
    • DOCA 3.1 运行时库(位于 /asapd-lite/3.1/):提供可编程拥塞控制 (PCC)、Data Direct 控制及 FlexIO 遥测收集。
    • apply_non_volatile_configs.sh(位于 /asapd-lite/ipam_scripts/):在设备初始化阶段通过 mstconfig 固件工具直接下发优化参数,配置网口与 IP 并消除 DAD 异常。
  4. 巨页缓存机制:其 YAML 资源清单中配置了 hugepages-2Mi: 8Gi 内存映射,表明底层 DPDK 包处理守护程序强依赖巨页作 DMA 物理缓冲区,故必须通过 Host 启动脚本预先完成 hugepage 分配。

应用为 TLinux 4/Rocky 9 深度适配过的 asapd-lite 资源文件(完整 YAML 配置请参见 附录 G):

# 部署 TLinux 适配版 asapd-lite 安装描述
kubectl apply -f yamls/asapd-lite-installer-tlinux.yaml

# 验证 Pod 状态,状态必须为 Running 且 liveness/readiness 均正常
kubectl get ds -n kube-system asapd-lite

4.2 【必须项】部署 NVIDIA DRA GPU 驱动 (v25.12.0)

组件职责与选型依据: 在自建 Kubernetes 集群管理 GB300 (Grace Blackwell NVL72) 平台时,GPU 相关组件的选型分工如下: 1. GPU 算力分配:使用传统的 nvidia-device-plugin。由于开源社区官方通过 DRA (Dynamic Resource Allocation) 机制直接管理和调度 GPU 算力资源的功能尚未完全 GA,使用 Device Plugin 进行标准的 nvidia.com/gpu 资源数额声明可以提供最佳的调度稳定性。 2. NVLink 通信管理:部署 nvidia-dra-driver-gpu (v25.12.0)。该驱动虽然不直接接管 GPU 算力分配,但提供了核心的 ComputeDomain 自定义资源控制器。它负责动态构建和管理 IMEX (Multi-Node NVLink / NVLink-Network) 拓扑。同机柜内的 18 台物理节点正是通过 ComputeDomain 才得以协调各自的 GPU 虚拟通道,构建出 1.8 TB/s 极速互联的 NVLink 物理域。若缺少此驱动,跨节点的 NCCL AllReduce 通信将无法走 NVLink 通信网,而会强制退化到普通的 RoCE 网卡(3200 Gbps),导致通信带宽断崖式下跌。

官方参考文档与依据: - 组件官方参考文档:关于使用 K8s DRA 驱动及 ComputeDomain CRD 来管理和配置多节点 NVLink (MNNVL) 拓扑的具体物理机制与架构细节,请参阅 NVIDIA 官方文档 NVIDIA DRA Driver for GPUs 与 Compute Domains and MNNVL。 - GKE 拓扑感知调度依据:该分层配置架构对应了 Google Cloud 官方关于高性能 AI 算力集群中对网卡与 GPU 进行声明式拓扑管理的推荐设计,具体可参见 Google Cloud 官方指南 Prepare GKE infrastructure for DRA workloads。

官方参考文档与依据: - 组件官方参考文档:关于使用 K8s DRA 驱动及 ComputeDomain CRD 来管理和配置多节点 NVLink (MNNVL) 拓扑的具体物理机制与架构细节,请参阅 NVIDIA 官方文档 NVIDIA DRA Driver for GPUs 与 Compute Domains and MNNVL。 - GKE 拓扑感知调度依据:该分层配置架构对应了 Google Cloud 官方关于高性能 AI 算力集群中对网卡与 GPU 进行声明式拓扑管理的推荐设计,具体可参见 Google Cloud 官方指南 Prepare GKE infrastructure for DRA workloads。

版本对齐说明:
对于 GB300 硬件支持,必须部署 NGC 发布的 25.12.0 版本 DRA 驱动以支持 Blackwell 架构特征及 CUDA 13.0 / 580+ 驱动栈。

注:本自建集群方案与官方 GKE 方案的核心差异:在 GKE 方案中,GKE 节点镜像(COS)不预装 nvidia-container-toolkit,因此 Pod 必须显式以 volume 方式挂载宿主机的 /home/kubernetes/bin/nvidia 到容器内;而在自建 K8s (TLinux/Rocky) 环境下,我们在 Appendix F 的系统脚本中安装了标准的 nvidia-container-toolkit,并通过 containerd 运行时自动注入 GPU 设备和库,业务 Pod 无需手动挂载任何物理机驱动目录。

安全配额保障 (ResourceQuota): 为防止系统级关键 Pod(如 DRA Driver 控制器)因意外抢占或调度过载导致集群级锁死,建议在部署的 Namespace 下创建对应的 ResourceQuota 限制,这也是官方 GKE A4X-Max 环境的核心合规实践之一:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: nvidia-dra-driver-quota
  namespace: gpu-operator
spec:
  hard:
    pods: "37" # 设为 (2 * NODE_COUNT + 1),对于 18 节点集群,此处推荐填 37
  scopeSelector:
    matchExpressions:
    - operator: In
      scopeName: PriorityClass
      values:
        - system-node-critical
        - system-cluster-critical
# 1. 部署 NVIDIA Device Plugin
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update
helm install nvidia-device-plugin nvdp/nvidia-device-plugin \
  --namespace gpu-operator \
  --create-namespace \
  --set-string nodeSelector."nvidia\.com/gpu\.present"=true \
  --wait --timeout 300s

# 2. 创建 ResourceQuota 配额限制
kubectl apply -f - << 'EOF'
apiVersion: v1
kind: ResourceQuota
metadata:
  name: nvidia-dra-driver-quota
  namespace: gpu-operator
spec:
  hard:
    pods: "37"
  scopeSelector:
    matchExpressions:
    - operator: In
      scopeName: PriorityClass
      values:
        - system-node-critical
        - system-cluster-critical
EOF

# 3. 部署 K8s DRA Driver (绑定 v25.12.0)
helm repo add nvidia-dra https://nvidia.github.io/k8s-dra-driver
helm repo update
helm install nvidia-dra-driver-gpu nvidia-dra/nvidia-dra-driver-gpu \
  --namespace gpu-operator \
  --version 25.12.0 \
  --set gpuResourcesEnabledOverride=true \
  --wait --timeout 300s

4.3 【必须项】部署 DRANET 网络分配器

dranet 用于调度并向 Pod 分配 MRDMA 设备文件:

helm install dranet oci://registry.k8s.io/networking/charts/dranet \
  --version v1.3.0 \
  -n kube-system

# 注册 DeviceClass 类名
cat << 'EOF' | kubectl apply -f -
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: rdma-devices
spec:
  selectors:
  - cel:
      expression: |
        device.driver == "dra.net" &&
        has(device.attributes["dra.net"].rdma) && device.attributes["dra.net"].rdma == true
EOF

4.4 【必须项】配置 Scheduler DRA RBAC 权限

在 Kubernetes 1.34 中,默认的 scheduler ClusterRole 缺少 DRA 资源的 informer 权限。如果使用 kubeadm 部署自管理集群,必须手动补齐以下 RBAC 绑定,否则 ComputeDomain daemon 容器在启动时会永久阻塞。

# 1. 补全 Scheduler 的 DRA RBAC 规则
cat <<'EOF' | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:kube-scheduler:dra
rules:
- apiGroups: ["resource.k8s.io"]
  resources: ["resourceclaims"]
  verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["resource.k8s.io"]
  resources: ["resourceclaims/status"]
  verbs: ["get", "update", "patch"]
- apiGroups: ["resource.k8s.io"]
  resources: ["resourceslices"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["resource.k8s.io"]
  resources: ["deviceclasses"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-scheduler:dra
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-scheduler:dra
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:kube-scheduler
EOF

# 2. 重启 K8s 控制面 Scheduler Pod 使权限重载生效
kubectl delete pod -n kube-system -l component=kube-scheduler
kubectl get pods -n kube-system -l component=kube-scheduler -w

4.5 【必须项】创建 ComputeDomain 亲和域资源

为了能够协调同一个 IMEX Clique (NVLink 域) 内的 18 台工作节点,必须为每个 Domain 声明一个 ComputeDomain CRD 实例,并将物理计算节点的 UIDs 与其绑定。这能使 NVIDIA DRA 驱动自动发现机架内的多机组播通道。

# 遍历创建所有 NVL72 对应的 ComputeDomain 实例并为节点打标
for d in $(seq 1 ${NUM_DOMAINS}); do
  CD_NAME="domain-${d}-cd"
  echo "=== Creating ComputeDomain: ${CD_NAME} ==="

  cat <<EOF | kubectl apply -f -
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: ${CD_NAME}
spec:
  numNodes: 0
  channel:
    allocationMode: Single
    resourceClaimTemplate:
      name: ${CD_NAME}-channel
EOF

  # 提取该 ComputeDomain 的 Kubernetes UID
  CD_UID=$(kubectl get computedomain ${CD_NAME} -o jsonpath='{.metadata.uid}')
  echo "ComputeDomain ${CD_NAME} UID: $CD_UID"

  # 为属于当前 Domain 的 18 个工作节点打标,绑定到当前通道
  for i in $(seq 0 $((NODES_PER_DOMAIN - 1))); do
    kubectl label node ${WORKER_PREFIX}-d${d}-w${i} \
      resource.nvidia.com/computeDomain=$CD_UID --overwrite
  done
done

# 验证 ComputeDomain 配置及守护进程运行状态
kubectl get computedomain -o wide
kubectl get pods -n gpu-operator -l app=computedomain-daemon

4.6 【可选项】Master 控制面虚拟机创建 (自测试沙盒参考)

对于内部验证或搭建隔离测试环境,可以通过以下步骤部署自管理的 Master 节点:

gcloud compute instances create ${CLUSTER_PREFIX}-master \
  --project=${PROJECT} \
  --zone=${ZONE} \
  --machine-type="n2-standard-16" \
  --image-family="rocky-linux-9" \
  --image-project="rocky-linux-cloud" \
  --boot-disk-type="pd-ssd" \
  --boot-disk-size="200GB" \
  --network-interface=network=${MGMT_NET},subnet=${MGMT_SUB} \
  --network-interface=network=${LUSTRE_NET},subnet=${LUSTRE_SUB}

4.7 【可选项】K8s 控制面初始化

登录自建的 Master VM,执行控制面初始化:

sudo kubeadm init --kubernetes-version=v1.34.9 \
  --pod-network-cidr=10.244.0.0/16 \
  --apiserver-advertise-address=10.200.0.2

4.8 【可选项】部署 Calico CNI (包含 MTU 1450 调优)

在控制面上部署 Calico 覆盖网络。为配合管理网络 bond0 (IDPF) 的 MTU 1500 并扣除 VXLAN 的 50 字节封装开销,Calico MTU 必须手动降低调优设置为 1450。如果不进行调优,默认的 1500 MTU 在跨节点发送大包时会被物理网卡分片丢弃,导致控制面连接超时。

curl -O https://raw.githubusercontent.com/projectcalico/calico/v3.29.3/manifests/calico.yaml
# 编辑 calico.yaml,将以下 MTU 配置项修改为 1450:
# - name: FELIX_IPINIPMTU
#   value: "1450"
# - name: FELIX_VXLANMTU
#   value: "1450"
# - name: veth_mtu
#   value: "1450"
kubectl apply -f calico.yaml

4.8.1 【可选项】修复 CoreDNS 阻断 GCP Metadata DNS 漏洞

在自建 K8s 中,CoreDNS 默认将未知的外部域名解析请求(如 metadata.google.internal)转发至 Master 节点的 /etc/resolv.conf(指向宿主机的 169.254.169.254)。然而在 GCP 网络环境中,来自于 Pod 虚拟网络 overlay 的 metadata 域名查询请求会触发 VPC 丢弃保护,导致容器内的 DNS 发生解析挂起。

为了彻底避免此问题,可在控制面上对 CoreDNS 进行打补丁,将其上游转发 DNS 直接修改为公网公共 DNS 或者是内网 VPC 可达 of 外部 DNS:

# 导出并打补丁将 resolv.conf 转发修改为公共 DNS (例如 8.8.8.8)
kubectl get configmap coredns -n kube-system -o yaml | \
  sed 's/forward . \/etc\/resolv.conf/forward . 8.8.8.8 8.8.4.4/' | \
  kubectl apply -f -

# 重启 CoreDNS 生效
kubectl rollout restart deployment/coredns -n kube-system

4.9 【可选项】Worker 节点加入集群

对于自建 K8s 验证环境,控制面初始化后,需生成 Token 将 Worker 节点加入集群。

  1. 在 Master 节点上生成 Join 命令:
kubeadm token create --print-join-command
  1. 在 Worker 节点上执行 Join: 以 root 权限登录 Worker 节点并执行上述命令(示例如下):
sudo kubeadm join 10.200.0.2:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --ignore-preflight-errors=SystemVerification

4.10 【可选项】部署 NVIDIA GPU Feature Discovery (GFD)

GFD 负责在 K8s 中自动检测节点上的 GPU 拓扑及 clique 等底层属性,并自动打上对应的节点标签。

helm upgrade --install gpu-feature-discovery nvdp/gpu-feature-discovery \
  --namespace gpu-operator \
  --set nfd.deploy=false \
  --wait

4.11 【可选项】部署 Kubeflow MPI Operator

MPI Operator 用于调度和管理跨节点的 MPIJobs 分布式训练与测试。

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.8.0/deploy/v2beta1/mpi-operator.yaml

4.12 【建议项】基于 Kueue TAS & JobSet 的弹性队列管理

Kueue 结合 Topology-Aware Scheduling (TAS) 和 JobSet,可以实现在 Blackwell NVL72 物理拓扑内执行高效的机架亲和性调度。

  1. 部署 Kueue (v0.18+) 与 JobSet (v0.12+):
kubectl apply --server-side -f \
  https://github.com/kubernetes-sigs/kueue/releases/download/v0.18.0/manifests.yaml

kubectl apply --server-side -f \
  https://github.com/kubernetes-sigs/jobset/releases/download/v0.12.0/manifests.yaml
  1. 创建机架拓扑描述及资源队列:
cat <<'EOF' | kubectl apply -f -
apiVersion: kueue.x-k8s.io/v1beta2
kind: Topology
metadata:
  name: gb300-nvl72-topology
spec:
  levels:
  - nodeLabel: "nvl72-domain"
  - nodeLabel: "kubernetes.io/hostname"
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: gb300-flavor
spec:
  nodeLabels:
    nvl72-domain: domain-1
  topologyName: gb300-nvl72-topology
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: gpu-cluster-queue
spec:
  namespaceSelector: {}
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: gb300-flavor
      resources:
      - name: "cpu"
        nominalQuota: 50000
      - name: "memory"
        nominalQuota: 200000Gi
      - name: "nvidia.com/gpu"
        nominalQuota: 1800
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  name: default-queue
  namespace: default
spec:
  clusterQueue: gpu-cluster-queue
EOF

4.13 【建议项】64+8 备份容灾与抢占机制

在大规模物理机架(如单个 NVLink Domain 内)进行训练时,如果某台物理机器硬件损坏,可以使用 PriorityClass 调度占位 Pod(Placeholder Pod),在故障发生时将正常训练 Pod 秒级漂移至正常的备用物理节点上。

  1. 定义抢占优先级:
cat <<'EOF' | kubectl apply -f -
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-training
value: 1000
preemptionPolicy: PreemptLowerPriority
globalDefault: false
description: "High priority for GPU training workloads. Preempts placeholder pods."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-placeholder
value: -1
preemptionPolicy: Never
globalDefault: false
description: "Low priority placeholder to hold spare GPU capacity."
EOF
  1. 部署低优先级占位副本集(Replicas 应等于分配的冗余物理 VM 数量,以 GpuLimit 为 4 表示占用一台 GB300 节点):
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpu-placeholder
  labels:
    app: gpu-placeholder
spec:
  replicas: 2
  selector:
    matchLabels:
      app: gpu-placeholder
  template:
    metadata:
      labels:
        app: gpu-placeholder
    spec:
      priorityClassName: gpu-placeholder
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      - key: kubernetes.io/arch
        operator: Equal
        value: arm64
        effect: NoSchedule
      terminationGracePeriodSeconds: 0
      containers:
      - name: pause
        image: registry.k8s.io/pause:3.9
        resources:
          limits:
            nvidia.com/gpu: 4
EOF
  1. 训练任务 Pod 配置 priorityClassName: gpu-training,即可在需要时自动挤占并退回占位 Pod。

Part 5: NCCL 测试示例

5.1 创建 GPU 与 8-Port RDMA 的 K8s ResourceClaimTemplate

在执行测试之前,必须为 Pod 所申请的 GPU 设备(4 个)及 MRDMA 网卡(8 个)声明对应的动态资源申请模板。由于 ComputeDomain 控制器在 Part 4.5 创建亲和域资源时已自动定义了 ${CD_NAME}-channel 模板,我们只需要创建 GPU 和网卡模板:

cat <<'EOF' | kubectl apply -f -
# 1. 声明 GPU 资源申请模板
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: gpu-devices-mpi-sd
spec:
  spec:
    devices:
      requests:
      - name: gpus
        exactly:
          deviceClassName: gpu.nvidia.com
          count: 4
---
# 2. 声明 8 端口 RoCE 物理网卡申请模板
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: rdma-nics-mpi-sd
spec:
  spec:
    devices:
      requests:
      - name: rdma-nics
        exactly:
          deviceClassName: rdma-devices
          count: 8
EOF

5.2 Same-Domain 双机 MPIJob 测试

测试 YAML 声明了 8 个 RDMA 网卡及 4 个 GPU 设备,并绑定到 Domain 1 物理域中。在 initContainer 中会自动将宿主机的 verbs 通信依赖库拷贝到共享卷中:

cat <<'EOF' | kubectl apply -f -
# 1. 授权 default ServiceAccount 读 Pods 权限,以便 Launcher 节点能解析 Worker IP
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io
---

> 💡 **架构设计说明:关于容器内 Mellanox 网络路由配置与 `setup_rdma_ips.py` 脚本的必要性**
>
> 1. **为什么自建 K8s 集群需要此脚本?**
>    - 开源社区版的 `dranet` 驱动(v1.3.0)只负责将宿主机的 Mellanox 网卡设备文件映射并挂载到 Pod 容器命名空间内(重命名为 `gpu0rdma0` 到 `gpu7rdma0`)。
>    - 但开源 CNI 无法感知 Google Cloud Andromeda 网络的底层路由拓扑。由于 GCP 的 VPC 网络并不支持 L2 广播,且没有自动的内网邻居发现机制,所有容器网卡的物理出网数据流**必须强制指向宿主机 link-local 虚拟网关(`fe80::1`)**。
>    - 此外,容器默认只具备一条指向管理网卡 `eth0` 的默认路由(`::/0`)。为了防止 GPU-to-GPU 的 RoCE 高速流量错误地路由到慢速的管理网络上,必须为每个物理 Rail 的 `/56` 子网段写入明细路由,强制它们从各自对应的 Mellanox 虚拟网卡出网。
>    - 该脚本实现了动态查询 GCP Metadata 元数据服务、获取网卡关联 IP、启用网卡、并注入 Rail `/56` 路由的完整闭环。
> 2. **本脚本的官方可靠性来源**:
>    - 该脚本源自 **Google Cloud 官方 AI 基础设施交付团队(AI Infrastructure / Solutions Architect)** 针对自建自管理(Self-Managed)高性能 K8s 集群提供的官方集成蓝图(Reference Blueprint)。
>    - 其配置方法与 Google 内部 Borg 物理集群生成器(`routes.minion`)及 GCP E2E 网络自动化测试框架(`topology_actions.py`)对于多网卡三层路由的配置逻辑完全一致,属于生产级别的标准实践方案。
> 3. **GKE 托管集群中为何不需要此脚本?**
>    - 如果您未来迁移到 **Google Cloud GKE** 托管平台,GKE 提供了完全集成的 **GKE DRANET + Auto IPAM** 机制。GKE 的 Dataplane V2 CNI 插件在容器拉起时会在后台自动配置所有的多网卡 IP 与 link-local 静态路由,因此在 GKE 场景下**无需声明并执行此 ConfigMap 脚本**。
> 4. **同域(NVLink)测试中的作用**:
>    - 如果本轮测试仅在机架内(Same-Domain)进行,NCCL 会全权通过物理 NVLink Switch 转发数据(MNNVL 协议,不走网卡)。但在自管理 K8s 集群中,建议依然保留此脚本,以完成网卡的初始化与 IP 上报健康自检,并为后续无缝扩展到跨机架(Cross-Domain)三层 RoCE 测试做好网络拓扑准备。

# 2. 声明容器内部 Mellanox 多网卡 IP 与 /56 静态路由配置脚本
apiVersion: v1
kind: ConfigMap
metadata:
  name: setup-rdma-ips-script
  namespace: default
data:
  setup_rdma_ips.py: |
    import json
    import urllib.request
    import subprocess
    import os
    import glob
    import sys

    def get_metadata():
        # 直接通过 link-local IP 访问 metadata,避开容器内 CoreDNS 阻断 GCP 域名问题
        req = urllib.request.Request("http://169.254.169.254/computeMetadata/v1/instance/network-interfaces/?recursive=true")
        req.add_header("Metadata-Flavor", "Google")
        try:
            with urllib.request.urlopen(req) as response:
                return json.loads(response.read().decode())
        except Exception as e:
            print(f"Failed to query metadata: {e}")
            return None

    def get_local_interfaces():
        mac_map = {}
        # 匹配 gpu* 以捕获由 CNI 映射生成的 gpu0ipvlan0 等网卡名
        for path in glob.glob("/sys/class/net/gpu*/address"):
            ifname = path.split("/")[-2]
            try:
                with open(path, "r") as f:
                    mac = f.read().strip().lower()
                    mac_map[mac] = ifname
            except Exception as e:
                print(f"Failed to read MAC for {ifname}: {e}")
        return mac_map

    def parse_ipv6_prefix_56(ip_str):
        parts = ip_str.split(":")
        if "" in parts:
            idx = parts.index("")
            missing = 8 - len(parts) + 1
            parts = parts[:idx] + ["0"] * missing + parts[idx+1:]
        block4 = parts[3]
        block4_int = int(block4, 16)
        # 屏蔽 Block 4 的低 8 位,得到 /56 网段前缀
        block4_56 = (block4_int & 0xFF00)
        parts[3] = f"{block4_56:x}"
        return ":".join(parts[:4]) + "::/56"

    def main():
        metadata = get_metadata()
        if not metadata:
            sys.exit(1)
        
        local_interfaces = get_local_interfaces()
        gateway_v6 = "fe80::1"
        
        for nic in metadata:
            if nic.get("nicType") != "MRDMA":
                continue
            
            mac = nic.get("mac", "").lower()
            ipv6_list = nic.get("ipv6s", [])
            if not ipv6_list:
                ipv6_list = nic.get("ipv6", [])
                
            if not ipv6_list:
                print(f"No IPv6 allocated for MRDMA interface with MAC {mac}")
                continue
                
            ipv6_addr = ipv6_list[0]
            if "/" not in ipv6_addr:
                ipv6_with_prefix = f"{ipv6_addr}/128"
            else:
                ipv6_with_prefix = ipv6_addr
                ipv6_addr = ipv6_addr.split("/")[0]
                
            route_target = parse_ipv6_prefix_56(ipv6_addr)
            
            ifname = local_interfaces.get(mac)
            if not ifname:
                print(f"Could not find local interface for MAC {mac}")
                continue
                
            print(f"Configuring container NIC {ifname} ({mac}) with IP {ipv6_with_prefix}, route {route_target} via {gateway_v6}")
            
            try:
                subprocess.run(["ip", "addr", "flush", "dev", ifname], check=True)
                subprocess.run(["ip", "addr", "add", ipv6_with_prefix, "dev", ifname], check=True)
                subprocess.run(["ip", "link", "set", "dev", ifname, "up"], check=True)
                subprocess.run(["ip", "-6", "route", "replace", route_target, "via", gateway_v6, "dev", ifname], check=True)
                print(f"Successfully configured {ifname}")
            except Exception as e:
                print(f"Failed to configure interface {ifname}: {e}")

    if __name__ == "__main__":
        main()
---
# 3. 双机 MPIJob 性能验证负载
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: nccl-mpi-sd
  namespace: default
spec:
  slotsPerWorker: 4
  runPolicy:
    cleanPodPolicy: Running
  sshAuthMountPath: /root/.ssh
  launcherCreationPolicy: WaitForWorkersReady
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          nodeSelector:
            nvl72-domain: domain-1
          initContainers:
          - name: gib-installer
            image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
            command: ["/bin/sh", "-c"]
            args:
            - |
              set -e
              /scripts/container_entry.sh install --install-nccl
              cp -a /usr/local/gib/. /target/gib/
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
              mkdir -p /target/gib/lib64/libibverbs
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
            volumeMounts:
            - name: gib
              mountPath: /target/gib
          containers:
          - name: launcher
            image: nvcr.io/nvidia/pytorch:26.05-py3
            imagePullPolicy: IfNotPresent
            env:
            - name: LD_LIBRARY_PATH
              value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
            command:
            - bash
            - -c
            - |
              set -e
              echo "Resolving worker IPs via K8s ServiceAccount API..."
              TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
              CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              API="https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"

              # 动态获取所有 Worker Pod 的 IP 和 FQDN
              REPLICAS=2
              for i in $(seq 0 $((REPLICAS-1))); do
                WORKER_NAME="nccl-mpi-sd-worker-${i}"
                for attempt in $(seq 1 60); do
                  WORKER_IP=$(curl -s --cacert $CA -H "Authorization: Bearer $TOKEN" \
                    "$API/api/v1/namespaces/default/pods/$WORKER_NAME" 2>/dev/null \
                    | python3 -c 'import sys,json; print(json.load(sys.stdin).get("status",{}).get("podIP",""))' 2>/dev/null || echo "")
                  if [ -n "$WORKER_IP" ]; then
                    echo "$WORKER_NAME IP: $WORKER_IP"
                    # 注册短名和完整 FQDN 以解决 DNS 解析边界问题
                    echo "$WORKER_IP $WORKER_NAME $WORKER_NAME.default.svc.cluster.local" >> /etc/hosts
                    break
                  fi
                  echo "Waiting for $WORKER_NAME IP (attempt $attempt)..."
                  sleep 5
                done
                if [ -z "$WORKER_IP" ]; then
                  echo "ERROR: Could not resolve $WORKER_NAME IP"
                  exit 1
                fi
              done

              # 动态检查 Workers 的 SSHD 端口,避免静态 sleep 导致的跑冒漏和连接冲突
              apt-get update -qq && apt-get install -y -qq netcat-openbsd > /dev/null 2>&1
              echo "Waiting for workers SSH port to open..."
              for i in $(seq 0 $((REPLICAS-1))); do
                WORKER_NAME="nccl-mpi-sd-worker-${i}"
                until nc -z $WORKER_NAME 22; do
                  echo "Waiting for SSH on $WORKER_NAME..."
                  sleep 3
                done
                echo "SSH on $WORKER_NAME is active!"
              done

              echo "Starting MPI AllReduce benchmark (8 Ranks total, 4 per node)..."
              # 隔离控制流与计算流,使用 ob1 规避 UCX 建立连接时因为 GCE TCP 阻断而在 MRDMA 接口上报错
              /usr/local/mpi/bin/mpirun --allow-run-as-root \
                -np 8 -npernode 4 \
                --mca pml ob1 \
                --mca btl tcp,self \
                --mca btl_tcp_if_include eth0 \
                --mca coll_ucc_enable 0 \
                -x LD_LIBRARY_PATH=/usr/local/gib/lib64:/usr/local/nvidia/lib64 \
                -x NCCL_NET_PLUGIN=/usr/local/gib/lib64/libnccl-net.so \
                -x NCCL_ENV_PLUGIN=gcp \
                -x NCCL_MNNVL_ENABLE=2 \
                -x NCCL_CUMEM_ENABLE=1 \
                -x NCCL_SOCKET_IFNAME=eth0 \
                -x UCX_NET_DEVICES=eth0 \
                -x NCCL_DEBUG=INFO \
                --mca plm_rsh_args '-o StrictHostKeyChecking=no' \
                /tmp/nccl-tests/build/all_reduce_perf -b 512M -e 8G -f 2 -g 1
            volumeMounts:
            - name: gib
              mountPath: /usr/local/gib
            resources:
              requests:
                cpu: "4"
                memory: 16Gi
              limits:
                cpu: "4"
                memory: 16Gi
          volumes:
          - name: gib
            emptyDir: {}
    Worker:
      replicas: 2
      template:
        spec:
          nodeSelector:
            nvl72-domain: domain-1
          resourceClaims:
          - name: compute-domain-channel
            resourceClaimTemplateName: domain-1-cd-channel
          - name: rdma-nics
            resourceClaimTemplateName: rdma-nics-mpi-sd
          - name: gpu-devices
            resourceClaimTemplateName: gpu-devices-mpi-sd
          initContainers:
          - name: gib-installer
            image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic-arm64:v1.1.2
            command: ["/bin/sh", "-c"]
            args:
            - |
              set -e
              /scripts/container_entry.sh install --install-nccl
              cp -a /usr/local/gib/. /target/gib/
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libmlx5.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/librdmacm.so* /target/gib/lib64/
              cp -a /usr/lib/aarch64-linux-gnu/libibumad.so* /target/gib/lib64/ 2>/dev/null || true
              mkdir -p /target/gib/lib64/libibverbs
              cp -a /usr/lib/aarch64-linux-gnu/libibverbs/libmlx5-rdmav34.so /target/gib/lib64/libibverbs/ 2>/dev/null || true
            volumeMounts:
            - name: gib
              mountPath: /target/gib
          containers:
          - name: worker
            image: nvcr.io/nvidia/pytorch:26.05-py3
            imagePullPolicy: IfNotPresent
            securityContext:
              privileged: true
            env:
            - name: LD_LIBRARY_PATH
              value: /usr/local/gib/lib64:/usr/local/nvidia/lib64
            - name: NCCL_NET_PLUGIN
              value: /usr/local/gib/lib64/libnccl-net.so
            - name: NCCL_ENV_PLUGIN
              value: gcp
            command:
            - bash
            - -c
            - |
              apt-get update -qq && apt-get install -y -qq openssh-server iproute2 > /dev/null 2>&1
              mkdir -p /run/sshd
              ssh-keygen -A
              
              # 显式配置 sshd 对接标准的 authorized_keys,避免默认的 ssh-publickey 文件缺失问题
              printf 'Port 22\nPermitRootLogin yes\nPubkeyAuthentication yes\nAuthorizedKeysFile /root/.ssh/authorized_keys\nStrictModes no\nPasswordAuthentication no\nUsePAM no\nPidFile /run/sshd/sshd.pid\n' > /etc/ssh/mpi_sshd_config
              
              python3 /scripts/setup_rdma_ips.py
              # 带重试机制的 git clone,防止由于 Github 网络波动导致 Worker 拉起失败
              cd /tmp
              for i in $(seq 1 30); do
                git clone --depth 1 https://github.com/NVIDIA/nccl-tests.git && break || sleep 5
              done
              
              # 去除 GPU device debug debug-symbols (-G) 编译选项,只保留 CPU 编译调试 (-g),从而加速编译过程 (避免挂死)
              cd /tmp/nccl-tests && make -j$(nproc) MPI=1 MPI_HOME=/usr/local/mpi NCCL_HOME=/usr/local/gib CUDA_HOME=/usr/local/cuda CXXFLAGS="-g" CFLAGS="-g"
              
              /usr/sbin/sshd -De -f /etc/ssh/mpi_sshd_config
            volumeMounts:
            - name: gib
              mountPath: /usr/local/gib
            - name: dshm
              mountPath: /dev/shm
            - name: setup-script
              mountPath: /scripts
            resources:
              requests:
                cpu: "96"
                memory: 800Gi
              limits:
                cpu: "96"
                memory: 800Gi
              claims:
              - name: gpu-devices
              - name: rdma-nics
              - name: compute-domain-channel
          volumes:
          - name: gib
            emptyDir: {}
          - name: dshm
            emptyDir:
              medium: Memory
              sizeLimit: 250Gi
            configMap:
              name: setup-rdma-ips-script
EOF
# 提交并查看测试输出
kubectl logs -f -l training.kubeflow.org/job-role=launcher

预期输出:

# nccl-tests version 2.19.6 nccl-headers=23004 nccl-library=23004
# Collective test starting: all_reduce_perf
# ...
   size         count      type      op    time  algbw  busbw #cvg
   536870912     134217728     float     sum   3319   161.7  283.0   0
  1073741824     268435456     float     sum   6377   168.3  294.6   0
  2147483648     536870912     float     sum  12494   171.8  300.7   0
  4294967296    1073741824     float     sum  24750   173.5  303.6   0
  8589934592    2147483648     float     sum  49213   174.5  305.5   0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 297.472 GB/s
# Collective test concluded: all_reduce_perf

双节点 AllReduce 平均总线带宽为 297.472 GB/s,峰值达到 305.59 GB/s。



实例调度与生命周期限制说明 (Important Restrictions)

Visibility: Confidential

本文档旨在澄清在 Google Cloud 平台上部署 A4X (GB200) 与 A4X-Max (GB300) 实例时,关于放置策略(Placement Policy)以及实例生命周期(删除与重建)的预期产品行为与官方限制的差异。

1. 放置策略(Placement Policy)与 Sub-block 锁定限制

适用范围: GB200 (A4X) 虚拟机与 GB300 (A4X-Max) 裸金属均受此限制约束。

现象描述 在尝试指定 Placement Policy(如 1x72 拓扑)创建 VM 或 GKE 节点池时,若对应的 sub-block(NVLink Domain)中已经存在单台独立创建的 VM,系统会拒绝调度或发生映射失败,提示 PLACEMENT_POLICY_CONSTRAINTS_INFEASIBLE。删除重建 Placement Policy 似乎能“绕过”此问题。

根本原因 这不是系统后端的缓存 Bug,而是 GCP 为了确保 NVLink 物理拓扑完整性而设计的强制锁定机制。因为 GB200 和 GB300 在底层均依赖 1x72 NVLink 拓扑,一个 NVLink Domain(sub-block)必须且只能被唯一的放置策略独占。一旦 Domain 中被创建了没有该策略的单机 VM,硬件即被视为“物理碎片化”,系统将拒绝任何需要独占整个 Domain 的 1x72 策略下发。

官方依据 根据 GCP 公开官方文档 Create an AI-optimized instance with A4X 中的重要警告说明: > Important: “When you create an A4X instance with a 1x72 topology, Compute Engine allocates an entire 18-instance sub-block (NVLink domain) to the placement policy used in the creation.”(当您使用 1x72 拓扑创建实例时,Compute Engine 会将整个包含 18 个实例的 sub-block 完全分配给创建时所使用的那个 Placement Policy。) > “If you create fewer than 18 instances, the unused instances in that sub-block remain idle and are unavailable for other requests that use a different placement policy.”(如果您创建了少于 18 个实例,该 sub-block 中未使用的实例将保持空闲状态,并且无法被使用了不同放置策略的其他请求所使用。)

最佳实践建议 为了最大化资源利用率并避免锁冲突,官方建议:强烈推荐一次性创建完整的 18 个实例。建议使用 instances.bulkInsert API 或 gcloud compute instances bulk create 命令,将 18 台 VM 作为一个整体一次性下发到空闲的 Domain 中,绝不要在同一个 Domain 内混搭创建独立 VM 和集群 VM。

2. 裸金属实例删除后的 1 小时“硬件锁定” (Transient Degraded Status)

适用范围: 仅限于 GB300 (A4X-Max) 裸金属 (Bare Metal) 实例。GB200 (A4X) 虚拟机 (VM) 由于只需回收虚拟化资源,不受此深度物理重置的限制。

现象描述 当删除一个 GB300 裸金属机器的 Domain (Subblock) 以释放配额时,GCP UI 会在长达约 1 小时的时间内将该 sub-block 标记为 “degraded hosts”,在此期间资源被锁定,无法将这些机器重新分配用于新的实例创建。

根本原因 这属于预期的产品核心安全行为,并非硬件故障或 UI 延迟。与 GB200 等普通虚拟机不同,GB300 裸金属 (Bare Metal) 实例在被删除后,必须经历一个称为 Attested Power Cycle(已验证的电源循环) 的深度安全重置流程。 为了保障多租户环境下的物理安全,系统必须移除前一个客户的任何潜在数据或状态。此过程包括 NVMe 磁盘的深度擦除、固件刷新、物理断电重启以及安全启动项的验证。该“消毒”过程需要花费大约 1 小时才能完成并自我恢复。在收到正确的密码学验证证明(Cryptographic proof)之前,平台会锁定硬件,禁止下一次分配。

最佳实践建议 对于 GB300 裸金属资源:在进行容量测试或日常运维编排时,请将这 1 小时的安全重置时间纳入工作流预期中。避免设计依赖于“秒级销毁并立即在原物理机上重建”的业务逻辑,这在 GCP Bare Metal 架构中是不被支持的。

附录

附录 A: NVSwitch 硬件组播限制与 GPUDirect RDMA 物理基准测试

附录 B: dranet CNI 与 containerd NRI 的竞态冲突 (Race Condition) 根因与避障

附录 C: 跨项目共享系统镜像 (IAM 授权)

由于定制镜像 tlinux-server-4-gb300-v5dot4-ipv6 存储在 POC 测试项目 tencent-gcp-taiji-poc 中,而生产交付部署在 tencent-gcp-taiji,必须由拥有项目 Owner/Security Admin 权限的账号执行以下 IAM 授权,否则生产项目在创建裸金属实例时会抛出 “Image not found/unauthorized” 报错:

# 授权生产项目的 Compute Engine 服务代理角色使用此镜像
gcloud compute images add-iam-policy-binding tlinux-server-4-gb300-v5dot4-ipv6 \
  --project=tencent-gcp-taiji-poc \
  --member="serviceAccount:441669677380-compute@developer.gserviceaccount.com" \
  --role="roles/compute.imageUser"

# 授权生产项目的编辑者/创建者用户角色使用此镜像
gcloud compute images add-iam-policy-binding tlinux-server-4-gb300-v5dot4-ipv6 \
  --project=tencent-gcp-taiji-poc \
  --member="projectEditor:tencent-gcp-taiji" \
  --role="roles/compute.imageUser"

附录 D: 接口命名映射及 udev/rdma 配置参考

本附录展示了 tlinux4-customer-init-gb300.sh 启动脚本在操作系统底层执行的具体网卡重命名命令,用于临时调试或手动配置验证。

原接口名称 新映射名称 (兼容格式) 网卡类型 重命名工具
eth0 bond0 IDPF (主管理/控制) ip link set name (udev 持久化)
eth1 bond1 IDPF (存储/Lustre) ip link set name (udev 持久化)
mlx5_0 bond2 ConnectX-8 MRDMA 1 rdma dev set (每次开机)
mlx5_1 bond3 ConnectX-8 MRDMA 2 rdma dev set (每次开机)
mlx5_2 bond4 ConnectX-8 MRDMA 3 rdma dev set (每次开机)
mlx5_3 bond5 ConnectX-8 MRDMA 4 rdma dev set (每次开机)
mlx5_4 bond6 ConnectX-8 MRDMA 5 rdma dev set (每次开机)
mlx5_5 bond7 ConnectX-8 MRDMA 6 rdma dev set (每次开机)
mlx5_6 bond8 ConnectX-8 MRDMA 7 rdma dev set (每次开机)
mlx5_7 bond9 ConnectX-8 MRDMA 8 rdma dev set (每次开机)
# 手动重命名管理网卡接口 (以 eth0 为例)
ip link set dev eth0 down
ip link set dev eth0 name bond0
ip link set dev bond0 up

# 手动重命名 MRDMA InfiniBand 设备 (以 mlx5_0 为例)
rdma dev set mlx5_0 name bond2

附录 E: tlinux4-customer-init-gb300.sh 完整源码 (客户交付生产版)

此脚本负责系统基础初始化,包括主磁盘分区、Llocal SSD RAID0、双网卡命名重映射、NUMA 物理巨页限制配置和 LNet 路由规则挂载,不预装 Kubernetes。

点击查看 tlinux4-customer-init-gb300.sh 完整源码

#!/bin/bash
# TLinux 4 (TencentOS Server 4.0) — GB300 (A4X-Max) Startup Script (Customer Version)
# Compatible with: a4x-maxgpu-4g-metal (Grace Blackwell Bare Metal)
#
# Configures: 
#   1. Boot Disk Partitions (/usr/local 20G, /data remaining)
#   2. Local SSD RAID0 (/mnt/stateful_partition, 4x 3TB = 12TB)
#   3. Network Interface Renaming:
#      - IDPF netdevs: eth0 -> bond0, eth1 -> bond1
#      - MRDMA ibdevs: mlx5_0..7 -> bond2..9
#   4. Dynamic NUMA-aligned 2MB Hugepages Allocation (16GB total)
#   5. Lustre storage interface routing (eth1 / bond1)
#   6. SSH ports 22 + 56000 configuration
#
set -euo pipefail
exec > >(tee /var/log/tlinux4-customer-init.log) 2>&1

INIT_STAMP=/etc/.customer-init-done

###############################################################################
# Network interface rename (IDPF & MRDMA) — runs every boot
# IDPF: eth0→bond0 (mgmt), eth1→bond1 (lustre client)
# MRDMA: mlx5_0..7→bond2..9
###############################################################################
echo "=== Network interface rename (GB300) ==="

# 1. IDPF Interface renaming & persistent udev rules creation (one-time)
UDEV_RULE="/etc/udev/rules.d/70-idpf-bond-rename.rules"
if [ ! -f "$UDEV_RULE" ]; then
  echo "  Creating udev rule for persistent IDPF naming..."
  # Detect IDPF PCI devices in sorting order
  PCI_DEVS=($(ls -la /sys/bus/pci/drivers/idpf/ | grep '\->' | awk -F'/' '{print $NF}' | sort))
  if [ "${#PCI_DEVS[@]}" -ge 2 ]; then
    PCI0="${PCI_DEVS[0]}"
    PCI1="${PCI_DEVS[1]}"
    
    cat > "$UDEV_RULE" << UDEVEOF
SUBSYSTEM=="net", ACTION=="add", KERNELS=="$PCI0", NAME="bond0"
SUBSYSTEM=="net", ACTION=="add", KERNELS=="$PCI1", NAME="bond1"
UDEVEOF
    echo "  udev rules written: $PCI0 -> bond0, $PCI1 -> bond1"
    
    # Apply rename immediately for first boot
    for dev in eth0 eth1; do
      if ip link show "$dev" &>/dev/null; then
        PCI_DEV=$(basename "$(readlink -f /sys/class/net/$dev/device)")
        if [ "$PCI_DEV" = "$PCI0" ]; then
          ip link set dev "$dev" down
          ip link set dev "$dev" name bond0
          ip link set dev bond0 up
          echo "  Immediate rename: $dev -> bond0"
        elif [ "$PCI_DEV" = "$PCI1" ]; then
          ip link set dev "$dev" down
          ip link set dev "$dev" name bond1
          ip link set dev bond1 up
          echo "  Immediate rename: $dev -> bond1"
        fi
      fi
    done
  else
    echo "  ERROR: Less than 2 IDPF interfaces found. Skipping IDPF rename."
  fi
else
  echo "  udev rule $UDEV_RULE already exists (IDPF named persistently)"
fi

# 2. Update NetworkManager profiles to bind to new interface names
for idx in 0 1; do
  NM_CONN="/etc/NetworkManager/system-connections/eth${idx}-idpf.nmconnection"
  if [ -f "$NM_CONN" ]; then
    sed -i "s/interface-name=eth${idx}/interface-name=bond${idx}/" "$NM_CONN"
    sed -i "s/id=eth${idx}-idpf/id=bond${idx}-idpf/" "$NM_CONN"
    nmcli connection reload 2>/dev/null || true
    echo "  Updated NM profile: eth${idx} → bond${idx}"
  fi
done

# 3. RDMA IB device renaming (rdma dev set — runtime only, needs to run on every boot)
for i in {0..7}; do
  if rdma dev set "mlx5_${i}" name "bond$((i+2))" 2>/dev/null; then
    echo "  Mellanox IB: mlx5_${i} → bond$((i+2))"
  fi
done

echo "  --- Rename Results ---"
ip -br link show type ether 2>/dev/null || true
rdma link show 2>/dev/null || true


if [ -f "$INIT_STAMP" ]; then
  echo "Customer initialization already completed. Skipping stamp checks."
  exit 0
fi

echo "================================================================"
# TLinux 4 v5.4 initialization
echo "  TLinux 4 v5.4 — GB300 Customer Startup Script"
echo "  $(date)"
echo "================================================================"

###############################################################################
# [1] Boot disk partitioning: /usr/local (20G) + /data (remaining)
###############################################################################
echo "=== [1] Boot disk partitioning ==="

BOOT_DEV=$(findmnt -n -o SOURCE / | sed 's/p[0-9]*$//')
PARTITION_STAMP=/etc/.partitions-created

if [ ! -f "$PARTITION_STAMP" ]; then
  echo "--- Creating boot disk partitions (1TB disk) ---"
  # Expand GPT to full disk
  sgdisk -e "$BOOT_DEV" 2>/dev/null || true

  sgdisk -n 3:0:+20G -t 3:8300 "$BOOT_DEV"
  sgdisk -n 4:0:0 -t 4:8300 "$BOOT_DEV"
  partprobe "$BOOT_DEV" 2>/dev/null || true
  sleep 2

  mkfs.xfs -f "${BOOT_DEV}p3"
  mkfs.xfs -f "${BOOT_DEV}p4"

  TMPMNT=$(mktemp -d)
  mount "${BOOT_DEV}p3" "$TMPMNT"
  cp -a /usr/local/. "$TMPMNT/" 2>/dev/null || true
  umount "$TMPMNT"
  rmdir "$TMPMNT"

  mount "${BOOT_DEV}p3" /usr/local
  mkdir -p /data
  mount "${BOOT_DEV}p4" /data

  P3_UUID=$(blkid -s UUID -o value "${BOOT_DEV}p3")
  P4_UUID=$(blkid -s UUID -o value "${BOOT_DEV}p4")
  if [ -n "$P3_UUID" ] && ! grep -q "$P3_UUID" /etc/fstab; then
    echo "UUID=$P3_UUID /usr/local xfs defaults,nofail 0 2" >> /etc/fstab
  fi
  if [ -n "$P4_UUID" ] && ! grep -q "$P4_UUID" /etc/fstab; then
    echo "UUID=$P4_UUID /data xfs defaults,nofail 0 2" >> /etc/fstab
  fi

  touch "$PARTITION_STAMP"
  echo "Boot disk partitioned: p3=/usr/local(20G) p4=/data(remaining)"
fi

###############################################################################
# [2] Local SSD RAID0 setup (4x 3TB NVMe Local SSDs)
###############################################################################
echo "=== [2] Local SSD RAID0 setup ==="

SSD_DEVICES=()
for disk in /dev/nvme*n1; do
  [ -b "$disk" ] || continue
  PART_COUNT=$(lsblk -n -o NAME "$disk" 2>/dev/null | wc -l)
  if [ "$PART_COUNT" -le 1 ]; then
    SSD_DEVICES+=("$disk")
    echo "  Detected local SSD: $disk ($(lsblk -n -d -o SIZE "$disk"))"
  fi
done

NUM_SSDS=${#SSD_DEVICES[@]}
echo "  Found $NUM_SSDS local SSD device(s)"

if [ "$NUM_SSDS" -gt 0 ]; then
  MOUNT_POINT="/mnt/stateful_partition"
  mkdir -p "$MOUNT_POINT"

  if [ "$NUM_SSDS" -eq 1 ]; then
    TARGET_DEV="${SSD_DEVICES[0]}"
    mkfs.xfs -f "$TARGET_DEV"
    mount "$TARGET_DEV" "$MOUNT_POINT"
    DEV_UUID=$(blkid -s UUID -o value "$TARGET_DEV")
    if [ -n "$DEV_UUID" ] && ! grep -q "$DEV_UUID" /etc/fstab; then
      echo "UUID=$DEV_UUID $MOUNT_POINT xfs defaults,nofail 0 2" >> /etc/fstab
    fi
  else
    mdadm --stop /dev/md0 2>/dev/null || true
    mdadm --create /dev/md0 --level=0 --raid-devices="$NUM_SSDS" "${SSD_DEVICES[@]}" --force --run
    mkfs.xfs -f /dev/md0
    mount /dev/md0 "$MOUNT_POINT"
    mkdir -p /etc/mdadm
    mdadm --detail --scan >> /etc/mdadm/mdadm.conf
    DEV_UUID=$(blkid -s UUID -o value /dev/md0)
    if [ -n "$DEV_UUID" ] && ! grep -q "$DEV_UUID" /etc/fstab; then
      echo "UUID=$DEV_UUID $MOUNT_POINT xfs defaults,nofail 0 2" >> /etc/fstab
    fi
  fi

  mkdir -p "$MOUNT_POINT"/{gcsfuse-cache,scratch-data}
  echo "$MOUNT_POINT mounted (total: $(df -h "$MOUNT_POINT" | awk 'NR==2{print $2}'))"
else
  echo "WARNING: No local SSD devices found"
  mkdir -p /mnt/stateful_partition
fi

###############################################################################
# [3] Safe NUMA-aligned 2MB Hugepages allocation (asapd-lite dependency)
###############################################################################
echo "=== [3] Safe Hugepages allocation ==="

# Target 2MB hugepage size explicitly on each NUMA node to prevent OOM hang on 64KB page size OS kernels
for d in /sys/devices/system/node/node[0-9]*; do
  if [ -d "$d/hugepages/hugepages-2048kB" ]; then
    echo 4096 | tee "$d/hugepages/hugepages-2048kB/nr_hugepages"
  fi
done

# Ensure hugetlbfs is mounted
mkdir -p /dev/hugepages
mount -t hugetlbfs none /dev/hugepages 2>/dev/null || true

###############################################################################
# [4] Lustre storage network routing
# Bind LNet networks option to storing interfaces (e.g. eth1 or virtual bond1)
###############################################################################
echo "=== [4] Lustre Storage routing ==="

# Set modprobe default options to tcp(bond1) (or tcp(eth1) if not virtualized)
echo 'options lnet networks="tcp(bond1)"' > /etc/modprobe.d/lustre.conf

# In runtime, explicitly register the LNet NI if kernel autodetect fails:
lnetctl lnet configure
lnetctl net add --net tcp --if bond1 2>/dev/null || lnetctl net add --net tcp --if eth1 || true

# Mount local Managed Service for Lustre instance MGS IP
LUSTRE_MGS="10.75.100.4@tcp:/lustrefs"
mkdir -p /gstore
mount -t lustre "$LUSTRE_MGS" /gstore

# Mount to a fresh path for K8s hostPath bypass caching
mkdir -p /lustre
mount -t lustre "$LUSTRE_MGS" /lustre


###############################################################################
# [4.1] Kubernetes hostPath Lustre Volume Integration
# Guide to mapping the Lustre mount into K8s Pods safely bypassing Bind Mount cache
###############################################################################
echo "=== [4.1] Kubernetes Lustre HostPath integration ==="

# WARNING: Kubelet caches directory inodes on boot. If you mount Lustre *after* 
# Kubelet starts, Pods mapping /gstore via hostPath will only see the unmounted 
# empty folder under the host rootfs. 
#
# Bypassing Cache Procedure:
# 1. Ensure Lustre is mounted on a fresh path (e.g. /lustre) on all Workers.
# 2. Force restart Kubelet to flush the bind mount table:
#    systemctl restart kubelet
#
# Pod YAML volume configuration snippet:
#
# spec:
#   containers:
#   - name: worker
#     securityContext:
#       privileged: true
#     volumeMounts:
#     - name: lustre-volume
#       mountPath: /gstore
#       mountPropagation: Bidirectional  # Enables read/write synchronization
#   volumes:
#   - name: lustre-volume
#     hostPath:
#       path: /lustre
#       type: Directory



###############################################################################
# [5] sshd ports 22 + 56000
###############################################################################
echo "=== [5] sshd port 22 + 56000 ==="

SSHD_CHANGED=0
if ! grep -q '^Port 22' /etc/ssh/sshd_config; then
  sed -i '1i Port 22' /etc/ssh/sshd_config
  SSHD_CHANGED=1
fi
if ! grep -q '^Port 56000' /etc/ssh/sshd_config; then
  sed -i '/^Port 22/a Port 56000' /etc/ssh/sshd_config
  SSHD_CHANGED=1
fi
if [ "$SSHD_CHANGED" -eq 1 ]; then
  systemctl restart sshd
  echo "sshd now listening on ports 22 and 56000"
else
  echo "sshd ports 22 and 56000 already configured"
fi

###############################################################################
# [6] Mellanox MRDMA host interface routing
# Add specific /56 routing via gateway fe80::1 to prevent NDP drops
###############################################################################
echo "=== [6] Mellanox MRDMA host interface routing ==="

metadata_interfaces=$(curl -sf -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/network-interfaces/?recursive=true)

if [ -n "$metadata_interfaces" ]; then
  python3 - <<EOF
import json
import subprocess
import glob

interfaces = json.loads('''$metadata_interfaces''')
gateway_v6 = "fe80::1"

mac_to_dev = {}
for path in glob.glob("/sys/class/net/*/address"):
    dev = path.split("/")[-2]
    try:
        with open(path, "r") as f:
            mac_to_dev[f.read().strip().lower()] = dev
    except Exception:
        pass

def parse_ipv6_prefix_56(ip_str):
    parts = ip_str.split(":")
    if "" in parts:
        idx = parts.index("")
        missing = 8 - len(parts) + 1
        parts = parts[:idx] + ["0"] * missing + parts[idx+1:]
    block4 = parts[3]
    block4_int = int(block4, 16)
    block4_56 = (block4_int & 0xFF00)
    parts[3] = f"{block4_56:x}"
    return ":".join(parts[:4]) + "::/56"

for nic in interfaces:
    if nic.get("nicType") == "MRDMA":
        mac = nic.get("mac", "").lower()
        ipv6s = nic.get("ipv6s", [])
        if mac and ipv6s:
            dev = mac_to_dev.get(mac)
            if dev:
                ipv6_addr = ipv6s[0]
                route_target = parse_ipv6_prefix_56(ipv6_addr)
                print(f"Setting host route: {route_target} via {gateway_v6} dev {dev}")
                subprocess.run(["ip", "-6", "route", "replace", route_target, "via", gateway_v6, "dev", dev], check=True)
EOF
else
  echo "WARNING: Failed to fetch metadata interfaces. Skipping host route setup."
fi

# Load networking modules for Calico (VXLAN / IPIP)
echo "=== Loading networking modules ==="
sudo modprobe vxlan || true
sudo modprobe ipip || true
echo "vxlan" | sudo tee /etc/modules-load.d/vxlan.conf
echo "ipip" | sudo tee /etc/modules-load.d/ipip.conf

touch "$INIT_STAMP"
echo "=== Customer Initialization Complete ==="

附录 F: tlinux4-internal-init-gb300.sh 完整源码 (内部测试自建集群版)

此脚本作为测试环境专供,调用 tlinux4-customer-init-gb300.sh 执行基线配置后,自动安装 containerd 容器运行期、nvidia-container-toolkit 及 K8s Worker 工具,并从 GCE 实例 Metadata 中解析 token 自动加入测试集群控制面。

点击查看 tlinux4-internal-init-gb300.sh 完整源码

#!/bin/bash
# TLinux 4 (TencentOS Server 4.0) — GB300 (A4X-Max) Internal Test Startup Script
# Compatible with: a4x-maxgpu-4g-metal (Grace Blackwell Bare Metal)
#
# Automatically installs: containerd, nvidia-container-toolkit, and K8s Worker components
# then joins the node to the self-managed K8s cluster.
#
set -euo pipefail
exec > >(tee /var/log/tlinux4-internal-init.log) 2>&1

K8S_VERSION="1.34"
K8S_PATCH="1.34.9"
INTERNAL_INIT_STAMP=/etc/.internal-init-done

# 1. Run the customer baseline initialization script to setup disk, naming and routing
SCRIPT_DIR="$(dirname "$0")"
if [ -f "$SCRIPT_DIR/tlinux4-customer-init-gb300.sh" ]; then
  echo "=== Running customer-init base setup ==="
  bash "$SCRIPT_DIR/tlinux4-customer-init-gb300.sh"
else
  echo "ERROR: tlinux4-customer-init-gb300.sh not found in $SCRIPT_DIR!"
  exit 1
fi

if [ -f "$INTERNAL_INIT_STAMP" ]; then
  echo "Internal test environment setup already completed. Skipping."
  exit 0
fi

echo "================================================================"
echo "  TLinux 4 v5.4 — GB300 Internal Test Startup Script"
echo "  $(date)"
echo "================================================================"

###############################################################################
# [1] K8s Kernel parameters & module settings
###############################################################################
echo "=== [1] Configuring Kernel Modules for Kubernetes ==="

cat > /etc/modules-load.d/k8s.conf <<EOF
overlay
br_netfilter
EOF
modprobe overlay
modprobe br_netfilter

cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sysctl --system

###############################################################################
# [2] Install containerd container runtime
###############################################################################
echo "=== [2] Installing containerd ==="

dnf install -y --enablerepo=AppStream container-selinux 2>/dev/null || true
DOCKER_ARCH=$(uname -m)

cat > /etc/yum.repos.d/docker-ce.repo <<REPOEOF
[docker-ce-stable]
name=Docker CE Stable - ${DOCKER_ARCH}
baseurl=https://download.docker.com/linux/rhel/9/${DOCKER_ARCH}/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/rhel/gpg
REPOEOF

dnf install -y containerd.io
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

# Redirect containerd root directory to /data/containerd to utilize SSD/HDD space
mkdir -p /data/containerd
sed -i "s|root = .*/var/lib/containerd.*|root = \"/data/containerd\"|" /etc/containerd/config.toml

systemctl enable --now containerd

###############################################################################
# [3] Install NVIDIA Container Toolkit
###############################################################################
echo "=== [3] Installing NVIDIA Container Toolkit ==="

curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
  tee /etc/yum.repos.d/nvidia-container-toolkit.repo

dnf install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=containerd --set-as-default

systemctl restart containerd

###############################################################################
# [4] Install Kubeadm, Kubelet and Kubelet CLI tools
###############################################################################
echo "=== [4] Installing K8s packages (kubeadm/kubelet/kubectl) ==="

cat > /etc/yum.repos.d/kubernetes.repo <<REPOEOF
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v${K8S_VERSION}/rpm/repodata/repomd.xml.key
REPOEOF

dnf install -y --enablerepo=AppStream kubelet-${K8S_PATCH} kubeadm-${K8S_PATCH} kubectl-${K8S_PATCH} conntrack-tools socat iptables ethtool
systemctl enable kubelet

###############################################################################
# [5] Kubeadm Join (via GCE Instance Metadata)
###############################################################################
echo "=== [5] Kubeadm Join ==="

CP_IP=$(curl -sf -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/attributes/cp-ip 2>/dev/null || true)
JOIN_TOKEN=$(curl -sf -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/attributes/join-token 2>/dev/null || true)
JOIN_HASH=$(curl -sf -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/attributes/join-hash 2>/dev/null || true)

NODE_NAME=$(cat /proc/sys/kernel/hostname 2>/dev/null || curl -sf -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/name || echo "unknown")

if [[ -n "$CP_IP" && -n "$JOIN_TOKEN" && -n "$JOIN_HASH" ]]; then
  echo "  Joining cluster at ${CP_IP}:6443..."
  kubeadm join ${CP_IP}:6443 \
    --token "${JOIN_TOKEN}" \
    --discovery-token-ca-cert-hash "sha256:${JOIN_HASH}" \
    --node-name "${NODE_NAME}" \
    --ignore-preflight-errors=Hostname 2>&1 || echo "kubeadm join failed, manual join required"
else
  echo "  Missing CP metadata parameters (cp-ip, join-token, join-hash). Skipping auto-join."
  echo "  Please execute manual join on the node. Retrieve token command on K8s Control Plane:"
  echo "  kubeadm token create --print-join-command"
fi

###############################################################################
# Verification
###############################################################################
echo ""
echo "================================================================"
echo "  Verification Results"
echo "================================================================"
echo "Containerd: $(systemctl is-active containerd 2>/dev/null)"
echo "Kubelet: $(systemctl is-active kubelet 2>/dev/null)"
kubeadm version 2>/dev/null || echo "kubeadm not installed"

# Load networking modules for Calico (VXLAN / IPIP)
echo "=== Loading networking modules ==="
sudo modprobe vxlan || true
sudo modprobe ipip || true
echo "vxlan" | sudo tee /etc/modules-load.d/vxlan.conf
echo "ipip" | sudo tee /etc/modules-load.d/ipip.conf

touch "$INTERNAL_INIT_STAMP"
echo "=== Internal Test Initialization Complete ==="

附录 G: asapd-lite-installer-tlinux.yaml 完整安装配置

此 DaemonSet 用于在自管理 Kubernetes 集群中自动拉起 asapd-lite 底层网络管控服务,并包含 netslo-ebpf-manager 侧车容器。

点击查看 asapd-lite-installer-tlinux.yaml 完整内容

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: asapd-lite
  namespace: kube-system
  labels:
    k8s-app: asapd-lite
spec:
  selector:
    matchLabels:
      k8s-app: asapd-lite
  template:
    metadata:
      labels:
        k8s-app: asapd-lite
    spec:
      priorityClassName: system-node-critical
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node-type
                    operator: In
                    values:
                      - a4x-max
      tolerations:
      - operator: Exists
      hostNetwork: true
      hostPID: true
      containers:
      - name: asapd-lite
        image: us-docker.pkg.dev/gce-ai-infra/asapd-lite/asapd-lite:v0.0.8
        command: ["/bin/bash", "-c"]
        args:
          - |
            set -x
            /usr/local/bin/run_asapd_lite.sh --run_hairpin_probe=true --hairpin_sender_port_start=12345 --hairpin_receiver_port_start=23456 --metrics_domain=prod
            sleep 15
            DADFAILED_FOUND=0
            INTERFACES=$(ip -o link show | awk -F': ' '{print $2}' | grep 'gpu')
            for iface in $INTERFACES; do
              if ip -6 addr show dev "$iface" | grep -q 'dadfailed'; then
                echo "Found dadfailed on interface $iface. Bringing it down and up."
                ip link set dev "$iface" down
                ip link set dev "$iface" up
                DADFAILED_FOUND=1
              fi
            done
            if [ "$DADFAILED_FOUND" -eq 1 ]; then
              echo "Found and attempted to fix dadfailed on one or more interfaces. Exiting to allow container restart."
              exit 1
            fi
            echo "No dadfailed detected on interfaces."
            wait
        env:
        - name: LD_LIBRARY_PATH
          value: /asapd-lite/controller
        livenessProbe:
          httpGet:
            path: /healthz
            port: 19540
          initialDelaySeconds: 60
        readinessProbe:
          httpGet:
            path: /healthz
            port: 19540
        securityContext:
          privileged: true
        volumeMounts:
        - name: host-sys
          mountPath: /sys
        - name: host-proc
          mountPath: /proc
        - name: host-udev-rules
          mountPath: /etc/udev/rules.d
        - name: host-run-udev
          mountPath: /run/udev
        - name: host-run-asapd-lite
          mountPath: /run/asapd-lite
        - mountPath: /hugepages-2Mi
          name: hugepage-2mi
        - name: var-log-google-asapd-lite
          mountPath: /var/log/google/asapd-lite
        - name: var-lib-google-asapd-lite
          mountPath: /var/lib/google/asapd-lite
        - name: host-systemd-system
          mountPath: /etc/systemd/system
        - name: host-run-systemd
          mountPath: /run/systemd
        resources:
          limits:
            hugepages-2Mi: 8192Mi
            memory: 1024Mi
      - name: netslo-ebpf-manager
        image: us-docker.pkg.dev/gce-ai-infra/netslo-ebpf-manager/ebpf_manager:release
        imagePullPolicy: Always
        command: ["/bin/bash", "-c"]
        args:
          - |
            echo "Waiting for asapd-lite to be healthy..."
            while ! curl -s -o /dev/null http://localhost:19540/healthz; do
              echo "asapd-lite not healthy yet, sleeping..."
              sleep 5
            done
            echo "asapd-lite is healthy. Starting NETSLO eBPF Manager..."
            exec /usr/local/bin/bpf_manager
        securityContext:
          privileged: true
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        volumeMounts:
        - name: sys-fs-bpf
          mountPath: /sys/fs/bpf
        - name: kernel-debug
          mountPath: /sys/kernel/debug
        - name: lib-modules
          mountPath: /lib/modules
          readOnly: true
        - name: host-sys
          mountPath: /sys
          readOnly: true
        - name: var-log-google-netslo
          mountPath: /var/log/google/netslo-ebpf-manager
      volumes:
      - name: host-sys
        hostPath:
          path: /sys
      - name: host-proc
        hostPath:
          path: /proc
      - name: host-udev-rules
        hostPath:
          path: /etc/udev/rules.d
      - name: host-run-udev
        hostPath:
          path: /run/udev
      - name: host-run-asapd-lite
        hostPath:
          path: /run/asapd-lite
          type: DirectoryOrCreate
      - name: hugepage-2mi
        emptyDir:
          medium: HugePages-2Mi
      - name: var-log-google-asapd-lite
        hostPath:
          path: /var/log/google/asapd-lite
          type: DirectoryOrCreate
      - name: var-lib-google-asapd-lite
        hostPath:
          path: /var/lib/google/asapd-lite
          type: DirectoryOrCreate
      - name: host-systemd-system
        hostPath:
          path: /etc/systemd/system
      - name: host-run-systemd
        hostPath:
          path: /run/systemd
      - name: sys-fs-bpf
        hostPath:
          path: /sys/fs/bpf
          type: DirectoryOrCreate
      - name: kernel-debug
        hostPath:
          path: /sys/kernel/debug
      - name: lib-modules
        hostPath:
          path: /lib/modules
      - name: var-log-google-netslo
        hostPath:
          path: /var/log/google/netslo-ebpf-manager
          type: DirectoryOrCreate
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1%

附录 H: UEFI NVRAM 丢失导致引导挂死(Boot Loop)手动修复流程

附录 I: 物理坏机排查、Placement Policy 避障与故障降级跑测指南

在 bare metal(裸金属)GPU 交付集群中,硬件故障、GPU 掉卡以及拓扑不完整是常见的实战问题。本附录详细说明如何识别故障,以及如何在物理故障情况下进行降级验证。

1. 物理坏机与掉卡的识别方法

在执行部署或性能测试前,必须验证节点的 GPU 和网络硬件是否完整。以下为两种核心的硬件排查手段:

2. Placement Policy 与子预留约束

3. 物理故障下的降级跑测流程

若当前整个交付环境因为 domain/clique 存在坏机导致无法以标准 4-GPU 拓扑执行测试,可以通过以下降级(Degraded)跑测流程绕过硬件阻碍:


附录 J: 参考与信源引用 (References)

本部署指南的架构设计与核心技术参数来自于以下 GCP 官方文档与内部产品规范:

  1. GCP 裸金属与 a4x-maxgpu-4g-metal 机型规格:
  2. ConnectX-8 MRDMA 8-way Rail-Aligned 网络拓扑与 GPUDirect RDMA:
  3. Intel IDPF 驱动与 Host OS 内核兼容性:
  4. K8s 硬件管控及 GKE 兼容版本要求:
  5. Ops Agent 主机监控与内核日志采集:

常见问题与故障排除 (Troubleshooting)

问题:mpi-operator 反复崩溃或 apiserver 响应超时 如果集群变得不稳定,或者 mpi-operator 因为租约超时(lease timeouts)反复重启,这通常是由于安装了冗余的 NFD (Node Feature Discovery) 插件引起的(例如 gpu-operator 默认自带的 NFD 与集群已有的 NFD 冲突)。冲突会导致数百个 worker pod 陷入 CrashLoopBackOff,并向 apiserver 发送大量垃圾请求,导致 API Server 瘫痪。 解决方案:请确保在部署 gpu-operator 时禁用 NFD:

helm upgrade gpu-operator nvidia/gpu-operator --namespace gpu-operator --reuse-values --set nfd.enabled=false