遇到问题

使用PaddleOCR进行多卡训练的时候, 显示错误

server not ready, wait 3 sec to retry...
not ready endpoints:['127.0.0.1:49741']
W0213 13:56:36.361328 41053 dynamic_loader.cc:258] You may need to install 'nccl2' from NVIDIA official website: https://developer.nvidia.com/nccl/nccl-downloadbefore install PaddlePaddle.
Traceback (most recent call last):
  File "tools/train.py", line 145, in <module>
    main(config, device, logger, vdl_writer)
  File "tools/train.py", line 44, in main
    dist.init_parallel_env()
  File "/home/xxx/.conda/envs/paddle/lib/python3.8/site-packages/paddle/distributed/parallel.py", line 225, in init_parallel_env
    parallel_helper._init_parallel_ctx()
  File "/home/xxx/.conda/envs/paddle/lib/python3.8/site-packages/paddle/fluid/dygraph/parallel_helper.py", line 42, in _init_parallel_ctx
    __parallel_ctx__clz__.init()
RuntimeError: (PreconditionNotMet) The third-party dynamic library (libnccl.so) that Paddle depends on is not configured correctly. (error code is libnccl.so: cannot open shared object file: No such file or directory)
  Suggestions:
  1. Check if the third-party dynamic library (e.g. CUDA, CUDNN) is installed correctly and its version is matched with paddlepaddle you installed.
  2. Configure third-party dynamic library environment variables as follows:
  - Linux: set LD_LIBRARY_PATH by `export LD_LIBRARY_PATH=...`
  - Windows: set PATH by `set PATH=XXX; (at /paddle/paddle/fluid/platform/dynload/dynamic_loader.cc:285)

INFO 2022-02-13 13:56:37,879 launch_utils.py:320] terminate process group gid:41058
INFO 2022-02-13 13:56:41,884 launch_utils.py:341] terminate all the procs
ERROR 2022-02-13 13:56:41,884 launch_utils.py:602] ABORT!!! Out of all 2 trainers, the trainer process with rank=[0] was aborted. Please check its log.
INFO 2022-02-13 13:56:45,888 launch_utils.py:341] terminate all the procs
INFO 2022-02-13 13:56:45,888 launch.py:311] Local processes completed.

推测错误是系统缺少nccl2导致的.

解决方法

  1. 下载与CUDA版本和linux系统版本符合的NCCL, URL(需要登录NVIDIA账号)

    • 查看cuda版本: nvidia-smi
    • 查看系统版本: lsb_release -a

    使用cuda 11.4, Ubuntu 18.04.6 LTS对应的 Local installer for Ubuntu 18.04

  2. 下载nccl的git项目

    git clone https://github.com/NVIDIA/nccl.git

    安装设置

    chmod 777 -R nccl
cd nccl

vim src/collectives/device/gen_rules.sh
:set ff=unix  # gen_rules.sh文件是dos格式文件(windows默认格式), 需要转换成unix格式

make -j48 src.build BUILDDIR=/home/xxx/Install/nccl/build CUDA_HOME=/usr/local/cuda NVCC_GENCODE="-gencode=arch=compute_80,code=sm_80"
    • j12表示使用12个核心，使用nproc查看总核心数，根据具体情况进行调整；
    • BUILDDIR表示编译后，一些文件的存储路径；默认是nccl/build；当然如果是root用户可以指定到/usr/local/ncc/；
    • CUDA_HOME表示CUDA的目录，默认就是/usr/local/cuda，但是我这边不加这个，会报错，很奇怪；
    • NVCC_GENCODE，如果不添加该字段，默认会编译支持所有架构；为了加速编译以及降低二进制文件大小，添加该字段，具体见：CUDA GPUs, **Matching CUDA arch and CUDA gencode for various NVIDIA architectures** (注意似乎目前nccl还不支持算力8.6, 使用8.0. NVCC_GENCODE="-gencode=arch=compute_80,code=sm_80")

    最后呢，编译完成的文件都在指定的**BUILDDIR**字段路径下，需要将其添加到环境变量；

    vim ~/.bashrc

    在打开的文件中，添加：

    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/xxx/Install/nccl/build/lib
export PATH=$PATH:/home/xxx/Install/nccl/build/include

    保存后，执行：

    source ~/.bashrc

    之后就能开始跑多卡训练了.