CUDA

遇到问题 使用PaddleOCR进行多卡训练的时候, 显示错误 server not ready, wait 3 sec to retry... not ready endpoints:['127.0.0.1:49741'] W0213 13:56:36.361328 41053 dynamic_loader.cc:258] You may need to install 'nccl2' from NVIDIA official website: https://developer.nvidia.com/nccl/nccl-downloadbefore install PaddlePaddle. Traceback (most recent call last): File "tools/train.py", line 145, in <module> main(config, device, logger, vdl_writer) File "tools/train.py", line 44, in main dist.init_parallel_env() File "/home/xxx/.conda/envs/paddle/lib/python3.8/site-packages/paddle/distributed/parallel.py", line 225, in init_parallel_env parallel_helper._init_parallel_ctx() File "/home/xxx/.conda/envs/paddle/lib/python3.8/site-packages/paddle/fluid/dygraph/parallel_helper.py", line 42, in _init_parallel_ctx __parallel_ctx__clz__.init() RuntimeError: (PreconditionNotMet) The third-party dynamic library (libnccl.so) that Paddle depends on is not configured correctly. (error code is libnccl.so: cannot open shared object file: No such file or directory) Suggestions: 1. Check if the third-party dynamic library (e.g. CUDA, CUDNN) is installed correctly and its version is matched with paddlepaddle you installed. 2. Configure third-party dynamic library environment variables as follows: - Linux: set LD_LIBRARY_PATH by `export LD_LIBRARY_PATH=...` - Windows: set PATH by `set PATH=XXX; (at /paddle/paddle/fluid/platform/dynload/dynamic_loader.cc:285) INFO 2022-02-13 13:56:37,879 launch_utils.py:320] terminate process group gid:41058 INFO 2022-02-13 13:56:41,884 launch_utils.py:341] terminate all the procs ERROR 2022-02-13 13:56:41,884 launch_utils.py:602] ABORT!!! Out of all 2 trainers, the trainer process with rank=[0] was aborted. Please check its log. INFO 2022-02-13 13:56:45,888 launch_utils.py:341] terminate all the procs INFO 2022-02-13 13:56:45,888 launch.py:311] Local processes completed. 推测错误是系统缺少nccl2导致的. ...