Cuda check failed (35 vs. 0): CUDA driver version is insufficient for CUDA runtime version
Coming from:
- src/openpose/utilities/cuda.cpp:getGpuNumber():47
- src/openpose/utilities/cuda.cpp:cudaCheck():36
- src/openpose/utilities/cuda.cpp:getGpuNumber():57
- ./include/openpose/wrapper/wrapper.hpp:configure():947
terminate called after throwing an instance of 'std::runtime_error'
what():
Error:
Cuda check failed (35 vs. 0): CUDA driver version is insufficient for CUDA runtime version
Coming from:
- src/openpose/utilities/cuda.cpp:getGpuNumber():47
- src/openpose/utilities/cuda.cpp:cudaCheck():36
- src/openpose/utilities/cuda.cpp:getGpuNumber():57
- ./include/openpose/wrapper/wrapper.hpp:configure():947
/var/spool/slurmd/job5483636/slurm_script: line 55: 154633 Aborted (core dumped) apptainer run --pwd /openpose-master /home/kojovic/openpose_orchestrator/openpose.simg build/examples/openpose/openpose.bin --no_display --net_resolution '-1x368' --scale_number 4 --scale_gap 0.25 --video /home/kojovic/8081/8081_CS3_Video_3_10_26_2022_1_38_52_PM_3.mp4 --disable_blending --write_coco_json /home/kojovic/8081/OUTPUT/8081_CS3_Video_3_10_26_2022_1_38_52_PM_3.mp4/json/keypoints_coco.json --write_video /home/kojovic/8081/OUTPUT/8081_CS3_Video_3_10_26_2022_1_38_52_PM_3.mp4/Black_Video.avi
real 0m3.798s
user 0m0.557s
sys 0m0.237s
you are using --nv and in your full example with the sbatch you aren’t using --nv. It is mandatory to use this option if you want to use a gpu. Please clarify if you use it or not.
Hi, I now included --nv flag, it runs, jobs are scheduled, and they seem to launch and finish, only outputs are incomplete. Pose estimation seems to stop after a minute or two. Here is one feedback I get:
Starting Processing at Wed Oct 18 11:17:44 CEST 2023
/home/kojovic/8081
8081_PP3_Video_2_9_20_2022_3_41_09_PM_2.mp4
Creating output folder /home/kojovic/8081/OUTPUT/8081_PP3_Video_2_9_20_2022_3_41_09_PM_2.mp4...
Running the following command:
apptainer run --nv --pwd /openpose-master /home/kojovic/openpose_orchestrator/openpose.simg build/examples/openpose/openpose.bin --no_display --net_resolution '-1x368' --scale_number 4 --scale_gap 0.25 --video /home/kojovic/8081/8081_PP3_Video_2_9_20_2022_3_41_09_PM_2.mp4 --disable_blending --write_coco_json /home/kojovic/8081/OUTPUT/8081_PP3_Video_2_9_20_2022_3_41_09_PM_2.mp4/json/keypoints_coco.json --write_video /home/kojovic/8081/OUTPUT/8081_PP3_Video_2_9_20_2022_3_41_09_PM_2.mp4/Black_Video.avi
Starting pose estimation demo.
Auto-detecting GPUs... Detected 2 GPU(s), using them all.
Starting thread(s)
Empty frame detected, frame number 45979 of 46044. In src/openpose/producer/producer.cpp:checkFrameIntegrity():155
Input images must be 3-channel BGR. Converting grey image into BGR.
Empty frame detected, frame number 45979 of 46044. In src/openpose/producer/producer.cpp:checkFrameIntegrity():155
Input images must be 3-channel BGR. Converting grey image into BGR.
Empty frame detected, frame number 45979 of 46044. In src/openpose/producer/producer.cpp:checkFrameIntegrity():155
Input images must be 3-channel BGR. Converting grey image into BGR.
Real-time pose estimation demo successfully finished. Total time: 7698.607437 seconds.
JOB Complete at Wed Oct 18 13:26:11 CEST 2023
edited by admin: please use code formating (pre formated text icon) for your logs in the forum, it is easier to read.
It seems to run more than one minute if you check the start : “Starting Processing at Wed Oct 18 11:17:44 CEST 2023” and the end “JOB Complete at Wed Oct 18 13:26:11 CEST 2023”. Job have run approximatly 2 hours. I don’t have much informations regarding the way this software is working.