Openpose with Singularity not working

Thank a lot!

I am having issues with jobs failing after 3 minutes so I might be doing something wrong. I used the following:

DOCKER_OPENPOSE_IMG_PATH="${HOME}/openpose_orchestrator/openpose.simg"
DOCKER_OPENPOSE_IMG_WORKING_DIRECTORY='/openpose-master'
DOCKER_OPENPOSE_IMG_OPENPOSE_BIN='build/examples/openpose/openpose.bin'
CMD="\
apptainer run --nv --pwd ${DOCKER_OPENPOSE_IMG_WORKING_DIRECTORY} ${DOCKER_OPENPOSE_IMG_PATH} ${DOCKER_OPENPOSE_IMG_OPENPOSE_BIN} \
--no_display \
--net_resolution '-1x368' \
--scale_number 4 \
--scale_gap 0.25 \
--video $INPUT_DIR/$FILENAME \
--disable_blending \
--write_coco_json $OUTPUT_FOLDER/json/keypoints_coco.json \
--write_video $OUTPUT_FOLDER/Black_Video.avi"

Would it be possible to have a small debug session? I am highly grateful!!!

Nada

Error:

Cuda check failed (35 vs. 0): CUDA driver version is insufficient for CUDA runtime version

Coming from:

- src/openpose/utilities/cuda.cpp:getGpuNumber():47
- src/openpose/utilities/cuda.cpp:cudaCheck():36
- src/openpose/utilities/cuda.cpp:getGpuNumber():57
- ./include/openpose/wrapper/wrapper.hpp:configure():947
terminate called after throwing an instance of 'std::runtime_error'
  what():  
Error:
Cuda check failed (35 vs. 0): CUDA driver version is insufficient for CUDA runtime version

Coming from:
- src/openpose/utilities/cuda.cpp:getGpuNumber():47
- src/openpose/utilities/cuda.cpp:cudaCheck():36
- src/openpose/utilities/cuda.cpp:getGpuNumber():57
- ./include/openpose/wrapper/wrapper.hpp:configure():947
/var/spool/slurmd/job5483636/slurm_script: line 55: 154633 Aborted                 (core dumped) apptainer run --pwd /openpose-master /home/kojovic/openpose_orchestrator/openpose.simg build/examples/openpose/openpose.bin --no_display --net_resolution '-1x368' --scale_number 4 --scale_gap 0.25 --video /home/kojovic/8081/8081_CS3_Video_3_10_26_2022_1_38_52_PM_3.mp4 --disable_blending --write_coco_json /home/kojovic/8081/OUTPUT/8081_CS3_Video_3_10_26_2022_1_38_52_PM_3.mp4/json/keypoints_coco.json --write_video /home/kojovic/8081/OUTPUT/8081_CS3_Video_3_10_26_2022_1_38_52_PM_3.mp4/Black_Video.avi

real	0m3.798s
user	0m0.557s
sys	0m0.237s

Dear @Nada.Kojovic please share your sbatch with us.

the code i run :
cd openpose_orchestrator/
python run_openpose_2023_BLACK_NK.py --input_videos /home/kojovic/FOLDER_NAME

Doing so, you are running your code in the login node instead of a compute node: this is forbidden.

You need to write an sbatch script and use a partition with GPUs and request a GPU.

sorry, i am indeed using sbatch inside this code the run_openpose* code calls this one:

#!/bin/bash

#SBATCH --output=%J-test_adf.log
#SBATCH --error=%J-test_adf.err
#SBATCH --partition=shared-gpu
#SBATCH --cpus-per-task=4
#SBATCH --time=12:00:00
#SBATCH --gres=gpu:2


### <https://hpc-community.unige.ch/t/tutorial-launch-openpose-with-gpu-support-through-singularity/593>
# module load GCCcore/8.2.0 Singularity/3.4.0-Go-1.12
echo "Starting Processing at `date`   "
cd INSERT_LOGS_DIRECTORY_HERE



FILENAME="INSERT_FILENAME_HERE"
INPUT_DIR="INSERT_INPUT_DIRECTORY_HERE"
OUTPUT_DIR="INSERT_OUTPUT_DIRECTORY_HERE"
OUTPUT_FOLDER=$OUTPUT_DIR/$FILENAME

echo $INPUT_DIR
echo $FILENAME

echo "Creating output folder ${OUTPUT_FOLDER}..."
rm -rf $OUTPUT_FOLDER
mkdir $OUTPUT_FOLDER
mkdir $OUTPUT_FOLDER/json

### ATTENTION, the original Docker image is available at
### <https://hub.docker.com/r/garyfeng/docker-openpose/>
###   NB, the "--pwd /openpose-master" option is mandatory since the
###       build/examples/openpose/openpose.bin is compiled with
###       relative-path dymanic libraries...
DOCKER_OPENPOSE_IMG_PATH="${HOME}/openpose_orchestrator/openpose.simg"
DOCKER_OPENPOSE_IMG_WORKING_DIRECTORY='/openpose-master'
DOCKER_OPENPOSE_IMG_OPENPOSE_BIN='build/examples/openpose/openpose.bin'
CMD="\
apptainer run --pwd ${DOCKER_OPENPOSE_IMG_WORKING_DIRECTORY} ${DOCKER_OPENPOSE_IMG_PATH} ${DOCKER_OPENPOSE_IMG_OPENPOSE_BIN} \
--no_display \
--net_resolution '-1x368' \
--scale_number 4 \
--scale_gap 0.25 \
--video $INPUT_DIR/$FILENAME \
--disable_blending \
--write_coco_json $OUTPUT_FOLDER/json/keypoints_coco.json \
--write_video $OUTPUT_FOLDER/Black_Video.avi"


echo "Running the following command:"
echo "$CMD"
echo
time \
 eval ${CMD}

echo "JOB Complete at `date`   "

# Convert result.avi to result.mp4
#ffmpeg -i $OUTPUT_FOLDER/result.avi  -strict 2 $OUTPUT_FOLDER/result.mp4
#rm -rf $OUTPUT_FOLDER/result.avi


exit

In your first example

you are using --nv and in your full example with the sbatch you aren’t using --nv. It is mandatory to use this option if you want to use a gpu. Please clarify if you use it or not.

Hi, I now included --nv flag, it runs, jobs are scheduled, and they seem to launch and finish, only outputs are incomplete. Pose estimation seems to stop after a minute or two. Here is one feedback I get:

Starting Processing at Wed Oct 18 11:17:44 CEST 2023   
/home/kojovic/8081
8081_PP3_Video_2_9_20_2022_3_41_09_PM_2.mp4
Creating output folder /home/kojovic/8081/OUTPUT/8081_PP3_Video_2_9_20_2022_3_41_09_PM_2.mp4...
Running the following command:
apptainer run --nv --pwd /openpose-master /home/kojovic/openpose_orchestrator/openpose.simg build/examples/openpose/openpose.bin --no_display --net_resolution '-1x368' --scale_number 4 --scale_gap 0.25 --video /home/kojovic/8081/8081_PP3_Video_2_9_20_2022_3_41_09_PM_2.mp4 --disable_blending --write_coco_json /home/kojovic/8081/OUTPUT/8081_PP3_Video_2_9_20_2022_3_41_09_PM_2.mp4/json/keypoints_coco.json --write_video /home/kojovic/8081/OUTPUT/8081_PP3_Video_2_9_20_2022_3_41_09_PM_2.mp4/Black_Video.avi

Starting pose estimation demo.
Auto-detecting GPUs... Detected 2 GPU(s), using them all.
Starting thread(s)
Empty frame detected, frame number 45979 of 46044. In src/openpose/producer/producer.cpp:checkFrameIntegrity():155
Input images must be 3-channel BGR. Converting grey image into BGR.
Empty frame detected, frame number 45979 of 46044. In src/openpose/producer/producer.cpp:checkFrameIntegrity():155
Input images must be 3-channel BGR. Converting grey image into BGR.
Empty frame detected, frame number 45979 of 46044. In src/openpose/producer/producer.cpp:checkFrameIntegrity():155
Input images must be 3-channel BGR. Converting grey image into BGR.
Real-time pose estimation demo successfully finished. Total time: 7698.607437 seconds.
JOB Complete at Wed Oct 18 13:26:11 CEST 2023

edited by admin: please use code formating (pre formated text icon) for your logs in the forum, it is easier to read.

Dear Nada,

It seems to run more than one minute if you check the start : “Starting Processing at Wed Oct 18 11:17:44 CEST 2023” and the end “JOB Complete at Wed Oct 18 13:26:11 CEST 2023”. Job have run approximatly 2 hours. I don’t have much informations regarding the way this software is working.

Best regards,