Problem with samtools and bowtie

Nathan.Evan · November 29, 2023, 9:19am

Primary informations

Username:evan9
Cluster: baobab

Description

I recently obtained my access and for some reasons things do not go for me as for my PI on his own session. When i try to run samtools or bowtie for my analysis, i get an error message telling me that some library file do not exist

Steps to Reproduce

/srv/beegfs/scratch/shares/brauns_lab/data/sequencing_data/nathan/BAM
type: sbatch makebam.sh

Expected Result

in BAM: a file called ***.raw.bam

Actual Result

I get an error message:

sbatch: error: Batch script is empty!
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = “fr_FR.UTF-8”
are supported and installed on your system.
perl: warning: Falling back to the standard locale (“C”).
/srv/beegfs/scratch/shares/brauns_lab/Tools/samtools-1.12/samtools: error while loading shared libraries: libbz2.so.1.0: cannot open shared object file: No such file or directory
(ERR): bowtie2-align died with signal 9 (KILL)
slurmstepd: error: Detected 1 oom_kill event in StepId=6021107.batch. Some of the step tasks have been OOM Killed.

I tried, as suggested by my PI, to use load module GCC/8.3.0 and SAMtools/1.10 and also for bowtie to see if it changes something.
I still get the same result

Thank you in advance

Gael.Rossignol · November 29, 2023, 9:42am

Dear Nathan,

It seems the bowtie2 software is not compiled correctly. But we propose a lot of precompiled bowtie software on the cluster

(baobab)-[root@login2 ~]$ ml spider bowtie

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Bowtie:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Description:
      Bowtie is an ultrafast, memory-efficient short read aligner. It aligns short DNA sequences (reads) to the human genome.

     Versions:
        Bowtie/1.1.2
        Bowtie/1.2.2
        Bowtie/1.3.0
        Bowtie/1.3.1

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "Bowtie" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider Bowtie/1.3.1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Bowtie2:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Description:
      Bowtie 2 is an ultrafast and memory-efficient tool for aligning sequencing reads to long reference sequences. It is particularly good at aligning reads of about 50 up to 100s or 1,000s of characters, and particularly good at
      aligning to relatively long (e.g. mammalian) genomes. Bowtie 2 indexes the genome with an FM Index to keep its memory footprint small: for the human genome, its memory footprint is typically around 3.2 GB. Bowtie 2 supports
      gapped, local, and paired-end alignment modes.

     Versions:
        Bowtie2/2.3.2
        Bowtie2/2.3.4.1
        Bowtie2/2.3.4.2
        Bowtie2/2.3.5.1
        Bowtie2/2.4.4
        Bowtie2/2.4.5
        Bowtie2/2.5.1

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "Bowtie2" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider Bowtie2/2.5.1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Could you please try again by using one of those, idealy the last one?

Best regards,

Nathan.Evan · November 29, 2023, 9:56am

I loaded bowtie2/2.5.1

now the erorr message is different:
sbatch: error: Batch script is empty!
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = “fr_FR.UTF-8”
are supported and installed on your system.
perl: warning: Falling back to the standard locale (“C”).
(ERR): bowtie2-align died with signal 9 (KILL)
[main_samview] fail to read the header from “-”.
slurmstepd: error: Detected 1 oom_kill event in StepId=6021708.batch. Some of the step tasks have been OOM Killed.

I can also add that when i run the command using only bash and not sbatch (with the sbatch in the script also commented) this seems to work but take too much time

Yann.Sagon · November 29, 2023, 1:34pm

Hi,
why are you using your own version of bowtie and samtools instead of the one we provide?
This is my recommendation:

(baobab)-[sagon@login2 ~]$ ml GCCcore/11.3.0 SAMtools/1.16.1 Bowtie2/2.4.5

Please share you sbatch.

But this is wrong: doing so, you are running your job on the login node instead of using a compute node and it is forbidden.

Nathan.Evan · November 29, 2023, 2:00pm

Oh sorry i didn’t know it was forbidden.

here is the code with the sbatch:

#!/bin/sh                                                                       

sbatch --open-mode=append --time=12:00:00 --mem=40G --partition=shared-cpu -n 1\
 -N 1 -c 8

/srv/beegfs/scratch/shares/brauns_lab/Tools/bowtie2-2.4.4/bowtie2 -p 8  --maxins 2000 -N 1 -x /srv/beegfs/scratch/shares/brauns_lab/data/\
indexes/Bowtie2/mm10/mm10 -1 /srv/beegfs/scratch/shares/brauns_lab/data/sequenc\
ing_data/nathan/CUTADAPT/6ESInh*_R1_001.fastq.gz -2 /srv/beegfs/scratch/shares/\
brauns_lab/data/sequencing_data/nathan/CUTADAPT/6ESInh*_R2_001.fastq.gz | /srv/\
beegfs/scratch/shares/brauns_lab/Tools/samtools-1.12/samtools view -b -h -o /sr\
v/beegfs/scratch/shares/brauns_lab/data/sequencing_data/nathan/BAM/6ESInh.raw.b\
am

I use the bowtie and samtools that way because it is how the pipeline of the lab is written (it is not my code)

admin edit: code formating

Gael.Rossignol · November 29, 2023, 3:54pm

Dear Nathan,

If you wish to use module Bowtie2 you have to correct the path of executable in your script :

(baobab)-[rossigng@login2 ~]$ ml GCC/12.2.0 Bowtie2/2.5.1
(baobab)-[rossigng@login2 ~]$ which bowtie2
/opt/ebsofts/Bowtie2/2.5.1-GCC-12.2.0/bin/bowtie2

So replace :

/srv/beegfs/scratch/shares/brauns_lab/Tools/bowtie2-2.4.4/bowtie2

By :

/opt/ebsofts/Bowtie2/2.5.1-GCC-12.2.0/bin/bowtie2

Or just :

bowtie2

Samtools is available in the cluster with the same process

(baobab)-[rossigng@login2 ~]$ ml spider samtools

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  SAMtools:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Description:
      SAM Tools provide various utilities for manipulating alignments in the SAM format, including sorting, merging, indexing and generating alignments in a per-position format.

     Versions:
        SAMtools/1.3
        SAMtools/1.5
        SAMtools/1.9
        SAMtools/1.10
        SAMtools/1.12
        SAMtools/1.13
        SAMtools/1.16.1

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "SAMtools" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider SAMtools/1.16.1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


ml GCCcore/11.3.0 SAMtools/1.16.1

Best regards,

Yann.Sagon · November 30, 2023, 8:20am

Dear @Nathan.Evan

Well… extra work for you:) At least you are aware that we provide those software and we keep them updated.

About your sbatch script:

Your are calling sbatch from inside your sbatch ? This is why you have the error about “empty sbatch” and none of the arguments are used.

The way to proceed is:

write an sbatch script my_sbatch.sh

#/bin/sh
#SBATCH --time=12:00:00
#SBATCH --mem=40G
#SBATCH --partition=shared-cpu
#SBATCH --cores-per-task=8

srun /srv/beegfs/scratch/shares/brauns_lab/Tools/bowtie2-2.4.4/bowtie2 <your arguments>

launch your sbatch: sbatch my_sbatch.sh

Yann.Sagon · November 30, 2023, 8:21am

Yes it is: hpc:best_practices [eResearch Doc]