Segmentation fault with valle-inclan dataset - bwa-mem2 2.2.1 avx512bw

[waltergallegog]

Hello,
I'm attempting to run bwa-mem2 with the valle-inclan dataset, but I get a segmentation fault.
I'm using the precompiled version 2.2.1 avx512:

Looking to launch executable "/home/wgallego/.local/bin/bwa-mem2.avx512bw", simd = .avx512bw
Launching executable "/home/wgallego/.local/bin/bwa-mem2.avx512bw"
2.2.1

Running on Linux Debian 4.19:

Linux ulisse 4.19.0-22-cloud-amd64 #1 SMP Debian 4.19.260-1 (2022-09-29) x86_64 GNU/Linux

I have tried running with 15, 10 and 5 threads and I always get the segmentation fault.

• The segmentation fault happens always at the same place. For example for the 15 threads run, it always happens at task 90.
• All of the runs always generate a sam output of around 33GB.
• The issue should not be related to RAM because the system has 256GB, and the runs usually reach around 60GB of usage.
• It should not be related to disk space or file size either. I have also tried piping the output of bwa-mem2 to samtools in order to generate a smaller .bam output, but the segfault happens anyway.

I have generated a coredump. Here is the output of the backtrace command:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  FMI_search::getSMEMsOnePosOneThread (this=0x7f5c7f01b072, enc_qdb=0x62 <error: Cannot access memory at address 0x62>,
    query_pos_array=0xffffffeb4644fb40, min_intv_array=0x7f5a403dc800, rid_array=0xffffffff9176c53c, numReads=19,
    batch_size=32828, seq_=0x7f5c2a880030, query_cum_len_ar=0x7f5a403dc800, max_readlength=151, minSeedLen=19,
    matchArray=0x7f5c7fbb7480, __numTotalSmem=0x7f5c177fdd88) at src/FMI_search.cpp:520

warning: Source file is more recent than executable.
520             int readlength = seq_[rid].l_seq;
[Current thread is 1 (Thread 0x7f5c177fe700 (LWP 24758))]
(gdb) bt
#0  FMI_search::getSMEMsOnePosOneThread (this=0x7f5c7f01b072, enc_qdb=0x62 <error: Cannot access memory at address 0x62>,
    query_pos_array=0xffffffeb4644fb40, min_intv_array=0x7f5a403dc800, rid_array=0xffffffff9176c53c, numReads=19,
    batch_size=32828, seq_=0x7f5c2a880030, query_cum_len_ar=0x7f5a403dc800, max_readlength=151, minSeedLen=19,
    matchArray=0x7f5c7fbb7480, __numTotalSmem=0x7f5c177fdd88) at src/FMI_search.cpp:520
#1  0x00000000004285be in mem_collect_smem (fmi=<optimized out>, opt=<optimized out>, seq_=<optimized out>, nseq=<optimized out>,
    matchArray=<optimized out>, min_intv_ar=<optimized out>, query_pos_ar=<optimized out>, enc_qdb=<optimized out>,
    rid=<optimized out>, tot_smem=<optimized out>) at src/bwamem.cpp:686
#2  mem_kernel1_core (fmi=<optimized out>, opt=<optimized out>, seq_=<optimized out>, nseq=<optimized out>,
    chain_ar=<optimized out>, seedBuf=<optimized out>, seedBufSize=<optimized out>, mmc=<optimized out>, tid=<optimized out>)
    at src/bwamem.cpp:958
#3  worker_bwt (data=0x7f5c7f01b072, seq_id=98, batch_size=1178925888, tid=1077790720) at src/bwamem.cpp:1125
#4  0x000000000046b0e7 in ktf_worker (data=0x7f5c7f01b072) at src/kthread.cpp:69
#5  0x00007f61e92b8fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f61e8ea906f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

You can download the short reads from ENA:
https://www.ebi.ac.uk/ena/browser/view/ERR2752450

The segmentation fault happens only when aligning ERR2752450_2.fastq.gz. (I have verified the md5sum of my local version).

I'm using as reference refdata-b37-2.1.0/fasta/genome.fa that you can download from:
http://cf.10xgenomics.com/supp/genome/refdata-b37-2.1.0.tar.gz

The commands I'm using:

bwa-mem2 index genome.fa
bwa-mem2 mem -t 15 genome.fa ERR2752450_2.fastq.gz > output.sam 2 > output.log

I hope you can reproduce the issue. If not I can try to share the coredump (although it is large, around 27 GB).

Attached you can find the log files of the runs. For the reference, I have used a slightly modified version that has some insertions.
15_threads_genome_ins.txt
10_threads_genome_ins.txt
5_threads_genome_ins.txt

To make sure the issue is not related to the modification of the reference I had tried aligning to the original one, but the segmentation fault is the same.
15_threads_genome_orig.txt

[waltergallegog]

I have tested using paired end mode and there is no segmentation fault in this case:

bwa-mem2 mem -t 15 genome.fa ERR2752450_1.fastq.gz ERR2752450_2.fastq.gz 2>output.log | samtools view -bS -o output.bam