Repository navigation
Falco 2.0.3 crashes with ≥72 threads and strange --mem behavior #133
Description
Hello!
I am testing Falco 2.0.3 on BAM files and found a reproducible crash related to the number of threads.
On my machine, falco --help reports:
-t, --threads [1] Threads for analysis (this machine supports: 112)
-m, --mem [256MiB] Input memory buffer size (G/M/K units ok)
However, with the same BAM input, Falco works with -t 71 but consistently crashes starting from -t 72. This error isn't related to the number of BAM files, I've tried using from 1 to 3. Also at -t 72, the error is consistently:
Segmentation fault (core dumped)
At higher thread counts, for example -t 99, the same input can fail with different errors in different runs, including:
munmap_chunk(): invalid pointer
free(): invalid size
double free or corruption (out)
Segmentation fault (core dumped)
Aborted (core dumped)
I also tried changing the input memory buffer size with --mem, but noticed that the verbose output always reports:
input memory buffer size: 1MiB
For example, both -m 64M/-m 1G still report 1MiB. Also, -m 256MiB is rejected with:
--mem: mib unit not recognized. Allowed values: {g,k,m}
although the help message shows the default as 256MiB.
Thank you for your time!
Activity
andrewdavidsmith commentedon Sep 30, 2026
@Alexandra2022kzn I'll check it out. Can you provide any info about file sizes and a rough sense of read length?
Edit. And now that I think about it (short on time for a couple hours) if you can try to run it with varying the arguments that allow a
--nolike--no-tiles, etc., it would help identify the problem.andrewdavidsmith commentedon Sep 30, 2026
Also, if the change is a difference between 2.0.2 and 2.0.3 that should be easy to pinpoint
andrewdavidsmith commentedon Sep 30, 2026
@Alexandra2022kzn I can't reproduce the crashes on high numbers of threads. I suspect your data files have some characteristic that I'm not thinking about at all. I did find several ways to cause crashes (e.g., make incorrect quality scores) but the issues I've been able to create aren't related to numbers of threads used. I will need more info.
Alexandra2022kzn commentedon Oct 5, 2026
Sorry for the late reply — I was away from my computer. I used small test files, so maybe the problem is related to their size, but I am not sure why.
Here is the log:
We used BAM files generated from the FASTQ files available here: https://github.com/hartwigmedical/testdata/tree/master/100k_reads_hiseq/TESTX. I can upload the BAM files here as well if needed.
andrewdavidsmith commentedon Oct 7, 2026
@Alexandra2022kzn It would help for you to upload the bams or you can email them to me (they seem small). I downloaded those fastq files and converted them to bam using samtools, but the resulting bams are running fine for me. Based on the filesizes, you might have also mapped the fastq files, and I don't know what mapping tool, parameters, etc., to use.
Edit: The main branch now has better error reporting within threads so you might get more reliable error reporting if you clone and build.
andrewdavidsmith commentedon Oct 13, 2026
I've been able to reproduce the crash at exactly 72 threads using the specified data for v2.0.3. Between v2.0.3 and 54af0b2 the problem seems to have been fixed, but the precise change that fixed the problem hasn't been identified.
Alexandra2022kzn commentedon Oct 28, 2026
Hello, @andrewdavidsmith! Sorry for the long reply
Yes, I can now run Falco on the three files we discussed above. However, I tried running it on a small test FASTQ file and got the following error. However, on the larger BAM file, everything worked fine.
falco -o /outputs -t 1 --verbose /input/TESTX_H7YRLADXX_S1_L001_R1_001.fastq.gz Falco v2.0.3 Run started: 2026-10-28 10:14:19 MSK Resources threads requested: 1 input memory buffer size: 1MB max analyzable read length: 160Kbp Analyses adapter content: ON sequence duplication: ON GC content: ON k-mer content: OFF sequence length: ON N content: ON overrep sequences: ON per-base quality: ON per-sequence quality: ON sequence composition: ON per-tile quality: ON Input files TESTX_H7YRLADXX_S1_L001_R1_001.fastq.gz FASTQ BGZF-compressed sequence data 1.8M Floating point exception (core dumped)I checked the
--memflag. The memory is set correctly in the BAM file, but in the example above with FASTQ, for some reason, it's still set to 1 MB.And one more question: is it possible to specify something like a seed for calculations in
Per tile sequence qualityandSequence Duplication Levelsso that they are reproducible between different runs?andrewdavidsmith commentedon Oct 28, 2026
@Alexandra2022kzn thanks. I think there's probably a division by zero somewhere due to some count not rising above zero, likely because of such small data. I should be able to find it. The
--memis misleading, and I can fix it. The value you are seeing is shrunk to be at most the size of the file (but apparently the floor in terms of units). There would not be a seed. The non-determinism results from the use of multiple cores in a non-deterministic order by the system. I'm pretty sure if you try it with one core it will be identical. Thanks again. I really appreciate this input. Stay tuned.andrewdavidsmith commentedon Oct 28, 2026
Ok, the
Floating point exceptionhas been fixed (change related to bgzf fastq input). Interestingly, I could only reproduce the problem in exactly your file. If I do gunzip on the file, the bgzip on it again, or gzip it again, it was working for me. So I'm very worried I have a bind-spot somewhere, and if so, there could be more problems I'm just not seeing... I really appreciate the help you've given.The message related to the
--memis as I intended: the reserved buffer can't be larger than the file. But it was rounding down in your case. Now that's fixed, but if you have a suggestion for a different message, I'll consider it. I'm trying to keep it concise, but it's obviously a bit confusing. That wasn't intended to reflect back the arg you provide, but rather the memory reserved for reading files.Anyway, I merged the changes into main and it's in the most recent binaries made from main the 'actions' list.