Repository navigation
Map the DP's block MAP_NORESERVE, so a banded alignment is not refused for the whole matrix - #15
Merged
Merged
Conversation
…d for the whole matrix The DP's block is sized for every row at full width, though a banded DP writes only the front of it. Linux, under its default overcommit policy, refuses an allocation larger than its memory and swap however little of it is touched, so pgrealign's banded alignments of long tandem-repeat alleles failed with "posix_memalign fail" on machines that could have run them: 62 kb against 70 kb reserves 128 GB and writes 5.5 GB. A block of 64 MB or more is now mapped with MAP_NORESERVE, which Linux does not count against memory and swap; a smaller one, or one on a platform without it, is allocated as before, and which one a block is follows from its size, so the struct and every other file are unchanged. The rows written, and so the alignments and the memory they take, are the same: byte-identical output and peak memory on 24 pgrealign alignments of 5 to 49 kb in every mode it uses (unbanded, banded, seeded, several sequences), and on cactus's evolver primates with abPOA in BAR (MAF and halStats byte-identical, 103 s both). tests/run_all.sh passes. The 62 kb pair now runs in 4 s and 5.5 GB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The DP's block is sized for every row at full width, though a banded DP writes only the front of it. Linux, under its default overcommit policy, refuses an allocation larger than its memory and swap however little of it is touched, so pgrealign's banded alignments of long tandem-repeat alleles failed with
posix_memalign failon machines that could have run them: 62 kb against 70 kb reserves 128 GB and writes 5.5 GB.A block of 64 MB or more is now mapped with
MAP_NORESERVE(abpoa_matrix_alloc,abpoa_matrix_freeinabpoa_simd.c), which Linux does not count against memory and swap; a smaller one, or one on a platform withoutMAP_NORESERVE, is allocated as before. Which one a block is follows from its size, so the struct and every other file are unchanged. A failed mapping exits as a failedposix_memaligndid, with[abpoa_matrix_alloc] mmap fail!and the size.tests/run_all.shpasses.avx2=1, as cactus builds it, it ran the 30-way HPRC rejoin (897,416 regions, up to a 1.74 Mb allele and a 69 GB region) and the 466-way chrX + chr15 rejoin on the cluster with no failure.🤖 Generated with Claude Code