feat: aligners & variant callers by adthrasher · Pull Request #282 · stjudecloud/workflows

adthrasher · 2025-12-03T19:23:37Z

Adds new WDL task implementations:

bwa-mem2
minimap2
~~hisat2~~
vg giraffe
DeepVariant / DeepSomatic
Strelka
Manta
Clair3
~~NGSEP~~

I also created a new evaluation directory under workflows (maybe it should be placed somewhere else) that has workflows for running a sample through multiple analyses in parallel for comparison purposes.

Before submitting this PR, please make sure:

You have added a few sentences describing the PR here.
The code passes all CI tests without any errors or warnings.
You have added tests (when appropriate).
You have added an entry in any relevant CHANGELOGs (when appropriate).
If you have made any changes to the scripts/ or docker/ directories, please ensure any image versions have been incremented accordingly!
You have updated the README or other documentation to account for these changes (when appropriate).

github-advanced-security · 2025-12-03T19:30:56Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

into feat/minimap2

a-frantz

phew 😮‍💨 a lot of code to go through. Still skimmed a good chunk of the code, but wanted to get another round of comments out. This asks a lot of questions again.

a-frantz · 2026-09-24T15:08:13Z

nothing off the shelf we could find for this one?

Biocontainers has a single old tag (4.0.1: https://quay.io/repository/biocontainers/ngsep?tab=tags) and the project, itself, just provides a Dockerfile (https://github.com/NGSEP/NGSEPcore/blob/master/Dockerfile).

I think this is another tool we can consider just dropping. I only implemented the SingleSampleVariantsDetector method, but it is an entire Java framework for next-gen sequencing data.

That seems fine to drop imo

a-frantz · 2026-09-24T15:11:06Z

+
+The following list is sorted alphabetically:
+
+## af-only-gnomad.hg38.chrY_chrM.vcf.gz


is this VCF suitable for general use? I don't see any reason in this description we can't plop it in the existing fictures/vcfs dir and use as a test in non-mutect2 tests (not suggesting you add it to existing tests, but if its in the repo a future PR might want to add it for an extra case)

It's gnomAD stripped to just the allele frequency field for hg38. I further stripped it down to chrY and chrM. In theory another tool could try to use it, but I think that would be a bad idea. It's a custom Broad reference file used for this specific pipeline

https://gatk.broadinstitute.org/hc/en-us/articles/360050722212-FAQ-for-Mutect2:
Finally, "af-only-gnomad.hg38.vcf" is a copy of the gnomAD VCF stripped of all unnecessary INFO fields.

a-frantz · 2026-09-24T15:50:35Z

+    }
+}
+
+task sort {


we don't already have this one?

It appears not. We have sort tasks in picard, sambamba, and mako.

Is there a reason you've added this instead of using one of those?

At this point I'd call the picard sort battle-tested, and given what we know of bioinformatics tools, I wouldn't be totally surprised if there's some niche edge case where samtools sort hogs resources or otherwise fails.

Not objected to adding the sort task for completeness, but I think it makes sense to use one of the other implementations in workflow calls. Picard for being reliable, sambamba or mako for performance. Basically I don't see any reason to use samtools sort over one of those.

a-frantz · 2026-09-25T15:16:11Z

+        output_format: {
+            description: "The output format for alignments",
+            options: [
+                "gam",
+                "gaf",
+                "json",
+                "tsv",
+                "SAM",
+                "BAM",
+                "CRAM",
+            ],
+        }
+        preset: {
+            description: "vg giraffe preset for alignment",
+            options: [
+                "chaining-sr",
+                "default",
+                "fast",
+                "hifi",
+                "r10",
+                "srold",
+            ],
+        }


these should be enums

a-frantz · 2026-09-25T15:19:22Z

+    }
+}
+
+task calmd {


I'm guessing there are other options not exposed here? Can we log that somewhere? Maybe a meta field acknowledging "minimum working implementation"?

a-frantz · 2026-09-25T15:21:56Z

+        bam: "Input BAM format file to sort"
+        sort_order: {
+            description: "Order by which to sort the input BAM",
+            choices: [


should we drop the choices convention when there's an enum?

To me, it depends on what sprocket doc does? If it in-lines the enum, then I'd favor removing choices, but if it doesn't, I think having the choices listed in the task doc is good. Though it means maintaining two lists.

sprocket dev doc --open will show you what it currently does (and any complaints, I am returning to work on the doc command soon so they can be addressed soon-ish)

It doesn't inline the variants, but it does link to the page with the enum definition

where the second column is a local link

a-frantz · 2026-09-25T15:22:57Z

+
+    requirements {
+        container: "google/deepvariant:1.10.0-gpu"
+        #container: "google/deepvariant:1.10.0"


Suggested change

#container: "google/deepvariant:1.10.0"

a-frantz · 2026-09-25T15:23:30Z

+        output_prefix: "Prefix for output VCF and gVCF files"
+        model_type: {
+            description: "Type of model to use for variant calling",
+            choices: [


same here about dropping the choices field

a-frantz · 2026-09-25T15:24:47Z

@@ -0,0 +1,83 @@
+version 1.3
+
+enum SortOrder[String] {


do these need doc comments, and should sprocket lint be checking for doc comments?

adthrasher · 2026-09-25T17:04:39Z

+    requirements {
+        container: "quay.io/biocontainers/vg:1.70.0--h9ee0642_0"
+        cpu: ncpu
+        memory: "120 GB"


Real world data without VCF files

index (1 attempt) #0 completed wall 54m00s queued 30.2s 4 cpu, 111.8 GiB peak 82.6 GiB, cpu 87m26s

I am also running with VCF files from 1000genomes.

In progress run with VCF files:

MEMLIMIT SWAPLIMIT 111.7 G 111.7 G MEMORY USAGE: MAX MEM: 98.9 Gbytes; AVG MEM: 18.5 Gbytes; MEM Efficiency: 88.54%

Note that our LSF submission ends up allocating less RAM than specified in the WDL.

adthrasher added 5 commits December 3, 2025 10:17

feat: minimap2 tool wrapper

f911ef4

refactor: handle optionally gzipped reference

fd2bca4

chore: fill in options

1ebe2a0

chore: add samtools to minimap2 image and convert to BAM

9533656

chore: lint

1b28075

adthrasher added 8 commits December 3, 2025 17:00

chore: add disk specification

0ac46db

feat: add bwa-mem2 task

5161669

chore: lint

af994e2

feat: add hisat2 task

00deed4

chore: change base image as other segfaults

4434666

feat: add vg indexing

b07d769

chore: localize fasta for indexing

cb47772

feat: add vg giraffe task

39ca901

adthrasher marked this pull request as draft December 9, 2025 16:12

adthrasher self-assigned this Dec 9, 2025

adthrasher added 10 commits December 9, 2025 14:23

chore: avoid writing intermediate SAM to disk

a921e17

Merge branch 'feat/minimap2' of https://github.com/stjudecloud/workflows

733368e

into feat/minimap2

chore: use database prefix

2dc478a

chore: bump resources for azure

1c832fc

chore: format+lint

bd84132

chore: remove memory oversubscribe

8abaf9c

feat: add strelka and manta wrappers

d42fbfb

feat: add clair3 wrapper

589190d

chore: fix invocation

efb1d50

feat: add NGSEP wrapper

25681c7

github-advanced-security AI found potential problems Dec 18, 2025

View reviewed changes

adthrasher added 3 commits December 18, 2025 17:27

feat: add deepsomatic and deepvariant wrappers with GPU support

1083326

chore: change hisat2 output to BAM

30d1159

chore: write to stdout instead of fifo

e58c2ed

adthrasher added 7 commits September 18, 2026 14:44

chore: update read_group data structure

0eb2b01

chore: use enum for clair3 model

5123b95

chore: lint and format

a077d74

chore: missing changes

a25ef84

chore: bump clair images and symlink

8f3961b

Merge branch 'feat/minimap2' of https://github.com/stjudecloud/workflows

48b8007

into feat/minimap2

Merge branch 'main' into feat/minimap2

d00557f

adthrasher requested review from DelaramR and a-frantz September 21, 2026 14:19

a-frantz reviewed Sep 24, 2026

View reviewed changes

Comment thread data_structures/read_group.wdl Outdated

a-frantz reviewed Sep 24, 2026

View reviewed changes

adthrasher added 12 commits September 24, 2026 17:03

chore: bubble None to the top-level

b014c06

chore: remove slow tags from manta tests

b53df5c

chore: fix samtools core calculations

7576ace

chore: add modify_memory_gb to bwamem2

e84c0a7

chore: apply feedback from PR

40094b6

chore: add maxRetries

46458f4

chore: grammar

a94874e

chore: apply changes from PR

a0a50bf

chore: apply changes from PR

1bfcec7

chore: remove unused NGSEP

3b1459e

chore: apply changes from PR

43f1e62

chore: remove NGSEP from eval workflow

f9f04ea

a-frantz reviewed Sep 25, 2026

View reviewed changes

adthrasher added 3 commits September 25, 2026 11:32

chore: apply changes from PR

857e2b2

chore: apply changes from PR

049e525

chore: apply changes from PR

1b8b75a

adthrasher commented Sep 25, 2026

View reviewed changes

adthrasher added 2 commits September 25, 2026 13:08

chore: apply changes from PR

abbce83

chore: address CI issues

7c0e337


		The following list is sorted alphabetically:

		## af-only-gnomad.hg38.chrY_chrM.vcf.gz

Conversation

adthrasher commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-advanced-security AI commented Dec 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

a-frantz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

adthrasher commented Dec 3, 2025 •

edited

Loading