v2.0.3: Overrepresented sequences and Adapter Content are graded PASS when FastQC's cutoffs give FAIL #145

Hello @andrewdavidsmith, y'all – and thanks for Falco!

To speed things up, we are keen to switch our nf-core/deepmutscan pipeline from FastQC to Falco. For this, I just compared the grades of FastQC 0.13.0, Falco 1.2.5 and Falco 2.0.3 (Bioconda) on the same files. The numbers in fastqc_data.txt agree. Two grades do not, however, and to me it looks like the reason lies in a mix of fractions versus percentages in the grading code.

Example. One public file of 50 000 amplicon reads: GID1A_input1_50k_1.fastq.gz from nf-core/test-datasets. Falco 2.0.3's own report, falco --groups -o out GID1A_input1_50k_1.fastq.gz, lists the top overrepresented sequence at 3.98 % of reads, and the Illumina Universal Adapter reaches 35.2 % at the read end. Both values are above FastQC's error cutoffs (1 % and 10 %).

Cause, as far as I understand it by glimpsing at the source code with Claude (line numbers at v2.0.3).

• Overrepresented: get_grade_overrepresented() (src/duplication_results.cpp:157–165) grades as_frac(max_n_obs, n_reads), a fraction. The default cutoffs warn = 0.10, fail = 1.00 (src/falco_grade.cpp:31) are FastQC's percentages, so in effect the cutoffs are 10 % and 100 %.
• Adapter: adapter_matcher::get_grade() (src/adapter_matcher.cpp:56–65) grades the largest count of reads in which an adapter starts at a single position. The report, like FastQC, shows the cumulative percentage (adapter at or before each position) and FastQC grades the maximum of that. In our file, adapter starts are spread over many positions: the most at any one position is 0.82 % of reads, while the cumulative value reaches 35 %.
• N content and duplication: get_grade_n_content() (src/falco_grade.cpp:200) and get_grade_duplication() (src/duplication_results.cpp:286) grade fractions against fractional defaults (0.05/0.20, 0.70/0.50). That works without a config file. But load_config_and_set_graders() (src/falco_config.cpp:109) takes the values from a limits file as written, in percent (5/20, 70/50), so N content can no longer warn and duplication always fails. The same applies to adapter (5/10) and overrepresented (0.1/1) when a config is given.

Possible fix (also via Claude). Grade all of these modules consistently in percent, as every other module; grade adapter content on the cumulative percentage; grade overrepresented sequences on the reported list (the top listed sequence, or zero if none passes the list's cutoffs, as FastQC does); also let a value equal to a cutoff pass (edge cases as FastQC). I have made a small patch for this (four files, +24/−18) against main, with the one expected checksum in data/test_data/md5sum.txt updated. All 17 tests pass, and the grades then match FastQC 0.13.0 on all four modules. I've opened PR #146 with it.

Best wishes,
Max Stammnitz