perf(concat): block-copy concatenation instead of SeqRecord round-trip (89x faster, -2.6 GB)

[sanjaynagi-eit]

Hey @gbouras13

PR here which should hopefully make a worthwhile difference in terms of speed and memory. You should be able to edit the PR, but let me know if you want me to make any changes!

---

Claude-ish below:

Problem

concatenate_single_fastq parses both input FASTQs into a python list of BioPython SeqRecords and then writes them out again:

records = []
records.extend(SeqIO.parse(handle, "fastq"))   # x2
SeqIO.write(records, handle, "fastq")

SeqRecord overhead is roughly 5-10x the file size, so concatenating the short reads of a hybrid run can transiently need tens of GB. concatenate_single_fasta does the same for contigs, and Plass.get_depth_long / Assembly.combine_input_fastas each carry their own copy of the list(SeqIO.parse(...)) habit.

Concatenating records requires no parsing at all.

Change

Copy the inputs in 1 MiB blocks, decompressing gzipped sources on the way through. Two details worth calling out:

• Newline safety: a source whose final line is unterminated would otherwise run straight into the next file's first header. _append_file appends a newline when the copied data does not end with one.
• Format validation: parsing every record used to catch a wrong-format input, and test_concat_single_fastq_bad depends on that. The first byte is now checked against the format's record marker (@ / >) — same guarantee, O(1) instead of O(file).

Plass.get_depth_long now calls concatenate_single_fasta instead of building a list of records. Assembly.combine_input_fastas genuinely has to parse (it renames contigs) but streams them rather than materialising the assembly as a list.

Measurements

573 MiB of ONT FASTQ, each implementation in a fresh process:

	before	after
time	28.44 s	0.32 s	89x
peak RSS	2672 MB	88 MB	-2.6 GB

Output is byte-identical, and the parsed (id, sequence, qualities) records match exactly.

---

One thing deliberately left alone

Assembly.combine_input_fastas contains:

for record in records:
    ...
    record.id = str(i)
    records[0].description = ""   # <- always index 0, inside the loop

That clears only the first plasmid's description while every later one keeps its own — almost certainly a copy-paste slip. It matters, because get_contig_circularity() decides circularity by looking for "circular" in the description. Fixing it would change results, so the behaviour is preserved exactly and flagged with a comment for a separate PR.

[codecov]

Codecov Report

❌ Patch coverage is 95.83333% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 47.32%. Comparing base (779d922) to head (236a5f5).
⚠️ Report is 48 commits behind head on main.

Files with missing lines	Patch %	Lines
src/plassembler/utils/plass_class.py	87.50%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main      #90       +/-   ##
===========================================
- Coverage   78.37%   47.32%   -31.05%     
===========================================
  Files          20       20               
  Lines        2044     2132       +88     
  Branches      257      271       +14     
===========================================
- Hits         1602     1009      -593     
- Misses        340     1064      +724     
+ Partials      102       59       -43     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.