perf(concat): block-copy concatenation instead of SeqRecord round-trip (89x faster, -2.6 GB)

[sanjaynagi-eit]

Problem

concatenate_single_fastq parses both input FASTQs into a python list of BioPython SeqRecords and then writes them out again:

records = []
records.extend(SeqIO.parse(handle, "fastq"))   # x2
SeqIO.write(records, handle, "fastq")

SeqRecord overhead is roughly 5-10x the file size, so concatenating the short reads of a hybrid run can transiently need tens of GB. concatenate_single_fasta does the same for contigs, and Plass.get_depth_long / Assembly.combine_input_fastas each carry their own copy of the list(SeqIO.parse(...)) habit.

Concatenating records requires no parsing at all.

Change

Copy the inputs in 1 MiB blocks, decompressing gzipped sources on the way through. Two details worth calling out:

• Newline safety: a source whose final line is unterminated would otherwise run straight into the next file's first header. _append_file appends a newline when the copied data does not end with one.
• Format validation: parsing every record used to catch a wrong-format input, and test_concat_single_fastq_bad depends on that. The first byte is now checked against the format's record marker (@ / >) — same guarantee, O(1) instead of O(file).

Plass.get_depth_long now calls concatenate_single_fasta instead of building a list of records. Assembly.combine_input_fastas genuinely has to parse (it renames contigs) but streams them rather than materialising the assembly as a list.

Measurements

573 MiB of ONT FASTQ, each implementation in a fresh process:

	before	after
time	28.44 s	0.32 s	89x
peak RSS	2672 MB	88 MB	-2.6 GB

Output is byte-identical, and the parsed (id, sequence, qualities) records match exactly.

One thing deliberately left alone

Assembly.combine_input_fastas contains:

for record in records:
    ...
    record.id = str(i)
    records[0].description = ""   # <- always index 0, inside the loop

That clears only the first plasmid's description while every later one keeps its own — almost certainly a copy-paste slip. It matters, because get_contig_circularity() decides circularity by looking for "circular" in the description. Fixing it would change results, so the behaviour is preserved exactly and flagged with a comment for a separate PR.

Tests

New tests/test_concat.py: record preservation for FASTQ and FASTA, gzipped input, empty input, the missing-trailing-newline case, wrong-format rejection for both directions, and that FASTA descriptions survive (circularity depends on them). All existing tests pass (76 non-slow).

[sanjaynagi-eit]

hey @gbouras13, hope you are well.

During my (LLM) exploration of the plassembler source, this was another finding I wondered if it would be worth scoping further and incorporating?

p.s You may notice there are other PRs in this fork, but I dont think they are necessarily worthwhile.

[gbouras13]

Sounds good and thanks for this - can you make a PR? I'll add some changes on top of it