Add .tbl NCBI feature table output for GenBank submission - #139
Open
gbouras13 wants to merge 3 commits into
Open
Add .tbl NCBI feature table output for GenBank submission#139gbouras13 wants to merge 3 commits into
gbouras13 wants to merge 3 commits into
Conversation
Closes #137. Phold already writes a GenBank file, but submitting to GenBank via table2asn wants an NCBI feature table. Users were converting the .gbk by hand. This adds <prefix>.tbl alongside the existing outputs, matching the .tbl Pharokka emits (pharokka.post_processing.create_tbl) so the two tools stay interchangeable in a submission pipeline. Format notes: * Coordinates are 1-based inclusive and written in transcription order -- minus-strand features are written high-coordinate first, which is how a feature table encodes strand (there is no strand column). * per_cds_df is the source rather than merged_df: write_genbank() has already normalised its coordinates for the genbank 0-index issue (#75/#77), and merged_df's left-joins can introduce null-padded rows for CDS with no Foldseek hit. * tRNA / tmRNA / CRISPR features are passed through from the input GenBank. Phold does not predict these, but a feature table that omitted them would describe fewer features than the .gbk written beside it. * Pharokka stores the tRNA name in a /trna qualifier, which is not valid NCBI and which table2asn requires as /product. write_genbank renames it in place; write_tbl does the same mapping itself rather than depending on call order. * Pipeline bookkeeping qualifiers (ID, locus_tag, score, source, isotype, ...) are blocklisted so they never reach a submission file, and null-valued qualifiers are skipped rather than written as bare lines, which table2asn treats as a parse error. * Every contig gets a >Feature header even with no features -- table2asn matches records by that header, so an omitted contig reads as absent rather than empty. Skipped for proteins-predict / proteins-compare, which have no contig coordinates to describe. Adds tests/unit/test_tbl.py (24 tests) pinning the exact line shape: tab counts, coordinate order on both strands, BioPython half-open to 1-based conversion, qualifier filtering and ordering, the /trna alias, multi-contig grouping, and that the writer does not mutate the SeqFeatures it shares with the GenBank writer. Verified end to end against tests/test_data/SAOMS1.gbk: 246 CDS + 3 tRNAs, no malformed lines.
Follow-up to the .tbl writer (#137). CDS that run off the edge of a contig now carry NCBI's incomplete-end markers, so a genome with truncated contigs produces a feature table table2asn will accept. Plumbing: * get_fasta_run_pyrodigal_gv() now records a Prodigal-style partial qualifier ("<left><right>") from gene.partial_begin/partial_end. Pharokka already writes the same qualifier on its GenBank CDS, so both input routes now reach the writer with one representation. * write_genbank() surfaces it as a per_cds_df column, with the same list-vs-bare-string handling as transl_table, and pins it to Utf8 so an input with no partial qualifiers cannot infer the column as Null. * The column is dropped at the merged_df join, so neither _per_cds_predictions.tsv nor any sub_db_tophits TSV gains a column. * Contig lengths are passed to write_tbl to place a minus-strand 5'-partial CDS against the contig edge. Two bugs in the initial writer, both found by driving the real pipeline rather than a hand-built dataframe: * Strand reaches write_tbl as "+"/"-" strings, because write_genbank converts them before returning per_cds_df. The writer was doing int(strand) == -1, which raises ValueError on "-". It now accepts both representations. * per_cds_df is ALREADY in transcription order for CDS -- write_genbank assigns start = location.end for a minus-strand CDS, so start > end there. The writer was re-orienting that pair and writing every reverse-strand CDS backwards. CDS coordinates are now passed through untouched; the non-CDS path still orients, because those come from BioPython locations which really are genomic min/max. On orientation: Prodigal's partial flag is two digits in genomic order, left edge then right edge, on both strands -- so the 5' end is the left digit on the plus strand and the right digit on the minus strand. Verified against pyrodigal's own GFF writer: reverse-complementing a sequence turns a left-edge "10" gene into a right-edge "01" gene. Unlike Pharokka, a CDS partial at both ends ("11") gets both markers rather than falling through the if/elif chain unmarked. An out-of-range codon_start warns instead of aborting -- this runs at the very end of a long pipeline. Tests: 43 in test_tbl.py, covering both strands, all four partial flags plus "11", codon_start on each strand, malformed flags, and a mirror-image case asserting a plus-strand partial and its reverse complement produce mirrored output. Verified end to end on both input routes: 246 CDS (78 reverse-strand) from tests/test_data/SAOMS1.gbk and a truncated NC_043029 through pyrodigal-gv, with zero misoriented coordinate lines and forward/revcomp runs producing exact mirrors.
Validated the generated feature tables with table2asn 1.28.1179 (bioconda,
osx-64). Three problems showed up, all of which would have reached anyone
trying to submit:
* /inference was written as the raw annotation_method ("foldseek",
"pharokka", "none"). INSDC requires a category from a controlled
vocabulary, so table2asn rejected EVERY CDS with "Qualifier had bad value"
-- 246 hard errors on the SAOMS1 test genome, 15/15 on a pyrodigal run.
This was inherited from Pharokka, which writes its GFF Method column here
and has the same problem.
Category chosen by measurement, not guesswork:
similar to AA sequence -> InvalidInferenceValue warning
similar to AA sequence:PHROG:1215 -> InvalidInferenceValue warning
("unrecognized database")
protein motif:PHROG:1215 -> clean
alignment:Foldseek:1215 -> clean
The "similar to ... sequence" categories are checked against NCBI's list
of recognised databases, which PHROG is not on. "protein motif" is not
checked, and is also the honest description -- a PHROG is a protein group
/ profile, and Phold transfers annotation from the group its structure
matched. CDS with no PHROG to cite get no /inference at all rather than a
bare category (which is itself flagged) or an invented accession.
* /anticodon was emitted as a bare 3-letter code for Pharokka < v1.8.2
input, which table2asn rejects -- NCBI wants a location,
"(pos:115194..115196,aa:Met)". write_genbank already warns about that
input, so the qualifier is now dropped with a warning rather than written
knowing it is invalid. The valid location form is passed through.
* 3'-partial CDS stopped at the last whole codon, leaving 1-2 trailing
bases, and table2asn warned PartialProblemNotSpliceConsensus3Prime ("3'
partial is not at end of sequence"). They are now extended to the contig
edge, mirroring the 5' codon_start handling.
After the fixes, the only findings left on all three test cases are the
submission metadata table2asn always requires (NoPubFound, NoTaxonID,
MissingPubRequirement -- supplied by the submitter's template.sbt and
organism), plus BadCDScomponentOverlapTRNA on SAOMS1, which is genuine in
that input: CDS_0193 (115129..115329) spans the tRNA at 115194..115265, and
Pharokka's own .tbl would report it identically. Zero feature or qualifier
findings attributable to the writer.
Test cases: SAOMS1.gbk via the Pharokka GenBank route (246 CDS, 78 reverse
strand, 3 tRNAs) and a truncated NC_043029 through pyrodigal-gv in both
orientations (15 CDS each, 2 partial), which is what exercises the
incomplete-end markers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #137.
Adds
<prefix>.tbl, an NCBI feature table, alongside the existing outputs ofphold runandphold compare, so a Phold-reannotated genome can go totable2asnwithout a hand conversion. Modelled on the.tblPharokka emits (pharokka.post_processing.create_tbl) so the two tools stay interchangeable in a submission pipeline.Not produced for
proteins-predict/proteins-compare, which have no contig coordinates.What's in it
.gbkwritten beside it.</>, withcodon_startwhere the reading frame requires it.partialis plumbed through from both input routes — newly recorded fromgene.partial_begin/partial_endon the pyrodigal path, and read from Pharokka's existing/partialqualifier on the GenBank path. The column is dropped at themerged_dfjoin, so no existing TSV gains a column.Validated against table2asn
Checked with table2asn 1.28.1179 (bioconda, osx-64). Three problems showed up, all of which would have hit anyone actually submitting:
/inferencewas rejected on every single CDS. It was being written as the rawannotation_method(foldseek,pharokka,none). INSDC requires a category from a controlled vocabulary, so table2asn returnedQualifier had bad value— 246 hard errors on SAOMS1, 15/15 on a pyrodigal run. Pharokka has the same problem, since it writes its GFF Method column here.Category picked by measurement rather than guesswork:
/inferencevaluesimilar to AA sequenceInvalidInferenceValuewarningsimilar to AA sequence:PHROG:1215protein motif:PHROG:1215alignment:Foldseek:1215The
similar to ... sequencecategories are checked against NCBI's recognised-database list, which PHROG isn't on.protein motifisn't checked, and is also the honest description: a PHROG is a protein group / profile, and Phold transfers annotation from the group its structure matched. This is the one judgement call worth your review — it's a single constant (_INFERENCE_CATEGORY) if you'd rather claim something else. CDS with no PHROG to cite get no/inferenceat all, rather than a bare category (itself flagged) or an invented accession./anticodonwas rejected for Pharokka < v1.8.2 input, which writes a bare 3-letter code where NCBI wants a location ((pos:115194..115196,aa:Met)).write_genbankalready warns about that input; the qualifier is now dropped with a warning instead of written knowing it's invalid. The valid location form passes through untouched.3'-partial CDS warned
PartialProblemNotSpliceConsensus3Prime. Gene callers stop at the last whole codon, leaving 1–2 trailing bases; these are now extended to the contig edge, mirroring the 5'codon_starthandling.After the fixes the only findings left are the submission metadata table2asn always wants (
NoPubFound,NoTaxonID,MissingPubRequirement— supplied by the submitter'stemplate.sbtand organism), plusBadCDScomponentOverlapTRNAon SAOMS1, which is genuine in that input: CDS_0193 (115129..115329) spans the tRNA at 115194..115265, and Pharokka's.tblwould report it identically. Zero feature or qualifier findings attributable to the writer.Two bugs caught mid-review
Both were in my own first pass, and both surfaced only once I drove the real
write_genbankinstead of a hand-built dataframe:"+"/"-"strings —write_genbankconverts before returningper_cds_df.int(strand) == -1would have raisedValueErroron every real run, at the final write step.per_cds_dfis already in transcription order for CDS (start = location.endfor minus strand, sostart > end). Re-orienting that pair wrote every reverse-strand CDS backwards — no crash, just a silently wrong submission file, 78 of 246 CDS on SAOMS1.Departures from Pharokka
"11") gets both markers, instead of falling through Pharokka'sif/elifchain unmarked.codon_startwarns instead of aborting — this runs at the very end of a long pipeline.Testing
57 unit tests in
tests/unit/test_tbl.pypinning the exact line shape: tab counts, coordinate order on both strands, BioPython half-open → 1-based conversion, all four partial flags plus"11",codon_starton each strand,/inferenceand/anticodonrules, qualifier filtering and ordering, multi-contig grouping, and that the writer doesn't mutate the SeqFeatures it shares with the GenBank writer. One test asserts a plus-strand partial and its reverse complement produce mirrored output.Full suite: 85 passed, 18 skipped. Ruff clean on new files; unchanged counts on the files touched.
End-to-end on both input routes: SAOMS1.gbk (246 CDS, 78 reverse strand, 3 tRNAs) and a truncated NC_043029 through pyrodigal-gv in both orientations — zero misoriented coordinate lines, forward/revcomp producing exact mirrors.
docs/output.mdupdated.