Skip to content

Add .tbl NCBI feature table output for GenBank submission - #139

Open
gbouras13 wants to merge 3 commits into
mainfrom
tbl-output
Open

Add .tbl NCBI feature table output for GenBank submission#139
gbouras13 wants to merge 3 commits into
mainfrom
tbl-output

Conversation

@gbouras13

Copy link
Copy Markdown
Owner

Closes #137.

Adds <prefix>.tbl, an NCBI feature table, alongside the existing outputs of phold run and phold compare, so a Phold-reannotated genome can go to table2asn without a hand conversion. Modelled on the .tbl Pharokka emits (pharokka.post_processing.create_tbl) so the two tools stay interchangeable in a submission pipeline.

Not produced for proteins-predict / proteins-compare, which have no contig coordinates.

What's in it

  • All CDS, plus the tRNA / tmRNA / CRISPR features carried over from the input GenBank. Phold doesn't predict those, but a feature table that omitted them would describe fewer features than the .gbk written beside it.
  • Incomplete CDS marked with < / >, with codon_start where the reading frame requires it. partial is plumbed through from both input routes — newly recorded from gene.partial_begin/partial_end on the pyrodigal path, and read from Pharokka's existing /partial qualifier on the GenBank path. The column is dropped at the merged_df join, so no existing TSV gains a column.
  • Contig lengths passed through so a minus-strand 5'-partial CDS can be placed against the contig edge.

Validated against table2asn

Checked with table2asn 1.28.1179 (bioconda, osx-64). Three problems showed up, all of which would have hit anyone actually submitting:

/inference was rejected on every single CDS. It was being written as the raw annotation_method (foldseek, pharokka, none). INSDC requires a category from a controlled vocabulary, so table2asn returned Qualifier had bad value — 246 hard errors on SAOMS1, 15/15 on a pyrodigal run. Pharokka has the same problem, since it writes its GFF Method column here.

Category picked by measurement rather than guesswork:

/inference value result
similar to AA sequence InvalidInferenceValue warning
similar to AA sequence:PHROG:1215 warning — "unrecognized database"
protein motif:PHROG:1215 clean
alignment:Foldseek:1215 clean

The similar to ... sequence categories are checked against NCBI's recognised-database list, which PHROG isn't on. protein motif isn't checked, and is also the honest description: a PHROG is a protein group / profile, and Phold transfers annotation from the group its structure matched. This is the one judgement call worth your review — it's a single constant (_INFERENCE_CATEGORY) if you'd rather claim something else. CDS with no PHROG to cite get no /inference at all, rather than a bare category (itself flagged) or an invented accession.

/anticodon was rejected for Pharokka < v1.8.2 input, which writes a bare 3-letter code where NCBI wants a location ((pos:115194..115196,aa:Met)). write_genbank already warns about that input; the qualifier is now dropped with a warning instead of written knowing it's invalid. The valid location form passes through untouched.

3'-partial CDS warned PartialProblemNotSpliceConsensus3Prime. Gene callers stop at the last whole codon, leaving 1–2 trailing bases; these are now extended to the contig edge, mirroring the 5' codon_start handling.

After the fixes the only findings left are the submission metadata table2asn always wants (NoPubFound, NoTaxonID, MissingPubRequirement — supplied by the submitter's template.sbt and organism), plus BadCDScomponentOverlapTRNA on SAOMS1, which is genuine in that input: CDS_0193 (115129..115329) spans the tRNA at 115194..115265, and Pharokka's .tbl would report it identically. Zero feature or qualifier findings attributable to the writer.

Two bugs caught mid-review

Both were in my own first pass, and both surfaced only once I drove the real write_genbank instead of a hand-built dataframe:

  1. Strand reaches the writer as "+"/"-" stringswrite_genbank converts before returning per_cds_df. int(strand) == -1 would have raised ValueError on every real run, at the final write step.
  2. per_cds_df is already in transcription order for CDS (start = location.end for minus strand, so start > end). Re-orienting that pair wrote every reverse-strand CDS backwards — no crash, just a silently wrong submission file, 78 of 246 CDS on SAOMS1.

Departures from Pharokka

  • A CDS partial at both ends ("11") gets both markers, instead of falling through Pharokka's if/elif chain unmarked.
  • An out-of-range codon_start warns instead of aborting — this runs at the very end of a long pipeline.

Testing

57 unit tests in tests/unit/test_tbl.py pinning the exact line shape: tab counts, coordinate order on both strands, BioPython half-open → 1-based conversion, all four partial flags plus "11", codon_start on each strand, /inference and /anticodon rules, qualifier filtering and ordering, multi-contig grouping, and that the writer doesn't mutate the SeqFeatures it shares with the GenBank writer. One test asserts a plus-strand partial and its reverse complement produce mirrored output.

Full suite: 85 passed, 18 skipped. Ruff clean on new files; unchanged counts on the files touched.

End-to-end on both input routes: SAOMS1.gbk (246 CDS, 78 reverse strand, 3 tRNAs) and a truncated NC_043029 through pyrodigal-gv in both orientations — zero misoriented coordinate lines, forward/revcomp producing exact mirrors.

docs/output.md updated.

Closes #137.

Phold already writes a GenBank file, but submitting to GenBank via table2asn
wants an NCBI feature table. Users were converting the .gbk by hand. This adds
<prefix>.tbl alongside the existing outputs, matching the .tbl Pharokka emits
(pharokka.post_processing.create_tbl) so the two tools stay interchangeable in
a submission pipeline.

Format notes:

* Coordinates are 1-based inclusive and written in transcription order --
  minus-strand features are written high-coordinate first, which is how a
  feature table encodes strand (there is no strand column).
* per_cds_df is the source rather than merged_df: write_genbank() has already
  normalised its coordinates for the genbank 0-index issue (#75/#77), and
  merged_df's left-joins can introduce null-padded rows for CDS with no
  Foldseek hit.
* tRNA / tmRNA / CRISPR features are passed through from the input GenBank.
  Phold does not predict these, but a feature table that omitted them would
  describe fewer features than the .gbk written beside it.
* Pharokka stores the tRNA name in a /trna qualifier, which is not valid NCBI
  and which table2asn requires as /product. write_genbank renames it in place;
  write_tbl does the same mapping itself rather than depending on call order.
* Pipeline bookkeeping qualifiers (ID, locus_tag, score, source, isotype, ...)
  are blocklisted so they never reach a submission file, and null-valued
  qualifiers are skipped rather than written as bare lines, which table2asn
  treats as a parse error.
* Every contig gets a >Feature header even with no features -- table2asn
  matches records by that header, so an omitted contig reads as absent rather
  than empty.

Skipped for proteins-predict / proteins-compare, which have no contig
coordinates to describe.

Adds tests/unit/test_tbl.py (24 tests) pinning the exact line shape: tab
counts, coordinate order on both strands, BioPython half-open to 1-based
conversion, qualifier filtering and ordering, the /trna alias, multi-contig
grouping, and that the writer does not mutate the SeqFeatures it shares with
the GenBank writer. Verified end to end against tests/test_data/SAOMS1.gbk:
246 CDS + 3 tRNAs, no malformed lines.
Follow-up to the .tbl writer (#137). CDS that run off the edge of a contig
now carry NCBI's incomplete-end markers, so a genome with truncated contigs
produces a feature table table2asn will accept.

Plumbing:

* get_fasta_run_pyrodigal_gv() now records a Prodigal-style partial
  qualifier ("<left><right>") from gene.partial_begin/partial_end. Pharokka
  already writes the same qualifier on its GenBank CDS, so both input routes
  now reach the writer with one representation.
* write_genbank() surfaces it as a per_cds_df column, with the same
  list-vs-bare-string handling as transl_table, and pins it to Utf8 so an
  input with no partial qualifiers cannot infer the column as Null.
* The column is dropped at the merged_df join, so neither
  _per_cds_predictions.tsv nor any sub_db_tophits TSV gains a column.
* Contig lengths are passed to write_tbl to place a minus-strand 5'-partial
  CDS against the contig edge.

Two bugs in the initial writer, both found by driving the real pipeline
rather than a hand-built dataframe:

* Strand reaches write_tbl as "+"/"-" strings, because write_genbank
  converts them before returning per_cds_df. The writer was doing
  int(strand) == -1, which raises ValueError on "-". It now accepts both
  representations.
* per_cds_df is ALREADY in transcription order for CDS -- write_genbank
  assigns start = location.end for a minus-strand CDS, so start > end there.
  The writer was re-orienting that pair and writing every reverse-strand CDS
  backwards. CDS coordinates are now passed through untouched; the non-CDS
  path still orients, because those come from BioPython locations which
  really are genomic min/max.

On orientation: Prodigal's partial flag is two digits in genomic order, left
edge then right edge, on both strands -- so the 5' end is the left digit on
the plus strand and the right digit on the minus strand. Verified against
pyrodigal's own GFF writer: reverse-complementing a sequence turns a
left-edge "10" gene into a right-edge "01" gene.

Unlike Pharokka, a CDS partial at both ends ("11") gets both markers rather
than falling through the if/elif chain unmarked. An out-of-range codon_start
warns instead of aborting -- this runs at the very end of a long pipeline.

Tests: 43 in test_tbl.py, covering both strands, all four partial flags plus
"11", codon_start on each strand, malformed flags, and a mirror-image case
asserting a plus-strand partial and its reverse complement produce mirrored
output. Verified end to end on both input routes: 246 CDS (78 reverse-strand)
from tests/test_data/SAOMS1.gbk and a truncated NC_043029 through
pyrodigal-gv, with zero misoriented coordinate lines and forward/revcomp
runs producing exact mirrors.
Validated the generated feature tables with table2asn 1.28.1179 (bioconda,
osx-64). Three problems showed up, all of which would have reached anyone
trying to submit:

* /inference was written as the raw annotation_method ("foldseek",
  "pharokka", "none"). INSDC requires a category from a controlled
  vocabulary, so table2asn rejected EVERY CDS with "Qualifier had bad value"
  -- 246 hard errors on the SAOMS1 test genome, 15/15 on a pyrodigal run.
  This was inherited from Pharokka, which writes its GFF Method column here
  and has the same problem.

  Category chosen by measurement, not guesswork:

      similar to AA sequence             -> InvalidInferenceValue warning
      similar to AA sequence:PHROG:1215  -> InvalidInferenceValue warning
                                            ("unrecognized database")
      protein motif:PHROG:1215           -> clean
      alignment:Foldseek:1215            -> clean

  The "similar to ... sequence" categories are checked against NCBI's list
  of recognised databases, which PHROG is not on. "protein motif" is not
  checked, and is also the honest description -- a PHROG is a protein group
  / profile, and Phold transfers annotation from the group its structure
  matched. CDS with no PHROG to cite get no /inference at all rather than a
  bare category (which is itself flagged) or an invented accession.

* /anticodon was emitted as a bare 3-letter code for Pharokka < v1.8.2
  input, which table2asn rejects -- NCBI wants a location,
  "(pos:115194..115196,aa:Met)". write_genbank already warns about that
  input, so the qualifier is now dropped with a warning rather than written
  knowing it is invalid. The valid location form is passed through.

* 3'-partial CDS stopped at the last whole codon, leaving 1-2 trailing
  bases, and table2asn warned PartialProblemNotSpliceConsensus3Prime ("3'
  partial is not at end of sequence"). They are now extended to the contig
  edge, mirroring the 5' codon_start handling.

After the fixes, the only findings left on all three test cases are the
submission metadata table2asn always requires (NoPubFound, NoTaxonID,
MissingPubRequirement -- supplied by the submitter's template.sbt and
organism), plus BadCDScomponentOverlapTRNA on SAOMS1, which is genuine in
that input: CDS_0193 (115129..115329) spans the tRNA at 115194..115265, and
Pharokka's own .tbl would report it identically. Zero feature or qualifier
findings attributable to the writer.

Test cases: SAOMS1.gbk via the Pharokka GenBank route (246 CDS, 78 reverse
strand, 3 tRNAs) and a truncated NC_043029 through pyrodigal-gv in both
orientations (15 CDS each, 2 partial), which is what exercises the
incomplete-end markers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add .tbl output like pharokka

1 participant