Skip to content

feat: update ETL infrastructure for metadata #901

Description

@jacobvjk

CURRENT SITUATION

When preparing pathway classifications, we currently use the following steps:

  • Manually source pathway raw data
  • Pathway raw data stored locally and here
  • Custom GPT run to create draft classification, classification based on cookbook, outputs returned in a standard pathway template
  • Manual steps
  • Finally, there is a set of scripts that can be used to inject the prepared data into the tool, but these are outdated and only saved locally thus far (prep_pathway_metadata.R and MetaData.sh)

EXPECTED OUTCOME

  • At least the final two scripts need to be updated to the latest schema changes and they should be stored in this repository, ideally including tests based on the pathway_data_prepared excel format
  • Any other improvements of the pipeline that are low-hanging fruits should at least be discussed and prioritized

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions