Skip to content

Support loading/streaming data from s3 locations #1

Description

@nihaljn

This is what's supported at the moment:

  • Local - JSON, JSONL, HuggingFace parquet
  • Remote
    • HuggingFace Hub - HuggingFace parquet

Supporting S3 loading/streaming would entail supporting the following:

  • Local - JSON, JSONL, HuggingFace parquet
  • Remote
    • HuggingFace Hub - HuggingFace parquet
    • S3 - JSON, JSONL, HuggingFace parquet

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions