Skip to content

Add an MS MARCO dataset importer #8

Description

@AshwinUgale

The problem / use case

Context: cli.py has import-squad and import-beir that emit docs.jsonl +
queries.jsonl with source-span gold. MS MARCO is a natural third target.

What: Add an import-msmarco subcommand that converts an MS MARCO passage/qrels
export into the same format.

Acceptance:

  • retrieval-lab import-msmarco … --out-dir … produces valid docs.jsonl + queries.jsonl
  • gold is documented as passage-level (coarser/stricter than answer-span), matching how
    BEIR import already frames it
  • a small fixture test in tests/test_datasets.py; CHANGELOG.md updated

Pointers: mirror import-beir (passage-level relevance → span over the whole passage).

Proposed idea

Add import-msmarco alongside the SQuAD/BEIR importers so users can benchmark on a well-known passage-ranking dataset; its passage-level relevance imports as coarser (passage-span) gold, stated honestly.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions