feat(bulk-import): add client-side parquet validation

[tim-pinecone]

Summary

• Adds index.bulk_import.validate(uri) and the standalone pinecone.validate_bulk_import(uri) to check parquet files for Pinecone bulk import compatibility before sending them to the server
• Schema validation reads only the parquet file footer (no data downloaded) — cheap even for large remote files
• Optional data sampling (sample_rows=100 by default) checks for null IDs, non-finite vector values, metadata JSON validity, and the 40 KB metadata size limit
• Returns BulkImportValidationResult with .is_valid, .errors, .warnings, .files_checked, .rows_sampled; the .uri field can be passed directly to index.bulk_import.start()
• Verbose mode prints per-file OK/BAD progress and a final summary
• Supports single .parquet files and directory/prefix URIs; handles s3://, gs://, and az:// via the pyarrow filesystem abstraction
• pyarrow is an optional dependency: pip install 'pinecone[parquet]'

Usage

# Standalone (no index needed)
from pinecone import validate_bulk_import

result = validate_bulk_import("s3://my-bucket/vectors/")
if result.is_valid:
    index.bulk_import.start(uri=result.uri)
else:
    for error in result.errors:
        print(error)

# Via the index object
result = index.bulk_import.validate(
    "s3://my-bucket/vectors/",
    dimension=1024,
    sample_rows=0,   # schema-only, no data download
)

# Verbose progress across many files
result = index.bulk_import.validate("s3://my-bucket/vectors/", verbose=True)
# [  1/50] OK   s3://my-bucket/vectors/part-00000.parquet
# [  2/50] OK   s3://my-bucket/vectors/part-00001.parquet
# ...
# Total: 50  OK: 50  BAD: 0  rows sampled: 5000

Test plan

• uv run pytest tests/unit/data/test_bulk_import_validator.py — 40 unit tests (schema validation, data sampling, end-to-end file I/O with real parquet files on disk)
• Validated manually against a real S3 bucket with 50 sparse parquet files
• uv run mypy pinecone passes clean

🤖 Generated with Claude Code

---

Note

Medium Risk
Introduces new file/URI inspection logic using pyarrow (including remote filesystem access) and surfaces it via public SDK APIs; failures or edge cases could affect user workflows, but it is additive and optional.

Overview
Adds a client-side parquet compatibility validator for bulk import, exposed as index.bulk_import.validate(...) (sync and asyncio) and the top-level pinecone.validate_bulk_import(...) helper.

The validator inspects parquet schema (footer-only by default) and optionally samples rows to catch common import blockers (missing/typed columns, dimension mismatches, null/empty IDs, non-finite vector values, and metadata JSON/size issues), returning a new BulkImportValidationResult with errors/warnings and counts.

Introduces a new optional extra pinecone[parquet] (adds pyarrow) plus mypy overrides for pyarrow, and adds a comprehensive unit test suite covering schema, sampling, and end-to-end parquet files.

^{Reviewed by Cursor Bugbot for commit 322b522. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d125b19. Configure here.}