Skip to content

fix: pin dependency versions and Python support range - #74

Open
maan-iitd2 wants to merge 1 commit into
blackrock:mainfrom
maan-iitd2:pr/packaging-pins
Open

fix: pin dependency versions and Python support range#74
maan-iitd2 wants to merge 1 commit into
blackrock:mainfrom
maan-iitd2:pr/packaging-pins

Conversation

@maan-iitd2

@maan-iitd2 maan-iitd2 commented Jul 15, 2026

Copy link
Copy Markdown

What

Pin dependency versions and Python support range in setup.py / requirements.txt. No code changes.

Change Reason
great_expectations>=0.18.9,<1.0 InGen uses the legacy ge.from_pandas(...).expect_* Dataset API, removed in GE 1.0. Unpinned, pip install can resolve a GE version that silently breaks every validation.
pandas>=2.2.2,<3.0 pandas 3.0 turns Copy-on-Write on by default and drops deprecated APIs ingen's formatters/readers use.
openpyxl>=3.1.0 (new) ingen/reader/file_reader.py and ingen/writer/writer.py call pd.read_excel/to_excel(engine='openpyxl') directly, but pandas does not install openpyxl by default — this was a missing, implicit dependency.
setup.py: python_requires='>=3.9,<3.13' numpy<2 (forced transitively by GE's legacy-API requirement) has no wheels for Python 3.13+, so an unconstrained install on 3.13+ falls into a from-source numpy build that needs a C compiler and fails with a confusing error. This makes pip reject it up front with a clear message.
setup.py: read_requirements() open('requirements.txt').read().splitlines() fed every line — including the new comments above — straight into install_requires. Replaced with a small filter that skips blank lines and # comments.
setup.py: resolve_version() The version string ships as the literal token {{VERSION_PLACEHOLDER}}, substituted by CI (.github/workflows/python-publish.yml) via sed at release time. Locally, the token is left as-is, which isn't a valid PEP 440 version and makes pip install -e . fail on a fresh checkout. Falls back to 0.0.0.dev0 when the token hasn't been substituted; the release path is untouched.

Testing

Built a from-scratch venv (Python 3.12, no pre-existing installs):

  • Before this change, pip install -e . on main fails outright — the unresolved {{VERSION_PLACEHOLDER}} isn't a valid version string, so setuptools raises InvalidVersion during dependency resolution.
  • After this change, pip install -e . succeeds cleanly.
  • Full suite: 393 passed / 33 failed on unmodified main → 404 passed / 22 failed with this change — the version pins fix 11 pre-existing failures (test_common_formatters.py, test_merger.py) caused by dependency drift on unpinned installs. The remaining 22 failures are pre-existing environment issues unrelated to packaging (missing GPG binary, Windows file locks, live network calls in test_api_reader/test_aiohttp_retry) — same failures, same count, on both branches.
  • Ran python -m ingen end-to-end against the fresh install (XML source → CSV output) to confirm the installed package actually runs.
  • Also confirmed python_requires does its job: installing into a Python 3.14 venv fails immediately with Package 'ingen-lib' requires a different Python: 3.14.0 not in '<3.13,>=3.9' instead of a numpy build failure.

- great_expectations<1.0: ingen uses the legacy Dataset API
  (ge.from_pandas().expect_*) that was removed in GE 1.0.
- pandas<3.0: pandas 3.0 flips Copy-on-Write on by default and drops
  deprecated APIs that ingen's formatters/readers rely on.
- numpy left unbounded on the low end but effectively capped below 2
  by great_expectations' own dependency; documented why in setup.py's
  python_requires so a 3.13+ install fails with a clear message
  instead of trying to build numpy from source.
- add python_requires='>=3.9,<3.13' so pip rejects incompatible
  interpreters up front.
- add openpyxl as an explicit dependency: ingen's own file_reader and
  writer call pandas' Excel read/write with engine='openpyxl', which
  pandas does not install by default.
- setup.py: parse requirements.txt properly (skip blank/comment
  lines) so the new inline comments above don't get fed to
  install_requires as garbage requirements; fall back to a valid dev
  version when the CI-only {{VERSION_PLACEHOLDER}} token hasn't been
  substituted, since an unresolved token isn't a valid PEP 440 version
  and breaks `pip install -e .` for every local/editable install.

Verified with a from-scratch venv (Python 3.12): `pip install -e .`
succeeds, and the full test suite goes from 393 passed/33 failed (as
on unmodified main) to 404 passed/22 failed -- the version pins fix
11 pre-existing failures (formatter and merger tests) caused by
dependency drift. Remaining 22 failures are pre-existing environment
issues unrelated to packaging (missing GPG binary, Windows file locks,
live network calls) and are unchanged in count/identity by this diff.
Also ran `python -m ingen` end-to-end against the fresh install.

Signed-off-by: maan-iitd2 <maan.iitd.ac.in@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants