Fix camel-tools dependency compatibility

[oasissan]

Summary

• declare PyTorch as a direct runtime dependency and bound Transformers to the range compatible with camel-tools 1.5.7
• remove the unused torchvision dependency from package metadata and the packaged environment file
• fix the sentence_tokenizer CLI so --text input is not split into characters
• add regression tests for dependency metadata and the CLI tokenizer behavior

Root cause

SinaTools imports torch directly from WSD/NER modules, but the previous compatibility change removed torchvision without adding torch explicitly. Loosening Transformers all the way to an unbounded dependency also allowed Transformers 5.x, which breaks current relation import behavior. camel-tools 1.5.7 requires transformers >=4.0,<4.44, so this PR uses that bounded range.

Validation

• python -m unittest discover -s tests -v
• python -m pytest -q tests
• python -m flake8 setup.py sinatools/CLI/utils/sentence_tokenizer.py tests
• git diff --check
• pip install --dry-run .
• pip install --dry-run . camel-tools==1.5.7
• clean venv: pip install . camel-tools==1.5.7 pytest; verified installed metadata, torch/transformers/camel_tools imports, NER module imports, pip check, and sentence_tokenizer CLI output

Notes

• Full flake8 sinatools tests still reports existing repository-wide style issues outside this patch.
• python setup.py test is still blocked by the legacy pytest-runner/setuptools test command path; direct pytest is passing.

[Copilot]

Pull request overview

This PR updates SinaTools packaging and CLI behavior to restore compatibility with the camel-tools/Transformers stack while fixing a regression in the sentence tokenizer CLI. It primarily adjusts runtime dependency metadata, removes an unused packaged dependency, and adds regression tests around the dependency declarations and --text CLI handling.

Changes:

• Add torch as a direct runtime dependency, bound transformers to <4.44, and remove torchvision from package/environment metadata.
• Fix the sentence tokenizer CLI so --text input is passed as a full string instead of being split character-by-character.
• Add regression tests for dependency declarations and sentence tokenizer CLI output; regenerate packaged metadata/build artifacts.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_sentence_tokenizer_cli.py	Adds a CLI regression test for --text sentence splitting behavior.
tests/test_dependency_metadata.py	Adds tests that inspect dependency declarations in setup.py.
sinatools/environment.yml	Updates the packaged environment pins to remove torchvision and downgrade transformers/tokenizers.
sinatools/CLI/utils/sentence_tokenizer.py	Refactors the CLI and changes how text/file input is assembled before tokenization.
setup.py	Declares torch directly and constrains transformers in install requirements.
setup.cfg	Removes the pytest-specific collect_ignore section and keeps egg-info settings.
build/lib/sinatools/environment.yml	Regenerated build artifact mirroring the environment dependency changes.
build/lib/sinatools/CLI/utils/sentence_tokenizer.py	Regenerated build artifact mirroring the CLI tokenizer changes.
SinaTools.egg-info/requires.txt	Regenerated requirement metadata reflecting the updated install requirements.
SinaTools.egg-info/SOURCES.txt	Regenerated source manifest including the new test files.
SinaTools.egg-info/PKG-INFO	Regenerated package metadata reflecting the updated dependency declarations and package version metadata.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.