Skip to content

Feature/dch notre environnement#158

Merged
lpi-tn merged 7 commits into
mainfrom
Feature/dch-notre-environnement
Jul 21, 2026
Merged

Feature/dch notre environnement#158
lpi-tn merged 7 commits into
mainfrom
Feature/dch-notre-environnement

Conversation

@lpi-tn

@lpi-tn lpi-tn commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

This pull request introduces a new scraping plugin for the "Notre Environnement" website and makes supporting changes to integrate it into the project. It also adds new dependencies, utility functions, and comprehensive unit tests to ensure the reliability of the new collector.

NotreEnvironnementCollector Plugin Integration

  • Added a new NotreEnvironnementCollector class in welearn_datastack/plugins/scrapers/notre_environnement.py for scraping and extracting metadata and content from the "Notre Environnement" website, including robust error handling and Dublin Core metadata extraction.
  • Registered NotreEnvironnementCollector in the scrapers plugin system (welearn_datastack/plugins/scrapers/__init__.py). [1] [2]

Testing Enhancements

  • Added a comprehensive test suite for NotreEnvironnementCollector covering successful runs, no-content scenarios, and HTTP errors (tests/document_collector_hub/plugins_test/test_notre_environnement.py).

Utility and Dependency Updates

  • Introduced a normalize_text utility function in scraping_utils.py for consistent text normalization.
  • Added trafilatura and extruct as dependencies in pyproject.toml to support content extraction and metadata parsing.

@lpi-tn
lpi-tn requested review from jmsevin and samonaisi July 20, 2026 15:30
Comment thread welearn_datastack/plugins/scrapers/notre_environnement.py
@lpi-tn
lpi-tn merged commit 9c432af into main Jul 21, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants