Skip to content

feat: scrape blog.python.org and store in PostgreSQL - #140

Open
aashitha20 wants to merge 1 commit into
UniCourt:mainfrom
aashitha20:homework
Open

feat: scrape blog.python.org and store in PostgreSQL#140
aashitha20 wants to merge 1 commit into
UniCourt:mainfrom
aashitha20:homework

Conversation

@aashitha20

Copy link
Copy Markdown

Workshop 1 Homework

  • Scrapes blog.python.org using Python and BeautifulSoup
  • Extracts blog title, URL, and publication date
  • Stores scraped data in PostgreSQL
  • Uses Docker and Docker Compose for the application and database
  • Adds PostgreSQL schema and duplicate handling
  • Uses environment variables for database configuration
  • Tested the scraper and PostgreSQL integration using Docker Compose

Copilot AI lite review requested due to automatic review settings August 27, 2026 15:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR turns the existing sample script into a Python web scraper intended to collect posts from blog.python.org and persist them into a PostgreSQL table, packaged for local execution via Docker Compose.

Changes:

  • Replaced the previous demo scraping logic with a multi-page blog scraper and PostgreSQL persistence (including deduping on URL).
  • Added a PostgreSQL schema (schema.sql) and inserts with ON CONFLICT DO NOTHING.
  • Updated containerization to use requirements.txt, and added a Postgres service + env-driven configuration in Docker Compose.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
web_scraping_sample.py Implements scraping, parsing, and PostgreSQL insert logic (plus env configuration).
schema.sql Defines the blog_posts table with a unique URL constraint for deduplication.
Dockerfile Builds a runnable scraper image based on requirements.txt.
docker-compose.yaml Adds PostgreSQL service + healthcheck and wires scraper env vars to the DB container.
.gitignore Ignores .env, virtualenv, and Python bytecode artifacts.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread web_scraping_sample.py
Comment on lines +128 to +130
def main():
logging.basicConfig(level=os.getenv("LOG_LEVEL", "INFO"))
posts = scrape_posts(BLOG_URL)
Comment thread web_scraping_sample.py
Comment on lines +107 to +108
with open("schema.sql", encoding="utf-8") as schema_file:
cursor.execute(schema_file.read())
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants