🤖 Automate Bot

A powerful automation tool that scrapes data from websites, processes it, and seamlessly uploads it to a MySQL database. Features a real-time web dashboard for monitoring and managing your automation tasks.

✨ Features

• Web Scraping: Efficiently scrapes data from websites using BeautifulSoup and lxml
• Data Processing: Cleans, processes, and transforms scraped data
• Database Integration: Posts data directly to MySQL database via API
• Smart Checkpointing: Remembers processed URLs to avoid duplicate scraping
• Web Dashboard: Real-time Flask-based monitoring interface
• Error Tracking: Automatically logs failed uploads for retry and debugging
• Thread-based Execution: Non-blocking background task execution
• Comprehensive Logging: Detailed logging to both console and file
• Docker Ready: Includes Dockerfile for containerized deployment
• Configuration-driven: Easy configuration via JSON

🛠️ Tech Stack

• Backend: Python 3.x
• Web Framework: Flask
• Database: MySQL
• Data Scraping: BeautifulSoup4, Requests, lxml
• Data Processing: Pandas
• Containerization: Docker
• Database Driver: mysql-connector-python

📦 Prerequisites

Before you begin, ensure you have the following installed:

• Python 3.7+
• MySQL Server 5.7+ or MySQL 8.0+
• pip (Python package manager)
• Git (for cloning the repository)

🚀 Installation

1. Clone the Repository

git clone https://github.com/yourusername/Automate_Bot.git
cd Automate_Bot

2. Create a Virtual Environment (Recommended)

# On Windows
python -m venv venv
venv\Scripts\activate

# On macOS/Linux
python3 -m venv venv
source venv/bin/activate

3. Install Dependencies

pip install -r requirements.txt

4. Set Up MySQL Database

CREATE DATABASE blog_web;

Ensure you have the necessary tables created in your MySQL database.

⚙️ Configuration

config.json

Create or update the config.json file with your credentials:

{
    "api_url": "http://localhost/api/post-data-insertion",
    "mysql_config": {
        "host": "localhost",
        "user": "root",
        "password": "your_password",
        "database": "blog_web"
    },
    "cookies": {
        "vr_session": "your_session_cookie",
        "vr_csrftoken": "your_csrf_token"
    }
}

Important: Never commit sensitive credentials to version control. Use environment variables or .env files for production.

📖 Usage

Running the Scraper & Uploader

python app.py

This will:

1. Read URLs from sitemap.csv
2. Scrape data from each URL
3. Post the data to your MySQL database
4. Log all activities to logs.txt
5. Track successful uploads in checkpoint.txt
6. Save failed uploads to failed_uploads.txt

Launching the Web Dashboard

python dashboard.py

The dashboard will be available at http://localhost:5000 with features to:

• View real-time logs
• Monitor scraping progress
• Start/stop the bot
• View checkpoint status
• Track failed uploads

Running with Docker

docker build -t automate-bot .
docker run -d -p 5000:5000 automate-bot

📁 Project Structure

Automate_Bot/
├── app.py                    # Main orchestrator script
├── scrapper_bot.py          # Web scraping module
├── poster_bot.py            # Data posting & database module
├── dashboard.py             # Flask web dashboard
├── config.json              # Configuration file (add to .gitignore)
├── requirements.txt         # Python dependencies
├── Dockerfile               # Docker configuration
├── checkpoint.txt           # Successfully processed URLs
├── failed_uploads.txt       # Failed uploads log
├── logs.txt                 # Application logs
├── sitemap.csv              # URLs to scrape
├── templates/               # Flask HTML templates
│   ├── index.html
│   └── edit_file.html
├── static/                  # Static assets
│   └── style.css
└── README.md               # This file

🏗️ Architecture

Component Overview

┌─────────────────────────────────────────────────┐
│          Web Dashboard (Flask)                   │
│        (dashboard.py - Port 5000)                │
└─────────────────────────────────────────────────┘
                      ↓
┌─────────────────────────────────────────────────┐
│         Main Application (app.py)                │
│   - Orchestrates scraping and posting flow       │
│   - Manages checkpointing                        │
│   - Handles category mapping                     │
└─────────────────────────────────────────────────┘
         ↙                                ↘
┌──────────────────────┐    ┌──────────────────────┐
│  Scraper Bot         │    │  Poster Bot          │
│ (scrapper_bot.py)    │    │ (poster_bot.py)      │
│ - Fetches HTML       │    │ - Posts to API       │
│ - Parses with BS4    │    │ - MySQL connection   │
│ - Extracts data      │    │ - Error handling     │
└──────────────────────┘    └──────────────────────┘
                      ↓
            ┌──────────────────┐
            │  MySQL Database  │
            │   (blog_web)     │
            └──────────────────┘

Data Flow

1. Initialization: Load URLs from sitemap.csv and previous checkpoints
2. Scraping: Fetch and parse web pages using BeautifulSoup
3. Processing: Extract and clean data
4. Categorization: Map content to predefined software categories
5. Posting: Send data to API endpoint
6. Storage: Save to MySQL database
7. Logging: Record success/failure with detailed timestamps
8. Checkpoint: Update checkpoint file to track progress

📊 Logging & Monitoring

Log Files

• logs.txt: Complete application logs with timestamps
• checkpoint.txt: List of successfully processed URLs (one per line)
• failed_uploads.txt: URLs that failed to upload

Viewing Logs

# View last 20 lines of logs
tail -20 logs.txt

# View all failed uploads
cat failed_uploads.txt

# Monitor logs in real-time
tail -f logs.txt

Log Format

2024-06-03 10:15:30,123 - INFO - Successfully scraped: https://example.com
2024-06-03 10:15:31,456 - ERROR - Failed to post data for URL: https://example.com

⚠️ Error Handling

The bot includes robust error handling:

• Connection Timeouts: Automatically logged and added to failed uploads
• Network Errors: Gracefully handled with retry capability
• Database Errors: Logged for investigation
• Parse Errors: Handled with fallback to failed uploads list
• Concurrent Connection Pool: Manages MySQL connection pooling efficiently

Recovering from Failed Uploads

# Check failed uploads
cat failed_uploads.txt

# Retry failed uploads (add URLs back to sitemap.csv)
# Then run: python app.py

🐳 Docker Support

Build Docker Image

docker build -t automate-bot:latest .

Run Container

# Interactive mode
docker run -it -p 5000:5000 automate-bot:latest

# Detached mode
docker run -d -p 5000:5000 --name my-bot automate-bot:latest

Docker Environment Variables

You can override config values using environment variables:

docker run -e MYSQL_HOST=db.example.com \
           -e MYSQL_USER=admin \
           -p 5000:5000 automate-bot:latest

🤝 Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Code Standards

• Follow PEP 8 style guide
• Add docstrings to functions and modules
• Write meaningful commit messages
• Add tests for new features

🐛 Troubleshooting

Issue: Database Connection Fails

Solution:

• Check MySQL server is running
• Verify credentials in config.json
• Ensure database exists

Issue: Scraper Hangs

Solution:

• Check internet connection
• Verify URLs in sitemap.csv
• Check timeout settings (default: 10s)

Issue: Dashboard not accessible

Solution:

• Ensure port 5000 is not in use
• Check firewall settings
• Verify Flask is running

📧 Support

For issues, questions, or suggestions, please open an issue on GitHub or contact the maintainer.

---

Made with ❤️ by Rij1