🕷️ Alt Product Scraper

A Python web scraper designed to extract product data from Altman.co.il online store. This tool crawls through the shop pages and extracts detailed product information including titles, descriptions, prices, and images.

✨ Features

• Smart Crawling: Automatically discovers and crawls product pages
• Product Data Extraction: Extracts titles, names, descriptions, prices, and images
• Resume Capability: Skips already crawled URLs in subsequent runs
• Shop-Only Focus: Only crawls URLs containing /shop/ to avoid irrelevant pages
• JSON Output: Saves structured data in JSON format
• Hebrew Support: Properly handles Hebrew text content
• Progress Tracking: Shows real-time crawling progress
• Error Handling: Robust error handling and recovery

📋 Requirements

• Python 3.7+
• Windows/Linux/MacOS
• Internet connection

🚀 Installation

Option 1: Automatic Setup (Windows)

1. Clone the repository:

   git clone https://github.com/yourusername/altman-scraper.git
   cd altman-scraper

2. Run the setup script:

   setup.bat

Option 2: Manual Installation

1. Clone the repository:

   git clone https://github.com/yourusername/altman-scraper.git
   cd altman-scraper

2. Install Python dependencies:

   pip install -r requirements.txt

3. Install Playwright browsers:

   playwright install

🎯 Usage

Basic Usage

Run the scraper with default settings (25 pages):

python alt_scan_v01.py

First Run Output

🕷️  Altman.co.il Product Scraper
============================================================
📋 Configuration:
   • Max pages to crawl: 25
   • Max depth: 15
   • Starting URL: https://www.altman.co.il/shop/
   • Crawled URLs file: crawled_urls.json
============================================================

Loaded 0 previously crawled URLs
Crawling: https://www.altman.co.il/shop/ (depth: 0)
✓ Product data saved to www_altman_co_il/magnesium_up_60_product_data.json
  Product: מגנזיום UP - Magnesium UP
  Price: ₪ 110.80
  Unit Price: ₪ 1.85
  Images found: 4
  Description: מינון גבוה של 450 מ"ג מגנזיום איכותי המופק ממעבדות בים המלח...

⚙️ Configuration

You can modify the scraper behavior by editing the configuration variables at the top of alt_scan_v01.py:

# CONFIGURATION SETTINGS - Update these values as needed
MAX_PAGES = 25        # Change this number to crawl more or fewer pages
MAX_DEPTH = 15        # Maximum depth to crawl (how many clicks deep from start page)
START_URL = "https://www.altman.co.il/shop/"  # Starting URL for crawling

# File to store crawled URLs across runs (to avoid re-crawling)
CRAWLED_URLS_FILE = "crawled_urls.json"

Configuration Options

Variable	Description	Default	Example Values
MAX_PAGES	Number of pages to crawl per session	25	10, 50, 100
MAX_DEPTH	Maximum clicks deep from start page	15	5, 10, 20
START_URL	URL to start crawling from	"https://www.altman.co.il/shop/"	Any shop URL
CRAWLED_URLS_FILE	File to store crawled URLs	"crawled_urls.json"	"progress.json"

Examples

Crawl only 10 pages:

MAX_PAGES = 10

Start from a specific category:

START_URL = "https://www.altman.co.il/shop/vitamins/"

Use a different progress file:

CRAWLED_URLS_FILE = "my_crawling_progress.json"

📁 Output Structure

The scraper creates the following file structure:

your-project/
├── www_altman_co_il/           # Main output directory
│   ├── product1_product_data.json
│   ├── product2_product_data.json
│   └── category_general_content.json
├── crawled_urls.json          # Progress tracking file
├── alt_scan_v01.py            # Main scraper script
└── requirements.txt           # Dependencies

Product Data Format

Each product JSON file contains:

{
    "url": "https://www.altman.co.il/shop/magnesium/magnesium-up-60/",
    "timestamp": "2025-01-07T15:13:33.005354",
    "product_title": "מגנזיום UP",
    "product_name": "Magnesium UP",
    "description": "מינון גבוה של 450 מ\"ג מגנזיום איכותי המופק ממעבדות בים המלח בתוספת אשלגן לפעילות משלימה.",
    "total_price": "₪ 110.80",
    "price_per_unit": "₪ 1.85",
    "image_links": [
        "https://www.altman.co.il/wp-content/uploads/batch_images/image1.webp",
        "https://www.altman.co.il/wp-content/uploads/batch_images/image2.webp"
    ]
}

🔄 Resume Functionality

The scraper automatically saves progress and can resume from where it left off:

• First run: Crawls up to 25 new pages
• Second run: Skips already crawled URLs, crawls 25 more new pages
• Progress file: crawled_urls.json tracks all crawled URLs

Reset Progress

To start fresh and re-crawl all pages:

# Delete the progress file
rm crawled_urls.json
# Or on Windows:
del crawled_urls.json

# Then run the scraper
python alt_scan_v01.py

🛠️ Troubleshooting

Common Issues

Issue: playwright._impl._errors.Error: Executable doesn't exist

# Solution: Install Playwright browsers
playwright install

Issue: ModuleNotFoundError: No module named 'crawl4ai'

# Solution: Install dependencies
pip install -r requirements.txt

Issue: Empty image_links in output

• This might be normal for some pages
• Images are filtered to exclude icons and theme assets
• Check if the page actually contains product images

Issue: total_price shows "₪ 0"

• The price pattern might need adjustment for specific products
• Some products might not have prices displayed
• Check the HTML structure of problematic pages

Debug Mode

To see more detailed output, you can modify the crawler settings in the code:

# In the crawler.arun() call, change:
verbose=True  # Already enabled for debug output

📊 Performance

• Speed: ~3-5 seconds per page (due to JavaScript loading delay)
• Memory: Minimal memory usage as data is saved to files immediately
• Storage: Each product JSON is typically 1-3 KB

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

⚖️ Legal Notice

This scraper is for educational and research purposes. Please ensure you comply with:

• Altman.co.il's Terms of Service
• robots.txt file
• Rate limiting (scraper includes delays)
• Data usage regulations

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support

If you encounter issues:

1. Check the Troubleshooting section
2. Ensure all dependencies are installed correctly
3. Verify your internet connection
4. Check that the target website is accessible

For bugs or feature requests, please open an issue on GitHub.

---

Happy Scraping! 🚀