Work27
Python Scraper
A Python web scraping and data enrichment pipeline that automates collecting business data from websites and syncs clean, structured records into a database.
- Role
- Backend
- Built for
- Work and Travel Guide GmbH (WTG)
- Status
- private
- Stack
- Python, Playwright, Pandas, aiohttp, Airtable API
The project
A set of Python web scrapers that collect business data from various websites and turn it into clean, structured records.
The pipeline uses headless browser automation to gather listings, then enriches and validates each entry with business details before syncing everything into a shared database.
Key features:
- Headless browser scraping with Playwright to collect and verify listings
- Data validation and cleanup, removing duplicates and incomplete records
- Concurrent async workers for faster large-batch processing
- Automatic retries, rate limiting, and progress logging to handle long-running jobs
- Data consistency checks (name/address matching) to filter out bad matches
- Syncs results directly into a shared database for downstream use
Next project
All work