Overview
Automated data collection and processing system for golf airmen database using headless browser automation, CSV parsing, and intelligent data pipeline with cron-based scheduling.
Key Features
- Automated web scraping with Puppeteer
- Headless Chrome browser automation
- CSV data parsing and transformation
- ZIP archive extraction and processing
- Cookie-based authentication handling
- XSS protection bypass for secure pages
- MySQL data persistence and querying
- Cron-based scheduling for regular updates
- Error handling and logging
- Data validation and cleaning
Technology Stack
Node.js, TypeScript, Puppeteer, MySQL, CSV Parser, ADM-Zip, Axios, Cron
