GB-Scraper

The problem

Comparing results across search engines means running the same query twice and reconciling two pages by hand. Doing it programmatically runs into the other problem: scraping is slow, and a user staring at a spinner with no feedback assumes the app has hung.

Approach

• 01Puppeteer drives headless Chrome against both Google and Bing, with the two scrapes running concurrently rather than in sequence.
• 02A Socket.IO channel pushes per-engine progress to the browser as each result batch lands, so the UI fills in continuously instead of blocking on the full job.
• 03Results are normalised into a single shape and persisted to MongoDB, which doubles as a cache for repeated queries.
• 04The scraping endpoint is the expensive one, so it sits behind JWT auth, per-query rate limiting, and CSRF protection — otherwise a single client can pin the box by firing concurrent jobs.

Outcome

A query returns aggregated, de-duplicated results from both engines with visible progress throughout, and the rate limiter keeps concurrent scraping jobs from exhausting the browser pool.