The latest articles on DEV Community by 핫구사 (@_246c4383449a51a8f8ff1). https://dev.to/_246c4383449a51a8f8ff1 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3649833%2Ff73cbbac-d69f-4fd5-9932-a739ac701b38.png https://dev.to/_246c4383449a51a8f8ff1 en 핫구사 Sun, 07 Dec 2025 04:55:52 +0000 https://dev.to/_246c4383449a51a8f8ff1/building-a-deal-aggregator-lessons-from-crawling-korean-e-commerce-sites-20kp https://dev.to/_246c4383449a51a8f8ff1/building-a-deal-aggregator-lessons-from-crawling-korean-e-commerce-sites-20kp <p>I've been using a Korean hot deal aggregator called <a href="https://www.hotgusa.com" rel="noopener noreferrer">Hotgusa</a> for a while now, and it got me thinking about the technical challenges behind building something like this.</p> <h2> The problem it solves </h2> <p>Korean shoppers check multiple community forums (Ppomppu, Clien, Ruliweb, FM Korea) for deals. Same product, different prices across platforms. Manually checking everything is a time sink.</p> <h2> Technical challenges I imagine they faced </h2> <p><strong>1. Multi-source crawling</strong></p> <p>Each community has different HTML structures. Some are SPAs requiring headless browsers like Puppeteer or Playwright.</p> <p><strong>2. Real-time updates</strong></p> <p>Hot deals sell out in minutes. The crawling interval needs to be aggressive but respectful to avoid getting blocked.</p> <p><strong>3. Deduplication</strong></p> <p>Same deal posted across multiple communities. Need similarity matching on title, price, and URL.</p> <p><strong>4. Ranking algorithm</strong></p> <p>They seem to use views, upvotes, and recency to surface the best deals first.</p> <p><strong>5. Push notifications</strong></p> <p>Keyword-based alerts via FCM for their mobile app. Users set keywords and get notified instantly.</p> <h2> Thoughts </h2> <p>Building an aggregator like this seems straightforward at first, but the edge cases are brutal. Rate limiting, changing HTML structures, duplicate detection...</p> <p>If anyone's built something similar, I'd love to hear about your architecture choices. What worked? What didn't?</p> webdev crawling sideprojects korea