How One Maker Used ChatGPT’s API to Normalize 5.3M Job Posts; Why It Matters

What happened

A Redditor (u/hamed_n) built Hiring.Cafe after getting fed up with ghost jobs and noisy listings on LinkedIn and Indeed (https://hiring.cafe/). Instead of relying on job boards, they crawled employer career pages and used ChatGPT’s API to normalize wildly different job descriptions into structured JSON. The result: 5.3 million scraped jobs (including ~273k remote roles) and an open-source prompt shared for others to inspect or contribute to (https://gist.github.com/hamedn/b8bfc56afa91a3f397d8725e74596cf2). The original Reddit post is here: https://www.reddit.com/r/ChatGPT/comments/1r23leo/update_i_scraped_53_million_jobs_with_chatgpt/

Why this is interesting

1. Practical LLM application, not just chat: This is a concrete example of using a large language model as a normalization layer, turning messy text from thousands of distinct HTML formats into consistent JSON (salary, years of experience, remote flag, role type, etc.).
2. Scale: 5.3M is not a toy project. It raises questions about rate limits, cost, deduplication and freshness, problems that are not solved by a neat prompt alone.
3. Product-market fit signals: Multiple comments are from users who found legitimate jobs and prefer the UX and filtering on Hiring.Cafe to major job boards.

According to top comments, the conversation clustered around three themes

• Data quality and “ghost jobs”: A top commenter asked whether the builder checked how many postings are fake or misleading, a persistent community worry (estimates range wildly). Quote: “If you’re up for it… check how many are fake or misleading.”
• Arms race in automation: Recruiters report an influx of AI-driven auto-applications and are countering with AI filters. “We are using AI on our end to filter it all out,” one hiring-side commenter said, a concise description of the automation loop forming between applicants and employers.
• Engineering challenges at scale: Experienced builders pointed out that LLM extraction is the easy part; long-term product quality requires expiration detection, deduplication, multilingual normalization, and keeping millions of URLs fresh.

Pause for a second.

LLMs excel at normalization but not at maintaining an index. If you can convert a single job post to JSON reliably, you still need systems engineering to:

• detect duplicates across domains and ATSs,
• determine whether a posting is stale or reposted,
• manage cost and rate limits for millions of API calls,
• handle multilingual and semantic edge cases,
• integrate proxying, politeness and legal constraints for crawling.

Context and risks

• Ethics and legality: Scraping employer sites is typically allowed if the content is public, but respect robots.txt, rate limits and terms of service. Large-scale crawling can create traffic and legal headaches if done irresponsibly.
• Cost: Running LLMs at scale for millions of pages is non-trivial, both in API cost and infrastructure. Several commenters asked about rate limits and operational details.
• False sense of authenticity: Employer career pages are not automatically more authentic than job boards; companies often forget to remove old listings, and staffing agencies still post via employer domains.

Takeaways

1. Practical LLMs are powerful for normalization, a clear high-value use case beyond chat.
2. Product success depends on systems engineering beyond the model: deduplication, freshness, multilingual support and fraud detection.
3. The hiring market is already reacting: applicants use AI to scale applications and recruiters use AI to filter them, so expect an automated arms race.

If you want to look deeper

• Live demo: https://hiring.cafe/
• Shared prompt: https://gist.github.com/hamedn/b8bfc56afa91a3f397d8725e74596cf2
• Reddit post: https://www.reddit.com/r/ChatGPT/comments/1r23leo/update_i_scraped_53_million_jobs_with_chatgpt/

Bottom line

This is an instructive, real-world application of LLMs solving data normalization at scale. The prompt is the hook; the long game is running and maintaining a high-quality, up-to-date index and navigating the ethical, technical and economic tradeoffs that come with it.