In the previous post, we covered the two-pass grading system. Now we turn to a component that sits alongside it: the link scraper. But this adds complexity, latency, and more constraints to the existing pipeline we discussed earlier. By the end of this article, we will explore the motivation behind adding the link extractor, how we built it, and how we managed it within the existing data grading pipeline.
The Motivation Behind the Link Scraper
Sharing links is an inclusive part of tech conversation. From our data, we found that roughly 40% of the total messages contain at least one link embedded in them. Constituting roughly 40% of the total data, this demands that we capture the data shared via these links as well. If the knowledge graph fails to capture the content of these links, it cannot truly be a knowledge graph in its essence, as it will miss out on all the important updates shared via links.
Secondly, the whole idea is that the user does not need to track everything. If the link extractor is not present, the user receiving a link in response to a query would have to read it all over again, make sense out of it, and figure out what is required from it on their own terms. They would realize this only after spending time consuming roughly half of the shared content.
Having known the reason for the existence of the link scraper as an integral tool, now let us focus on how we made it.
How We Made It
From the analysis of real data, we found links can be broadly classified into three categories: GitHub, YouTube videos, and articles/blogs. The original constitution is about GitHub at 10%, YouTube videos at 5%, and articles at 85%.
One interesting observation is that as articles contribute largely to the links as a whole, most of them were easily accessible without having to log in.
| Link Type | Method | Why This Method |
|---|---|---|
| Articles | Playwright + enhancer | Works when there is no auth wall; captures page content |
| GitHub | REST API | Playwright only captures README and file tree; API returns structured repo data |
| YouTube | youtube-transcript-api + Supadata fallback | Content is in the video, not the page |
For article data, we are using Playwright, but only if the article does not have a login wall or bot detection. If the content is directly in the writing of the article, using Playwright to copy all the contents covers the majority of the data.
But this is the case when the article is not blocked by an authorization wall. To solve for the blocked case, we just record that link along with the reason as a blocked link. The user can then either paste the contents of the link manually or skip that link (i.e., removing it from the list). For now, it is saved in a JSON file, and we will upgrade it to use a database in the future.
The above methods rely on the assumption that the link is not blocked by bot detection or an auth wall, and that the contents are present on the opening page itself. On GitHub, copying all the contents will not help us, as it will only cover the file structure along with the README and noise. For YouTube, the content is not on the page but in the video, so it will fail badly in these cases.
Using the GitHub REST API for GitHub links, we first verify via regex if it is a GitHub link, and after verification, we use the REST API for extracting the contents of the repository. For YouTube videos, since the content is in the video, we first rely on the youtube-transcript-api library. As a fallback, we use Supadata, which mostly provides the transcript of the video.
Is This All?
No. Till now, we have only managed to get the data from the links or external sources shared. However, we cannot still use it to make sense of it, and it is not advisable to just dump it into our knowledge graph. So, what is missing?
Let's refer back to the motivation section, where we mentioned that the user had to manually track everything. If they forget the content, they have to go through the link again, understand how it can be used, and relate it to their work. It is too much friction and time-consuming just to gather data that was already shared.
Now, the idea is not only to get the link's data but to provide all the important details about the data present, so we don't face these issues when we have our knowledge graph. That is where we create the metadata of the link via the enricher module, so the user can just glance over it and decide if it is worth their time.
This module takes the raw data, uses an LLM (here we are using a Local LLM only, but external LLMs may be used as per the user's preferences), and extracts the intention of the links, topics, and entities covered to give an idea. Finally, it covers the summary: what problem it solves, how useful it is, and what it is.
We also ensure none of the details are too long for the user (so they don't end up writing a whole blog from the metadata itself). We do this by using limited output tokens and explicit guardrails via prompts to ensure it remains a concise overview of the link.
How Are We Managing It With the Existing Grading Pipeline?
We realized that link extraction does not contribute anything to the second grade pass. Since the second grade pass is only for messages with a grade quality of 3, and the link extractor works with messages containing links, we separated them. The first grade pass segregates the link-embedded messages to quality 4-5, as explicitly mentioned in the system prompt. As grade pass 1 completes, the link extractor and second grade pass work in parallel, and are finally merged before ingesting.
We achieved this with the help of LangGraph. We wanted to have persistence, because during our tests, we experienced first-hand how the whole system had to fire up and start from scratch even when the work was halfway done. That is why we chose LangGraph; we chose it for its simplicity and easy compatibility with persistence, parallel routing, and merging states.
Counter Arguments
One might object to using Firecrawl or any other scraping agents. We are not focused on building a universal scraper, but rather a scraper that covers most of the content shared. We also encourage users to try different techniques once the repo is out. The whole idea is to get the system personalized to how you prefer to work. We agree that using external services might be easier, but we started with minimalistic tools to get the job done without running into the hassle of balancing credits and relying heavily on external services.
Another objection may come from GitHub rate limits and YouTube videos whose transcripts are not available. Using unauthenticated GitHub requests sure puts a lot of load on the API, but we are using authenticated requests which provide us with roughly 5k requests per hour. This is more than enough for what we require, and since it will be run on your hardware, you can choose the option you prefer. We gracefully degrade to Playwright scraping when the rate limit is exhausted or we receive a 403 request. From our original data, GitHub shares roughly 10%, so it can be processed well under this limit. For videos without transcript, that's where fallback supadata comes into play.
Conclusion
The link scraper was not added as an afterthought or a single, universal tool. It was built as three domain-specific extractors, enriched by a Local LLM to provide actual context, running in parallel with the second grading pass.
The broader lesson here is that a knowledge graph is only as good as the context it holds. By capturing not just the text of the message but the rich data hidden behind the links, we eliminate the friction of re-reading and ensure the user gets the complete picture at a glance.
What's Your Experience?
Have you built a link extraction system for a knowledge graph or a data pipeline? How did you handle the different link types, and what did you use for the ones that resisted simple scraping?
If you have worked with GitHub or YouTube APIs, did you run into rate limits or transcript availability issues? How did you work around them?
Drop your thoughts in the comments below. I read every response, and I am especially curious about cases where a different extraction approach won out for you. If you have written about your own pipeline design, share a link. I would love to learn from your approach.
Let's make this a conversation, not just a case study.
Top comments (0)