<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Siamese_Cat_Dev</title>
    <description>The latest articles on DEV Community by Siamese_Cat_Dev (@lovecatisgoodsudo).</description>
    <link>https://dev.to/lovecatisgoodsudo</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4058618%2F3ba17eab-dad6-4dbb-b6d8-6c69d341e32b.png</url>
      <title>DEV Community: Siamese_Cat_Dev</title>
      <link>https://dev.to/lovecatisgoodsudo</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/lovecatisgoodsudo"/>
    <language>en</language>
    <item>
      <title>Open Canvas: A Free Open Source Infinite Canvas for Storyboarding, Film Planning and Visual Thinking</title>
      <dc:creator>Siamese_Cat_Dev</dc:creator>
      <pubDate>Fri, 14 Aug 2026 17:00:40 +0000</pubDate>
      <link>https://dev.to/lovecatisgoodsudo/open-canvas-a-free-open-source-infinite-canvas-for-storyboarding-film-planning-and-visual-thinking-919</link>
      <guid>https://dev.to/lovecatisgoodsudo/open-canvas-a-free-open-source-infinite-canvas-for-storyboarding-film-planning-and-visual-thinking-919</guid>
      <description>&lt;p&gt;Most creative projects do not begin with a perfectly organized document.&lt;/p&gt;

&lt;p&gt;They begin with scattered ideas.&lt;/p&gt;

&lt;p&gt;A reference image here. A camera angle there. A rough scene description. A character concept. A location photo. Maybe a few arrows connecting things that only make sense inside your head.&lt;/p&gt;

&lt;p&gt;Traditional documents are not particularly good at this kind of thinking.&lt;/p&gt;

&lt;p&gt;That is exactly the problem DJAI Open Canvas is trying to solve.&lt;/p&gt;

&lt;p&gt;DJAI Open Canvas is a free, open-source, local-first infinite canvas built for visual thinking, storyboarding, film pre-production, mood boards, diagrams and shot planning.&lt;/p&gt;

&lt;p&gt;Instead of forcing your ideas into pages, slides or rows, it gives you an open space where you can arrange information visually.&lt;/p&gt;

&lt;p&gt;For filmmakers, video creators, designers and people experimenting with AI filmmaking, that can be surprisingly useful.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is an infinite canvas?
&lt;/h2&gt;

&lt;p&gt;Imagine putting a giant whiteboard on your wall.&lt;/p&gt;

&lt;p&gt;You could place a storyboard in one corner, reference images beside it, notes underneath, camera ideas on the right and a rough flowchart somewhere else.&lt;/p&gt;

&lt;p&gt;You could then zoom out and see the entire project.&lt;/p&gt;

&lt;p&gt;An infinite canvas brings that idea into software.&lt;/p&gt;

&lt;p&gt;There is no traditional page boundary telling you where your ideas need to fit.&lt;/p&gt;

&lt;p&gt;That makes it useful when the relationship between things matters as much as the things themselves.&lt;/p&gt;

&lt;p&gt;A filmmaker might organize a project like this:&lt;/p&gt;

&lt;p&gt;Opening scene&lt;/p&gt;

&lt;p&gt;Character references&lt;/p&gt;

&lt;p&gt;Location inspiration&lt;/p&gt;

&lt;p&gt;Camera angles&lt;/p&gt;

&lt;p&gt;Lighting ideas&lt;/p&gt;

&lt;p&gt;Storyboard frames&lt;/p&gt;

&lt;p&gt;Shot order&lt;/p&gt;

&lt;p&gt;Visual effects references&lt;/p&gt;

&lt;p&gt;Production notes&lt;/p&gt;

&lt;p&gt;Instead of keeping all of those things in separate folders and applications, they can become part of one visual workspace.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why I think this is especially interesting for filmmaking
&lt;/h2&gt;

&lt;p&gt;Film production is extremely visual.&lt;/p&gt;

&lt;p&gt;A written description such as "wide shot of the character entering a dark warehouse" communicates an idea, but a director, cinematographer, editor and production designer may still imagine four completely different scenes.&lt;/p&gt;

&lt;p&gt;A storyboard helps turn that abstract idea into something everyone can see.&lt;/p&gt;

&lt;p&gt;That is why visual planning is such an important part of pre-production.&lt;/p&gt;

&lt;p&gt;DJAI Open Canvas takes the idea further by giving creators space not only for storyboard frames, but also for the information surrounding those frames.&lt;/p&gt;

&lt;p&gt;You could place a character reference next to a scene.&lt;/p&gt;

&lt;p&gt;Put lighting references underneath it.&lt;/p&gt;

&lt;p&gt;Add notes about camera movement.&lt;/p&gt;

&lt;p&gt;Then connect that scene to the next part of the story.&lt;/p&gt;

&lt;p&gt;You are effectively building a visual map of the production.&lt;/p&gt;

&lt;p&gt;For independent filmmakers and small teams, that can be much easier to understand than a collection of disconnected documents.&lt;/p&gt;

&lt;h2&gt;
  
  
  It could also be useful for AI filmmaking
&lt;/h2&gt;

&lt;p&gt;AI video generation has made visual planning even more important.&lt;/p&gt;

&lt;p&gt;Generating a good-looking eight-second clip is one problem.&lt;/p&gt;

&lt;p&gt;Generating 30 clips that actually look like they belong in the same film is a completely different problem.&lt;/p&gt;

&lt;p&gt;You need to think about character consistency, locations, shot progression, costumes, lighting, camera direction and the visual language of the overall production.&lt;/p&gt;

&lt;p&gt;An infinite storyboard canvas gives you somewhere to organize those decisions before you start generating everything.&lt;/p&gt;

&lt;p&gt;For example, you could create a section for each scene and keep the following information together:&lt;/p&gt;

&lt;p&gt;Reference image&lt;/p&gt;

&lt;p&gt;Storyboard frame&lt;/p&gt;

&lt;p&gt;Prompt concept&lt;/p&gt;

&lt;p&gt;Character appearance&lt;/p&gt;

&lt;p&gt;Environment&lt;/p&gt;

&lt;p&gt;Camera angle&lt;/p&gt;

&lt;p&gt;Camera movement&lt;/p&gt;

&lt;p&gt;Lighting&lt;/p&gt;

&lt;p&gt;Expected duration&lt;/p&gt;

&lt;p&gt;Transition into the next shot&lt;/p&gt;

&lt;p&gt;Suddenly AI video generation stops being a random prompt-and-hope process and starts looking more like actual pre-production.&lt;/p&gt;

&lt;p&gt;That is where I think tools like DJAI Open Canvas become particularly interesting.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why open source matters
&lt;/h2&gt;

&lt;p&gt;There are already plenty of online whiteboards, design platforms and storyboard applications.&lt;/p&gt;

&lt;p&gt;DJAI Open Canvas is interesting for another reason: it is open source.&lt;/p&gt;

&lt;p&gt;The repository is publicly available on GitHub.&lt;/p&gt;

&lt;p&gt;That matters because developers are not limited to whatever features a SaaS company decides to provide.&lt;/p&gt;

&lt;p&gt;You can inspect the project.&lt;/p&gt;

&lt;p&gt;You can modify it.&lt;/p&gt;

&lt;p&gt;You can build on top of it.&lt;/p&gt;

&lt;p&gt;You can contribute improvements.&lt;/p&gt;

&lt;p&gt;You can adapt the concept to your own workflow.&lt;/p&gt;

&lt;p&gt;And because the project describes itself as local-first, it is designed around the idea that your working environment does not always need to revolve around sending everything to somebody else's cloud service.&lt;/p&gt;

&lt;p&gt;For developers and technical teams, that opens up some interesting possibilities.&lt;/p&gt;

&lt;p&gt;A production company could eventually adapt the interface around its own workflow.&lt;/p&gt;

&lt;p&gt;A university could experiment with it as a visual teaching tool.&lt;/p&gt;

&lt;p&gt;A developer could extend it for game planning.&lt;/p&gt;

&lt;p&gt;An AI filmmaker could build integrations around image or video generation.&lt;/p&gt;

&lt;p&gt;A creative agency could create its own internal planning environment.&lt;/p&gt;

&lt;p&gt;Open source changes the question from "Can this software do what I want?" to "Can I make this software do what I want?"&lt;/p&gt;

&lt;p&gt;That is a much more interesting question.&lt;/p&gt;

&lt;h2&gt;
  
  
  You do not need to be a developer to understand the idea
&lt;/h2&gt;

&lt;p&gt;Open source projects sometimes make themselves sound unnecessarily complicated.&lt;/p&gt;

&lt;p&gt;The basic idea here is actually simple.&lt;/p&gt;

&lt;p&gt;DJAI Open Canvas gives you a large visual workspace where you can organize a project spatially instead of forcing everything into documents.&lt;/p&gt;

&lt;p&gt;If you have ever used sticky notes on a wall to plan something, you already understand the concept.&lt;/p&gt;

&lt;p&gt;The difference is that your wall can now keep expanding.&lt;/p&gt;

&lt;p&gt;For a filmmaker, that wall becomes a storyboard.&lt;/p&gt;

&lt;p&gt;For a designer, it becomes a mood board.&lt;/p&gt;

&lt;p&gt;For a developer, it could become a system diagram.&lt;/p&gt;

&lt;p&gt;For a content creator, it might become a video planning board.&lt;/p&gt;

&lt;p&gt;For an AI filmmaker, it could become the control center for references, prompts and generated scenes.&lt;/p&gt;

&lt;p&gt;That flexibility is the interesting part.&lt;/p&gt;

&lt;h2&gt;
  
  
  A useful open-source alternative in a world of subscriptions
&lt;/h2&gt;

&lt;p&gt;Creative software is increasingly moving toward subscription models.&lt;/p&gt;

&lt;p&gt;Sometimes those subscriptions are worth paying for.&lt;/p&gt;

&lt;p&gt;But there is also value in having open-source alternatives that people can learn from, modify and contribute to.&lt;/p&gt;

&lt;p&gt;DJAI Open Canvas is still a young project, and it should be viewed that way.&lt;/p&gt;

&lt;p&gt;This is not about pretending that a community project automatically replaces every mature commercial product.&lt;/p&gt;

&lt;p&gt;The exciting part is that the foundation is open.&lt;/p&gt;

&lt;p&gt;People can experiment with it.&lt;/p&gt;

&lt;p&gt;Developers can improve it.&lt;/p&gt;

&lt;p&gt;Creators can test new workflows.&lt;/p&gt;

&lt;p&gt;And the community can help determine where it goes next.&lt;/p&gt;

&lt;p&gt;That is how interesting open-source tools often begin.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where to find DJAI Open Canvas
&lt;/h2&gt;

&lt;p&gt;The project is available on GitHub under:&lt;/p&gt;

&lt;p&gt;lovecatisgood-sudo/djai-open-canvas&lt;/p&gt;

&lt;p&gt;If you are interested in open-source storyboard software, infinite canvas tools, visual project planning, AI filmmaking workflows or film pre-production software, it is worth checking out.&lt;/p&gt;

&lt;p&gt;Clone it if you want to experiment.&lt;/p&gt;

&lt;p&gt;Fork it if you want to build something different.&lt;/p&gt;

&lt;p&gt;Open an issue if you find something that could be better.&lt;/p&gt;

&lt;p&gt;Contribute if you have an idea worth adding.&lt;/p&gt;

&lt;p&gt;And if you simply like seeing free creative tools being developed in public, give the repository a star.&lt;/p&gt;

&lt;p&gt;DJAI Open Canvas is still early, but the concept makes a lot of sense: give creators an open visual space, let them organize ideas their own way, and make the underlying project open enough for developers to take it further.&lt;/p&gt;

&lt;p&gt;For filmmakers trying to organize increasingly complicated visual workflows, especially people experimenting with AI-generated film and video, that could become very useful.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>vibecoding</category>
      <category>opensource</category>
      <category>web</category>
    </item>
    <item>
      <title>The turning point is here, real acceleration in AI redefining humanity.</title>
      <dc:creator>Siamese_Cat_Dev</dc:creator>
      <pubDate>Mon, 10 Aug 2026 04:34:13 +0000</pubDate>
      <link>https://dev.to/lovecatisgoodsudo/the-turning-point-is-here-real-acceleration-in-ai-redefining-humanity-3clf</link>
      <guid>https://dev.to/lovecatisgoodsudo/the-turning-point-is-here-real-acceleration-in-ai-redefining-humanity-3clf</guid>
      <description>&lt;h1&gt;
  
  
  AI Is Starting to Change How Scientific Research Works
&lt;/h1&gt;

&lt;p&gt;Most of the attention around AI still goes to things that are easy to see. A new image model comes out and suddenly the images look more realistic. A coding model gets better and someone posts a video of it building an app in ten minutes.&lt;/p&gt;

&lt;p&gt;I follow all of that because I work with AI and software myself, but lately I’ve been much more interested in what is happening in scientific research.&lt;/p&gt;

&lt;p&gt;Some AI systems are starting to do more than search papers, summarize information or answer questions. They are beginning to take part in the research process itself.&lt;/p&gt;

&lt;p&gt;Google’s Co-Scientist is a good example. It can generate scientific hypotheses, but generating ideas is not really the impressive part anymore. Modern language models can come up with plenty of plausible ideas if you give them a problem.&lt;/p&gt;

&lt;p&gt;The difficult part is figuring out which ideas are actually worth testing.&lt;/p&gt;

&lt;p&gt;Co-Scientist approaches this by having different parts of the system generate ideas, criticize them, compare similar hypotheses and gradually improve the stronger ones. I find this much more interesting than simply asking an AI for its best answer, especially when language models can still produce something completely wrong while sounding very confident.&lt;/p&gt;

&lt;p&gt;In science, eventually someone has to test the idea and see if it actually works.&lt;/p&gt;

&lt;p&gt;Researchers have already explored systems like this in areas including cancer research and drug repurposing. This doesn't mean AI has suddenly cured cancer, and I think headlines sometimes make these results sound much more dramatic than they really are.&lt;/p&gt;

&lt;p&gt;What matters is that AI can help researchers search through possibilities.&lt;/p&gt;

&lt;p&gt;There are an enormous number of drugs, proteins, mutations, cell types and combinations that scientists could investigate. Testing everything in a laboratory is impossible. Experiments cost money and take time.&lt;/p&gt;

&lt;p&gt;If AI can help move an unusual but promising candidate higher up the list, researchers can spend more of their time testing better guesses.&lt;/p&gt;

&lt;p&gt;That alone could be extremely useful.&lt;/p&gt;

&lt;p&gt;Other systems are starting to go further.&lt;/p&gt;

&lt;p&gt;Robin, for example, has been used in research where the AI doesn't simply suggest something to test. It can also work with experimental results, help write analysis code, create visualizations and use what happened in one experiment to help decide what should be investigated next.&lt;/p&gt;

&lt;p&gt;This is where I think things get particularly interesting.&lt;/p&gt;

&lt;p&gt;Scientific research is a loop. You start with an idea, run an experiment, collect data, analyze what happened and then decide what to try next.&lt;/p&gt;

&lt;p&gt;A lot of time is spent between those steps.&lt;/p&gt;

&lt;p&gt;If AI can shorten that process, researchers may be able to try more ideas in the same amount of time. An experiment that creates another question can quickly lead to another experiment, and several possibilities could potentially be explored at once.&lt;/p&gt;

&lt;p&gt;The important development may not be an AI suddenly producing one brilliant scientific answer. It may simply be that researchers can move through this cycle much faster.&lt;/p&gt;

&lt;p&gt;Another system I’ve been following is Mammal, which tries to learn relationships between different kinds of biological information.&lt;/p&gt;

&lt;p&gt;That makes sense because biology doesn't really care about the categories humans use to organize it. A molecule interacts with a protein, that protein affects a cellular pathway, gene expression changes and the cell behaves differently.&lt;/p&gt;

&lt;p&gt;Those relationships cross several areas of biology.&lt;/p&gt;

&lt;p&gt;A model that can work across those layers may notice useful connections that would otherwise take researchers much longer to find.&lt;/p&gt;

&lt;p&gt;Mammal has been tested on things like predicting how cancer cell lines respond to drugs and generating parts of antibody designs. Again, these are early research results. A drug working on cells in a laboratory is very different from that drug working safely in a human patient.&lt;/p&gt;

&lt;p&gt;But that isn't really the point.&lt;/p&gt;

&lt;p&gt;The interesting part is that AI can help researchers decide where to look.&lt;/p&gt;

&lt;p&gt;When you put systems like these together, you can start to imagine how scientific research might change.&lt;/p&gt;

&lt;p&gt;A researcher begins with a problem. AI searches existing research and suggests possible explanations. Another model helps rank compounds or generate potential designs. Researchers test the most promising candidates. The resulting data goes back into the system, which helps analyze what happened and suggests what might be worth trying next.&lt;/p&gt;

&lt;p&gt;Then the process repeats.&lt;/p&gt;

&lt;p&gt;Laboratory automation could eventually make that loop even faster.&lt;/p&gt;

&lt;p&gt;I don't think this means scientists disappear. Someone still needs to understand whether a result makes sense, decide which questions are worth asking and recognize when something unexpected deserves further investigation.&lt;/p&gt;

&lt;p&gt;AI is useful here because humans have limits. Researchers specialize, nobody can read every paper ever published, and there are simply too many possible experiments to test everything.&lt;/p&gt;

&lt;p&gt;The machine doesn't need to become the scientist.&lt;/p&gt;

&lt;p&gt;It can help scientists search a much larger space.&lt;/p&gt;

&lt;p&gt;There are obviously risks as these systems become more capable. Models hallucinate, datasets contain biases and experimental results don't always reproduce. Something that works in cultured cells can fail in animals, and something that works in animals can still fail in humans.&lt;/p&gt;

&lt;p&gt;Biology is also an area where increasingly capable AI systems need serious safety controls and human oversight.&lt;/p&gt;

&lt;p&gt;So I don't think every new AI biology paper should be treated as a breakthrough.&lt;/p&gt;

&lt;p&gt;But I do think the overall direction is worth paying attention to.&lt;/p&gt;

&lt;p&gt;A few years ago, the interesting question was whether AI could answer scientific questions.&lt;/p&gt;

&lt;p&gt;Now researchers are beginning to explore whether AI can help decide which question should be asked next.&lt;/p&gt;

&lt;p&gt;For me, that is a much more important development.&lt;/p&gt;

&lt;p&gt;The biggest impact of AI on science may not eventually be one famous AI-discovered drug. It may be thousands of smaller improvements to the way research is done, allowing scientists to test ideas, learn from the results and move on to the next experiment faster.&lt;/p&gt;

&lt;p&gt;We are still very early, but AI becoming part of the scientific process isn't really a future prediction anymore.&lt;/p&gt;

&lt;p&gt;It has already started.&lt;/p&gt;

&lt;p&gt;Full video : &lt;a href="https://www.youtube.com/watch?v=rwujwQn3wOg&amp;amp;t=1102s" rel="noopener noreferrer"&gt;https://www.youtube.com/watch?v=rwujwQn3wOg&amp;amp;t=1102s&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I build and experiment with AI and software to help my Siamese Cat Cafe :&lt;a href="https://siamesecat.cafe/" rel="noopener noreferrer"&gt;https://siamesecat.cafe/&lt;/a&gt; which could help you saving hundreds of dollar every year on SEO : &lt;a href="https://www.youtube.com/watch?v=DNlpgX7MAv4&amp;amp;t=5s" rel="noopener noreferrer"&gt;https://www.youtube.com/watch?v=DNlpgX7MAv4&amp;amp;t=5s&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I also created the [Siamese Cat Dev course at DJAI Academy : &lt;a href="https://djai.academy/siamese_cat/dev/course" rel="noopener noreferrer"&gt;https://djai.academy/siamese_cat/dev/course&lt;/a&gt;, where I focus on learning how to actually build with AI by connecting models, code, data and tools instead of simply learning whichever AI product happens to be popular right now.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>vibecoding</category>
      <category>claude</category>
      <category>chatgpt</category>
    </item>
    <item>
      <title>Building a Free, Local-First SEO Crawler for Developers and Small Businesses</title>
      <dc:creator>Siamese_Cat_Dev</dc:creator>
      <pubDate>Sun, 02 Aug 2026 03:26:08 +0000</pubDate>
      <link>https://dev.to/lovecatisgoodsudo/building-a-free-local-first-seo-crawler-for-developers-and-small-businesses-em4</link>
      <guid>https://dev.to/lovecatisgoodsudo/building-a-free-local-first-seo-crawler-for-developers-and-small-businesses-em4</guid>
      <description>&lt;p&gt;Screaming Toad , not a frog. An alternative to Screaming Frog and Ashrefs have been built. Opensource and self hosted.&lt;/p&gt;

&lt;p&gt;A technical SEO crawler can save hours of manual work.&lt;/p&gt;

&lt;p&gt;It can find broken links, missing titles, indexing problems, duplicate pages, redirect issues, and pages that search engines may struggle to understand.&lt;/p&gt;

&lt;p&gt;The problem is that many professional SEO tools require a recurring subscription.&lt;/p&gt;

&lt;p&gt;That may be reasonable for a large SEO agency, but it can be difficult to justify for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Independent developers&lt;/li&gt;
&lt;li&gt;Students&lt;/li&gt;
&lt;li&gt;Small agencies&lt;/li&gt;
&lt;li&gt;Local businesses&lt;/li&gt;
&lt;li&gt;Open-source projects&lt;/li&gt;
&lt;li&gt;Founders managing only a few websites&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This was the motivation behind &lt;strong&gt;SEO Screaming Toad — Not Frog&lt;/strong&gt;, an open-source technical SEO crawler designed to run locally.&lt;/p&gt;

&lt;p&gt;The goal is simple:&lt;/p&gt;

&lt;p&gt;Crawl a website, identify technical problems, preserve the evidence, and help the user understand what should be fixed.&lt;/p&gt;

&lt;p&gt;The project combines a Go crawler, SQLite persistence, a React dashboard, a local API, a command-line interface, and an MCP server for AI-assisted workflows.&lt;/p&gt;

&lt;p&gt;But the main benefit is not the technology.&lt;/p&gt;

&lt;p&gt;The benefit is being able to inspect and improve a website without paying for every crawl or uploading the crawl database to a third-party platform.&lt;/p&gt;

&lt;p&gt;Why Build Another SEO Crawler?&lt;/p&gt;

&lt;p&gt;There are already excellent commercial SEO tools available.&lt;/p&gt;

&lt;p&gt;They are mature, powerful, and widely used.&lt;/p&gt;

&lt;p&gt;However, many developers and small businesses do not need an entire enterprise SEO platform.&lt;/p&gt;

&lt;p&gt;Sometimes they only need answers to a few practical questions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Are there broken links?&lt;/li&gt;
&lt;li&gt;Can search engines reach the important pages?&lt;/li&gt;
&lt;li&gt;Are page titles missing or duplicated?&lt;/li&gt;
&lt;li&gt;Are canonical tags configured correctly?&lt;/li&gt;
&lt;li&gt;Are redirects creating unnecessary chains?&lt;/li&gt;
&lt;li&gt;Is important content visible only after JavaScript runs?&lt;/li&gt;
&lt;li&gt;Did the latest website update introduce new problems?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A local crawler can answer these questions without requiring a cloud account or recurring subscription.&lt;/p&gt;

&lt;p&gt;It also gives developers more control over the data and the implementation.&lt;/p&gt;

&lt;p&gt;Testing the Crawler on Real Websites&lt;/p&gt;

&lt;p&gt;The crawler was tested on two websites with very different purposes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;djai.academy&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;siamesecat.cafe&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;DJAI Academy provides AI education, software-development services, and coding-related content.&lt;/p&gt;

&lt;p&gt;Siamese Cat Café is a local business in Bangkok that depends on customers discovering its location, services, menu, and activities through search.&lt;/p&gt;

&lt;p&gt;At first, both websites appeared functional.&lt;/p&gt;

&lt;p&gt;The pages loaded, the navigation worked, and the content was visible.&lt;/p&gt;

&lt;p&gt;But crawling the sites revealed issues that were easy to miss during normal development.&lt;/p&gt;

&lt;p&gt;These included:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Broken and outdated links&lt;/li&gt;
&lt;li&gt;Weak or duplicated page metadata&lt;/li&gt;
&lt;li&gt;Pages that were difficult for search engines to discover&lt;/li&gt;
&lt;li&gt;Indexing and sitemap inconsistencies&lt;/li&gt;
&lt;li&gt;Internal linking problems&lt;/li&gt;
&lt;li&gt;Page titles that did not clearly describe the content&lt;/li&gt;
&lt;li&gt;Technical differences between raw HTML and rendered content&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;After reviewing and fixing these issues, both websites showed better technical SEO health.&lt;/p&gt;

&lt;p&gt;The improvements included:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Fewer broken links&lt;/li&gt;
&lt;li&gt;Cleaner indexing signals&lt;/li&gt;
&lt;li&gt;Better internal navigation&lt;/li&gt;
&lt;li&gt;Improved SEO audit scores&lt;/li&gt;
&lt;li&gt;Stronger search visibility&lt;/li&gt;
&lt;li&gt;Higher rankings for relevant pages&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The main lesson was that a website can look correct to a visitor while still sending confusing signals to search engines.&lt;/p&gt;

&lt;p&gt;The Real Value of a Technical Audit&lt;/p&gt;

&lt;p&gt;An SEO audit score is useful, but the score itself does not improve a business.&lt;/p&gt;

&lt;p&gt;The value comes from what the score represents.&lt;/p&gt;

&lt;p&gt;For example, fixing a broken link may lead to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Fewer visitors reaching dead pages&lt;/li&gt;
&lt;li&gt;More users reaching a service or product page&lt;/li&gt;
&lt;li&gt;Better internal distribution of link authority&lt;/li&gt;
&lt;li&gt;Easier crawling for search engines&lt;/li&gt;
&lt;li&gt;A more professional user experience&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Fixing an indexing problem may allow a page that already exists to finally appear in search results.&lt;/p&gt;

&lt;p&gt;Improving a title may help both search engines and users understand the page more clearly.&lt;/p&gt;

&lt;p&gt;The benefit is not simply “better SEO.”&lt;/p&gt;

&lt;p&gt;The benefit is helping the right person find the right page.&lt;/p&gt;

&lt;p&gt;Local-First by Design&lt;/p&gt;

&lt;p&gt;The crawler stores its working data locally using SQLite.&lt;/p&gt;

&lt;p&gt;This means the crawl database stays on the user’s machine.&lt;/p&gt;

&lt;p&gt;That provides several advantages:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;No required cloud account&lt;/li&gt;
&lt;li&gt;No URL-credit system&lt;/li&gt;
&lt;li&gt;No subscription needed to reopen previous crawl data&lt;/li&gt;
&lt;li&gt;Better control over client or internal website information&lt;/li&gt;
&lt;li&gt;Easier experimentation for developers&lt;/li&gt;
&lt;li&gt;A simpler setup than running several external infrastructure services&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;SQLite was selected because the application is intended to be easy to run locally.&lt;/p&gt;

&lt;p&gt;A user should not need to configure Redis, PostgreSQL, Kafka, or a distributed queue before checking a small or medium-sized website.&lt;/p&gt;

&lt;p&gt;SQLite also makes the application portable.&lt;/p&gt;

&lt;p&gt;The crawler can store:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Discovered URLs&lt;/li&gt;
&lt;li&gt;Fetch state&lt;/li&gt;
&lt;li&gt;Page metadata&lt;/li&gt;
&lt;li&gt;Links&lt;/li&gt;
&lt;li&gt;Audit findings&lt;/li&gt;
&lt;li&gt;Crawl events&lt;/li&gt;
&lt;li&gt;Checkpoints&lt;/li&gt;
&lt;li&gt;Comparison data&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Write operations are controlled carefully, while the crawling workers can continue fetching pages concurrently.&lt;/p&gt;

&lt;p&gt;Recovering Long Crawls&lt;/p&gt;

&lt;p&gt;One of the more important design goals was recovery.&lt;/p&gt;

&lt;p&gt;A crawler should not lose all progress because:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The application closed&lt;/li&gt;
&lt;li&gt;The machine restarted&lt;/li&gt;
&lt;li&gt;The network failed&lt;/li&gt;
&lt;li&gt;The crawl was paused&lt;/li&gt;
&lt;li&gt;The user cancelled the job&lt;/li&gt;
&lt;li&gt;A rendering worker crashed&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The crawler persists its frontier and crawl state.&lt;/p&gt;

&lt;p&gt;It supports:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Pause&lt;/li&gt;
&lt;li&gt;Resume&lt;/li&gt;
&lt;li&gt;Cancel&lt;/li&gt;
&lt;li&gt;Checkpoints&lt;/li&gt;
&lt;li&gt;Interrupted-run recovery&lt;/li&gt;
&lt;li&gt;Crawl history&lt;/li&gt;
&lt;li&gt;Recrawl comparison&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This makes the crawl database more than a temporary cache.&lt;/p&gt;

&lt;p&gt;It becomes a record of what was discovered, what was processed, and what still remains.&lt;/p&gt;

&lt;p&gt;For larger websites, that is much more useful than restarting from zero after every interruption.&lt;/p&gt;

&lt;p&gt;Raw HTML and Rendered Content Are Different&lt;/p&gt;

&lt;p&gt;Modern websites often rely heavily on JavaScript.&lt;/p&gt;

&lt;p&gt;A search crawler may receive one version of a page from the server and a different version after JavaScript executes.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The server response may contain no title&lt;/li&gt;
&lt;li&gt;JavaScript may insert a title later&lt;/li&gt;
&lt;li&gt;Canonical tags may be added client-side&lt;/li&gt;
&lt;li&gt;Links may appear only after rendering&lt;/li&gt;
&lt;li&gt;Important content may not exist in the original HTML&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It is tempting to render the page and treat the result as the only version.&lt;/p&gt;

&lt;p&gt;However, that can hide useful evidence.&lt;/p&gt;

&lt;p&gt;SEO Screaming Toad keeps raw and rendered evidence separate.&lt;/p&gt;

&lt;p&gt;This allows the user to inspect:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;What the server originally returned&lt;/li&gt;
&lt;li&gt;What appeared after rendering&lt;/li&gt;
&lt;li&gt;Which version triggered an audit finding&lt;/li&gt;
&lt;li&gt;Whether JavaScript improved or damaged the page’s SEO signals&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This distinction is especially useful when debugging React, Vue, Next.js, and other JavaScript-heavy websites.&lt;/p&gt;

&lt;p&gt;Evidence Instead of Only Scores&lt;/p&gt;

&lt;p&gt;Many audit tools present a list of problems, but it may not always be obvious why a page was flagged.&lt;/p&gt;

&lt;p&gt;The crawler attempts to preserve evidence for each finding.&lt;/p&gt;

&lt;p&gt;A finding can include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The affected URL&lt;/li&gt;
&lt;li&gt;Rule identifier&lt;/li&gt;
&lt;li&gt;Rule version&lt;/li&gt;
&lt;li&gt;Severity&lt;/li&gt;
&lt;li&gt;Extracted evidence&lt;/li&gt;
&lt;li&gt;Suggested remediation&lt;/li&gt;
&lt;li&gt;Known limitations&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example, a duplicate-title finding should not only say that duplicates exist.&lt;/p&gt;

&lt;p&gt;It should identify the affected pages and preserve the titles that caused the rule to trigger.&lt;/p&gt;

&lt;p&gt;A canonical issue should show the observed canonical value and the target involved.&lt;/p&gt;

&lt;p&gt;This makes the audit easier to verify.&lt;/p&gt;

&lt;p&gt;It also reduces the risk of blindly applying automated recommendations.&lt;/p&gt;

&lt;p&gt;Audit Coverage&lt;/p&gt;

&lt;p&gt;The current audit engine checks several common areas of technical SEO.&lt;/p&gt;

&lt;p&gt;These include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;HTTP errors&lt;/li&gt;
&lt;li&gt;Broken internal links&lt;/li&gt;
&lt;li&gt;Redirects&lt;/li&gt;
&lt;li&gt;Page titles&lt;/li&gt;
&lt;li&gt;Meta descriptions&lt;/li&gt;
&lt;li&gt;Heading structure&lt;/li&gt;
&lt;li&gt;Canonical tags&lt;/li&gt;
&lt;li&gt;Indexability&lt;/li&gt;
&lt;li&gt;Robots directives&lt;/li&gt;
&lt;li&gt;Robots.txt&lt;/li&gt;
&lt;li&gt;Sitemap coverage&lt;/li&gt;
&lt;li&gt;Hreflang&lt;/li&gt;
&lt;li&gt;Duplicate content signals&lt;/li&gt;
&lt;li&gt;Image alt attributes&lt;/li&gt;
&lt;li&gt;Broken image resources&lt;/li&gt;
&lt;li&gt;Crawl depth&lt;/li&gt;
&lt;li&gt;Internal linking&lt;/li&gt;
&lt;li&gt;Mixed content&lt;/li&gt;
&lt;li&gt;Selected security headers&lt;/li&gt;
&lt;li&gt;JSON-LD structured data&lt;/li&gt;
&lt;li&gt;Raw-versus-rendered differences&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The project does not claim that every warning automatically affects ranking.&lt;/p&gt;

&lt;p&gt;Technical findings should be reviewed in context.&lt;/p&gt;

&lt;p&gt;For example, a missing page from a sitemap may be intentional. A duplicate title may be acceptable for certain filtered pages. A blocked page may be blocked by design.&lt;/p&gt;

&lt;p&gt;The tool should provide evidence, not pretend to replace human judgment.&lt;/p&gt;

&lt;p&gt;MCP Support for AI Agents&lt;/p&gt;

&lt;p&gt;The project also includes a Model Context Protocol server.&lt;/p&gt;

&lt;p&gt;This allows compatible AI clients to interact with the crawler through bounded tools.&lt;/p&gt;

&lt;p&gt;An AI client can perform tasks such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Create a crawl profile&lt;/li&gt;
&lt;li&gt;Preview crawl scope&lt;/li&gt;
&lt;li&gt;Start a crawl&lt;/li&gt;
&lt;li&gt;Pause or resume a job&lt;/li&gt;
&lt;li&gt;List pages&lt;/li&gt;
&lt;li&gt;Inspect links&lt;/li&gt;
&lt;li&gt;Review findings&lt;/li&gt;
&lt;li&gt;Explain a rule&lt;/li&gt;
&lt;li&gt;Compare two crawls&lt;/li&gt;
&lt;li&gt;Generate an export&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The important word is &lt;strong&gt;bounded&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;The MCP server does not provide:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;General shell execution&lt;/li&gt;
&lt;li&gt;Arbitrary SQL queries&lt;/li&gt;
&lt;li&gt;Unrestricted filesystem access&lt;/li&gt;
&lt;li&gt;Unrestricted HTTP requests&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Instead, it exposes specific SEO and crawl operations.&lt;/p&gt;

&lt;p&gt;This allows an AI assistant to help analyze an audit without giving it unnecessary access to the host machine.&lt;/p&gt;

&lt;p&gt;A possible workflow could look like this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Start a bounded crawl.&lt;/li&gt;
&lt;li&gt;Wait for completion.&lt;/li&gt;
&lt;li&gt;List high-severity findings.&lt;/li&gt;
&lt;li&gt;Group issues by page type.&lt;/li&gt;
&lt;li&gt;Explain the most common rule.&lt;/li&gt;
&lt;li&gt;Export the results.&lt;/li&gt;
&lt;li&gt;Compare the crawl after fixes are deployed.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The goal is not to let AI control everything.&lt;/p&gt;

&lt;p&gt;The goal is to give AI structured access to trustworthy crawl evidence.&lt;/p&gt;

&lt;p&gt;Crawl Safety Matters&lt;/p&gt;

&lt;p&gt;A crawler is also a network tool, so safety controls are important.&lt;/p&gt;

&lt;p&gt;The project includes controls for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Robots.txt enforcement&lt;/li&gt;
&lt;li&gt;Per-host politeness&lt;/li&gt;
&lt;li&gt;Scope restrictions&lt;/li&gt;
&lt;li&gt;URL limits&lt;/li&gt;
&lt;li&gt;Depth limits&lt;/li&gt;
&lt;li&gt;Response-size limits&lt;/li&gt;
&lt;li&gt;Redirect validation&lt;/li&gt;
&lt;li&gt;Timeout budgets&lt;/li&gt;
&lt;li&gt;Retry limits&lt;/li&gt;
&lt;li&gt;DNS and IP validation&lt;/li&gt;
&lt;li&gt;TLS hostname verification&lt;/li&gt;
&lt;li&gt;Crawler-trap protection&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These controls help prevent a simple website audit from turning into an unbounded or unsafe crawl.&lt;/p&gt;

&lt;p&gt;Users should only crawl websites they own or are authorized to inspect.&lt;/p&gt;

&lt;p&gt;What About the 100M+ URL Claim?&lt;/p&gt;

&lt;p&gt;The repository includes an experimental architecture for segmented crawling campaigns.&lt;/p&gt;

&lt;p&gt;The idea is to avoid treating a very large site as one enormous in-memory crawl.&lt;/p&gt;

&lt;p&gt;Instead, work can be divided into bounded segments and processed over time.&lt;/p&gt;

&lt;p&gt;Synthetic persistence and analysis campaigns have completed at 1 million and 5 million URLs.&lt;/p&gt;

&lt;p&gt;However, the 100M+ figure should be understood carefully.&lt;/p&gt;

&lt;p&gt;It is a theoretical architectural target.&lt;/p&gt;

&lt;p&gt;It is not:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A verified live crawl of 100 million URLs&lt;/li&gt;
&lt;li&gt;A guarantee&lt;/li&gt;
&lt;li&gt;A laptop benchmark&lt;/li&gt;
&lt;li&gt;A claim that every website can be crawled at that scale&lt;/li&gt;
&lt;li&gt;A promise of commercial support&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Real-world capacity depends on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Storage&lt;/li&gt;
&lt;li&gt;CPU&lt;/li&gt;
&lt;li&gt;Memory&lt;/li&gt;
&lt;li&gt;Network speed&lt;/li&gt;
&lt;li&gt;Rendering requirements&lt;/li&gt;
&lt;li&gt;Website behavior&lt;/li&gt;
&lt;li&gt;Crawl limits&lt;/li&gt;
&lt;li&gt;Politeness settings&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The large-scale architecture is experimental and should be evaluated separately from the normal local crawler.&lt;/p&gt;

&lt;p&gt;What Improved on the Real Websites?&lt;/p&gt;

&lt;p&gt;The most useful outcome came from applying the crawler to real projects.&lt;/p&gt;

&lt;p&gt;For &lt;code&gt;djai.academy&lt;/code&gt;, the audit helped identify technical problems affecting content discovery and page clarity.&lt;/p&gt;

&lt;p&gt;Fixing those issues improved:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Broken-link coverage&lt;/li&gt;
&lt;li&gt;Page metadata&lt;/li&gt;
&lt;li&gt;Internal linking&lt;/li&gt;
&lt;li&gt;Indexability&lt;/li&gt;
&lt;li&gt;Search visibility&lt;/li&gt;
&lt;li&gt;Rankings for relevant pages&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For "siamesecat.cafe", the improvements were connected directly to local discovery.&lt;/p&gt;

&lt;p&gt;A local business website needs search engines to clearly understand:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;What the business is&lt;/li&gt;
&lt;li&gt;Where it is located&lt;/li&gt;
&lt;li&gt;What services it offers&lt;/li&gt;
&lt;li&gt;Which pages are important&lt;/li&gt;
&lt;li&gt;How users can visit or contact it&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Fixing technical problems gave the site a stronger foundation for appearing in relevant local searches.&lt;/p&gt;

&lt;p&gt;The crawler did not create the business value by itself.&lt;/p&gt;

&lt;p&gt;It revealed where existing value was being hidden by technical mistakes.&lt;/p&gt;

&lt;p&gt;Reflection:&lt;/p&gt;

&lt;p&gt;Building and using the crawler changed the way I think about website maintenance.&lt;/p&gt;

&lt;p&gt;A site is not finished when it is deployed.&lt;/p&gt;

&lt;p&gt;Every new page, redesign, framework update, CMS change, or content edit can introduce problems.&lt;/p&gt;

&lt;p&gt;Technical SEO should therefore be part of the development cycle.&lt;/p&gt;

&lt;p&gt;A useful workflow is:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Crawl the current site.&lt;/li&gt;
&lt;li&gt;Save the results.&lt;/li&gt;
&lt;li&gt;Make a change.&lt;/li&gt;
&lt;li&gt;Crawl it again.&lt;/li&gt;
&lt;li&gt;Compare the findings.&lt;/li&gt;
&lt;li&gt;Fix regressions before they affect users or search visibility.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This is similar to testing code.&lt;/p&gt;

&lt;p&gt;Developers would not normally deploy major changes without checking whether the application still works.&lt;/p&gt;

&lt;p&gt;The same idea should apply to website structure, metadata, indexing, and internal links.&lt;/p&gt;

&lt;p&gt;Project Status&lt;/p&gt;

&lt;p&gt;SEO Screaming Toad is currently a release candidate.&lt;/p&gt;

&lt;p&gt;It is not a stable replacement for every commercial SEO platform.&lt;/p&gt;

&lt;p&gt;It also does not provide the proprietary keyword, competitor, and backlink databases available in broader tools such as Ahrefs.&lt;/p&gt;

&lt;p&gt;Its purpose is narrower:&lt;/p&gt;

&lt;p&gt;Provide an open, local, inspectable way to crawl websites and understand technical SEO problems.&lt;/p&gt;

&lt;p&gt;The project is MIT licensed and open to technical review.&lt;/p&gt;

&lt;p&gt;Repository:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/lovecatisgood-sudo/Free-Opensource-SEO-Screaming-Toad-not-Frog-tool-with-100million-url-crawl-potential" rel="noopener noreferrer"&gt;https://github.com/lovecatisgood-sudo/Free-Opensource-SEO-Screaming-Toad-not-Frog-tool-with-100million-url-crawl-potential&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Websites used for real-world testing:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://djai.academy" rel="noopener noreferrer"&gt;https://djai.academy&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://siamesecat.cafe" rel="noopener noreferrer"&gt;https://siamesecat.cafe&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Final Thoughts&lt;/p&gt;

&lt;p&gt;Small technical problems can quietly reduce the value of an otherwise good website.&lt;/p&gt;

&lt;p&gt;A business may already have useful content, good services, and a strong product.&lt;/p&gt;

&lt;p&gt;But if links are broken, pages are difficult to index, or search engines receive unclear signals, that value becomes harder to discover.&lt;/p&gt;

&lt;p&gt;The most important benefit of a crawler is not producing more data.&lt;/p&gt;

&lt;p&gt;It is showing developers and business owners where their existing work is being wasted.&lt;/p&gt;

&lt;p&gt;A free, local-first crawler lowers the barrier to finding those problems.&lt;/p&gt;

&lt;p&gt;And once the problems are visible, they become much easier to fix.&lt;/p&gt;

&lt;h1&gt;
  
  
  opensource` #seo #webdev #golang
&lt;/h1&gt;

</description>
      <category>opensource</category>
      <category>seo</category>
      <category>software</category>
      <category>tools</category>
    </item>
  </channel>
</rss>
