<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: vianel</title>
    <description>The latest articles on DEV Community by vianel (@vianeltxt).</description>
    <link>https://dev.to/vianeltxt</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F23242%2F653cfd6e-86bd-4f13-8ade-90569a108d01.jpg</url>
      <title>DEV Community: vianel</title>
      <link>https://dev.to/vianeltxt</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/vianeltxt"/>
    <language>en</language>
    <item>
      <title>Why I'm sticking with herdr over Orca as my ADE</title>
      <dc:creator>vianel</dc:creator>
      <pubDate>Tue, 13 Oct 2026 15:06:20 +0000</pubDate>
      <link>https://dev.to/vianeltxt/why-im-sticking-with-herdr-over-orca-as-my-ade-4a43</link>
      <guid>https://dev.to/vianeltxt/why-im-sticking-with-herdr-over-orca-as-my-ade-4a43</guid>
      <description>&lt;p&gt;We're all in the same boat right now: trying to figure out the best development setup. We've always done this in the past with VS Code extensions, LSPs, dotfiles but the question has changed. Now it's &lt;em&gt;how do I connect, optimize and orchestrate a bunch of coding agents on my machine?&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;As a Vim/Neovim user, my first pick for an ADE (agentic development environment) was &lt;a href="https://herdr.dev/" rel="noopener noreferrer"&gt;herdr&lt;/a&gt;. I've been using it for a few weeks and it's been working great. Then a new competitor showed up, &lt;a href="https://www.onorca.dev/" rel="noopener noreferrer"&gt;Orca&lt;/a&gt;, so I gave it a fair shot: I read the docs, installed it, and ran my usual work through it.&lt;/p&gt;

&lt;p&gt;I'm still sticking with herdr. Here's why.&lt;/p&gt;

&lt;h2&gt;
  
  
  Orca is genuinely good
&lt;/h2&gt;

&lt;p&gt;Let me be fair first. Orca's GUI is polished and friendly. Seeing Claude and Codex usage and rate-limit resets at a glance is great, worktrees are front and centre, and features like fanning one prompt out to several agents and annotating AI diffs line by line are excellent.&lt;/p&gt;

&lt;p&gt;But I live in the terminal. Neovim, tmux, CLIs is just home for me. Even though Orca's GUI does a good job of feeling like a terminal, it &lt;em&gt;isn't&lt;/em&gt; my terminal, and switching to a separate app always felt slightly off for me.&lt;/p&gt;

&lt;h2&gt;
  
  
  What herdr gives me that fits my workflow
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;It runs inside the terminal I already use.&lt;/strong&gt; herdr is a single Rust binary, no Electron. It runs in my terminal next to Neovim, with my fonts, my theme, my keybindings.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Coming from tmux, there was nothing to learn.&lt;/strong&gt; Prefix keys, splits, &lt;code&gt;ctrl+b q&lt;/code&gt; to detach. My muscle memory just worked on day one. (And the mouse works too when I feel like clicking.)&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Agents can drive it, not just me.&lt;/strong&gt; This is the big one. herdr exposes a CLI that agents can use to spawn panes, prompt each other, and wait until another agent finishes or gets blocked. My main Claude Code session hands mechanical work to Codex on its own, based on routing rules in my &lt;code&gt;CLAUDE.md&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;| long-running or needs supervision | gpt-5.5 | herdr pane — see "herdr delegation" |

&lt;span class="gu"&gt;### herdr delegation&lt;/span&gt;
&lt;span class="p"&gt;-&lt;/span&gt; Prompt with &lt;span class="sb"&gt;`herdr agent prompt &amp;lt;name&amp;gt; "…" --wait`&lt;/span&gt;. Never sleep/poll.
&lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="sb"&gt;`blocked`&lt;/span&gt; = approval/question UI → show me what it's asking; don't approve it yourself.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Claude opens a Codex pane, waits on its state (no polling, no wasted tokens), and stops to ask me whenever Codex hits an approval prompt.&lt;/p&gt;

&lt;p&gt;Here in this small screenshot the codex panes were open by claude&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Foax1s18ktqeetljg1j0b.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Foax1s18ktqeetljg1j0b.png" alt=" " width="800" height="794"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;I always know which agent is stuck.&lt;/strong&gt; Every pane is marked working, blocked or idle. When something needs me, herdr tells me (I even gave Codex its own sound).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Closing the window doesn't kill the work.&lt;/strong&gt; Sessions live on a background server, so I can detach, close my laptop or drop my SSH connection, and everything keeps running. &lt;code&gt;herdr&lt;/code&gt; reattaches.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;My whole setup is a TOML file.&lt;/strong&gt; It lives in my dotfiles, so moving from my Mac to Linux, or onto a remote box over SSH, is completely seamless. Same keys, same popups, same everything.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It doesn't care which agent I use.&lt;/strong&gt; Claude Code, Codex, Gemini, Cursor, OpenCode and a long list more. herdr doesn't wrap them; it just owns their terminals.&lt;/p&gt;

&lt;h2&gt;
  
  
  Stealing Orca's best ideas
&lt;/h2&gt;

&lt;p&gt;Trying Orca wasn't wasted time. I took its best ideas and rebuilt them on top of herdr.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Fan-out.&lt;/strong&gt; One prompt, several agents, each in its own git worktree, then compare and merge the winner. In herdr that's a Claude Code skill: it creates the worktrees, starts the agents with &lt;code&gt;herdr agent start&lt;/code&gt;, prompts them in parallel, runs the tests in each, and gives me a comparison table. Nothing gets merged until I approve.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;/fanout “add retry with backoff to the S3 uploader” &lt;span class="nt"&gt;--agents&lt;/span&gt; codex,codex,opus
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Diff review.&lt;/strong&gt; Orca's annotate the diff feature is chef's kiss. My version is a small script bound to a herdr popup. It opens the diff in Neovim, I add &lt;code&gt;#!&lt;/code&gt; comments under the lines I care about, and on save they're sent to whichever agent I pick, with file and line references.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight toml"&gt;&lt;code&gt;&lt;span class="nn"&gt;[[keys.command]]&lt;/span&gt;
&lt;span class="py"&gt;key&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"prefix+alt+r"&lt;/span&gt;
&lt;span class="py"&gt;type&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"popup"&lt;/span&gt;
&lt;span class="py"&gt;command&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"herdr-review"&lt;/span&gt;
&lt;span class="py"&gt;width&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"90%"&lt;/span&gt;
&lt;span class="py"&gt;height&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"90%"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I use the same popup trick for &lt;code&gt;gh dash&lt;/code&gt; and &lt;code&gt;lazygit&lt;/code&gt;, so PRs and diffs are one keystroke away without leaving the terminal.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Phone notifications.&lt;/strong&gt; Doable with a hook and a push service, but honestly I don't want it. When I'm away from my desk, I want to be away and disconnect so checking on agents from my phone isn't a feature I'm missing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Design Mode.&lt;/strong&gt; Clicking a UI element to send its HTML and CSS to an agent is clever. I'm not doing frontend work lately i feel like i could use it in the future for sure and a browser MCP gets me close enough when I need it.&lt;/p&gt;

&lt;h2&gt;
  
  
  So which one should you use?
&lt;/h2&gt;

&lt;p&gt;If you like a GUI, want everything built in, and prefer to steer agents by hand, Orca is a great choice.&lt;/p&gt;

&lt;p&gt;If you live in the terminal, want agents that can orchestrate other agents, and like owning your setup as config files, herdr is hard to beat. The features it doesn't ship with, you can build in an afternoon.&lt;/p&gt;

&lt;p&gt;For me, the deciding question was simple: do I want a new app, or a better terminal? I wanted a better terminal.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;What's your ADE setup? I'd love to hear what's working for you.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>terminal</category>
      <category>neovim</category>
      <category>productivity</category>
    </item>
    <item>
      <title>We scanned 69 random Shopify stores for accessibility. 97% had detectable failures.</title>
      <dc:creator>vianel</dc:creator>
      <pubDate>Wed, 30 Sep 2026 16:18:42 +0000</pubDate>
      <link>https://dev.to/vianeltxt/we-scanned-69-random-shopify-stores-for-accessibility-97-had-detectable-failures-mnb</link>
      <guid>https://dev.to/vianeltxt/we-scanned-69-random-shopify-stores-for-accessibility-97-had-detectable-failures-mnb</guid>
      <description>&lt;p&gt;Most accessibility studies look at the biggest sites. We wanted ordinary Shopify stores, and not just home pages.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Method:&lt;/strong&gt; we shuffled the .com and .us domains in the Tranco top-1M list (V349N), found 130 live Shopify stores selling at least one product, and scanned four pages of each (home, a collection, a product, the cart) with axe-core 4.13 and the WCAG 2.0/2.1/2.2 A and AA rules. 69 stores loaded at least 3 of 4 pages; most of the rest were cut off by Shopify's per-visitor rate limit.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What we found:&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Problem&lt;/th&gt;
&lt;th&gt;Stores&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Low color contrast&lt;/td&gt;
&lt;td&gt;88%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Links with no accessible name&lt;/td&gt;
&lt;td&gt;61%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Invalid or misused ARIA&lt;/td&gt;
&lt;td&gt;51%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tap targets too small (WCAG 2.2)&lt;/td&gt;
&lt;td&gt;51%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Images with no text alternative&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Keyboard problems&lt;/td&gt;
&lt;td&gt;35%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Buttons with no accessible name&lt;/td&gt;
&lt;td&gt;25%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Form fields with no label&lt;/td&gt;
&lt;td&gt;22%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pinch-to-zoom turned off&lt;/td&gt;
&lt;td&gt;20%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;97% had at least one detectable failure (WebAIM Million: 95.9% of home pages).&lt;/li&gt;
&lt;li&gt;Product pages (91%) and cart pages (90%) were no cleaner than the home page.&lt;/li&gt;
&lt;li&gt;All 11 stores running an accessibility overlay still had failures. Small sample, but an overlay runs on top of the page and leaves the markup as it was.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Limits:&lt;/strong&gt; automated checks find only part of the problem, and a failure isn't a legal finding. We don't name any store.&lt;/p&gt;

&lt;p&gt;Full write-up, including where each problem is usually fixed in Shopify: &lt;a href="https://adatriage.com/shopify-accessibility-study-2026" rel="noopener noreferrer"&gt;https://adatriage.com/shopify-accessibility-study-2026&lt;/a&gt;&lt;/p&gt;

</description>
      <category>a11y</category>
      <category>shopify</category>
      <category>webdev</category>
      <category>ecommerce</category>
    </item>
    <item>
      <title>Flow to Source implementation with Akka streams</title>
      <dc:creator>vianel</dc:creator>
      <pubDate>Mon, 03 Aug 2020 23:58:30 +0000</pubDate>
      <link>https://dev.to/vianeltxt/flow-to-source-implementation-with-akka-streams-1gbh</link>
      <guid>https://dev.to/vianeltxt/flow-to-source-implementation-with-akka-streams-1gbh</guid>
      <description>&lt;p&gt;&lt;strong&gt;NOTE:&lt;/strong&gt; This post requires previous knowledge of scala using the &lt;a href="https://akka.io/"&gt;Akka&lt;/a&gt; framework specifically &lt;a href="https://doc.akka.io/docs/akka/current/stream/index.html"&gt;Akka streams&lt;/a&gt;.&lt;/p&gt;

&lt;h1&gt;
  
  
  Akka streams structure
&lt;/h1&gt;

&lt;p&gt;The Akka streams module of Akka provides a beautiful DSL that will help you to move data in an asynchronous way. The base of an Akka stream is composed of 3 main components:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source ~&amp;gt; Flow ~&amp;gt; Sink
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;Source:  Which is in charge of emitting the data through the stream, only have outputs ports&lt;/li&gt;
&lt;li&gt;Flow: receive the data, transform it and send it to the next component&lt;/li&gt;
&lt;li&gt;Sink: The end of the stream it will be the last step of any stream, only have input ports.&lt;/li&gt;
&lt;/ul&gt;

&lt;h1&gt;
  
  
  The problem
&lt;/h1&gt;

&lt;p&gt;So far so good but what happens when we need to get the data from a source and this receives the key on the fly? i.e we need to download a file but we don't know the name of the file which arrives frow a previous step and we are using a library to download the file which is a source.&lt;/p&gt;

&lt;h3&gt;
  
  
  FlatMapConcat FTW
&lt;/h3&gt;

&lt;p&gt;The FlatMapConcat flow allows us to transform any input element into a source this means we can connect a flow with a source, something like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Source ~&amp;gt; Flow ~&amp;gt; Source ~&amp;gt; Sink
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;A simple sample would look like this&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1    //Simple Source
2    val src = Source(List(1, 2, 3))
3
4    val fromSrc = b.add(Flow[Int].log("from-Src"))
5
6    //Emit new list per element received
7    val flow = b.add(Flow[Int].flatMapConcat{ element =&amp;gt;
8      Source(List(4, 5, 6))
9    })
10
11    //¯\_(ツ)_/¯ nothing to do, just for sample purposes
12    val map = b.add(Flow[Int].map{ element =&amp;gt;
13      element
14    })
15
16    src ~&amp;gt; fromSrc ~&amp;gt;flow ~&amp;gt; map ~&amp;gt; sink
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;At line 16 in the main flow, we can see the 3rd step of our process is a regular flow but digging into it at line 8 we notice this flow is actually a Source that receives data from the flow FromSrc and emits new data to the map flow. 🙌🏻&lt;/p&gt;

&lt;p&gt;That's it we just connected a source in the middle of a stream that emits new data based on the received data. &lt;/p&gt;

&lt;p&gt;...&lt;br&gt;
In case you want to run the sample check out my gist &lt;a href="https://gist.github.com/vianel/e9a0ac0ae8c542fe0a1709a1501cafc9"&gt;here&lt;/a&gt; &lt;/p&gt;

</description>
      <category>akka</category>
      <category>scala</category>
      <category>functionalprograming</category>
      <category>streams</category>
    </item>
    <item>
      <title>How to build a Web Scraper using golang with colly</title>
      <dc:creator>vianel</dc:creator>
      <pubDate>Sat, 18 Jul 2020 02:49:57 +0000</pubDate>
      <link>https://dev.to/vianeltxt/how-to-build-a-web-scraper-using-golang-with-colly-18lh</link>
      <guid>https://dev.to/vianeltxt/how-to-build-a-web-scraper-using-golang-with-colly-18lh</guid>
      <description>&lt;p&gt;I have seen a lot of examples of how to build a web scraper in lots of programming languages mostly in python specifically using &lt;a href="https://scrapy.org/"&gt;scrapy&lt;/a&gt; tool but only a few in &lt;a href="https://golang.org/"&gt;golang&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;As many golangs fans know, golang has tons of benefits when we talk about concurrency and parallelism, all of these features combined with a modern framework allow us to scratch the web in an easy and fastest way, but first of all, let's start with what a web scraper does?&lt;/p&gt;

&lt;h2&gt;
  
  
  Explaining Web scraper like I'm five
&lt;/h2&gt;

&lt;p&gt;Quoting &lt;a href="https://en.wikipedia.org/wiki/Web_scraping"&gt;Wikipedia&lt;/a&gt; definition:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. The web scraping software may access the World Wide Web directly using the Hypertext Transfer Protocol or through a web browser&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;So a web scraping is a technique used to extract data from websites using HTTP, think of this a web scraper is basically a robot that can read the data from a website like the human brain can read this post, a web scraper can get the text from this post, extract the data from the HTML and it can use them for many purposes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Web scraper vs Web crawler
&lt;/h2&gt;

&lt;p&gt;In order to keep this short, a web crawler is a bot that can browse the web so a search engine like google can index new websites and a web scraper is responsible of extract the data from that website.&lt;/p&gt;

&lt;h2&gt;
  
  
  Back to business
&lt;/h2&gt;

&lt;p&gt;Now we know what we are building let's start to get our hands dirty, our first step will be to create a simple server in golang with a ping endpoint, using the standard lib it will look like this&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;package main

import (
    "log"
    "net/http"
)

func ping(w http.ResponseWriter, r *http.Request) {
    log.Println("Ping")
    w.Write([]byte("ping"))
}

func main() {
    addr := ":7171"

    http.HandleFunc("/ping", ping)

    log.Println("listening on", addr)
    log.Fatal(http.ListenAndServe(addr, nil))
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;That's it we just created a simple server with golang 👏🏻 to test it just run it like any other golang program using &lt;code&gt;go run main.go&lt;/code&gt; you will see a log that our server is listening to the port &lt;code&gt;7171&lt;/code&gt;. To test it with curl just run &lt;code&gt;curl -s 'http://127.0.0.1:7171/ping'&lt;/code&gt; &lt;/p&gt;

&lt;p&gt;Thanks to the http lib we can easily create a server just calling the &lt;code&gt;ListenAndServe&lt;/code&gt; function specifying a port and using the &lt;code&gt;HandleFunc&lt;/code&gt; we can manage different endpoints for our API. &lt;/p&gt;

&lt;p&gt;Now we have a server up and running. Let's create another endpoint, this one will extract some data from the colly website.&lt;/p&gt;

&lt;p&gt;First of all, we need to install the colly dependency to do this I highly recommend to use &lt;a href="https://blog.golang.org/using-go-modules"&gt;go module&lt;/a&gt; just run &lt;code&gt;go mod init &amp;lt;project-name&amp;gt;&lt;/code&gt; this will generate the &lt;code&gt;go.mod&lt;/code&gt; file where all dependencies used in the project will be. Open the &lt;code&gt;go.mod&lt;/code&gt; and add the colly dependency in the require section&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;require (
github.com/gocolly/colly v1.2.0
)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;and that's it go module will take care of download the dependency to your local machine.&lt;/p&gt;

&lt;p&gt;We are all set to extract all the data from the websites so let's create a function to get all links from any website&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;func getData(w http.ResponseWriter, r *http.Request) {
//Verify the param "URL" exists
    URL := r.URL.Query().Get("url")
    if URL == "" {
        log.Println("missing URL argument")
        return
    }
    log.Println("visiting", URL)

//Create a new collector which will be in charge of collect the data from HTML
    c := colly.NewCollector()

//Slices to store the data
    var response []string

//onHTML function allows the collector to use a callback function when the specific HTML tag is reached 
//in this case whenever our collector finds an
//anchor tag with href it will call the anonymous function
// specified below which will get the info from the href and append it to our slice
    c.OnHTML("a[href]", func(e *colly.HTMLElement) {
        link := e.Request.AbsoluteURL(e.Attr("href"))
        if link != "" {
            response = append(response, link)
        }
    })

//Command to visit the website
    c.Visit(URL)

// parse our response slice into JSON format
    b, err := json.Marshal(response)
    if err != nil {
        log.Println("failed to serialize response:", err)
        return
    }
// Add some header and write the body for our endpoint
    w.Header().Add("Content-Type", "application/json")
    w.Write(b)
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;This function will extract all links from any website specified in the &lt;code&gt;url&lt;/code&gt; param.&lt;/p&gt;

&lt;p&gt;And finally, let's add our new endpoint to our server&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;func main() {
    addr := ":7171"

    http.HandleFunc("/search", getData)
    http.HandleFunc("/ping", ping)

    log.Println("listening on", addr)
    log.Fatal(http.ListenAndServe(addr, nil))
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Done we have our new search GET endpoint which receives the &lt;code&gt;url&lt;/code&gt; param and it will extract all links from the website specified. For test it using curl you can use &lt;code&gt;curl -s 'http://127.0.0.1:7171/search?url=http://go-colly.org/'&lt;/code&gt; it will response something like&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;["http://go-colly.org/","http://go-colly.org/docs/","http://go-colly.org/articles/","http://go-colly.org/services/","http://go-colly.org/datasets/","https://godoc.org/github.com/gocolly/colly","https://github.com/gocolly/colly","http://go-colly.org/","http://go-colly.org/","http://go-colly.org/docs/","http://go-colly.org/articles/","http://go-colly.org/services/","http://go-colly.org/datasets/","https://godoc.org/github.com/gocolly/colly","https://github.com/gocolly/colly","https://github.com/gocolly/colly","http://go-colly.org/docs/","https://github.com/gocolly/colly/blob/master/LICENSE.txt","https://github.com/gocolly/colly","http://go-colly.org/contact/","http://go-colly.org/docs/","http://go-colly.org/services/","https://github.com/gocolly/colly","https://github.com/gocolly/site/","http://go-colly.org/sitemap.xml"]
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;h1&gt;
  
  
  Conquer all the web
&lt;/h1&gt;

&lt;p&gt;Congrats! you just created a web scraper API in golang 🙌🏻.&lt;br&gt;
before conquering all the web remember to check the robots.txt file of the website to ensure you can extract that data. For those who are not aware of what &lt;a href="http://www.robotstxt.org/"&gt;robots.txt&lt;/a&gt; is.&lt;br&gt;
Is a file on all the websites where it specifies the instructions to the robots like web crawler or web scraper how to handle their data or what endpoint the robots can use. For example, in google will be &lt;code&gt;https://www.google.com/robots.txt&lt;/code&gt; &lt;/p&gt;

&lt;h2&gt;
  
  
  Next step
&lt;/h2&gt;

&lt;p&gt;Keep going further with &lt;a href="http://go-colly.org/"&gt;colly&lt;/a&gt; documentation, it has a lot of examples of web scraper and web crawlers. &lt;/p&gt;

</description>
      <category>go</category>
      <category>scraper</category>
      <category>web</category>
      <category>likeimfive</category>
    </item>
  </channel>
</rss>
