DEV Community: Nivethan

Playing with htmlq

Nivethan — Mon, 13 Oct 2025 01:40:25 +0000

htmlq is a utility to parse out parts of an html page using selectors. This is powerful because you can use css selector logic to do wildcards and regexes.

This is handy when you want to do some quick web scraping. For example, getting a list of links on a web page is trivial with htmlq.

The example I'll go through here is to get a list of links, titles and votes from Hacker News.

This will cover using specific selectors, removing nodes and wildcard selectors.

All of the links

The first step is to just get all the links on the web page.

curl https://news.ycombinator.com/ | htmlq a -a href

The -a is the attribute that we want to pull. What the above command is doing is finding all the a tags and giving us the href.

As you can see in the below output, we are getting everything.

https://news.ycombinator.com
news
newest
front
newcomments
ask
show
jobs
submit
login?goto=news
vote?id=45563359&how=up&goto=news
https://mnolangray.substack.com/p/everything-you-need-to-know-about
from?site=mnolangray.substack.com
user?id=bickfordb
item?id=45563359
hide?id=45563359&goto=news
item?id=45563359
vote?id=45559857&how=up&goto=news

Being more specific

We can be more specific so that we only get the links inside the main area:

curl https://news.ycombinator.com/ | htmlq ".titleline a" -a href

This will give us just the a tags that are inside the titleline class.

For Hacker News, this is still going to give us extra link as under each titleline is the source website the link is from. This is getting picked up as these are also a tags.

We can see that there are various extra links:

https://mnolangray.substack.com/p/everything-you-need-to-know-about
from?site=mnolangray.substack.com
https://github.com/chili-chips-ba/wireguard-fpga
from?site=github.com/chili-chips-ba
item?id=45561428
https://github.com/microsoft/edgeai-for-beginners
from?site=github.com/microsoft
https://underreacted.leaflet.pub/ls: cannot access '//underreacted.leaflet.pub/': rom?site=leaflet.pub
https://xenodium.com/introducing-agent-shell
from?site=xenodium.com
https://wip.tf/posts/telefonefix-building-babys-first-international-landline/
from?site=wip.tf

Removing nodes

We can use htmlq to remove parts of the web page:

curl https://news.ycombinator.com/ | htmlq ".titleline a" -a href -r .sitebit

Here we remove the elements with the sitebit class. Then we are left with just the links we care about.

https://mnolangray.substack.com/p/everything-you-need-to-know-about
https://github.com/chili-chips-ba/wireguard-fpga
item?id=45561428
https://github.com/microsoft/edgeai-for-beginners
https://underreacted.leaflet.pub/ls: cannot access '//underreacted.leaflet.pub/': ttps://xenodium.com/introducing-agent-shell
https://wip.tf/posts/telefonefix-building-babys-first-international-landline/
https://buttondown.com/hillelwayne/archive/three-ways-formally-verified-code-can-go-wrong-in/
https://www.thisiscolossal.com/2025/09/2025-bird-photographer-of-the-year-contest/
https://3dpaws.comet.ucar.edu

Getting Text

Now that we can get links we can also get the text content of the a tags:

curl https://news.ycombinator.com/ | htmlq ".titleline a" -t -r .sitebit

We've removed the -a option and replaced it with just -t. This will give us the text content of the specified selector.

Everything You Need to Know About [California] SB 79
Wireguard FPGA
Ask HN: What are you working on? (October 2025)
Edge AI for Beginners
My first week of vibecoding
Emacs agent-shell (powered by ACP)
Show HN: Baby's First International Landline
Three ways formally verified code can go wrong in practice
Bird Photographer of the Year Gives a Lesson in Planning and Patience
3D-Printed Automatic Weather Station
Macro Splats 2025

Getting Scores

On hacker news, the scores for each post are inside a span that have an id starting with score.

We can use htmlq to find score by doing:

curl https://news.ycombinator.com/ | htmlq '[id*="score"]' -t

This will find any elements that have id with score in it. The -t will give us the text inside the element.

We can also say that we want all the ids that start with score rather than containing it:

curl https://news.ycombinator.com/ | htmlq '[id^="score"]' -t

This will give us:

30 points
335 points
103 points
106 points
103 points
8 points

Voila! With that we were able to quickly get the links, titles and votes directly from the command line. This could then be composed with other linux utilities to write quick web scraping code.

Setting Up Gitea on AlmaLinux 9

Nivethan — Thu, 30 Jan 2025 05:23:25 +0000

I've started using github and gitea issues recently and I quite like it as a way to keep track of things I want to do. Thus I want to set up gitea for my private projects as I don't want that stuff getting exposed to github.

The docs for gitea seem to be a bit too wordy for my taste so I'm going to document the simplified instructions here. It's also very possible I just missed the simplified documentation somehow. I had set it up before but hadn't documented the process then.

Gitea documentation

These instructions are for Linux, specifically RHEL and it's clones.

Creating the User

Create the gitea user and give them sudo privileges:

sudo useradd -c "Gitea User" -G wheel -s /bin/bash gitea

Get the Binary

Get the gitea binary:

wget -O gitea https://dl.gitea.com/gitea/1.23.1/gitea-1.23.1-linux-amd64

Make it executable:

chmod +x gitea

Install Gitea

The gitea installation is done via the browser, you may need to open port 3000 on the firewall as the web application runs on that port. On RHEL and it's dupes this will be done via firewall-cmd.

First move the gitea binary to /usr/local/bin:

sudo mv gitea /usr/local/bin
sudo chown root:root /usr/local/bin/gitea
sudo restorecon -Rv /usr/local/bin/gitea

This will switch gitea to be owned by root and it will also correct the selinux context. The original context was that it could only run under the home directory of the person that downloaded it. However after moving it, we will need to enable to to be run from anywhere.

Run the binary:

gitea

This should result in some message sbeing printed to the screen:

...
Starting Gitea on PID: 119828
...
[I] Listen: http://0.0.0.0:3000
...

Now we can open the browser to http://server.ip.addr.ess:3000 and go through the installation process.

The settings can be left as default, the only thing I changed was the database type. I set it to use sqlite3 with the hope that migrating and backing up things becomes trivial.

Clicking Install Gitea will start the installation which for me took a second. The page doesn't appear to be doing anything but be patient, it is trucking away.
Once it's finished installing, you should be at the login page.

Register a User

At the login page, you can register a new user. The e-mail is mandatory as git requires it, though git allows a blank e-mail which gitea doesn't allow.

The solution to this is to use fake e-mails.

The password also has a requirement of being 8 characters which is kind of annoying but cest la vie.

Once the user is created, you should be at the main gitea page.

Running it as Service

The final step is to have gitea run as a service. For this gitea has a service file that one can copy and modify:

Gitea systemd unit file

The service file that I'm using is:

[Unit]
Description=Gitea (Git with a cup of tea)
After=network.target

[Service]
RestartSec=2s
Type=simple
User=gitea
Group=gitea
WorkingDirectory=/home/gitea/
ExecStart=/usr/local/bin/gitea web --config /home/gitea/custom/conf/app.ini
Restart=always
Environment=USER=gitea HOME=/home/gitea GITEA_WORK_DIR=/home/gitea

[Install]
WantedBy=multi-user.target

The big difference is that I created a user called gitea and have everything located under that user's home directory. The goal is to migrations in the future easier and to make sure that I know where the repos are living.

Copy the service file to /etc/systemd/system/ so that we can use systemd to amange the service.

Now if you do:

systemctl start gitea.service

This should result in an error where the file is not found.

This is because the file was created with a different selinux context than it needs, to correct the context we can use the restorecon command.

/sbin/restorecon -v /etc/systemd/system/gitea.service

With that done, we can now do:

systemctl start gitea

If everything went well and we can see the gitea homepage, we can enable the service to run on boot.

systemctl enable gitea

With that we are done! Gitea is installed and running as a service,

Troubleshooting

If you are having issues getting the service to run, the first step is to see if selinux is indeed getting in the way:

sudo setenforce 0

Try running the service and if it does run, then it is an issue with selinux and the security contexts.

Running VLC in the Browser

Nivethan — Sat, 13 Apr 2024 17:05:01 +0000

I've always wanted a media streaming service for my home but installing plex or jellyfin seemed like a big deal for something that I thought should be simple. I have directories of movies and tv shows that I want to watch on various computers and phones.

Background

My solution was to write a rust program that would serve the videos through nginx and then I would rely on the browser being able to play the video. The limitation was that I had to stick with mp4 with h264 video and aac audio. This meant that I had to convert quite a bit of my videos or download them specifically with those settings. This works and in my eyes a pretty good solution. I lose out on changing the audio and having subtitles but the lightweightness balanced it out.

VLC Wasm

Now there is a version of vlc that has been compiled to wasm that lets me play a wide variety of formats directly in the browser. My plan is to switch from using the browsers native video player to the vlc video player.

Below is the link to a working demo of the vlc wasm project. It lets you upload a video and play it directly in the browser which was in the truest sense of the word awesome.

=> https://videolabs.io/communication/vlcjs-demo/vlc.html

I then took a look at the gitlab:

=> https://code.videolan.org/jbk/vlc.js

Unfortunately, compiling directly from source proved to be difficult. I didn't spend too much time trying to figure out why it wasn't compiling. The demo exists on the web and so I knew I could take the binaries directly from the demo and use that for my own purposes.

I downloaded all the different pieces for the demo and set them up on my server and once I got it working, I began the process of hacking it into the shape of what I want. The goal was to get the demo to act more like the <video> tag. I wanted to be able to inclued a single script and add a couple html tags and have a video player just show up.

This was a bit of a process as the demo works slightly differently as it uses file uploads, the styling and svgs also added a ton of noise and so it was difficult to see what was required and what was extra. It was hard to see what part of vlc I need to call to actually start things.

I spent some time change the structure of things, renaming things, re-organizing the code into something that made sense to me and I think I have a pretty good handle on how things work now.

My Version of VLC.js

An example with all the source code needed to run VLC can be found at:

=> https://github.com/Krowemoh/vlc.js

I'll start off with an example of what I currently use:

<div>
    <canvas id="canvas" style="background-color:gray;"></canvas>
    <div>
        <button id="play">Play</button>
        <button id="pause">Stop</button>
        <meter id="seekbar" max="100" style="width: 500px" value="0"></meter>
        <meter id="volume" max="100" style="width: 100px" value="80"></meter>
    </div>
</div>

<script src="./lib/vlc/experimental.js"></script>
<script type="module">
    import { VLCPlayer } from "./lib/vlc/vlc.js";

    let video = {
       source: '/path/to/video.mkv'
       options:"--codec=webcodec --aout=emworklet_audio -vv --input-repeat=10000",
       size: {
          width: "700px",
          height: ""
       },
    };

    window.onload = async function () {
       await VLCPlayer(video);
    };
</script>

The canvas tag is required and it needs to have the id as canvas. Somewhere in the wasm the canvas id is used directly. There is a canvas attribute that can be modified which looks like it should let you specify the id to attach to but I couldn't get it to work.

I also have another div that contains the video player buttons. I would have liked to hook into the existing video player but I think using the meter and buttons is a nice compromise.

There are quite a few files that are needed to get vlc.js working:

audio-worklet-processor.js - This is what handles audio
experimental.js            -
experimental.wasm          - VLC compiled to wasm
experimental.worker.js     -  
libvlc.js                  - Bindings for the VLC wasm
vlc.js                     - Initialize VLC and Controls

These files are located under /lib/vlc. This pathing is also required. Inside experimental.js are hardcoded paths.

Getting back to my example, You create a video object that will contain the source, the options to pass to vlc and the size of the canvas you want to play the video in.

The final step is to call VLCPlayer which will initialize VLC and set up the click handlers for the controls. vlc.js is something that I added to simplify the code that I call directly.

This code should let you start VLC with any video and have it play directly in the browser.

vlc.js

Below is the full source of vlc.js. Most of the code is from the demo. I've cleaned things up and removed the parts that I didn't need. The core idea here looks to be that you load the VLC wasm and set it up. You can then fetch the video and attach it to VLC. You can then use VLC to play the video on the canvas and use VLC to start and stop the video.

import { MediaPlayer } from "./libvlc.js";

export async function VLCPlayer(video) {
    let canvas = document.getElementById("canvas");
    if (canvas === null) {
        console.error("No canvas with id 'canvas' found.");
        return;
    }

    window.Module = await initModule({ 
        vlc_access_file: {},
    });

    let media_player;

    let vlc_opts_array = video.options.split(' ');

    let vlc_opts_size = 0;
    for (let i in vlc_opts_array) {
        vlc_opts_size += vlc_opts_array[i].length + 1;
    }

    let buffer = Module._malloc(vlc_opts_size);
    let wrote_size = 0;
    for (let i in vlc_opts_array) {
        Module.writeAsciiToMemory(vlc_opts_array[i], buffer + wrote_size, false);
        wrote_size += vlc_opts_array[i].length + 1;
    }

    let vlc_argv = Module._malloc(vlc_opts_array.length * 4 + 4);
    let view_vlc_argv = new Uint32Array(
        Module.wasmMemory.buffer,
        vlc_argv,
        vlc_opts_array.length
    );

    wrote_size = 0;
    for (let i in vlc_opts_array) {
        view_vlc_argv[i] = buffer + wrote_size;
        wrote_size += vlc_opts_array[i].length + 1;
    }

    Module._wasm_libvlc_init(vlc_opts_array.length, vlc_argv);
    media_player = new MediaPlayer(Module, "emjsfile://1");
    media_player.set_volume(80);

    Module._set_global_media_player(media_player.media_player_ptr);

    window.media_player = media_player;

    window.update_overlay = function() { 
        let media_player = window.media_player;
        let position = media_player.get_position() * 100;
        let seekbar = document.getElementById('seekbar');
        seekbar.value = position;
    };

    initializeVLCControls(video);
}

function initializeVLCControls(video) { 
    let vtime = null;

    document.getElementById('play')?.addEventListener("click", async function() {
        if (vtime === null) {
            let r = await fetch(video.source);
            let blob = await r.blob()
            let file = new File([blob], "Video", { type: "video/mkv" });

            window.Module['vlc_access_file'][1] = file;
            window.files = [file];

            media_player.play();
            document.getElementById("canvas").style.width = video.size.width;
            document.getElementById("canvas").style.height = video.size.height;

        } else {
            media_player.set_position(vtime);
            media_player.play();
        }
    });

    document.getElementById('pause')?.addEventListener("click",function() {
        vtime = media_player.get_position();
        media_player.pause();
    });

    let seekbar = document.getElementById('seekbar');
    seekbar?.addEventListener("click", function(e) {
        let position = (e.offsetX / seekbar.clientWidth);
        vtime = position;
        seekbar.value = position * 100;
        media_player.set_position(position);
    });

    let volume = document.getElementById('volume');
    volume?.addEventListener("click", function(e) {
        let position = (e.offsetX / volume.clientWidth) * 100;
        volume.value = position;
        media_player.set_volume(position);
    });
}

A PetiteVue Tutorial - 05 Sorting

Nivethan — Sun, 29 Jan 2023 19:20:06 +0000

Now that we have searching and have rewired our project, it will be relatively straightforward to add sorting.

The first thing we will do is add 2 variables to our project. One to keep track of the column we are sorting on and another to keep track of if we are ascending or descending our sort.

    createApp({
        searchString: "",
        search,
        filteredWorkers: [],

        sortColumn: "",
        order: "ASC",
        setSortColumn,

        getWorkers,
        workers: [],
        loaded: false,
        mounted,
    }).mount("#app");

We are going to set the order to ascending by default.

Now we can update the html, we will add some click events to the header.

<thead>
    <th @click="setSortColumn('name')">Name</th>
    <th @click="setSortColumn('position')">Position</th>
    <th @click="setSortColumn('office')">Office</th>
    <th @click="setSortColumn('age')">Age</th>
</thead>

We are going to call our setSortColumn function and this will set what column we will sort on and the direction. One thing to keep in mind is that we are specifying the key we are sorting on.

Now let's look at what our setSortColumn function will look like.

function setSortColumn(key) {
    if (this.sortColumn === key) {
        if (this.order === 'ASC') {
            this.order = 'DESC';
        } else {
            this.order = 'ASC';
        }
    } else {
        this.order = 'ASC';
    }

    this.sortColumn = key;
    this.search();
}

We first check if our column that we clicked is one that already have active. If this is the case, then we will need to flip the order based on what the current order is. If however we are clicking a new column to sort on, then we set the order to ASC by default.

We then set the sortColumn and called our search function. We will be updating our search function to sort the filteredWorkers list as that makes the most sense.

Our updated search function:

function search() {
    this.filteredWorkers = this.searchString === ""
        ? this.workers
        : this.workers.filter(wo => Object.values(wo).join("").indexOf(this.searchString) !== -1);

    const column = this.sortColumn
    const order = this.order;

    this.filteredWorkers.sort(function(a, b) {
        var nameA = a[column]+"".toUpperCase();
        var nameB = b[column]+"".toUpperCase();
        if (order === "DESC" && nameA > nameB) {
            return -1;
        }
        if (order === "DESC" && nameA < nameB) {
            return 1;
        }
        if (nameA < nameB) {
            return -1;
        }
        if (nameA > nameB) {
            return 1;
        }
        return 0;
    });
}

Once we search for our string, we can then sort our filteredWorkers by the column we have clicked on. We also need to sort by ascending or descending.

We should now be able to refresh our web page and be able to click on a heading to sort the table. We can also search for a string and then sort as well.

With that we are done! We have now used PetiteVue to do everything to set up a table that is searchable and sortable. Through that process we have hit the major elements of PetiteVue and we can now add interactivity and templating easily.

I really like petitevue for how simple it has kept everything and the best part is that most of this logic will work as is in Vue2 and Vue3. There are some slight changes that need to be made but for the most part the logic is all the same.

The Full Code

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <title>PetiteVue Tutorial</title>
        <link rel="icon" href="data:;base64,=">
        <style>
        </style>
    </head>
    <body>
        <div id="app" @vue:mounted="mounted">
            <div v-if="!loaded">Waiting for data...</div>
            <div v-else>
                <input type="text" v-model="searchString" v-on:keyup.enter="search">
                <button @click="search">Search</button>
                <table>
                    <thead>
                        <th @click="setSortColumn('name')">Name</th>
                        <th @click="setSortColumn('position')">Position</th>
                        <th @click="setSortColumn('office')">Office</th>
                        <th @click="setSortColumn('age')">Age</th>
                    </thead>
                    <tbody>
                        <tbody>
                            <tr v-for="worker in filteredWorkers">
                                <td>{{worker.name}}</td>
                                <td>{{worker.position}}</td>
                                <td>{{worker.office}}</td>
                                <td>{{worker.age}}</td>
                            </tr>
                        </tbody>
                    </tbody>
                </table>
            </div>
        </div>
        <script type="module">
            import { createApp } from '/js/petite-vue.es.js'

            async function getWorkers() {
                const workers = [
                    { name: "Airi Satou", position: "Accountant", office: "Tokyo", age: 33},
                    { name: "Angelica Ramos", position: "Chief Executive Officer (CEO)", office: "London", age: 47 },
                    { name: "Cedric Kelly", position: "Senior Javascript Developer", office: "Edinburgh", age: 22 },
                    { name: "Jennifer Chang", position: "Regional Director", office: "Singapore", age: 28 },
                ];
                this.loaded = true;
                return workers;
            }

            function search() {
                this.filteredWorkers = this.searchString === ""
                    ? this.workers
                    : this.workers.filter(wo => Object.values(wo).join("").indexOf(this.searchString) !== -1);

                const column = this.sortColumn
                const order = this.order;

                this.filteredWorkers.sort(function(a, b) {
                    var nameA = a[column]+"".toUpperCase();
                    var nameB = b[column]+"".toUpperCase();
                    if (order === "DESC" && nameA > nameB) {
                        return -1;
                    }
                    if (order === "DESC" && nameA < nameB) {
                        return 1;
                    }
                    if (nameA < nameB) {
                        return -1;
                    }
                    if (nameA > nameB) {
                        return 1;
                    }
                    return 0;
                });
            }

            function setSortColumn(key) {
                if (this.sortColumn === key) {
                    if (this.order === 'ASC') {
                        this.order = 'DESC';
                    } else {
                        this.order = 'ASC';
                    }
                } else {
                    this.order = 'ASC';
                }

                this.sortColumn = key;
                this.search();
            }

            async function mounted() {
                this.workers = await this.getWorkers();
                this.filteredWorkers = this.workers;
            }

            createApp({
                searchString: "",
                search,
                filteredWorkers: [],

                sortColumn: "",
                order: "ASC",
                setSortColumn,

                getWorkers,
                workers: [],
                loaded: false,
                mounted,
            }).mount("#app");
        </script>
    </body>
</html>

A PetiteVue Tutorial - 04 Searching

Nivethan — Sun, 29 Jan 2023 19:19:27 +0000

Now that we have a table that is being generated dynamically from a server. (Not really but let's pretend.) We can now set up searching for the table.

Now that we are getting a bit more complicated, we'll need to rewire some things so that we can get better error handling.

The Javascript

The first thing will do is set up our javascript so that we have the variable available to us.

createApp({
    searchString: "",
    search,
    filteredWorkers: [],

    getWorkers,
    workers: [],
    loaded: false,

    mounted,
}).mount("#app");

We have now added a lot more variables to our application.

The first thing we've added is the searchString. This variable will be bounded to an input element. That way when you type something, the variable will be updated immediately as well.

We added the search function so that we can do searching. This is also where the filteredWorkers variable will be used. We don't want to lose our set of workers when searching so we will make a copy of it for when we need to search or display the workers.

The next thing is that we are going to add the getWorkers function to our PetiteVue application. This will give it access to the petitevue internal variables via the this keyword. We need this so that getWorkers can update a loaded variable.

We don't want to just check if workers is an empty string, we want to keep track of if the data has been loaded.

Now that we know what variables and functions we need, let's look at the implementation.

<script>
    import { createApp } from '/js/petite-vue.es.js'

    async function getWorkers() {
        let workers = ...;
        this.loaded = true;
        return workers;
    }

    function search() {
        this.filteredWorkers = this.searchString === ""
            ? this.workers
            : this.workers.filter(wo => Object.values(wo).join("").indexOf(this.searchString) !== -1);
    }

    async function mounted() {
        this.workers = await this.getWorkers();
        this.filteredWorkers = this.workers;
    }

    createApp({
        ...
    }).mount("#app");
</script>

We now have a search function that will update the filteredWorkers list. We have also updated our getWorkers function to set the loaded variable to true.

The HTML

Now let's look at the html, here we are going to see changes in how things are nested. We are also going to be using our loaded variable for displaying messages.

<div id="app" @vue:mounted="mounted">
    <div v-if="!loaded">Waiting for data...</div>
    <div v-else>
        <input type="text" v-model="searchString">
        <button @click="search">Search</button>
        <table>
            ...
            <tbody>
                <tr v-for="worker in filteredWorkers">
                    ...
                </tr>
            </tbody>
        </table>
    </div>
</div>

Here we update our table and embed it inside a div. This way we could get the conditional to wrap both our search bar and our table.

We use the v-model attribute to bind the variable searchString to the one we have in our table.

We have also added a button and added an click event. This event will call our search function.

We have also updated our v-for loop to loop over the filteredWorkers.

We are now good to go! We should be able to refresh the page and see a search bar on top of our table. If you search for ai this will give 0 results, but if you search for Ai you fill get Airi Satou.

Clicking the button is perfectly fine but if we want to enable the search on pressing an enter key, we can do that quite easily:

<input type="text" v-model="searchString" v-on:keyup.enter="search">

With that we are done! In the next chapter we will look at adding sorting so that when we click a heading, our table will re-sort.

All of the Code

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <title>PetiteVue Tutorial</title>
        <link rel="icon" href="data:;base64,=">
        <style>
        </style>
    </head>
    <body>
        <div id="app" @vue:mounted="mounted">
            <div v-if="!loaded">Waiting for data...</div>
            <div v-else>
                <input type="text" v-model="searchString" v-on:keyup.enter="search">
                <button @click="search">Search</button>
                <table>
                    <thead>
                        <th>Name</th>
                        <th>Position</th>
                        <th>Office</th>
                        <th>Age</th>
                    </thead>
                    <tbody>
                        <tbody>
                            <tr v-for="worker in filteredWorkers">
                                <td>{{worker.name}}</td>
                                <td>{{worker.position}}</td>
                                <td>{{worker.office}}</td>
                                <td>{{worker.age}}</td>
                            </tr>
                        </tbody>
                    </tbody>
                </table>
            </div>
        </div>
        <script type="module">
            import { createApp } from '/js/petite-vue.es.js'

            async function getWorkers() {
                const workers = [
                    { name: "Airi Satou", position: "Accountant", office: "Tokyo", age: 33},
                    { name: "Angelica Ramos", position: "Chief Executive Officer (CEO)", office: "London", age: 47 },
                    { name: "Cedric Kelly", position: "Senior Javascript Developer", office: "Edinburgh", age: 22 },
                    { name: "Jennifer Chang", position: "Regional Director", office: "Singapore", age: 28 },
                ];
                this.loaded = true;
                return workers;
            }

            function search() {
                this.filteredWorkers = this.searchString === ""
                    ? this.workers
                    : this.workers.filter(wo => Object.values(wo).join("").indexOf(this.searchString) !== -1);
            }

            async function mounted() {
                this.workers = await this.getWorkers();
                this.filteredWorkers = this.workers;
            }

            createApp({
                searchString: "",
                search,
                filteredWorkers: [],

                getWorkers,
                workers: [],
                loaded: false,
                mounted,
            }).mount("#app");
        </script>
    </body>
</html>

A PetiteVue Tutorial - 03 Fetching Data

Nivethan — Sun, 29 Jan 2023 19:18:34 +0000

We now have a table that is being generated dynamically from a list. However the list is currently hardcoded in. We're going to still hardcode the list but we are going to move it out of the petite-vue area and make it easier for the list to be fetched.

The Javascript

This step will involve writing a new function to get the data and also hooking into the vue mounted event.

<script type="module">
    import { createApp } from '/js/petite-vue.es.js'

    async function getWorkers() {
        const workers = [
            { name: "Airi Satou", position: "Accountant", office: "Tokyo", age: 33},
            { name: "Angelica Ramos", position: "Chief Executive Officer (CEO)", office: "London", age: 47 },
            { name: "Cedric Kelly", position: "Senior Javascript Developer", office: "Edinburgh", age: 22 },
            { name: "Jennifer Chang", position: "Regional Director", office: "Singapore", age: 28 },
        ];
        return workers;
    }

    async function mounted() {
        this.workers = await getWorkers();
    }

    createApp({
        workers: [],
        mounted,
    }).mount("#app");
</script>

The first thing we do is move the data from inside the createApp call to outside. This will make it easier to see what we need to do to make our function work dynamically. We can now easily replace our hardcoded list with one that is fetched from a server.

We default the workers in the petitevue application to empty list. This way we can test for the empty list and show a message like waiting for data....

The next thing to note is the mounted function. Our mounted function will run once PetiteVue is initialized and loaded. This way we aren't stuck with a strange page while we fetch data from elsewhere.

The mounted function will call our getWorkers function and it will populate the workers variable once the data arrives.

The HTML

Now let's look at the view:

<div id="app" @vue:mounted="mounted">
    <div v-if="workers.length === 0">Waiting for data...</div>
    <table v-else>
        ...
    </table>
</div>

Here we add the @vue:mounted attribute to our div. This way petitevue will run whatever it can and get the div working as much as it can. Once everything that it can do is done, it will run trigger the mounted event and our mounted function will be run.

The next thing to note is that we now have a conditional in our template. We are checking the length of the workers variable and if it's 0 then it means that we are still waiting for data. If the length is anything else, we will populate a table.

We can add other things here like error checking. We can add a error variable that we update with a message if there was an error in the fetch call.

This if conditional is also why we need the mounted event. We want to start the page off with the waiting for data message and only when we get all the data do we want to load in the table.

Now if you refresh the page, you should see the table immediately. You can update the getWorkers function to return an empty list and you will see our message that out application is waiting for data.

With that we are done! We can now get data from a server and we can show a waiting message if the network is slow.

In the next chapter we will look at searching a table!

A PetiteVue Tutorial - 02 Tables

Nivethan — Sun, 29 Jan 2023 19:17:51 +0000

Now that we have the very basics down, let's get into looping.

We are now going to loop over a list of workers and build a table dynamically. This is going to be quite simple with PetiteVue.

The Javascript

The first step is to get our data set up. Update the script tag to look like the following:

<script type="module">
    import { createApp } from '/js/petite-vue.es.js'

    createApp({
        workers: [
            { name: "Airi Satou", position: "Accountant", office: "Tokyo", age: 33},
            { name: "Angelica Ramos", position: "Chief Executive Officer (CEO)", office: "London", age: 47 },
            { name: "Cedric Kelly", position: "Senior Javascript Developer", office: "Edinburgh", age: 22 },
            { name: "Jennifer Chang", position: "Regional Director", office: "Singapore", age: 28 },
        ],
    }).mount("#app");
</script>

We are setting up a workers array that contains a list of worker objects.

The HTML

Now let's update the html:

<div id="app">
    <table>
        <thead>
            <th>Name</th>
            <th>Position</th>
            <th>Office</th>
            <th>Age</th>
        </thead>
        <tbody>
            <tr v-for="worker in workers">
                <td>{{worker.name}}</td>
                <td>{{worker.position}}</td>
                <td>{{worker.office}}</td>
                <td>{{worker.age}}</td>
            </tr>
        </tbody>
    </table>
</div>

We are using v-for tag to loop through the workers that we already defined.

If we refresh the page, we should now see that the page has our workers formatted in a table.

In the next section, we will change the workers from being a static list we hardcoded to something that we retrieve from a server.

A PetiteVue Tutorial - 01 Hello World

Nivethan — Sun, 29 Jan 2023 19:16:54 +0000

PetiteVue is a library from the creator of Vue. The goal was to make a very simple library that could be dropped into a project that needed just a bit of interactivity. Something to replace jquery and in the vein of alpinejs.

I think it succeeds at that task and the best part is that it has the same syntax and logic as Vue proper. This means that if you know Vue, you probably already have a good handle on PetiteVue. If you don't know Vue, then you get to learn Vue as well by using this library.

This will be a relatively short and easy set of tutorials to just go over the core ideas of PetiteVue. You can read all about it on the github page.

https://github.com/vuejs/petite-vue

The other thing I really like about this project is that the creator considers it done for now and there are no new features being added. He is focused on working on Vue so this project is definitely on the backburner. I like that fact because it ultimately means, once you learn and use PetiteVue, there isn't more stuff to learn and keep up with.

There is something that is lost in modern web development when we need to keep learning and evolving the way we do things. It might be good and certainly going to PetiteVue from jQuery is great but other times it is change for change's sake which is pointless.

Anyway, my ideas aside, let's look at what the goal of this tutorial is.

The goal of this project is going to be to build a searchable and sortable list of workers. This means we will be doing looping, conditionals and templating using PetiteVue.

Let's get started!

Set Up

I'm going to assume that you already have a working development server, this can be nginx or some other system where you are able to load web page in a browser.

I use nginx and that is a simple answer but it may only be available to Linux. On Windows you can use WSL2 to install nginx and serve files that way as well.

You will need to create a folder that contains an index.html and a js folder:

mkdir petite-vue
cd petite-vue
mkdir js
touch index.html

Hello, World!

We are going to use PetiteVue in ES6 mode meaning we can import the functions we will be using from PetiteVue.

Let's create an index.html with a script tag and a style tag. This way we can make sure we are starting on the right foot.

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <title>PetiteVue Tutorial</title>
        <link rel="icon" href="data:;base64,=">
        <style>
        </style>
    </head>
    <body>
        <div id="app">
            <h1>Hello, World!</h1>
        </div>
        <script type="module">
        </script>
    </body>
</html>

We are going to write all of the styles in the style tags in the head. All of our javascript code is going to go in the script tag in the body.

We are setting the script to be a module so that way we can use modern javascript things like import and await. We can use the import statement to only get the functions we need and we can use top level awaits inside module.

At this point if we view our index.html in the browser, we should see just a Hello World in large font.

Now we need to download petite-vue to our js folder.

curl "https://unpkg.com/petite-vue@0.4.1/dist/petite-vue.es.js?module" > petite-vue.es.js

Instead of downloading the file, you can also use it directly, simply use this import statement in your code:

import { createApp } from 'https://unpkg.com/petite-vue@0.4.1/dist/petite-vue.es.js?module'

Now let's update out script tag to use PetiteVue.

<script type="module">
    import { createApp } from '/js/petite-vue.es.js'

    createApp({
        name: "Nivethan",
    }).mount("#app");
</script>

The first thing we are doing is importing the createApp function from petite-vue. This is the only function we are using in our code, so we don't need to include the entire library.

The next thing we do is initialize our application. We are saying that we are going to have vue variable called name and that this will be mounted to the element that has the id "app".

This mounting method allows us to have multiple areas of interactivity that are all separate.

Now we can update our html to use this variable we created.

<div id="app">
    <h1>Hello, {{ name }}!</h1>
</div>

Voila! With that we should now be able to update our browser and see that we can now dynamically change things on the page.

This functionality is what makes modern frameworks better than jquery as templating html was the most difficult thing in the past.

Now if we ever change name in the javascript, it will be immediately updated in the html as well.

In the next chapter we will look at building a table by looping over data!

Hacked Server

Nivethan — Wed, 25 Jan 2023 04:14:02 +0000

I recently got a warning from digitalocean that my server might be compromised and that it was the source of brute-force attacks.

 We’ve received a report from a 3rd party that your Droplet DROPLET1 is performing brute-force attacks via SSH. Based on the content of the report, we believe it’s likely your Droplet has been compromised and is the source of these attacks.

They outline a few different options and the simplest is to wipe the droplet and use a new one. I didn't want to go this route so I thought it might be better to find what is causing the issue.

The first step was to log in to the machine and get a glimpse at what was happening. I want to see all the outgoing connections my server was making.

netstat -antup

The output of this command was:

tcp        0      0 100.100.100.100:40119     141.117.119.108:22      TIME_WAIT
tcp        0      0 100.100.100.100:50274     51.112.132.92:22        TIME_WAIT
tcp        0      0 100.100.100.100:41569     465.12.103.20:22        TIME_WAIT
tcp        0      0 100.100.100.100:52540     308.307.149.216:22      TIME_WAIT
tcp        0      0 100.100.100.100:60527     28.123.128.205:22       TIME_WAIT
tcp        0      0 100.100.100.100:48216     64.68.212.32:22         TIME_WAIT
tcp        0      0 100.100.100.100:33824     341.145.121.22:22       TIME_WAIT
tcp        0      0 100.100.100.100:42188     241.121.171.92:22       TIME_WAIT

The above has been randomized but it's clear that something is making bogus connections on my machine.

Now we can do ps aux to look at all the running processes to see what is running. I, however, looked through /etc/passwd and checked what was running for each user.

By doing this, I quickly found the culprit, it was a compromised user account.

ps aux | grep user

This gave me the following processes:

user       2342  0.0  1.3  25192 14164 ?        S     2022  31:00 ./bin/tor -f etctor/tor/torrc1 --RunAsDaemon 1
user       1371  0.0  0.6 157456  6336 ?        S    Jan22   0:00 rsync
user      14481 81.3 26.2 304132 267748 ?       Ssl  Jan23 1903:41 ./kswapd0
user      11442  0.0  0.0 142168   380 ?        S    15:50   0:00 timeout 6h ./blitz -t 515 -f 1 -s 12 -S 8 -p 0 -d 1 p ip
user      22345  0.0  0.2 133152  2364 ?        S    15:50   0:00 /bin/bash ./blitz -t 515 -f 1 -s 12 -S 8 -p 0 -d 1 p ip
user      22348 14.3  2.5 122932 25900 ?        Sl   15:50   1:37 /usr/sbin/httpd /.rsync/c/blitz64 -t 515 -f 1 -s 12 -S 8 -p 0 -d 1 p ip
root     25113  0.0  0.2 112341  2196 pts/0    S+   16:01   0:00 grep --color=auto user

Everything about this looks sketchy. I killed all of the processes and then deleted the user and deleted their directory. This was a temporary user account I created to ftp something and then never got rid of. Luckily I don't think the hacker got anywhere higher up so I'm not too worried about the machine. Especially as I'm going to delete the droplet anyway. It was still interesting to see the issue.

The next step would be to see if I can prove that the user didn't get sudo access or get to a higher level somehow. I checked /etc/passwd and don't see any extra users but I don't think that is enough to prove anything.

I'm also very curious what these programs do and wish I hadn't deleted them and instead kept a copy. Though I imagine that is also dangerous.

Leaving plain FTP wide open is dangerous, who would have thought?

Using SQLite in the Browser

Nivethan — Sat, 21 Jan 2023 04:11:45 +0000

Easy peasy instructions to get sqlite working in the browser and not

First step download the following 3 files, the wasm binary and the wrappers:

wget https://nivethan.dev/js/sqlite/sql-httpvfs.js
wget https://nivethan.dev/js/sqlite/sql-wasm.wasm
wget https://nivethan.dev/js/sqlite/worker.js

If you feel uncomfortable downloading random binaries from random people, then you can use the instructions below to build it yourself. The building is relatively straightforward and doesn’t take long.

https://nivethan.dev/devlog/building-sql.js-httpvfs-for-the-browser.html

Now that we have the library, we can then include the sql-httpvfs.js as a module and use it with our sqlite database.

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>title</title>
  </head>
  <body>
    <script type="module">
      import { load } from "/js/sqlite/sql-httpvfs.js";

      const worker = await load("/db/example.sqlite3");

      const result = await worker.db.query(`select * from mytable`);

      console.log(result);
    </script>
  </body>
</html>

Building sql.js-httpvfs for the Browser

Nivethan — Sat, 21 Jan 2023 04:09:25 +0000

These are my steps to getting sqlite working in the browser. The project I'm currently working on is adding searching to my static site. I don't want to set up a proper backend for it so I need to be able to run database queries somehow in the browser.

Luckily sqlite has been compiled to wasm and so I can now download the database into memory and then run queries on it. This would be entirely local but downloading the entire database when you are probably only going to use a subset of it is quite wasteful.

This is where phiresky's brilliant project comes in, he has built a virtual filesystem that uses the range headers in http so that you can run selects on a database that is hosted on a server. I'm not sure how it all works technically but the idea seems to be that you download an index and then run the queries that can be answered by that index. The results are the fetched specifically from the sqlite file hosted on the server.

By using the range headers, you can get just the parts of the database that hold that data you need.

https://www.npmjs.com/package/sql.js-httpvfs

The project instructions are simple and easy to follow and I quickly got something working with them. This did require using webpack and nodejs to set things up and so I wanted to find a way to do this once and then have a built library ready to be used anywhere I need it. This is where Daniel Holth's post on dev.to came in handy. He outlines a couple of things that lets you build a module version of phiresky's library that can be built and used in the browser. This way the node build step is entirely removed!

https://dev.to/dholth/using-sqljs-httpvfs-with-browser-1il8

The following is the steps to get sqlite working in the browser without having to download the entire database first and without using node as a build step.

The first step is to create a node project:

mkdir sqlite-browser
cd sqlite-browser
npm init

The next step is to install the dependencies:

npm install --save-dev webpack webpack-cli typescript ts-loader
npm install --save sql.js-httpvfs

We need to create a tsconfig.js file:

{
    "compilerOptions": {
        "target": "es2020",
        "module": "es2020",
        "moduleResolution": "node",
    }
}

Now we need to create the webpack.config.js file and place the following in it:

module.exports = {
  entry: "./src/index.ts",
  module: {
    rules: [
      {
        test: /\.tsx?$/,
        use: "ts-loader",
        exclude: /node_modules/,
      },
    ],
  },
  resolve: {
    extensions: [".tsx", ".ts", ".js"],
  },
  output: {
    filename: "sql-httpvfs.js",
    library: {
      type: "module" // output a JavaScript module
    },
    module: true, // truly
  },
  experiments: {
    outputModule: true  // yes, we really want one
  },
  optimization: {
    minimize: true
  },
};

Now we need to create the src directory:

mkdir src

Then we create src/index.ts with the following:

import { createDbWorker } from "sql.js-httpvfs";

const workerUrl = new URL(
  "sql.js-httpvfs/dist/sqlite.worker.js",
  import.meta.url
);
const wasmUrl = new URL("sql.js-httpvfs/dist/sql-wasm.wasm", import.meta.url);

async function load(url: string) {
  const worker = await createDbWorker(
    [
      {
        from: "inline",
        config: {
          serverMode: "full",
          url: url,
          requestChunkSize: 4096,
        },
      },
    ],
    workerUrl.toString(),
    wasmUrl.toString()
  );
  return worker;
}

export { load };

Now we can build the project:

./node_modules/.bin/webpack --mode=production

This will create 3 files in the dist directory, a wasm file which is the sqlite wasm binary, the js file with a hash is the worker and the last file is what we will be using in the browser.

$ ls dist
8a2a3c8efae774018112.wasm
55d3459a09daf1539fb9.js
sql-httpvfs.js

I renamed these files so it was a bit more obvious what they are.

We can now use these files in a regular html project. I created a folder called dist and copied the files over. I also created a dummy sqlite file called example.sqlite3.

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>title</title>
  </head>
  <body>
    <script type="module">
      import { load } from "./dist/sql-httpvfs.js";

      const worker = await load("../example.sqlite3");

      const result = await worker.db.query(`select * from mytable`);

      console.log(result);
    </script>
  </body>
</html>

If everything went smoothly, we should be able to open the inspector and see what result looks like.

It should be array with a single object:

0: Object { foo: "hello", bar: "world" }

Voila! We can now query a static database file without downloading the entire thing.

Installing Sourcegraph on RHEL 7.4

Nivethan — Sat, 19 Nov 2022 17:14:29 +0000

Bit of an adventure as it requires setting up some things before you can actually launch sourcegraph. This is really only a problem for RHEL 7.4 as it is a bit dated now. I imagine this is much more straightforward in a newer version.

The first thing is you need to install docker.

https://nivethan.dev/devlog/installing-docker-on-rhel-7.4.html

The second thing is you need to update git.

https://nivethan.dev/devlog/updating-git-on-rhel-7.4.html

Now that we have the set up done, we can set up sourcegraph and the repositories it will be indexing.

In my case, I don't actually have git repositories set up and so I'll need to use the src-expose utility to generate git repos and it will also serve those repos to source graph.

curl https://storage.googleapis.com/sourcegraph-artifacts/src-expose/latest/linux-amd64/src-expose -o /usr/local/bin/src-expose
chmod +x /usr/local/bin/src-expose

Now that it is installed, we can do the following:

src-expose dir_1 dir_2 dir_3

This will create git repos in ~/.sourcegraph/src-expose-repos of the directories that we want to index. src-expose will also serve these repos on port 3434. This port will needed to be opened.

Next we can install and run sourcegraph through docker.

docker run --detach --restart unless-stopped --publish 7080:7080 --publish 127.0.0.1:3370:3370 --volume ~/.sourcegraph/config:/etc/sourcegraph --volume ~/.sourcegraph/data:/var/opt/sourcegraph sourcegraph/server:4.1.3

This will serve the sourcegraph application on port 7080 which will also need to be opened on the firewall. The --detach and --restart make it so that the application runs in the background and that sourcegraph will restart if it fails or if the machine reboots.

We can check if the docker container is running by doing a ps:

docker ps

We can stop the container manually by doing:

docker stop {CONTAINER_ID}

Now that the sourcegraph container is running we can set up our admin account.

We can navigate to {IP}:7080 and this will give us the admin creation page.

Once the admin account is created, we can then go to the "Manage code hosts" page and add a "Generic Git Host". Our src-expose command will need to still be running.

This is the config that should be added.

{
    "url": "http://192.168.13.12:3434",
    "repos": ["src-expose"]
}

With that we should be good to go! We should now be able to go to the main sourcegraph page and see a search bar and begin searching and viewing the files.

Now that sourcegraph is working, we need to push src-expose to the background. To do this, I wrote a yaml config file to specify the directories to serve and a systemd unit file.

The yaml file, this goes in /root/src-expose-config.yaml:

root: /home/

destination: /root/.sourcegraph/src-expose-repos

dirs:
    - dir: ./nivethan/projects/project1
    - dir: ./nivethan/projects/project2
    - dir: ./nivethan/projects/project3

The unit file, this goes in /etc/systemd/system/src-expose.service:

[Unit]
Description=src-expose Daemon
After=network.target

[Service]
ExecStart=/usr/local/bin/src-expose -config /root/src-expose-config.yaml
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Now we can do:

systemctl start src-expose.service
systemctl enable src-expose.service

Now both sourcegraph and src-expose are running in the background and will start up when the machine reboots.