DEV Community: Christian

ZenGPT: a simple ChapGPT alternative frontend

Christian — Wed, 18 Oct 2023 09:21:47 +0000

Originally posted on cri.dev

I've been playing around with a home-made, super simple ChatGPT UI clone, mainly with the excuse to try out htmx and Alpine.js

It's a fun little project that I've been working on for a few days.

I've been programming it on an iPad Pro (as my main device), and it's been a fun experience.

In this post I want to get deeper into the technical details of the project, and share some of the things I've learned while working on it.

node.js server and bare functions as views

The other day I wrote about functions as views, and how I've been using them in this project.

Namely, I'm using the native http node.js module, a simple home made router with if statements and a few functions that return HTML strings.

The functions are called with optional additional data and they return a string that is sent back to the client.

E.g.

if (req.url === '/') {
  res.setHeader('Content-Type', 'text/html')
  return res.end(mainView(messages, listing()))
}

Integrating with OpenAI's API

The Completions API is pretty straightforward too:

You can use the chat.completions.create method to send the conversation to the API, and get back a text completion (llm response/message).

const completion = await ai.chat.completions.create({
  model: 'gpt-3.5-turbo',
  messages: messages
              .concat([{ role: 'user', content: newUserMessage }]),
})

let llmText = completion.choices[0].message.content.trim()

htmx

As mentioned before, the main reason I started this project was to try out htmx (and Alpine.js)

The coolest thing I refreshed during this excursus was the concept of rethinking what we consider RESTful APIs (spoiler: they are actually HTTP JSON RPC APIs), HATEOAS, hypertext, and much more honestly. The htmx website is a goldmine of resources.

In short: our websites and "RESTful" APIs should be way more discoverable (for humans, not machines) and self-contained.

By making use of existing powerful technology like HTML and HTTP, with sprinkles of JavaScript, to make the web more accessible, lightweight, more reliable and easier to use.

But let's got back to htmx.

build modern user interfaces with the simplicity and power of hypertext

The emphasis here is leveraging the power of HTML.

E.g. in this small project, the client is loaded with a simple HTML page, preloaded with messages from the server.

The rest (sorry for the poor choice of words) is done by the client, that makes requests to load small snippets of HTML, and updates the DOM with the response.

This is an oversimplification, but it's the gist of it.

By using a declarative approach on the HTML, you can get a quite robust and powerful UI, with very little code.

The main focus of ZenGPT is the UI, this is the input and conversation part:

<div id="messages" hx-swap="scroll:bottom">
  ${renderMessages(messages)}
</div>
<input
  name="message"
  hx-post="https://cri.dev/chat"
  hx-trigger="keyup[keyCode==13]"
  hx-target="#messages"
  hx-swap="beforeend scroll:bottom"
  hx-indicator="#loading-message"
  hx-on:htmx:before-request="this.disabled=true"
  hx-on:htmx:after-request="this.disabled=false;setTimeout(() => this.focus(), 20)"
  x-bind:disabled="messageDisabled"
  x-ref="message"
  x-model="message"
  x-on:keyup.enter="setTimeout(() => {message = '';pristineChat = false}, 10)"
  class="my-message" autofocus type="text" placeholder="your message">
<div style="position:fixed;bottom:3em;right:2em;" class="htmx-indicator" id="loading-message">
  <svg xmlns="http://www.w3.org/2000/svg" version="1.1" width="64" height="64" viewBox="0 0 24 24">
    <path fill="none" stroke="#000" stroke-width="2" stroke-linecap="round" d="M12 2 L12 6 M12 18 L12 22 M4.93 4.93 L7.76 7.76 M16.24 16.24 L19.07 19.07 M2 12 L6 12 M18 12 L22 12"></path>
</div>

This is so-called "no-javascript", where the JS is simply hidden (remember you need to include a <script src="https://cri.dev//unpkg.com/htmx.org">). I think it's a refreshing approach.

Alpine.js

In ZenGPT, Alpine.js is used to manage the state of the UI, and to make client-side only UI changes and updates.

It is used to add interactivity to the UI.

E.g. it handles the display state of the action buttons in the header

<div style="display:flex">
  <div style="flex:1";><h1>zengpt</h1></div>
  <div x-show="!pristineChat" style="flex:1;";>
    <button style="display:block;padding:1rem;font-size:1.5rem;" hx-delete="https://cri.dev/chat" hx-target="#messages" x-on:click="$refs.message.focus();messageDisabled=false;pristineChat=true">
      new chat
    </button>
  </div>
  <div x-show="!pristineChat" style="flex:1;";>
    <button style="display:block;padding:1rem;font-size:1.5rem;" hx-post="https://cri.dev/chats" hx-target="#messages" x-on:click="$refs.message.value = '';messageDisabled=false;pristineChat=true">
      save chat
    </button>
  </div>
  <div x-show="viewingPreviousChat" style="flex:1;";>
    <button style="display:block;padding:1rem;font-size:1.5rem;" hx-get="https://cri.dev/chat" hx-target="#messages" x-on:click="$refs.message.value = '';messageDisabled=false;">
      go back
    </button>
  </div>
</div>

open source

You can find the project on github.com/christian-fei/zengpt

Originally posted on cri.dev

Simple Static site/blog search in 14 Lines of JavaScript

Christian — Sun, 11 Apr 2021 10:11:08 +0000

Originally posted on cri.dev

This is my approach to a purely client side search feature for static blogs and sites.

I am currently using this under /posts to let readers search through my blog posts.

Read more below about how to integrate it in your site.

Backstory

In 2015 I did a similar thing for Jekyll sites, namely Simple-Jekyll-Search

It's gotten quite a bit of attention and merged various Pull-Requests (64!) from others.

I ditched Jekyll a few years ago, to use a JavaScript based static site generator:

My home-made static site generator devblog, and settled with Eleventy

How it works

Under /posts I render the full list of blog posts.

(Not with the full content but with an excerpt/description.)

In my Nunjucks template I render the posts like this:

<div class="searchable">
  <input type="text" autofocus placeholder="🔍 Search posts"/>
  {% set postslist = collections.post | reverse %}
  {% for post in postslist %}
    <div class="searchable-item" data-search="{{ post.data.title | escape }} {{ post.data.tags | json | escape }}">
    ...
    </div>
  {% endfor %}
</div>

The idea is the following:

Within a .searchable element

look for a input element and attach an event listener to it (keyup event)
when searching, parse all .searchable-items data-search attribute and test a RegExp on it
make items that match visible, and make others disappear

Code

This is the full code for the search functionality on this blog:

;(function search () {
  ;[...document.querySelectorAll('.searchable')].forEach(makeSearchable)
  function makeSearchable ($searchable) {
    const $searchableItems = [...$searchable.querySelectorAll('.searchable-item')]
    const $search = $searchable.querySelector('input')
    $search.addEventListener('keyup', (e) => {
      $searchableItems.forEach(function ($el) {
        const text = $el.getAttribute('data-search') || $el.innerText
        const show = new RegExp(e.target.value, 'i').test(text)
        $el.style.display = show ? '' : 'none'
      })
    })
  }
})()

How to integrate it on your site

Grabbed the JavaScript above and put it in a <script> tag.

Define a minimal markup to enable the search functionality:

<div class="searchable">
  <input type="text" autofocus placeholder="🔍 Search posts"/>
  <!-- Loop through your blog posts -->
  <div class="searchable-item">
  ...
  </div>
</div>

Make sure you have the wrapper element .searchable, containing an input field and your post items with the .searchable-item class.

By default the search function will match the blog posts to filter with the element's innerText.

Optionally, apply a data-search attribute to each .searchable-item and set the value of your blog post title, tags, short description etc.

Full example

Check out a full example on CodePen if you need to better understand how to structure your markup.

Originally posted on cri.dev

5$/month Self Hosted VPN with WireGuard

Christian — Thu, 01 Apr 2021 11:07:34 +0000

Originally posted on cri.dev

This will take you less than 5 minutes to get a private VPN on your own Ubuntu server.

What you need to get started:

a VPS (Ubuntu 20.04 LTS preferrably)
docker and docker-compose installed

Setting up docker-compose

On the VPS, I suggest to do the following:

Create a folder wireguard/config in your $HOME:

mkdir -p wireguard/config

Inside the wireguard folder (next to the config folder), create the docker-compose file:

vim docker-compose.yml

With the following contents:

version: "2.1"
services:
  wireguard:
    image: linuxserver/wireguard
    container_name: wireguard
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Rome
      - SERVERURL=YOUR_IP_OR_DNS_NAME_OF_YOUR_SERVER #optional
      - SERVERPORT=51820
      - PEERS=5
    volumes:
      - /home/YOUR_USERNAME/wireguard/config:/config
      - /lib/modules:/lib/modules
    ports:
      - 51820:51820/udp
    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1
    restart: always

Simply change the SERVERURL variable, or delete that line if you want to use the server IP.

Additionally, change the location of your wireguard config path in the volumes section. (use pwd to get the current path you are in).

For more info and environment variables, check out the official linuxserver/wireguard doc.

I've used restart: always so that WireGuard comes up after a system restart.

Changed PEERS to 5, so that I have 5 configurations available for my devices.

Launching the container

Start the container in the background with

docker-compose up -d

This will create the needed configurations in the wireguard/config folder.

The structure of your wireguard folder looks something like this

.
├── config
│   ├── coredns
│   │   └── Corefile
│   ├── peer1
│   │   ├── peer1.conf
│   │   ├── peer1.png
│   │   ├── privatekey-peer1
│   │   └── publickey-peer1
.............................
│   ├── server
│   │   ├── privatekey-server
│   │   └── publickey-server
│   ├── templates
│   │   ├── peer.conf
│   │   └── server.conf
│   └── wg0.conf
└── docker-compose.yml

Firewall configuration

If you're using ufw, simply enable the port 51820 so that you can connect to your server from outside:

ufw enable 51820
ufw reload

If you run sudo ufw status you should see:

Status: active

To                         Action      From
--                         ------      ----
....................................................
51820                      ALLOW       Anywhere
51820 (v6)                 ALLOW       Anywhere (v6)

Connecting devices

Now you can cat or scp the configurations individually to your devices.

The configurations are located in the config/peerX folder, where X represents the peer number.

E.g. I could cat/scp the configuration in wireguard/config/peer1/peer1.conf and put it in /etc/wireguard/wg0.conf on my host machine.

View the configuration for peer 1 on your server with

cat /home/YOUR_USERNAME/wireguard/config/peer1/peer1.conf

and place it in /etc/wireguard/wg0.conf on your host machine.

If you want to use scp, you could run the following on your local machine:

scp USER@SERVER:/home/YOUR_USERNAME/wireguard/config/peer1/peer1.conf /etc/wireguard/wg0.conf

To connect to your newly created WireGuard VPN from one of your devices, you'll need to install wireguard-tools.

E.g. apt install wireguard-tools , pacman -S wireguad-tools based on your distro

Here you can find a more detailed explanation

Now you can simply run wg-quick up wg0 and you're connected to your VPN.

Test it out by running curl ipinfo.io and inspect the output.

Connecting mobile device with QR code

On your mobile device, install the WireGuard client.

Then add a new WireGuard tunnel by creating a new configuration scanning a QR code.

On your VPS run the following to output a QR code on the terminal that you can scan on your mobile device:

docker exec -it wireguard app/show-peer 1

Inspecting connections

If you want to understand who is connected and which profiles are in use, simply run the following on your VPS:

docker exec -it wireguard wg

This will give you more information about your connections with the following output:

interface: wg0
  public key: (redacted)
  private key: (hidden)
  listening port: 51820

peer: (redacted)
  endpoint: (redacted):51820
  allowed ips: (redacted)/32
  latest handshake: 27 seconds ago
  transfer: 5.04 MiB received, 172.64 MiB sent

...

peer: (redacted)
  allowed ips: (redacted)/32

Conclusion

This was most definitely the easiest way I found to connect computers and mobile devices to your own WireGuard VPN.

Let me know if you had troubles setting it up yourself!

Originally posted on cri.dev

Self-hosted Miniflux + Wallabag

Christian — Thu, 01 Apr 2021 11:06:58 +0000

Originally posted on cri.dev

Found the perfect combination for self-hosting my reading stack: Miniflux and Wallabag, you may now kiss!

Note: This is an update to "My reading stack"

Why?

Primarily for keeping the stuff I read private by self-hosting alternative services.

Additionally, to do all my reading from a web browser, without third party services and apps involved.

Another thing I wanted to experiment with was the interoperability between Miniflux and Wallabag.

How does it work?

Miniflux acts as my RSS feed reader, which integrates with the Wallabag API.

Wallabag is the alternative I use to Pocket, especially for longer reads.

In Miniflux I set up the integration to my Wallabag instance, so that when I hit "Save article", it sends it to Wallabag so that I can read it later (if it's a long read).

I access both services (under practical dns names) on my own domain (both from phone and PC).

Benefits

Both Miniflux and Wallabag can be "installed" as a PWA on your mobile device and accessed through a web browser on your PC.

No profiling, no tracking involved.

Extremely lightweight stack.

You use FOSS software, to which you can contribute to (or fork it).

Downsides

Minimum cost of 5$/month for hosting it on Linode or DigitalOcean.

You need to make sure your VPS is secured, duh.

So far I didn't see any other downsides.

How to self-host

Both Miniflux and Wallabag have docker-compose files in their repositories, so the installation is quite trivial.

You'll need to figure out the desired environment variables depending on your needs.

I host mine through caprover, through a Web UI it's a few clicks away.

I plan on writing a detailed how to in the future.

Originally posted on cri.dev

Resources to learn Elixir - A study path

Christian — Thu, 01 Apr 2021 11:05:05 +0000

Originally posted on cri.dev

Below you can find resources for getting started with Elixir.

It's what I am following and it could be helpful for new programmers that want to learn Elixir.

Getting started

Official guide

The official "Getting started" guide by Elixir-lang.org

elixir-lang.org official Getting started

You can find my notes here

Books

Programming Elixir by Dave Thomas is the go-to book if you want to learn all about Elixir.

Elixir in Action by Saša Jurić is also highly recommended.

Crash course about syntax

This is another official resource I can highly recommend for learning about Elixir's syntax

elixir-lang.org syntax crash course

Elixir school

This course will get you through the basic concepts (similar to the guide on elixir-lang) and more advances topics about Elixir.

It also goes briefly into Ecto and other specific module often used in Elixir applications.

Visit elixirschool.com to learn more.

Elixir Videos

Alchemist Camp

If you want code tutorials in form of video, Alchemist Camp can be a great place to start.

You can also find the videos on YouTube.

Talks

Saša Jurić's talk at GOTO 2019 about "The Soul of Erlang and Elixir" is highly recommended

Paid video courses

To learn about the basics of the language, you could check out learnelixir.tv.

For 18$ you can watch 30 episodes, as of March 2021.

A more complete bundle of video courses can be elixircasts.io.

At the time of writing there are around 130 episodes that guide you through various scenarios and examples on using Elixir.

This video library can be watched for 19$ / month.

awesome-elixir

github.com/h4cc/awesome-elixir is an infinite list of Elixir libraries & resources

Books & resources

A more exhaustive list of other resources to learning Elixir can be found on elixir-lang.org

Erlang

Learn you some Erlang

If you want to get your hands dirty understand concepts in Erlang, Learn you some Erlang can be a highly valuable resource.

In-depth

There is also an official Erlang course which can be used for learning more about Erlang and important concepts of the language and VM.

It will usually take four days to complete.

You'll learn more about the History of Erlang, Sequental / Concurrent Programming, Error handling and other advanced topics.

Originally posted on cri.dev

Bitcoin is trademarked. Is it really?

Christian — Sat, 20 Feb 2021 10:08:56 +0000

Originally posted on cri.dev

Useless to say, but here we go

I'm not a lawyer.
This is not legal advice.
This is for informational and educational purposes only.
This is my opinion.
May contain incorrect interpretations of the law.
May contain traces of peanut.

Bitcoin, the cryptocurrency invented in 2008 by an unknown person or group of people using the name Satoshi Nakamoto, is trademarked? Is it really?

I did some "research" and this is what I found.

United Kingdom

First of all, the term "bitcoin" for clothing/food/drinks, seems to be registered in the United Kingdom through the Intellectual Property Office (IPO).

The registration of the trademark dates back to December 22, 2017 (source).

Apparently a merchant on etsy.co.uk was selling t-shirts with the bitcoin logo on it.

This is the original Reddit post under /r/Bitcoin.

They received an adorable cease-and-desist letter from a company "holding the trademark", namely "A.B.C. IPHOLDINGS SOUTH WEST LLC". (source)

The lovely cease-and-desist letter is also still available. With a heartwarming "Yours sincerely," at the end of the letter.

Do a quick search for the term "A.B.C. IPHOLDINGS SOUTH WEST LLC" and look through the 30k+ results for this company.

The goods and services covered by this trademark are from the classes 25, 32 and 33.

Here some examples (just for giggles):

Adhesive bras
Ballroom dancing shoes
Bowties
Camouflage jackets
Flip-flops
Kimonos
Judo suits
Mittens
Silk scarves
Taekwondo suits
Waterskiing suits
Wooden bodies for Japanese style clogs
Yoga pants
Zoot suits

And this was only class 25, representing Clothing, footwear, headgear. (source).

Class 32 represents non-alcoholic beverages including beers. (source).

Class 33 represents Alcoholic beverages excluding beers. (source).

I'm not a lawyer. But this seems fishy.

What's the point of registering a trademark for which you don't have any business to?

Could the seller have gotten away with it? Or even sue them for false litigation?

What is this? Getting a trademark registration for the sole purpose of following legal action to who "abuses" it?

This seems to be pure "trademark trolling".

Here is the opinion of "The Bitcoin foundation" on the matter

It seems the TM application has been dropped, although it seems to still be registered and active on UK IPO official website.

Other parts of the world

Looking through trademarks.justia.com you can find more registration (I think at an international level).

There are 14 registrations for the word "Bitcoin" in various categories.

Two of which are currently marked with the status "Registered".

The other 12 were either express abandoned or suspended due to inactivity by the party.

The 2 ones that are currently registered are regarding

No but seriously, WTF?

The latter one is registered by a german company that produces confectionery products.

Conclusion

The Bitcoin logo itself is released with the public domain license.

Therefore it is free to use for both commercial and non-commercial use.

I don't know what these companies were thinking, and if they are in good or bad faith.

I mean, how ridiculous do you have to be to trademark "bitcoin" for coffee, tea, salt mustard, or alcoholic beverages?

Are you serious?

Additionally this is what I found when looking for the bitcoin logo:

On Wikipedia it is explicitly stated

This work is ineligible for copyright and therefore in the public domain because it consists entirely of information that is common property and contains no original authorship.

But at the same time regarding Trademark

This work includes material that may be protected as a trademark in some jurisdictions. If you want to use it, you have to ensure that you have the legal right to do so and that you do not infringe any trademark rights

Useful resources

Resources I found useful about this topic

Originally posted on cri.dev

A retrospective of 100k yearly pageviews

Christian — Sat, 20 Feb 2021 10:08:08 +0000

Originally posted on cri.dev

Incredible!

70k visitors / 100k pageviews (excluding those with analytics blockers installed) reached this very website in the last year.

Around July 2020 I switched to Plausible Analytics and made my dashboard public.

In 2020 a lot happened:

SEO improvements and mistakes
Domain name change from christianfei.com to cri.dev
HackerNews "virality" and backlinks
Reddit experiments
93 blog posts in 2020
Numerous design iterations
Ethical ads

Let me explain in detail.

SEO improvements

Learned quite a bit in the last year about effective SEO.

Even wrote a short ebook about how to improve SEO for your site.

Sometimes syndicating content to dev.to.

Most people coming from search engines land by looking for:

an Ansible error (ansible undefined variable)
Node.js 14 optional chaining + nullish coalescing
Puppeteer issue (err_invalid_argument)
HTTP error 431 in Node.js (431 request header fields too large nodejs)

This means a lot to me, since I started this blog to document my findings and learnings, and how to solve programming related issues.

Still, I need to learn a lot about effective SEO and keyword planning/strategies.

SEO mistakes

As mentioned above, around February 2020 (I think) I switched domain from christianfei.com to cri.dev

During this procedure, I made the mistake of not properly configuring the Canonical URL!

This affected the discovery quite a bit, for a month or so.

Then after figuring out the issue and implementing a fix, things got back to normal.

Google Search Console was vital to get this right.

HackerNews and Reddit

HackerNews and Reddit are both a blessing and a curse sometimes..

You post something you think could be interesting, and nobody checks it out.

Other times, you submit your post, and it blows up.

Don't let me get started on the discussion on these boards.

People on there might comment with kind words and appreciation, and in the same thread find your greatest enemy and hater.

I guess it's like in real life ^^

Anyways, here is the blog post that brought most visitors on my site through HackerNews.

Namely Raspberry Pi as a local server for self hosting applications

This seriously blew up: at the peak I had around 250+ realtime visitors, and the wave lasted for days.

To this day, this blog post brought over 24k visitors and 35k pageviews, and loads of backlinks.

No Hug-of-death fortunately, since this site is purely static, hosted on AWS with CloudFlare in front..

Here are some other posts that got people interested:

93 blog posts

I sat down and wrote about whatever I learned or felt like was useful to share, almost every 4 days on average.

Shared some in my newsletter over the year.

Writing these blog posts gives me the ability to share what I learned, and to declutter my head.

This is a very useful practice that I definitely want to continue throughout 2021.

Design iterations

Tried different designs, various CSS frameworks, bare handwritten CSS at some point, changed static site generator etc.

I think this is quite common for personal blogs.

At some point I had a sidebar in the post view. Now I removed it because it hurt the reading experience.

I tried to improve the navigation header over time, and now it has a proper mobile view with all the relevant links.

Ethical ads

Switched from carbonads to ethicalads a month ago.

It has pros and cons.

Pros: there is no targeting, option to show textual ads, no cookies being placed. GDPR hooray!

Cons: payout is once 50$ are reached (instead of monthly payout).

Currently I'm making around 5$ a month, so it's going to take some time to see the first payout.

Next steps

This year I want to get my SEO game to the next level.

More quality content, less quick fixes and short how tos.

Consistency in writing blog posts regularly, but with more focus on quality.

If you want to stay in touch, do so by subscribing to my newsletter

Originally posted on cri.dev

From Mac to Linux

Christian — Sun, 03 Jan 2021 02:03:45 +0000

Originally posted on cri.dev

I am switching from Mac to Linux.

This is how the adventure started to find the best linux distro, with a seemless user experience and minimum maintenance.

In the past months I experimented with various distros:

bare debian
Pop!_OS
Ubuntu
Elementary OS

Read below to read my experience and to which distro I settled using.

Experience recap

with bare debian

Driver support for WiFi and Bluetooth were absent out of the box.

Still too much to adjust and tweak to get a decent user experience.

Not for me.

with Pop!_OS

Seemed like a fresh approach to the linux distro landscape, but I had major hiccups when it comes to display drivers.

Weird issues like blank screen after switching to HDMI external display. Troublesome boots when display was attached.

Weird glitches and graphical artifacts when resizing windows.

No issues with drivers.

I bet it works great on System76 machines though.

with Ubuntu

Glitches when working with external monitors.

Blank screen after booting connected via HDMI.

Weird behaviour when putting laptop in clamshell mode. Either not turning on, or shortly showing the login screen and then going to sleep.

No issues with drivers.

Recurrent issues with the confusing Software center versions not compatible with the applications on it.

Amazon EKS keeped popping up as a "promoted" application, without the possibility to remove it.

Bye Ubuntu.

Elementary OS

No issues with drivers.

No issues with external monitor. Wake and sleep with HDMI works like expected.

No blank screen when booting from external monitor.

Decent software center (with donations built in), also using snap was great.

Slight issues when disconnecting the external monitor and using the laptop, works after a few tries.

So far so good.

Elementary OS is good enough for me

The user experience is stunningly similar to what I was used to with a Mac.

Switching windows is intuitive.

If you forget a shortcut, just press <SUPER>.

With snap it almost feels like using brew and brew cask.

Using it in clamshell mode works like a charm.

The system even wakes up from sleep using a Bluetooh logitech keyboard with mouse (namely the Logitech k400 I think).

Still need to find a better Terminal emulator, because I'm not really fond of the one built-in. Perhaps time to try Alacritty?

This is how I am writing this very blog post.

Originally posted on cri.dev

Mapping historic market data

Christian — Sat, 02 Jan 2021 13:33:57 +0000

Originally posted on cri.dev

Market data is generally represented as an array of arrays, and each entry is a data point that represents a "snapshot" of the market in a given period of time.

Each data point contains information like the timestamp, open/close price, high/low price and the volume of transactions in this period.

Here is a recent sample of a few data points for the currency pair LTC-EUR on the market gdax:

const marketData = [
  [
    1513025761,     // time
    161,            // low
    163.11,         // high
    162.98,         // open
    162.67,         // close
    823.99626973    // volume
  ],
  [
    1513024062,
    157.22,
    164.01,
    157.5,
    163,
    5343.671050650007
  ]
]

map to array of objects

To get a more pleasant array of objects to work with, you can map the array structure to an Object with named property like this:

marketData.map(h => ({
  time: h[0],
  low: h[1],
  high: h[2],
  open: h[3],
  close: h[4],
  volume: h[5]
}))

And the result would be the following:

[
  {
    "time": 1513025761,
    "low": 161,
    "high": 163.11,
    "open": 162.98,
    "close": 162.67,
    "volume": 823.99626973
  },
  {
    "time": 1513024062,
    "low": 157.22,
    "high": 164.01,
    "open": 157.5,
    "close": 163,
    "volume": 5343.671050650007
  }
]

Map to array of objects grouped by key

It can also be interesting to have the data grouped by key.

You can achieve this, reducing each key to an array of the corresponding values:

Object.keys(marketData[0]) // ['time','low','high','open','close','volume']
.reduce((acc, key => ({
  ...acc,
  [key]: marketData.map(data => data[key]) // returns an array of values for each key
}), {})


{
  "time": [1513024062, 1513025761],
  "low": [157.22, 161],
  "high": [164.01, 163.11],
  "open": [157.5, 162.98],
  "close": [163, 162.67],
  "volume": [5343.671050650007, 823.99626973]
}

Discuss these thoughts with me on twitter!

Originally posted on cri.dev
➜ christian-fei.github.io git:(master)

Functional JavaScript: Write your own negate/not function

Christian — Sat, 02 Jan 2021 13:32:36 +0000

Originally posted on cri.dev

Let's say you want to filter odd numbers from a list of numbers, like [1,2,3,4,5,6] should become [1,3,5].

One way to solve this problem in a functional fashion is to provide an odd filter function, that is internally composed by a negation of an even filter on those numbers.

So, let's define these functions step by step:

function odd(number){
  return not(even)
}

function not(predicate){
  return function(){
    return !predicate.apply(this, arguments)
  }
}

function even(number){
  return number % 2 == 0
}

The even function needs no explaination, where the functional magic ® happens is in the odd and not functions.

odd takes a number and uses the even filter, by negating it with not, to determine whether a number is odd or not.

not just wraps the predicate function passed in and negates its application. Note that this application makes sense with functions that return boolean values, thus the name predicate.

One use case could be in combination with Array.prototype.filter:

[1,2,3,4,5,6].filter(not(even))

// or

[1,2,3,4,5,6].filter(odd)

Originally posted on cri.dev
➜ christian-fei.github.io git:(master)

What is a retrospective?

Christian — Thu, 31 Dec 2020 05:25:44 +0000

Originally posted on cri.dev

What is it?

Comes from the necessity to regularly reflect about how project / team is going
It’s an interactive way to focus on past things and see if there is a way we can improve them through a team effort and collaboration

The Prime Directive

Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.

What do I get from it?

Discussion about important topics
- already a step in the right direction
Actions to improve in the future
Grow, connect as a team

4 questions retrospective

What went well?
What didn’t go so well?
What have I learned?
What still puzzles me?

We look at the past to improve in the future

1. What went well?

we start with stating positive things to ensure we continue on doing those positive things in the future
State a fact
Saying thx is cool
You can go into details

2. What didn’t go so well?

Difficulties, issues, dissatisfactions
“Where better to look to make improvements than where things are not going so well?”
Something less than ideal
Give observations, not judgement
“We spent a long time making the decision on the ordering process”, rather than “it shouldn’t take us long to decide the ordering process”

3. What have I learned?

Positive + negative things
> Observations on the past Create Opportunities for the Future

4. What still puzzles me?

Questions!
Concerns
Doubts
Worries

Turning 4 questions into improvements

We have some insights now
- Identify actions to improve in the future
- Volunteer for action
Remember that the volunteer is not necessarily the person doing the actions, they should remind the team about the action and be mindful about respecting the action

Ultimate web scraping with browserless, puppeteer and Node.js

Christian — Mon, 28 Dec 2020 16:36:21 +0000

Originally posted on cri.dev

Browser automation built for enterprises, loved by developers.

browserless.io is a neat service for hosted puppeteer scraping, but there is also the official Docker image for running it locally.

I was amazed when I found out about it 🤯!

Find the whole source code on Github christian-fei/browserless-example!

Running browserless in docker

A one-liner is enough to have a full puppeteer backend, with configured concurrency etc., to leverage using puppeteer.

You can connect to a browserless backend by passing the option browserWSEndpoint like this:

async function createBrowser () {
  return puppeteer.connect({ browserWSEndpoint: 'ws://localhost:3000' })
}

To start the backend you can use the following command, using the docker image browserless/chrome:

docker run \
  -e "MAX_CONCURRENT_SESSIONS=15" \
  -e "MAX_QUEUE_LENGTH=0" \
  -e "PREBOOT_CHROME=true" \
  -e "DEFAULT_BLOCK_ADS=true" \
  -e "DEFAULT_IGNORE_HTTPS_ERRORS=true" \
  -e "CONNECTION_TIMEOUT=600000" \
  -p 3000:3000 \
  --rm -it browserless/chrome

Source code

Find the whole source code on Github christian-fei/browserless-example!

You'll find a web crawler with puppeteer!

git clone https://github.com/christian-fei/browserless-example.git
cd browserless-example
npm i

npm run start-browserless
node crawl-with-api.js https://christianfei.com

Puppeteer using browserless docker backend

You simply connect to the Browser WebSocket Endpoint ws://localhost:3000 and you're connected to the browserless backend!

Here is a short example of getting all links <a> on christianfei.com:

const puppeteer = require('puppeteer')

main(process.argv[2])
  .then(err => console.log('finished, exiting') && process.exit(0))
  .catch(err => console.error(err) && process.exit(1))

async function main (url = 'https://christianfei.com') {
  const browser = await createBrowser()
  const page = await browser.newPage()
  await page.goto(url)
  console.log('title', await page.title())
  const links = await page.evaluate(selector => [...document.querySelectorAll(selector)], 'a')
  console.log('links.length', links.length)
}
async function createBrowser () {
  return puppeteer.connect({ browserWSEndpoint: 'ws://localhost:3000' })
}

An example video: