DEV Community: Ray-D-Song

I built a Github user analysis and ranking website

Ray-D-Song — Mon, 25 Nov 2024 05:08:34 +0000

https://github-persona.pages.dev

Why

Sometimes I treat participating in Github as a game, like I will follow some big guys, see what they are doing recently, whether they have any new projects, any new ideas.

Then I realized that submitting code, participating in discussions, and solving problems can be quantified. I can make a website to analyze Github users, rank them, just like in the game.

Tech Stack

Cloudflare Pages
Cloudflare D1
Cloudflare AI

I use Cloudflare Worker to develop it, because it has convenient CI/CD service and free quota.

One of the requirements is to use large models to summarize users. Cloudflare AI perfectly meets this requirement.

Conclusion

This project is completely open source, you can view the source code on Github.

Welcome to join and improve it together.

How to detect code language in browser

Ray-D-Song — Fri, 22 Nov 2024 01:47:57 +0000

Repository: https://github.com/ray-d-song/guesslang-js

Demo: https://ray-d-song.github.io/guesslang-js/

Recently, I'm working on a project called EchoRSS, and I have a very wanted feature, which is to intercept external links in subscriptions (read full text, quote, etc.) and display them directly on the current page.

There is a problem that the returned HTML code block loses the language annotation (or the language was not annotated on the pre and code tags in the original code block), so it cannot be highlighted using tools like shiki or prism.js.

I found three solutions to detect code language:

1. linguist

This is a Ruby project deployed on the server, and Github uses it to detect the language composition of the repository. If you need extremely high accuracy and can be calculated on the server, this is the best solution.

2. hljs

highlight.js is a very famous web code highlighting library, and it is also the only library that provides automatic code detection.

The principle is very simple, which is to enumerate the keywords of the language, and then match them one by one with the text, and finally see which one has the highest matching degree.

hljs has four problems.

It requires a very long code length, and most languages require at least 300 characters to achieve a relatively good accuracy.
The part that detects the language is not a separate module, but tightly coupled with the parser and render, and the code is also very imperative, making it difficult to extract useful parts.
If you don't extract the detection module, the original format (line breaks and indentation) of the code will be lost when using hljs to highlight.
It requires a lot of regular matching, the performance is poor, and because of reason 2, it cannot be run in a web worker.

3. guesslang

guesslang is a machine learning project based on tensorflow.js.

Microsoft ported this project to node.js in 2021 and added the automatic language detection function to vscode.

A Vietnamese guy hieplpvip three years ago also ported this project to the browser, but there are also three problems:

Memory leak, memory leak...
Only supports the <script> tag to introduce the umd format, does not support esm, does not support bundle
Similarly, because of reason 2, it does not support web worker

The guy has not maintained this project, and the feat request to support esm in March has not been replied.

So I extracted the detection module from hljs, and forked guesslang-js to fix the above problems, and in the end guesslang won, the result is this:
https://github.com/ray-d-song/guesslang-js

I think it's too much to talk about, maybe someone will need it in the future, so I'll post it.

If someone knows tensorflow.js, I hope they can recommend some learning materials, I want to further modify it to web gpu calculation to improve efficiency.

The difference between Error and Exception in JavaScript

Ray-D-Song — Sun, 17 Nov 2024 14:59:57 +0000

https://github.com/Ray-D-Song

Error and exception are concepts born from practice, aiming to handle "programmable errors".

Error

From a code perspective, error tends to be manually handled precisely.

For example, fnA calls fnB and fnC. Both methods may encounter errors, and the processing code is roughly as follows:

function fnA() {
  const { err: bErr, res: bRes } = fnB()
  if (bErr) {
    // ...
    // error handling
  }

  const { err: cErr, res: cRes } = fnC()
  if (cErr) {
    // ...
    // error handling
  }
  // normal logic
}

The key to "error" is to return an object or array from the function, with one field representing "an error occurred". As long as this field is not empty, programmers know that the normal flow has been interrupted.

JavaScript has an internal Error object and constructor, but the field representing the error is not required to be an Error object. Instead, the Error object is more often used in exception handling.

Exception

We already have error handling, why do we need exception?

Imagine a scenario where you have a button. When the button is clicked, it triggers function A. After going through multiple layers of calls (perhaps 10 layers), an error occurs in function X. You don't want to tell the user "unknown error", but rather want to provide specific information about what went wrong.

You can achieve this effect with errors, but you need to write ten times this code:

function fnA() {
  const { err, res } = fnB()
  if (err) {
    // display error to user
    showErr(err)
  }
}

function fnB() {
  const { err, res } = fnC()
  if (err)
    // propagate error
    return { err, null }
}

// ... 10 similar passes

function fnY() {
  const { err, res } = fnX()
  if (err)
    // propagate error
    return { err, null }
}

This kind of boilerplate code is very inefficient. A better method is to use exceptions.

You only need to throw an exception when an error occurs in fnY. At the top level, you can catch it.

function fnA() {
  try {
    fnB()
  } catch (e) {
    showErr(e)
  }
}

// ...

function fnY() {
  const { err, res } = fnX()
  if (err)
    // 抛出
    throw err
}

In this way, no matter where an error occurs, it can be caught at the top level, and the code in other layers is not affected.

Avoid polluting the entire code structure with errors at one place.

Why distinguish between the two?

We have already explained why we need exceptions, but why do we need to distinguish between errors and exceptions?

The best practice is to strictly distinguish between the two. If an error does not need to be passed up layer by layer, it should be handled directly in the current layer. For example, the error of fnC does not need to be used in fnA, so it should be handled as an error directly in B.

Assume that all errors are handled at the top level, then all logic is piled up in the catch block at the top level, which is difficult to maintain.

function main() {
  try {
    task1()
    task2()
    task3()
  } catch(e) {
    switch(e) {
      case "type A":
        //...
        break;
      case "type B":
        //...
        break;
      case "type C":
        //...
        break;
    }
  }
}

I built a full-stack web archive tool running on Cloudflare

Ray-D-Song — Sun, 10 Nov 2024 03:47:41 +0000

Project address: https://github.com/ray-d-song/web-archive

Why build this tool

I have been a loyal user of ArchiveBox for a long time. ArchiveBox is a very good web archiving tool, but it requires self-hosting and has high server requirements (requires headless browser). I used a Raspberry Pi before, and the performance was not good.

And for websites like x and Medium, which require login, ArchiveBox needs to manually configure tokens or cookies, which is troublesome.

So I thought, can there be a web archiving tool that doesn't require self-hosting, doesn't require headless browser, has no requirements for server, and can be cross-platform? Then I can access my archived pages anywhere, anytime, on any device.

Why Cloudflare

Cloudflare's Workers service is very powerful and free, with plenty of D1 databases and R2 storage buckets, which is very suitable for building this tool.

More importantly, Cloudflare's ecosystem is complete, supports one-click deployment and data migration. Cloudflare's global CDN service can also be used.

What can this tool do

[x] Folder classification
[x] Page preview image
[x] Title keyword search
[x] Showcase, share the pages you captured
[x] Mobile support
[x] Tag classification system
[x] Read mode

How it works

web-archive is composed of the following parts:

Browser extension: Save the page as a webpage snapshot and upload it to the server.
Server: Receive the snapshot and metadata uploaded by the browser extension, and store them in the database and storage bucket.
Web client: Query the snapshot and display it.

I used SingleFile's open-source code to save the page as a single html file (even including images and videos).

The server is completely based on Cloudflare's Workers service, with D1 database for storing metadata and R2 storage bucket for storing snapshots.

Although the number of interfaces is not small, I did not use ORM, actually I tried prisma and drizzle, because they caused a lot of trouble for deployment, so they were not used in the end.

The web client is built with React, Vite, TailwindCSS, and shadcn/ui, and the packaged size is astonishingly small, only 1.5MB. The packaged product will be embedded in the assets folder of the server, so it does not need to be deployed separately when deploying the server.

Limitations

I really like Cloudflare's free services, but there are some limitations.

The CPU calculation time of a single request cannot exceed 10 milliseconds, otherwise it will be forcibly terminated. (I was surprised to find that the paid account is 30 seconds 😂)
The memory usage cannot exceed 256MB, otherwise it will be forcibly terminated.

These limitations have affected the construction of the website to some extent, such as ssr or dom parsing during crawling.

However, no matter how it is said, thank you, Cloudflare!