DEV Community: Toby Chui

Zoraxy vs Nginx Proxy Manager

Toby Chui — Sat, 23 Nov 2024 11:26:01 +0000

Recently I got a lot of questions from Nginx Proxy Manager users asking how to switch over to Zoraxy. As the developer of Zoraxy, this sometime bothers me as Zoraxy, at its core, it is not powered by the same engine as Nginx Proxy Manager, but an independently developed reverse proxy core named "Dynamic Proxy" (or dpcore for short).

For those who have never heard of Zoraxy, here is a brief introduction of Zoraxy.

🌐 What is Zoraxy?

Zoraxy is a general purpose HTTP reverse proxy and forwarding tool, with tons of different network utilities and management feature that suitable for a small homelab setup.

Although it provides a lot of function, the main purpose for this project is to act as the gateway of my own homelab, which provide web services for my ArozOS instance (web desktop system that I can access all my storage nodes) and reverse proxy to my Apache web server which is hosting my blog and personal homepage. But most people find its reverse proxy tool and ACME tool helpful, and promoting it is a Nginx Proxy Manager replacement.

More details can be found on the project homepage

Reverse Proxy Server | Zoraxy

A reverse proxy server and cluster network gateway for noobs

zoraxy.aroz.org

Source code is available on my Github

tobychui / zoraxy

A general purpose HTTP reverse proxy and forwarding tool. Now written in Go!

Zoraxy

A general purpose HTTP reverse proxy and forwarding tool. Now written in Go!

Features

Simple to use interface with detail in-system instructions
Reverse Proxy (HTTP/2)
- Virtual Directory
- WebSocket Proxy (automatic, no set-up needed)
- Basic Auth
- Alias Hostnames
- Custom Headers
Redirection Rules
TLS / SSL setup and deploy
- ACME features like auto-renew to serve your sites in https
- SNI support (and SAN certs)
- DNS Challenge for Let's Encrypt and these DNS providers
Blacklist / Whitelist by country or IP address (single IP, CIDR or wildcard for beginners)
Global Area Network Controller Web UI (ZeroTier not included)
Stream Proxy (TCP & UDP)
Integrated Up-time Monitor
Web-SSH Terminal
Utilities
- CIDR IP converters
- mDNS Scanner
- Wake-On-Lan
- Debug Forward Proxy
- IP Scanner
Others
- Basic single-admin management mode
- External permission management system for easy system integration
- SMTP config for password reset

Downloads

Windows / Linux (amd64) / Linux (arm64)

For other systems or architectures…

View on GitHub

⚖️ The core difference of Zoraxy and Nginx Proxy Manager (NPM)

The core difference of Zoraxy and NPM is that NPM is based on a stable, old reverse proxy product named "Nginx". The NPM technically is just a PHP web interface that help you inject configurations into the Nginx server and do some basic management work for you like restarting it after the configuration is written. NPM has been here for long enough that basically all open source project use the Nginx proxy server behavior as the de-facto standard, even if it is sometime not following latest HTTP standard due to legacy reasons.

Meanwhile in Zoraxy, everything is done differently. The core reverse proxy feature of Zoraxy is powered by its "Dynamic Proxy", which as its name suggested, works dynamically instead of statically. Compare to Nginx, where any config change will require a service restart, Zoraxy's config can be dynamically changed in runtime with no downtime, together with some smart logic to handle domain sniffing, runtime protocol switch and behavior adjustment due to some other outdated open source project weird behavior, make it one of the best option for beginner who have no idea what they are doing and want to tinker around.

📘 Quick Comparison

Here is a quick comparison between Zoraxy and NPM

	Zoraxy	Nginx Proxy Manger
HTTP(s) Reverse Proxy	Yes	Yes
WebSocket support	Yes	Yes
Stream Proxy (TCP / UDP)	Yes	Yes
Beginner Friendly UI	Yes	Yes
ACME Support	Yes	Yes
SNI Support	Yes	No (Require manually linking certificate to host name)
Access Control	Easy (IP / GeoIP based access control rule, adjustable from web UI)	Advance (Manually input nginx settings)
Request handling speed	Slower	Faster
Restart required on config change	No	Yes
Level of Customization	Low	High
Level of Automation	High	Low
Require Docker	No	Yes
Runs on Windows	Yes (Natively)	Kinda (with some magic)
Reverse Proxy Engine	DPCore (Modify from Golang offical http proxy)	Nginx

🤔 Which one should I choose for my homelab?

This question really depends on your own knowledge in homelab networking and self-hosting.

Nginx Proxy Manager

More users: if you have any question, you can easily find someone to help you
Faster: Nginx is a stable and old reverse proxy server that have been intensively optimized for performance. With its static configuration design, all request can be handled in faster speed and allowing much better resources utilization. Suitable for use cases where you are tight on computing power or hardware resources like hosting on a single Raspberry Pi.
Higher customization: NPM gives you much better control on what to do with the routing. If you have some specific setup that require detail fine tuning, NPM (with custom configuration) will be a better option here.

Zoraxy

Newer: Zoraxy is less than 3 years old, making its UI and UX design much more modern and user friendly.
Automation: Zoraxy have tons of automation logic in place, allowing it to do runtime adjustment with minimal setup input. For example, it has automatic TLS sniffing (for automatic upstream configuration), SNI (picking the correct certificate for you base on request hostname) and automatic websocket proxy (no need to manually setup websocket proxy per hostname. Zoraxy will do an automatic protocol switch based on header sniffing & matching).
No docker: Zoraxy runs without docker, that means with a single binary file, you can easily deploy it everywhere easily without going through the docker setup. It even works on Windows!

Summary of Zoraxy vs. Nginx Proxy Manager

Zoraxy is a modern HTTP reverse proxy and forwarding tool designed for small homelab setups. Its distinguishing feature is its Dynamic Proxy Core (DPCore), enabling real-time configuration changes without restarts. It offers tools like ACME support, SNI, and access control while being highly automated and beginner-friendly. Zoraxy doesn't require Docker and runs natively on Windows, making it accessible for users with minimal technical knowledge. However, it has lower customization options and slower request handling compared to Nginx Proxy Manager (NPM).

Nginx Proxy Manager (NPM) relies on the robust, mature Nginx engine with a PHP-based UI for configuration management. Known for its speed and high resource efficiency, NPM excels in performance and fine-tuning for advanced setups. However, its static configuration requires service restarts for changes, and it lacks modern automation features.

Key Differences

Zoraxy: Dynamic, automated, Docker-free, beginner-friendly, runs on Windows.

NPM: Faster, highly customizable, optimized for resource-constrained environments, requires Docker.

Choose NPM for performance and customization, especially on limited hardware like Raspberry Pi. Opt for Zoraxy for simplicity, automation, and a modern interface. Other than that, there are also other open source reverse proxy project like Caddy, which are also worth taking a look into if you are into a more professional solution.

That is all for today, see you in the future posts :)

Writing the best custom header UX for Zoraxy

Toby Chui — Mon, 17 Jun 2024 13:33:49 +0000

Recently, I am working on improving the Zoraxy (my open source reverse proxy server written in Golang). As this is not a post about reverse proxy but UX, I am gonna sum up this project in one sentence: Zoraxy is a reverse proxy server that uses UX first design.

And one of the most challenging part I recently encounter is designing a UI for the header rewrite rule sets.

What is a header rewrite rule?

A header rewrite rule set is a few lines of header key-value that is gonna be injected into the HTTP request header when the proxy is processing the request. The rewrite can happens in two directions. One is when the request is proxying from downstream to upstream (aka from client like user's browsers to origin server like web server), another one is from upstream to downstream. Not to mention each direction there will be two operations, either you can add a new header to the request header, or delete a header from the current request.

Design for beginners vs professionals

For those who have networking background, it is a really simple thing to understand. However, Zoraxy is aiming for users who have little to no networking background, how to express the correct meaning to user gives me some great headache in designing the UI for this function.

UI for adding custom headers

Eventually, I come up with a solution. So first I created a web form that try to avoid the upstream and downstream technical terms and favor for a more easily understandable diagrams. Next, adding the next step of the operations as a new sets of buttons below the direction choosing buttons.

Then now, we have a very clear and easy to operate web form for creating and editing custom headers. Next, how can we list the current existing custom headers?

UI for Listing Current Custom Headers

To list the custom headers with direction indication, we can reuse the cognitive awareness that the user learnt while creating the custom header. So in the table of custom headers, I uses the same color and direction arrows to indicate the header add / remove directions. I also pick the same add and remove icon as the operations buttons. So now, we can easily get a sense on how and when the header is being rewritten.

And well, sometime user forgets their experience in creating custom headers. They might be setting this up last year and now revising the table and forgotten what those arrows means. So I added two lines of instruction at the bottom of the table. For the "add" and "remove" icon, I think that is common sense enough that I don't need to explain them so I didn't left a remarks for them.

Permission-Policy

Permission-Policy is a HTTP header that defines what policy / sensors that your website can access. It is a terribly long header that looks something like this



Permissions-Policy: accelerometer=*, ambient-light-sensor=(self),
 autoplay=*, battery=(), camera=*, cross-origin-isolated=(), 
display-capture=*, document-domain=(self), encrypted-media=*, 
execution-while-not-rendered=*, fullscreen=(self), geolocation=*, 
gyroscope=(), keyboard-map=(self), magnetometer=*, microphone=*,
 midi=*, navigation-override=*, payment=*, picture-in-picture=(),
 publickey-credentials-get=*, screen-wake-lock=*,
 sync-xhr=(self), usb=*, web-share=*, xr-spatial-tracking=*,
 clipboard-read=*, clipboard-write=(self), gamepad=(),
 speaker-selection=(self), conversion-measurement=*,
 focus-without-user-activation=*, hid=(), idle-detection=*,
 interest-cohort=*, serial=(), sync-script=(self),
 trust-token-redemption=*, unload=(self), window-placement=*,
 vertical-scroll=*

If I am developing for professionals, I could have just left a text-area and call it a day. However, Zoraxy is design for beginner and if I leave a text-area in place, people are gonna fill in some weird things and open bug issues claiming Zoraxy is not working and buggy. That is why I designed a web form for it again.

At the bottom of the list, there are a separately color section for experimental policies. This can help notify the user that these might not be supported by all browsers.

And this is probably the longest web form I have developed for this project. But at least this details here can make sure no one fill in invalid stuffs into the Permission-Policy header field (I hope)

Anyway, this project really challenge my UX design experience. I guess nowadays most devs will just use framework and make the UI looks good, but UI looking good is not the same as good UX. Sometime attention to details like this might helps a lot in allow user to stick with your software and willing to contribute.

Lastly, here is the git repo if you are looking for an easy to use reverse proxy server and you are interested to give it a try.

tobychui / zoraxy

A general purpose HTTP reverse proxy and forwarding tool. Now written in Go!

Zoraxy

A general purpose HTTP reverse proxy and forwarding tool. Now written in Go!

Features

Simple to use interface with detail in-system instructions
Reverse Proxy (HTTP/2)
- Virtual Directory
- WebSocket Proxy (automatic, no set-up needed)
- Basic Auth
- Alias Hostnames
- Custom Headers
Redirection Rules
TLS / SSL setup and deploy
- ACME features like auto-renew to serve your sites in https
- SNI support (and SAN certs)
- DNS Challenge for Let's Encrypt and these DNS providers
Blacklist / Whitelist by country or IP address (single IP, CIDR or wildcard for beginners)
Global Area Network Controller Web UI (ZeroTier not included)
Stream Proxy (TCP & UDP)
Integrated Up-time Monitor
Web-SSH Terminal
Utilities
- CIDR IP converters
- mDNS Scanner
- Wake-On-Lan
- Debug Forward Proxy
- IP Scanner
Others
- Basic single-admin management mode
- External permission management system for easy system integration
- SMTP config for password reset

Downloads

Windows / Linux (amd64) / Linux (arm64)

For other systems or architectures…

View on GitHub

That is all for today. Have a nice day :)

Zoraxy v3 - The brand new Reverse Proxy Server for Noobs

Toby Chui — Thu, 18 Apr 2024 04:04:00 +0000

Recently I have been working on quite some Zoraxy updates. For those who have never heard of Zoraxy, it is my open source reverse proxy server with features more than enough to run your tiny cluster in your homelab or server room.

TL;DR

If you don't care how Zoraxy works and just want a noobs friendly reverse proxy server that works on Windows and Linux, click here.

What is new?

Well, if you have used Zoraxy v2 before, every time you open the web management portable you will find yourself feeling depressed. It is because during the development of v2, I didn't have time to consider the color scheme or how to make it looks more attractive (as an open source project wise).

This is how the v3 UI looks like.

Other than UI updates, there are also tons of new feature added to optimize the proxy core function (or I would say, trying to be as similar to Nginx Proxy Manager (NPM) as possible due to user requests). Generally speaking, this is not a NPM replacement, but more like a system that lets you easier to switch between services for testing and debugging. But anyway, I don't mind people asking for new features that feels like NPM.

Default Site

In Zoraxy v3, we added support for Default Site and multiple hostnames support.
The predecessor of Zoraxy (which is called "Web Proxy", a sub-service of the ArozOS system) was designed to handle only one domain reverse proxy only, the v2 design of adding additional support for subdomain and other host name was messy. People are confusing about "Proxy", "Subdomain Proxy" and "proxy Root". That is why in the v3 design, a new interface and setup logic was introduced. With Default Site logic, now people can easily adapt Zoraxy just like Nginx Proxy Manager.

The default site provide 4 options to choose from. For starters, you might want to use the Internal Static Web Server and put an "index.html" file as your website homepage using the "Static Web Server" function which Zoraxy also provides. This is more like a traditional Apache kind of experience, where if no routing is match in the apache.conf, all the routing goes into the build in static web server and start serving files from your /var/www/html folder.

Redirect and 404 Not Found are also quite straight forward. For redirect, you can enter a target domain / ip address to redirect. This is useful when you are pointing your old (sub)domains to a new one or just straight out blocking those requests for unknown / out-dated subdomains that used to exists.

Certificate for wildcard domains / SNI

In v3, we introduced SNI in TLS/SSL certificate lookup logic. What difference of Zoraxy SNI and other implementation is that it do not require user input for "linking" a certificate with a given host name.

In v2, users need to manually set each certificates to a matching domain. For example, if you have a certificate that cover a.example.com and b.example.com, you need to manually set it the matching keyword to "example.com" in order for it to work. Now with the Zoraxy v3 automatic certificate lookup logic, you dont need to link anything. Just upload your certificate (or use the build in ACME tool to generate one from your prefered CA) and Zoraxy will resolve it for you automatically. The only catch is that it will be slightly slower for certificates containing multiple hostnames (e.g. domain.com and anotherdomain.com, might take O(n) time complexity to check all the certificates in the system), but for personal / homelab purpose it is good enough.

Access Filters

In the Zoraxy v3.0.2, a host rule independent access filter was introduced. This is yet another cool function in Zoraxy where each proxy rule have their own access filters, which the filter itself also include blacklist / whitelist of country code and/or ip address.

Compare to v2 where the global filter rule is applied, the v3 uses a per proxy host rule design, which allow each proxy hostname to have different access rule filter. I personally don't use it as I only host a single domain with my server, but seems many user request for this for more than a year now. So I added this in to make other's life easier.

For those who still love to use global access rule design, you can modify the "default" filter instead. The default filter works exactly like the previous global access filter and all proxy host by default use the "default" filter as the access control policy.

Alias

Another cool feature we added in Zoraxy v3.0.2 is the ability to create alias of a given hostname. People are complaining about cannot create alias host name on Zoraxy. So now we got support for alias host names! You can edit an existing Proxy host name and add alias for router matching.

But there is a trade-off. Using alias is a bit slower than a direct hit in host name resolving (which obviously make sense as it needs to iterate through and check if any hostname or hostname with wildcard matches the request header). So personally, as I only have one domain name, I will just add more proxy entries or redirection instead.

Redirection Regex Support

Also, due to high demands for complex rewrite rules, in Zoraxy v3.0.1 we introduced the redirection regex support. If enabled, you can use regular expression in setting up redirection URL.

It is not useful for me (as I am the kind of person think regex is like magic spells), but maybe someone will find it useful.

Forward Proxy

It is interesting to see Zoraxy can also be used as a forward proxy! Well if you don't want to get a VPN setup and only want to access a few site when you are traveling, you can use Zoraxy as a web-vpn-server kind of thing. This feature is called "Forward Proxy" and you can find Forward Proxy setting in most modern browser. Just make sure you have enabled port forward on your NAT router (and docker container) so you can connect to the forward proxy server with the port you assigned.

Wake-On-LAN

In the Zoraxy v3, the Wake-On-LAN helper is also introduced. This helps with kick starting your cluster in case there is a power loss and your UPS notifiy all servers to shut down. This function is design for people who have a setup that they uses a small form factor PC as software router which it will not be powered off during power outage (powered via UPS maybe?). This node will be used as the restarting node for kick starting the cluster when power resume.

Using this function and, if you have a few nodes with "ArozOS" installed, you can easily add all nodes into the Zoraxy WoL table and kick start them one by one remotely.

Summary

Anyway, that is all for today! Zoraxy v3 offers so many new features that now it has become one of my major services powering my distributed cluster. If you are interested to know more or get your hands dirty by digging into the source code, you can find everything you need on my Github repo and project homepage.

tobychui / zoraxy

A general purpose HTTP reverse proxy and forwarding tool. Now written in Go!

Zoraxy

A general purpose HTTP reverse proxy and forwarding tool. Now written in Go!

Features

Simple to use interface with detail in-system instructions
Reverse Proxy (HTTP/2)
- Virtual Directory
- WebSocket Proxy (automatic, no set-up needed)
- Basic Auth
- Alias Hostnames
- Custom Headers
Redirection Rules
TLS / SSL setup and deploy
- ACME features like auto-renew to serve your sites in https
- SNI support (and SAN certs)
- DNS Challenge for Let's Encrypt and these DNS providers
Blacklist / Whitelist by country or IP address (single IP, CIDR or wildcard for beginners)
Global Area Network Controller Web UI (ZeroTier not included)
Stream Proxy (TCP & UDP)
Integrated Up-time Monitor
Web-SSH Terminal
Utilities
- CIDR IP converters
- mDNS Scanner
- Wake-On-Lan
- Debug Forward Proxy
- IP Scanner
Others
- Basic single-admin management mode
- External permission management system for easy system integration
- SMTP config for password reset

Downloads

Windows / Linux (amd64) / Linux (arm64)

For other systems or architectures…

View on GitHub

Reverse Proxy Server | Zoraxy

A reverse proxy server and cluster network gateway for noobs

zoraxy.arozos.com

See you in the next dev post!

How I grow my projects based on my OWN needs

Toby Chui — Sun, 24 Sep 2023 11:26:39 +0000

As a master student working in a lab that research networking technology, I embarked on a journey that combined my passion for coding and my desire for self-convenience. What began as a personal project for fun soon evolved into something much more significant—a versatile, low-power, and highly customizable reverse proxy server that I named Zoraxy.

To be honest, I have no idea what I am writing (or trying to create), but it seems it is a fun thing to write my journey in a post that not sure how many people will read, just to let you know it is ok to create something that might be useless other than yourself.

tobychui / zoraxy

General purpose (reverse) proxy and forwarding tool for networking noobs. Now written in Go!

Zoraxy

General purpose request (reverse) proxy and forwarding tool for low power devices. Now written in Go!

Features

Simple to use interface with detail in-system instructions
Reverse Proxy
- Subdomain Reverse Proxy
- Virtual Directory Reverse Proxy
Redirection Rules
TLS / SSL setup and deploy
Blacklist by country or IP address (single IP, CIDR or wildcard for beginners)
Global Area Network Controller Web UI (ZeroTier not included)
Integrated Up-time Monitor
Web-SSH Terminal
Utilities
- CIDR IP converters
- mDNS Scanner
- IP Scanner
Others
- Basic single-admin management mode
- External permission management system for easy system integration
- SMTP config for password reset

Build from Source

Require Go 1.20 or above

git clone https://github.com/tobychui/zoraxy
cd ./zoraxy/src/
go mod tidy
go build
sudo ./zoraxy -port=:8000

Usage

Zoraxy provide basic authentication system for standalone mode. To use it in standalone mode, follow the instruction below for your desired deployment platform.

Standalone Mode

Standalone mode is the default mode for…

View on GitHub

Why it is named Zoraxy?

5 years ago I have another projected named ArozOS and I in need for a solution to set it up with my already existing infrastructure (Apache + wordpress). My first thought is just use Apache as a reverse proxy server, but it turns out to be a bad idea as every time I want to change something or test a new local site, I need to dig deep into the apache config and change a text file that might crash my own system during service restart.

Nginx proxy manager is something I would want to try, but it require dockers (and probably used a lot of glue code within it), I kind of drop that options when I trying to understand their project.

That is why I decided to write my own, modified from a really old project of mine named "Reverse Proxy", which is an ArozOS subservice design to, as the name suggest, reverse proxy another applications with web interface but not supporting iframe embedding in ArozOS web desktop interface.

So later on, as I am adding more and more features to it, I decided to separate it into another project that makes maintaining it more easy. That is why I came up with the name Zoraxy - reverse proXY for ArOZ (reversed, so "Zora")

What Make Zoraxy Special?

One of the standout features of Zoraxy is its user-friendly interface, complete with detailed in-system instructions. It's designed with both beginners and experienced users in mind, ensuring that anyone, regardless of their level of expertise, can harness its power. But what truly sets Zoraxy apart are the plethora of functionalities it offers. If you have a features that you don't know how to use, you can always find a small text under the button, a side bar with instruction or a collapsible section that explain how things work. That means even without a proper documentation, user still know how to operate them with ease.

At its core, Zoraxy is a robust reverse proxy server (which I copied some of the http transport settings from Caddy). It excels in handling subdomain and virtual directory reverse proxy configurations, making it easy to route incoming requests to the appropriate destinations.

It also got tons of other features that I am just too lazy to type them out here. If you are interested on the functionality perspective of this project, go take a look at the official website over here.

How a Reverse Proxy server become a Swiss knife?

To be honest I have no idea. I just add in things that I need (and people needs) into the project and it started to become larger and larger.

At first I was like "Maybe I can add a new feature that shows the up time of each of my subdomain target", and then when there are a lot of people visiting my site, I was wondering "Where do all these people came from?" From that, I added in a statistic collector and find a lot of them actually bots and crawler scripts. Then I added in access control system that blocks some of the bots and crawler from wasting my bandwidth.

Another issue is that I have a few nodes placed in the university where there is way to reach them via public internet. That is why I added in the GAN support (zerotier based) and a lot of other tools to help with the routing and networking of such complex environment.

After a year of developments and tons of feature added, a few collaborator who are interested in this project hopped in and help with the ACME / TLS auto renew modules, which gives it a much higher popularity among the homelab community.

What I learn from creating yet another open source project

In the realm of software development, we often find ourselves in situations where existing tools and software solutions don't quite fit our needs. It's a common predicament faced by both beginners and seasoned professionals alike. But what if I told you that this situation could be an incredible opportunity for personal and professional growth? The answer lies in a simple yet profound piece of advice: if you can't find the right software, consider building it yourself.

Because of the Zoraxy project, I got the chances to share my project on different open source conference, events and a new chat topic with my friends in CS. Now, I understand that the prospect of developing your own software can be daunting, especially if you're new to programming. However, it's precisely this challenge that makes it an invaluable learning experience. Writing your own software forces you to dive deep into the world of code, algorithms, and problem-solving. It's a crash course in programming that no tutorial or online course can replicate.

By tackling a software project from scratch, you open doors to networking and mentorship opportunities you might not have encountered otherwise. You'll meet individuals who are equally passionate about technology, innovation, and problem-solving. These connections can lead to collaborations, job offers, or simply a broader network of peers who can provide insights and support throughout your career.

So, the next time you find yourself frustrated by the limitations of existing software, consider building it yourself :)

Reverse Proxy Server for noobs (not Nginx / Apache!)

Toby Chui — Sun, 04 Jun 2023 03:41:43 +0000

I guess most of the web devs out there know what is a reverse proxy server. It is especially useful when you have multiple services deployed on a single homelab rack or a cluster of servers and want to expose them to the internet using a single gateway server with different subdomains. In this blog post, I will simply share my experience on why I wrote my own reverse proxy server in Go to replace apache / nginx in my distributed homelab setup.

What is Reverse Proxy Server?

Imagine you're at a music festival, grooving to your favorite tunes, and suddenly you need a refreshing drink. You don't want to miss a beat, so you send your friend, the reverse proxy, to fetch it for you. Your friend navigates through the crowd, stands in line, and brings back that ice-cold beverage without interrupting your dance moves. In this scenario, the reverse proxy server acts as your go-between, shielding you (the user) from the chaos of the festival (the backend servers) and delivering the content you desire.

Modern reverse proxy do more things just to relay and fetch webpage for you. For example, load balancing, failover, some managers like Nginx Proxy Manager (NPM, not the blackhole npm) even have build in automatic SSL certificates renew and so on. That is why reverse proxy is usually considered the "heart" of a homelab system (or any kind of web systems)

Why I am writing my own Reverse Proxy Server?

If you have never experience with the pain of writing an apache or nginx config as a senior dev-ops, it is a really memorable experience. You try copy things from the internet, modify them according to your knowledge and figure out its routing is unpredictable. Sometime the routing only works in testing environment but not the productions, and many web services that depends on some special settings in rewrite rules and headers. It is just pain in the *** to make all the service running.

Besides, it is just inconvenience to ssh into your gateway server and modify the config when you just want to add a temporary rules for proxying a testing environment or systems.

There are alternatives or solutions to these issue in the open source world. Like NPM is the one that most recommended by Redditors in r/selfhost. However, it have been outdated and it is known to break a lot of stuffs with the latest updates. That is why I decided to write my own to solve my issue in the foreseeable future.

That is why Zoraxy was born

tobychui / zoraxy

A general purpose HTTP reverse proxy and forwarding tool. Now written in Go!

Zoraxy

A general purpose HTTP reverse proxy and forwarding tool. Now written in Go!

Features

Simple to use interface with detail in-system instructions
Reverse Proxy (HTTP/2)
- Virtual Directory
- WebSocket Proxy (automatic, no set-up needed)
- Basic Auth
- Alias Hostnames
- Custom Headers
Redirection Rules
TLS / SSL setup and deploy
- ACME features like auto-renew to serve your sites in https
- SNI support (and SAN certs)
- DNS Challenge for Let's Encrypt and these DNS providers
Blacklist / Whitelist by country or IP address (single IP, CIDR or wildcard for beginners)
Global Area Network Controller Web UI (ZeroTier not included)
Stream Proxy (TCP & UDP)
Integrated Up-time Monitor
Web-SSH Terminal
Utilities
- CIDR IP converters
- mDNS Scanner
- Wake-On-Lan
- Debug Forward Proxy
- IP Scanner
Others
- Basic single-admin management mode
- External permission management system for easy system integration
- SMTP config for password reset

Downloads

Windows / Linux (amd64) / Linux (arm64)

For other systems or architectures…

View on GitHub

Zoraxy is a Go-Powered Reverse Proxy and Forwarding Tool for Web Developers

Looking for a versatile and user-friendly reverse proxy and forwarding tool? Look no further than Zoraxy! Built from the ground up using the power of Go, Zoraxy is a go-to solution for low-power devices and beyond. With its comprehensive set of features and straightforward interface, Zoraxy empowers both senior and casual web developers to effortlessly manage their web services and enhance their online presence.

Let's delve into the exciting world of Zoraxy and explore its impressive feature set. First and foremost, Zoraxy shines as a reverse proxy, allowing you to seamlessly route incoming requests to the appropriate backend servers. Whether you need subdomain reverse proxy capabilities or virtual directory reverse proxy functionality, Zoraxy has got you covered.

But that's not all! Zoraxy goes above and beyond with its extensive array of features. Need to set up redirection rules? Zoraxy has you covered.

Looking to implement TLS/SSL for enhanced security? Zoraxy has your back there too.

Worried about unwanted traffic? Zoraxy offers the ability to blacklist specific countries or IP addresses, providing you with granular control over your network.

Zoraxy doesn't stop at just being a reverse proxy either. It comes packed with handy utilities like CIDR IP converters, mDNS Scanner, and IP Scanner, making your web development tasks a breeze.

And for those times when you need remote access, Zoraxy even provides a convenient Web-SSH Terminal.

Want to know who are visiting your site? Zoraxy also got a build in statistic viewer where you can get an overview on the traffics on your site without the need of cookies (non-personalized tracking)

Managing Zoraxy is a breeze thanks to its simple-to-use interface and in-depth instructions built right into the system. Whether you're a seasoned developer or just starting out, Zoraxy offers a basic single-admin management mode (like a home router) that ensures a hassle-free experience. If you already got an upstream reverse proxy server with permission management set-up, you can even run Zoraxy in no-auth mode, making it integrate with your current infrastructure with ease.

Work In Progress

As this was just open source a few weeks ago, I still got a few more functions that I wanted to add and it is working in progress. One of the most requested features are the ACME automatic certificate update. However, I am still working on the details on how ACME works.

Meanwhile, there are a few experimental functions that is nearly completed. Like the TCP Proxy and ZeroTier controller integrations.

Know More

If you are interested to learn more about this open source project, feel free to take a look at our website at

Reverse Proxy Server | Zoraxy

A reverse proxy server and cluster network gateway for noobs

zoraxy.arozos.com

or our github repo over at

tobychui / zoraxy

A general purpose HTTP reverse proxy and forwarding tool. Now written in Go!

Zoraxy

A general purpose HTTP reverse proxy and forwarding tool. Now written in Go!

Features

Simple to use interface with detail in-system instructions
Reverse Proxy (HTTP/2)
- Virtual Directory
- WebSocket Proxy (automatic, no set-up needed)
- Basic Auth
- Alias Hostnames
- Custom Headers
Redirection Rules
TLS / SSL setup and deploy
- ACME features like auto-renew to serve your sites in https
- SNI support (and SAN certs)
- DNS Challenge for Let's Encrypt and these DNS providers
Blacklist / Whitelist by country or IP address (single IP, CIDR or wildcard for beginners)
Global Area Network Controller Web UI (ZeroTier not included)
Stream Proxy (TCP & UDP)
Integrated Up-time Monitor
Web-SSH Terminal
Utilities
- CIDR IP converters
- mDNS Scanner
- Wake-On-Lan
- Debug Forward Proxy
- IP Scanner
Others
- Basic single-admin management mode
- External permission management system for easy system integration
- SMTP config for password reset

Downloads

Windows / Linux (amd64) / Linux (arm64)

For other systems or architectures…

View on GitHub

If you like this project, please give us a star so we know there are users waiting and looking forward to this project to enter Beta phrase. Thanks for the support :)

Hack the Golang httputil Reverse Proxy settings to handle more requests

Toby Chui — Sat, 03 Jun 2023 12:36:24 +0000

Recently I run into a problem where the Golang offical httputil reverse proxy get stuck when testing my open source reverse proxy server Zoraxy.

tobychui / zoraxy

A general purpose HTTP reverse proxy and forwarding tool. Now written in Go!

Zoraxy

A general purpose HTTP reverse proxy and forwarding tool. Now written in Go!

Features

Simple to use interface with detail in-system instructions
Reverse Proxy (HTTP/2)
- Virtual Directory
- WebSocket Proxy (automatic, no set-up needed)
- Basic Auth
- Alias Hostnames
- Custom Headers
Redirection Rules
TLS / SSL setup and deploy
- ACME features like auto-renew to serve your sites in https
- SNI support (and SAN certs)
- DNS Challenge for Let's Encrypt and these DNS providers
Blacklist / Whitelist by country or IP address (single IP, CIDR or wildcard for beginners)
Global Area Network Controller Web UI (ZeroTier not included)
Stream Proxy (TCP & UDP)
Integrated Up-time Monitor
Web-SSH Terminal
Utilities
- CIDR IP converters
- mDNS Scanner
- Wake-On-Lan
- Debug Forward Proxy
- IP Scanner
Others
- Basic single-admin management mode
- External permission management system for easy system integration
- SMTP config for password reset

Downloads

Windows / Linux (amd64) / Linux (arm64)

For other systems or architectures…

View on GitHub

Have you ever try to open a website that is under heavy load? The connection just freeze there and do nothing until other objects in your site is loaded and free up some connections, then your content continue to load. This should not be happening as Go is known to be able to handle many requests in the same time using its magical go-routine. That is why I started to question about the implementation of the httputil.reverseproxy library.

If you don't know there is an official http reverse proxy in the Golang source code, this is the one I am talking about.

https://go.dev/src/net/http/httputil/reverseproxy.go

And within the source code, you will notice there is a line in the reverse proxy struct object that make use of something called the http.Transport



// The transport used to perform proxy requests.

// If nil, http.DefaultTransport is used.

Transport http.RoundTripper

After debugging a few days and tracing the Go source, I notice the issue was that the http.DefaultTransport pre-set a relatively low limits on the maximum connections counts. This setting is reasonable as most of them time this reverse proxy is used to serve single user instead of using it as a replacement of nginx or apache reverse proxy.

That is why in order to fix it, I included a modified version of the reverse proxy into Zoraxy project with the following modifications:



//Hack the default transporter to handle more connections
thisTransporter := http.DefaultTransport
thisTransporter.(*http.Transport).MaxIdleConns = 3000
thisTransporter.(*http.Transport).MaxIdleConnsPerHost = 3000
thisTransporter.(*http.Transport).IdleConnTimeout = 10 * time.Second
thisTransporter.(*http.Transport).MaxConnsPerHost = 0

Now, my reverse proxy can serve much more request in a given time compare to the http.DefaultTransport settings used in the original reverse proxy utilities!

---------- Updates -------------
After a few months of further investigation and testing, I found that it would be better for each of the reverse proxy object (aka it will only have 1 host handled by each Transport object) to have the following configurations



//Hack the default transporter to handle more connections
thisTransporter := http.DefaultTransport
optimalConcurrentConnection := 32
thisTransporter.(*http.Transport).MaxIdleConns = optimalConcurrentConnection * 2
thisTransporter.(*http.Transport).MaxIdleConnsPerHost = optimalConcurrentConnection
thisTransporter.(*http.Transport).IdleConnTimeout = 30 * time.Second
thisTransporter.(*http.Transport).MaxConnsPerHost = optimalConcurrentConnection * 2
thisTransporter.(*http.Transport).DisableCompression = true

Compare to the previous settings which uses around 85% of my bandwidth, this settings can utilize up to around 90 - 92% of my outbound bandwidth. I have no idea where these numbers come from, but it works great.

Announcing ArozOS 2.0 - 5 years journey into my own Web Desktop OS

Toby Chui — Fri, 12 May 2023 03:58:45 +0000

This is a story about a poor student who wants to build his own storage solution with cheap computers. It all starts with just a few php scripts serving my mp3 files from my server, and now it become one of the most complex and (maybe the) best designed web desktop system out there in the open source world.

2 years ago, I have written another post sharing the v1.0 release and the back story of the system. If you are interested in the evolution of this system starting from 2018, you can check out the following post.

I write my own web desktop OS for 3 years and this is what it looks like now

Toby Chui ・ Apr 4 '21

#webdev #go #linux #ux

And this is how it looks like now.

tobychui / arozos

Web Desktop Operating System for low power platforms, Now written in Go!

Features

User Interface

Web Desktop Interface
Ubuntu remix Windows style startup menu and task bars
Clean and easy to use File Manager (Support drag drop, upload etc)
Simplistic System Setting Menu
No-bull-shit module naming scheme

Networking

Basic Realtime Network Statistic
Static Web Server (with build in Web Editor!)
mDNS discovery + SSDP broadcast
UPnP Port Forwarding
WiFi Management (Support wpa_supplicant for Rpi or nmcli for Armbian)

File / Disk Management

Mount Disk Utilities
- Local File Systems (ext4, NTFS, FAT etc)
- Remote File Systems (WebDAV, SMB, SFTP etc)
Build in Network File Sharing Servers
- FTP, WebDAV, SFTP
- Basic Auth based simple HTTP interface for legacy devices with outdated browser
Virtual File System + Sandbox Architecture
File Sharing (Similar to Google Drive)
Basic File Operations with Real-time Progress (Copy / Cut / Paste / New File or Folder etc)

Security

oAuth
LDAP
IP White / Blacklist
Exponential login timeout

Extensibility

ECMA5 (JavaScript…

View on GitHub

Why v2.0?

The ArozOS v1 is a nice, stable system that enable me to access my files anywhere with internet and a browser from any kind of devices. It perfectly fits my needs when I am a university student studying in a local university. However, as I have started my Master research in another country, now I need to worry about redundancy and remote management issues.

That is why I have rewritten ArozOS to support mounting and sandbox network file shares into virtual drives. As this is not a technical post, I will simply summarize the update as: I rewrote the base of the whole system so that I can have multiple servers around the world and access them through one single gateway nodes. Just like your PC with mounted network drives!

How ArozOS looks like after 5 years

After 5 years of development, it finally matches the definition of modern OS design in all sense.

We rewritten the UI for the File Manager, which I will describe in details in later sections.

As usually, the system comes with build in players for all kind of media files. From music, video and photo to professional formats like psd and raw (support via 3rd party plugins)

We continue to work on a simple web development tools that allows users with little to no web development experience to create and host their website on ArozOS with a few clicks.

For professional use cases, we also provided a new tools that provide Serverless API script powered by ECMA5 (JavaScript like stuffs). So you can host some basic API services with your ArozOS running on your home network. More details below.

File Manager

The largest update of ArozOS v2 is the File Manager. The File Manage now contain much better RWD interface for mobile and desktop devices with different screen sizes.

As requested by a lot of users, the File Manager have a new "Detail" viewing mode. This mode allow a simple overview of all file properties within the current folder.

A properties sidebar is added to provide detail views over the selected item. This is very helpful when you do not want the "Details" / "Grid" view in a folder containing tons of files while wanting to have a quick way to check the properties of the selected file.

I also toke a lot of time optimizing the minor places to make user experience much better than the old versions. Including a dynamic adjusted folder path viewer (instead of the old version where it is hard coded to show only 3 folder name at a time)

And it even support inline editing function!

We optimized the rename workflow for desktop users. Instead of a popup window asking for the new file name, now we got inline filename editing as well.

Operation popups are moved to the side. So as to optimize for mobile interfaces as well as making the css less painful to maintain due to the old vertical align center design.

Storage Pools

In ArozOS v2, we get a brand new design in the Storage Pool setting interface as well as the backend architecture. In v1.0, you can only mount local disk into ArozOS as "virtual drive", which is basically a mount point on ArozOS, to v2.0's file system abstraction driver design, where you can mount anything that resemble a file system. From local disk, ram disk, network file sharing protocols like SMB, WebDAV, SFTP and FTP, all of them can be mounted as a "disk" in the new ArozOS system.

You can easily add new disk into ArozOS via the (+) button. By mounting another ArozOS server into your gateway server, you can access all ArozOS storage pools within a single server, making it much easier to manage.

We also redesigned the File System Handler properties editor to make it more simplistic and automated. Now many settings are default hidden and only show if you pick something related to it. Making it much more user friendly.

File Servers

What happens if you want to share files from ArozOS to your devices? In v2.0, you can do it in a lot of ways. For starters, you can use the Share API to share a file from File Manager.

Share API

In the new Share API, we added new Share modes so you can share to a particular user group or people you want.

The share interface got a bit update as well.

Under each of the preview interface, if it is a web compatible file, you will see an automatically generated embed link, where you can use it in your website like wordpress.

In order to make the share looks pretty on social network platforms, I added open graph support to the share API, so when you share the link to others, like discord for example, the preview will show up as a background rendered summary image file.

Network File Servers

If you want to access the file on other computers without browser, you can use the build in file server functions.

The build in file server support WebDAV, SFTP and FTP, where if you have a corresponding client like WinSCP or MacOS build in "Connect to server" function, you can access your file remotely from any devices.

Legacy Browsers Support

Legacy support is always important for ArozOS because even it is 2023, there are still people using Windows 7 or even Windows XP. That is why having backward compatibility is important. In v2.0, I have added a "Directory Server" function in which you can use basic auth to login to your ArozOS and download files from them.

This interface provide the most basic css and html where it takes little to no resources to load. I guess it might even works on NDS's browser. Note that basic auth is generally considered not safe. After you are done with it, you should turn it off.

Web Development and Serverless

The last update is about web development and serverless. Many of the ArozOS client are design professionals who seeks for a webpage for showcasing their work. That is why in ArozOS, we included some basic web development tools to get people started with their self hosted webpage.
Personal webpage can be enabled in System Settings

A few of my friends who are using ArozOS have a bit of programming skills. With request from them and my interest to try out Serverless, we added the Serverless app, where you can insert a ECMA5 code and let it run when a GET or POST request is received.

He also insert another script into scheduler so the currency exchange rage will be updated every night just to make his Telegram currency conversion bot works. In my opinion this is just another cool hacky way to use ArozOS to make things works!

More?

In this update, we added a lot of minor changes to make the UX better from the v1 releases. From Network Statistic Graphs

to background tasks viewers

and brand new document viewer

If you want to experience this wonderful system yourself, the system can be easily set up using a Raspberry Pi, old PC or laptop with a few line of bash command.

Hope you like this project! We are continuing updating the modules of this system to better fit our use cases. If you are interested to try it out or even contribute to this project, feel frees to find the source code and give us a star 🌟 in the attached Github link below.

tobychui / arozos

Web Desktop Operating System for low power platforms, Now written in Go!

Features

User Interface

Web Desktop Interface
Ubuntu remix Windows style startup menu and task bars
Clean and easy to use File Manager (Support drag drop, upload etc)
Simplistic System Setting Menu
No-bull-shit module naming scheme

Networking

Basic Realtime Network Statistic
Static Web Server (with build in Web Editor!)
mDNS discovery + SSDP broadcast
UPnP Port Forwarding
WiFi Management (Support wpa_supplicant for Rpi or nmcli for Armbian)

File / Disk Management

Mount Disk Utilities
- Local File Systems (ext4, NTFS, FAT etc)
- Remote File Systems (WebDAV, SMB, SFTP etc)
Build in Network File Sharing Servers
- FTP, WebDAV, SFTP
- Basic Auth based simple HTTP interface for legacy devices with outdated browser
Virtual File System + Sandbox Architecture
File Sharing (Similar to Google Drive)
Basic File Operations with Real-time Progress (Copy / Cut / Paste / New File or Folder etc)

Security

oAuth
LDAP
IP White / Blacklist
Exponential login timeout

Extensibility

ECMA5 (JavaScript…

View on GitHub

Vanilla JS for selecting a local text file and reading its content

Toby Chui — Sun, 05 Mar 2023 09:28:20 +0000

Recently I am working a small side project that aims to build a mechanical 4-keys macropad with the lowest cost possible. To make it more user friendly, I need to develop a website that let user load their previous config file (JSON) and allow them to continue their "graphical programming". I search online for a Javascript only solution that do the trick but it took me some time to reach what I need.

To save other's time (and make sure myself won't forget this in the future), here is how you allow user to select a text file, upload it using a File Selection Dialog and read its contents in Vanilla JS



let input = document.createElement('input');
input.type = 'file';
input.multiple = true;
input.onchange = e => { 
    let files = e.target.files; 
    for (var i = 0; i < files.length; i++){
        let read = new FileReader();
        read.readAsBinaryString(files[i]);
        read.onloadend = function(){
            //Content of the file selected
            console.log(read.result);
        }
    }
}
input.click();

Cheers!

I Wrote a simple Up-time Monitor in Golang for My PHP Server

Toby Chui — Fri, 30 Dec 2022 13:57:35 +0000

Up-time monitors are crucial because they help you keep track of your servers' availability and can help you detect outages quickly. This can help prevent prolonged downtime and ensure that your users are able to access your services as much as possible.

If you want to get started with an up-time monitor, there are plenty of options available like Pingdom, Datadog, and NewRelic. You can also choose to write one yourself if you know how to code.

As I only need a really simple and effective way to oversee my servers, and I already got a PHP server running for my blog, I decided to make one with PHP.

One problem: How to make PHP ping a website every x minutes?

As my server was built from an old Windows 7 PC, I decided to write a small Go program to make the website status check and store the data in RAM. The data was never written to disk unless it is required because writing data constantly to an SSD will quickly use up all its R/W count.

Then you might be wondering, how could I pass information to PHP for displaying then? To access the results by the PHP script, I setup a simple http listener on the Go program and allow php to request the buffered online status from the go program using HTTP request.

This up-time monitor is simple to use and easy to integrate to your existing LAMP infrastructure. Simply download the binary and the php script, move them to a suitable location and start the binary. Your php will then show the uptime information of your servers every time you request that script.

I am only using this for my web servers. However, it is designed to be extendable in which different protocols can also be implemented without the need to change anything on the front-end side. If you want to try it out, here is the Github link for the project. Feel free to PR this project if you want more protocols to be supported!

tobychui / imusutm

Basic service up-time monitor written in Golang

imusutm

Baisc service up-time monitor and TOTP generator written in Golang

Usage

Double click the binary executable to run the up-time monitor.

Build from Source

git clone https://github.com/tobychui/imusutm
cd imusutm
go mod tidy
go build
./utm
(or .\utm.exe)

Setup Monitoring Server

Create a file named "config.json" and place it in the same folder to the binary executable.

{
 "Targets": [
    {
        "ID": "imus_homepage", //ID of the target
        "Name": "imuslab Homepage", //Name to show on the UI
        "URL": "https://imuslab.com", //URL to request
        "Protocol": "https" //Protocol to check online
    },
    //More endpoint here
 ],
 "Interval": 300, //Update interval in seconds
 "LogToFile": false, //Log results to file
 "RecordsInJson": 288 //How many records to keep, in this example, 300s (5 min) x 288 records per target = 1 day
}

To start the…

View on GitHub

Universal File Access Gateway - How I turn my own Web Desktop OS into a gateway for all my file servers

Toby Chui — Mon, 21 Nov 2022 13:42:03 +0000

I have been working on my own Web Desktop OS called "ArozOS" for nearly 4+ years now. I have been setting up servers to store my data and provide different kind of services, all using my trusty Web Desktop OS for simple server management. However, separating the resources into so many servers are sometime a bit difficult to manage.

In this post, I will show you how I design a new file system for ArozOS and make it link up to all my servers. Creating a truly private cloud experience that pretty much make my life much easier in finding files, backup or even music collections in my sea of hard disks.

If this is your first time reading my post, I would recommend you to take a look at my old post over here so you can have a glance on how my web desktop OS looks like in general.

I write my own web desktop OS for 3 years and this is what it looks like now

Toby Chui ・ Apr 4 '21 ・ 9 min read

#webdev #go #linux #ux

Writing Local Program is Simple

Writing local programs that use local disk as storage is pretty straight forward. You can use io/ioutil or the os package to access anything you need locally. This is how the original ArozOS File System was designed. For every file you saw on the web desktop interface, it correspond to a local file in your server's hard disk.

However, I always want to bridge the file systems between servers together using some kind of protocol. Some of the ArozOS users already bridging their own servers using SMB / Samba, WebDAV (with davfs2 on Linux) or SFTP mounting. These solution are quick and dirty way to get things done, however this causes some issues as this created platform dependencies on the arozos system.

I didn't plan to solve this (As 1st rule of programming: If it works, don't touch it) until a feature request pop up on our Github repo asking for a new function to be implemented: Allows users to mount a webdav folder into arozos as vroot.

And of course we can use rsync or davfs2 to get the job done. However, as I am kind of a person who don't like hacky stuffs run on a 24/7 server, I started to think of a proper solution to this issue.

Virtual File System Layer

During the old beta development of the web desktop system, we did encounter a lot of security concerning path escape issues. That is why we developed the Virtual File System Layer on the ArozOS 1.0. This VFS layer provide an virtualized path for each file access and make the underlying infrastructure modification much easier.

Thus, in this update, I designed something that slip between the old VFS layer and Physical File System layer provided by the OS and call it the "File System Abstraction" layer.

File System Abstraction Driver

To allow ArozOS to support different kind of file system clients, a driver needed to be developed for each of the file system types. For the most basic one, it is the local disk driver. It basically re-map the function from os and ioutil into a special interface structure (which I will show you later) and allow the interface to be used to access local file system.
Other network file system or protocol can also be used this way. By developing a driver and adapt to the arozos file system interface, now any network file system protocol can be used as a local virtual drive in your ArozOS system, making them transparent for the WebApp that lies on top of the Virtual File System layer.

Replacing os, ioutil and more

As local file system is no longer the only file system type that can be used in ArozOS, we need to develop a special interface that allows file operations transparency between the Virtual File System and the target file system. Hence, we will need to define our own file system access interface for all the legacy virtual file system functions. This is where we referenced Afero, a VFS implementation that use for unit testing.

Afero provide a really similar file system methods to the os package but lacking a lot of features regarding file operation itself.

That is why we decided to combine the os.File functions with the afero file system functions. And here are what we came up with.

File Abstraction Interface

The file abstraction interface is pretty much the same as os.File Method but in interface{}. This allow any struct that provide the same functions be used as File in the arozos system.

type File interface {
    Chdir() error
    Chmod(mode fs.FileMode) error
    Chown(uid, gid int) error
    Close() error
    Name() string
    Read(b []byte) (n int, err error)
    ReadAt(b []byte, off int64) (n int, err error)
    Readdirnames(n int) (names []string, err error)
    ReadFrom(r io.Reader) (n int64, err error)
    Readdir(n int) ([]fs.FileInfo, error)
    Seek(offset int64, whence int) (ret int64, err error)
    Stat() (fs.FileInfo, error)
    Sync() error
    Truncate(size int64) error
    Write(b []byte) (n int, err error)
    WriteAt(b []byte, off int64) (n int, err error)
    WriteString(s string) (n int, err error)
}

File System Abstraction Interface

The File System Abstraction Interface is the most important part of the design. This allow us to perform file operation with ease between two file system types. For example, with these functions, we can do direct data streaming between SMB server and WebDAV server using one of the ArozOS server as middle-man. Or we can easily convert a file that only serve via SMB into a HTTP Stream.

type FileSystemAbstraction interface {
    //Fundamental Functions
    Chmod(string, os.FileMode) error
    Chown(string, int, int) error
    Chtimes(string, time.Time, time.Time) error
    Create(string) (arozfs.File, error)
    Mkdir(string, os.FileMode) error
    MkdirAll(string, os.FileMode) error
    Name() string
    Open(string) (arozfs.File, error)
    OpenFile(string, int, os.FileMode) (arozfs.File, error)
    Remove(string) error
    RemoveAll(string) error
    Rename(string, string) error
    Stat(string) (os.FileInfo, error)
    Close() error

    VirtualPathToRealPath(string, string) (string, error)
    RealPathToVirtualPath(string, string) (string, error)
    FileExists(string) bool
    IsDir(string) bool
    Glob(string) ([]string, error)
    GetFileSize(string) int64
    GetModTime(string) (int64, error)
    WriteFile(string, []byte, os.FileMode) error
    ReadFile(string) ([]byte, error)
    ReadDir(string) ([]fs.DirEntry, error)
    WriteStream(string, io.Reader, os.FileMode) error
    ReadStream(string) (io.ReadCloser, error)
    Walk(string, filepath.WalkFunc) error
}

You might notice the first section of the interface is pretty close to what Afero offers. For the second half, it is a mix of different functions that commonly available in utilities packages. For example, you will see Glob and Walk from filepath, or ReadDir from ioutil.

Streaming between File System Abstractions

The most important function that make the abstraction interface flexible is the WriteStream and ReadStream function.

The WriteStream and ReadStream function allow us to provide and write a byte stream into the target virtual driver, no matter the target virtual drive is a local or remote FS.

With these two functions, we can do something like doing real-time streaming between two remote FS using different protocol

or converting a remote FS protocol that is not supported by your device into something that support by basically every devices

I got myself a universal gateway to all my files!

Currently I am still rewriting majority of my system to support this kind of virtual file system architecture. I guess it might take another 4 - 6 months to complete. But after the hard works, I think I will be getting one of my best written piece of software - A universal gateway to all my files from all my servers, accessible with just a single browser tab! 😀

Upload a huge file with little RAM & space in Go

Toby Chui — Wed, 22 Jun 2022 11:17:40 +0000

This is an updated version of the previous implementation regarding uploading a file larger than RAM size in Go. If you didn't read it before, you can check it out in the following link.

Upload a file larger than RAM size in Go

Toby Chui ・ Jan 15 '21

#go #raspberrypi #upload #filesystem

In last blog post, I tackled the the Go upload method using websocket file chunking implementation to handle cases where the uploaded file is much larger than available RAM on the device. This implementation is really helpful when you are developing applications for some cheap SBCs with as little RAM as 512MB.

Recently, I encountered another issue when I am trying to migrate my whole Google Drive to my own ARM powered DIY NAS. The issue was that my NAS only have 512MB + 32GB (microSD card) as OS drive, while I have 2 x 512GB HDD attached to the SBC to store files. Uploading a file with size >32GB will causing the system to run out of space and crashing my ArozOS NAS OS .

In the previous implementation, in order to upload a 1GB file, you will need 1GB space in your tmp folder (i.e. the SD card) in order for it to buffer the file chunks received via websocket. In the latest implementation, a new "huge file mode" was added to handle cases where uploading file > tmp folder space by directly writing the upload file chunks to target disk while minimizing the maximum required space on all the system disk. Before I show you the code on how it works, this is the logic for me to decide when to enter "huge file mode"

Logic for Optimizing Both Upload Space & Time Occupancy

If a file is smaller than 4MB, upload with FORM POST (to reduce overhead, fastest)
Else if the file is smaller than "The remaining space on tmp" / 16 - 1KB, the file is buffered into the tmp folder (tmp folder should be in fast medium like NVME SSDs or RAM Disk, slower than FORM POST but still fast)
Otherwise, the file chunks are directly buffered to disk (slowest, but provide us the most space to work with)

File Merging Procedures

In the previous implementation, the file merging procedures happens like this

Create the destination file and open it
Iterate through each chunks, append it to the opened destination file
Delete all the chunk files

*However, this would take 2x the space of the file being upload. * It works fine for medium sized files, but not good for huge files. To solve this, I have changed the implementation to the followings.

Create the definition file and open it
Iterate through each chunks, append each chunks to the opened destination file, confirm the copy is success and remove the source chunks

In simple words, by deleting file on the fly, the new upload logic only takes up (x + c) bytes size, where x is the file size and c is the chunk size. In my design, c is 512KB.

The Code

There is no change to the front end code except there is an extra GET parameter when opening the websocket to define if the current upload is huge file upload. The following is an example implementation for the websocket object

 let hugeFileMode = "";
if (file.size > largeFileCutoffSize){
       //Filesize over cutoff line. Use huge file mode
       hugeFileMode = "&hugefile=true";
}

let socket = new WebSocket(protocol + window.location.hostname + ":" + port + "/system/file_system/lowmemUpload?filename=" + encodeURIComponent(filename) + "&path=" + encodeURIComponent(uploadDir) + hugeFileMode);

And here is the Go backend side implementation. Note the
isHugeFile flag and //Merge the file section.

targetUploadLocation := filepath.Join(uploadPath, filename)
if !fs.FileExists(uploadPath) {
    os.MkdirAll(uploadPath, 0755)
}

//Generate an UUID for this upload
uploadUUID := uuid.NewV4().String()
uploadFolder := filepath.Join(*tmp_directory, "uploads", uploadUUID)
if isHugeFile {
    //Upload to the same directory as the target location.
    uploadFolder = filepath.Join(uploadPath, ".metadata/.upload", uploadUUID)
}
os.MkdirAll(uploadFolder, 0700)

//Start websocket connection
var upgrader = websocket.Upgrader{}
upgrader.CheckOrigin = func(r *http.Request) bool { return true }
c, err := upgrader.Upgrade(w, r, nil)
if err != nil {
    log.Println("Failed to upgrade websocket connection: ", err.Error())
    w.WriteHeader(http.StatusInternalServerError)
    w.Write([]byte("500 WebSocket upgrade failed"))
    return
}
defer c.Close()

//Handle WebSocket upload
blockCounter := 0
chunkName := []string{}
lastChunkArrivalTime := time.Now().Unix()

//Setup a timeout listener, check if connection still active every 1 minute
ticker := time.NewTicker(60 * time.Second)
done := make(chan bool)
go func() {
    for {
        select {
        case <-done:
            return
        case <-ticker.C:
            if time.Now().Unix()-lastChunkArrivalTime > 300 {
                //Already 5 minutes without new data arraival. Stop connection
                log.Println("Upload WebSocket connection timeout. Disconnecting.")
                c.WriteControl(8, []byte{}, time.Now().Add(time.Second))
                time.Sleep(1 * time.Second)
                c.Close()
                return
            }
        }
    }
}()

totalFileSize := int64(0)
for {
    mt, message, err := c.ReadMessage()
    if err != nil {
        //Connection closed by client. Clear the tmp folder and exit
        log.Println("Upload terminated by client. Cleaning tmp folder.")
        //Clear the tmp folder
        time.Sleep(1 * time.Second)
        os.RemoveAll(uploadFolder)
        return
    }
    //The mt should be 2 = binary for file upload and 1 for control syntax
    if mt == 1 {
        msg := strings.TrimSpace(string(message))
        if msg == "done" {
            //Start the merging process
            break
        } else {
            //Unknown operations

        }
    } else if mt == 2 {
        //File block. Save it to tmp folder
        chunkFilepath := filepath.Join(uploadFolder, "upld_"+strconv.Itoa(blockCounter))
        chunkName = append(chunkName, chunkFilepath)
        writeErr := ioutil.WriteFile(chunkFilepath, message, 0700)

        if writeErr != nil {
            //Unable to write block. Is the tmp folder fulled?
            log.Println("[Upload] Upload chunk write failed: " + err.Error())
            c.WriteMessage(1, []byte(`{\"error\":\"Write file chunk to disk failed\"}`))

            //Close the connection
            c.WriteControl(8, []byte{}, time.Now().Add(time.Second))
            time.Sleep(1 * time.Second)
            c.Close()

            //Clear the tmp files
            os.RemoveAll(uploadFolder)
            return
        }

        //Update the last upload chunk time
        lastChunkArrivalTime = time.Now().Unix()

        //Check if the file size is too big
        totalFileSize += fs.GetFileSize(chunkFilepath)
        if totalFileSize > max_upload_size {
            //File too big
            c.WriteMessage(1, []byte(`{\"error\":\"File size too large\"}`))

            //Close the connection
            c.WriteControl(8, []byte{}, time.Now().Add(time.Second))
            time.Sleep(1 * time.Second)
            c.Close()

            //Clear the tmp files
            os.RemoveAll(uploadFolder)
            return
        }
        blockCounter++

        //Request client to send the next chunk
        c.WriteMessage(1, []byte("next"))

    }
}

//Try to decode the location if possible
decodedUploadLocation, err := url.QueryUnescape(targetUploadLocation)
if err != nil {
    decodedUploadLocation = targetUploadLocation
}

//Do not allow % sign in filename. Replace all with underscore
decodedUploadLocation = strings.ReplaceAll(decodedUploadLocation, "%", "_")

//Merge the file
out, err := os.OpenFile(decodedUploadLocation, os.O_CREATE|os.O_WRONLY, 0755)
if err != nil {
    log.Println("Failed to open file:", err)
    c.WriteMessage(1, []byte(`{\"error\":\"Failed to open destination file\"}`))
    c.WriteControl(8, []byte{}, time.Now().Add(time.Second))
    time.Sleep(1 * time.Second)
    c.Close()
    return
}

for _, filesrc := range chunkName {
    srcChunkReader, err := os.Open(filesrc)
    if err != nil {
        log.Println("Failed to open Source Chunk", filesrc, " with error ", err.Error())
        c.WriteMessage(1, []byte(`{\"error\":\"Failed to open Source Chunk\"}`))
        return
    }
    io.Copy(out, srcChunkReader)
    srcChunkReader.Close()

    //Delete file immediately to save space
    os.Remove(filesrc)
}

out.Close()

//Return complete signal
c.WriteMessage(1, []byte("OK"))

//Stop the timeout listner
done <- true

//Clear the tmp folder
time.Sleep(300 * time.Millisecond)
err = os.RemoveAll(uploadFolder)
if err != nil {
    log.Println(err)
}

//Close WebSocket connection after finished
c.WriteControl(8, []byte{}, time.Now().Add(time.Second))
time.Sleep(300 * time.Second)
c.Close()

And now you can have infinite file upload size

So there you have it. Now you can upload infinitely large file into your system as soon as you have enough disk space to store it. Notes that this upload method is very slow. It takes more than 2 time the speed than the previous method to actually merge the file as both of the reading file chunks and writing destination file are on the same disk. But for my use case, at least it works well enough for files that is too large to fit into the system RAM or the tmp/ folder.

I have no idea who will ever find this useful other than myself working on the ArozOS project. People with these issues nowadays usually just dump the file to AWS or whatever cloud service provider providing large file storage. But if you find it useful or you got even better implementation, feel free to let me know so we can further improve the design :)

tobychui / arozos

Web Desktop Operating System for low power platforms, Now written in Go!

Features

User Interface

Web Desktop Interface
Ubuntu remix Windows style startup menu and task bars
Clean and easy to use File Manager (Support drag drop, upload etc)
Simplistic System Setting Menu
No-bull-shit module naming scheme

Networking

Basic Realtime Network Statistic
Static Web Server (with build in Web Editor!)
mDNS discovery + SSDP broadcast
UPnP Port Forwarding
WiFi Management (Support wpa_supplicant for Rpi or nmcli for Armbian)

File / Disk Management

Mount Disk Utilities
- Local File Systems (ext4, NTFS, FAT etc)
- Remote File Systems (WebDAV, SMB, SFTP etc)
Build in Network File Sharing Servers
- FTP, WebDAV, SFTP
- Basic Auth based simple HTTP interface for legacy devices with outdated browser
Virtual File System + Sandbox Architecture
File Sharing (Similar to Google Drive)
Basic File Operations with Real-time Progress (Copy / Cut / Paste / New File or Folder etc)

Security

oAuth
LDAP
IP White / Blacklist
Exponential login timeout

Extensibility

ECMA5 (JavaScript…

View on GitHub

Update any application binary with this launcher!

Toby Chui — Tue, 08 Feb 2022 12:27:45 +0000

Recently, I was working on a software updater for my project and I found that there is no easy way for OTA update that is ready to use and work cross platform. So I decided to write one that is so universal, it works on all kind of application binaries or even interpreted languages, which I later called it the "Automatic Update Launcher"

tobychui / Automatic-Update-Launcher

A general purpose updater for updating program binaries when update folder exists

Automatic Update Launcher

A general purpose updater for updating (web) applications server when update folder exists

What is this?

The Automatic update launcher only do one simple task, which is to upgrade your application to the latest version if it finds a later version of such application in the ./updates folder.

Its logic basically goes like this

If updates/ folder exists
1. Backup the old application files and data
2. Copy the updates for application from updates folder to current folder
3. Remove the updates folder
Start the application
Check for crash, if crash happens after update
1. Restore the old application files and data
2. Remove the backup folder
Restart the whole process unless crash retry count > max allowed

Build

Require Go 1.17 or above

go mod tidy
go build

or just grab a copy from the release page.

Usage

Configure the launcher with launcher.json. See the example below

{
    "version":"1.0"
    "start":"helloworld*",
    "backup":

…

View on GitHub

Current Implementations

❔ How normal Desktop Programs Updates

Most desktop applications will use some kind of updater to update the main program by start and detaching the updater and allow it to do its things. This work on most desktop platforms but it will not works with systemd where the PID of the running application is important. (i.e. cannot change or otherwise systemd will assume it is dead and try to restart it)

❔ How embedded devices do OTA updates

Embedded devices usually uses two partitions (one for current running application, the other one for writing update files to) and a bootloader to do the switching. One common example ESP8266 which some projects utilize its bootloader's feature to do OTA update via WiFi or Bluetooth.

🌟 How the Automatic Update Launcher (auLauncher) works 🌟

The auLauncher works in a much simpler way following the logic described in the following diagram.

and the configuration of each steps and file names can be changed easily in a json file located next to the launcher's executable. Here is an example for that:

{
    "version":"1.0",
    "start":"helloworld*",
    "backup": ["./*.config","./*.exe"],
    "verbal": true,
    "resp_port": 25576,
    "max_retry": 3,
    "crash_time": 3
}

All arguments that passed to the launcher is then passed through to the main binary. So you can easily slide this in between your starting script and your target binary without the need for changing anything.

❓ How do my program update itself then?

It is simple! If you program want to update itself, just download the update files to the ./updates folder and exit itself with code 0. This way, the launcher will start overwriting the application files based on the defination in the json file and the contents in the ./updates folder and restart your application with given start path.

This method works for most Go compiled application, but in theory should also work with other compiled binaries and some interpret language like python with a correct starting bash / batch script.

If you are interested to know more, or just wanna try this launcher and make your application OTA update a new features, feel free to take a look at this repo and left some comments :))

tobychui / Automatic-Update-Launcher

A general purpose updater for updating program binaries when update folder exists

Automatic Update Launcher

A general purpose updater for updating (web) applications server when update folder exists

What is this?

The Automatic update launcher only do one simple task, which is to upgrade your application to the latest version if it finds a later version of such application in the ./updates folder.

Its logic basically goes like this

If updates/ folder exists
1. Backup the old application files and data
2. Copy the updates for application from updates folder to current folder
3. Remove the updates folder
Start the application
Check for crash, if crash happens after update
1. Restore the old application files and data
2. Remove the backup folder
Restart the whole process unless crash retry count > max allowed

Build

Require Go 1.17 or above

go mod tidy
go build

or just grab a copy from the release page.

Usage

Configure the launcher with launcher.json. See the example below

{
    "version":"1.0"
    "start":"helloworld*",
    "backup":

…

View on GitHub