To make an operating system, you need to know the following definitions.
- Hardware - Physical items you can touch. Think CPU, GPU, RAM, MB, etc.
- Firmware - Adds the functionality behind the hardware.
- Kernel - Gateway for software to talk to the firmware.
- Software - Code that doesn't work with hardware directly.
- Userland - The OS backend, how it works.
- Middleware - Talks to userland/kernel so applications can run.
- Applications - Compilations of code that the user interacts with.
These definitions hold up a lot to keep the OS running.
- Based OS - Based on a kernel, such how Debian is based on Linux.
- Derivative OS - Like how FreeBSD derives from Unix
- Source OS - You build everything, kernel up. Like SerenityOS. or ReactOS.
These types will tell you how difficult everything is. Here is a table that labels difficulty per type.
The rest of this article takes in a lot of FOSS and open-source. Closed-source paid operating systems are going to be difficult to distribute and maintain as you agreed to work alone from Source as most derivatives you will be able to use will often require the same license or rarely choice with a similar license. Often this is a form of GPL-v2.
Doing closed-source development will not only require a harder development, but you will also need to find teammates who can work within your budget, without community support. So for the sake of making everything nice and easy, as well as fair and powerful, open-source (of any license) will be our license type of choice.
How to do this?
Well, remember, a base is just the Kernel, everything else you do by your self. I recommend doing Linux as a base, as it is popular and easy working. Books like Linux From Scratch will also be an amazing beginner's guide to the craft.
This means to take an already existing OS (active or not), like Ubuntu, Debian, Arch, Void, FreeBSD, BeOS, and using that to get your OS off the ground. If you want a challenge, work with an OS not currently active, you can find many at ArchiveOS.
Doing this, I would recommend not using a live-system. A live-system is one that is currently installed. Let's say you have Ubuntu 20.04 and are running that to read this, that is a live-system. This isn't to avoid stolen data, that is not difficult to get around. The issue is that there are no "good" active tools. Many seem to be either inactive (like Systemback or Remastersys), halfway dead (like Linux Respin), or active but not well working or documented (like Linux-Live). While some might be usable now, they might not be in the future.
There are tools like the Ubuntu Budgie ISO-Builder, used by Ubuntu Cinnamon, Ubuntu Deepin, and Ubuntu Lumina. It also can do much more than a simple desktop switch, given a little time and effort.
There is an entire Wiki for this at OSDev but the issue is not that it is not well documented, but OSDev itself states the following quotes:
Not all "make it" in this field, many don't even pass the "Hello World" of OS development
developing an operating system is probably one of the most challenging things you can do on a computer
...it returns very little to no "instant gratification"
Writing an OS is not a beginner's task. In fact, writing an OS is usually considered the most difficult programming task.
You will need above-average programming skills before even considering a project like this. Failure to comply will make you look silly.
Those were from three pages of the Wiki, cherry-picked from the first five sentences in the paragraphs. If you want to jump in, they give a small and reasonable list of things you will not only need proficiency in but above-average proficiency in. You can find that here.
If you desperately want to make an OS from source, then look into OSDev.
Don't: Use free software, and call it open-source software. Just because it is free, and the source is available to use in some areas. That doesn't mean it is FOSS. For example Vivaldi and Brave. Partly closed source web browsers that multiple Linux OS's uses. Free and partially open-source isn't FOSS.
Don't: Pick software because everyone else uses it. I would love to see the day where systemd isn't killing my dreams. Also not see any new versions of the worst desktop environment as of current: GNOME.
Don't: Use easy software if it slows everything down. Look back again at systemd.
Don't: Listen to recommendations if it goes against what you want. That will kill your OS faster than if you run
rm -r /* --no-preserve-root on your machine right now. ALSO, DO NOT DO THAT!! THAT COMMAND ERASES EVERYTHING AND I VERY MUCH MEAN EVERYTHING.
Don't: Over-work. Burnout kills anything.