Note: this is a tech person's guide, for sharing with other tech folk. I'm not attempting to make this completely newbie friendly - but I'd be happy to see its points used in something that is.
What is the Purpose?
The goal here is to create a fully-functional forever-updateable system. So the two things that this is not:
- it is not just a “live boot”;
- this is not just something to then use to install Linux onto the internal hard drive of a computer.
Rather, this is Linux ready to plug into the USB port of any computer that can be told to boot from that. Every time you boot this system, it will inspect and deal with the hardware that it finds.
This way, the external USB drive becomes your "system", and the computer is just the commodity used at the time. Depending on your use cases:
- a physically small drive is easy to hide for security;
- a drive can be more portable than travelling with a computer;
- a drive can be cloned for backup with instant usabilty;
- drive capacity is a simple purchasing choice - I currently use 1TB drives;
- it's quite feasible to make lots of these e.g. 16GB flash memory thumb drives are handy;
- stating the obvious a little, being Free Open Source Software, there are no license counts to worry about.
While I'll try to explain the principles, it is however just a description of how I make such a USB drive using, and for use on, my fairly regular, slightly-old laptop.
Overview
Why write this, when there are plenty of guides already published online?
Well, there are two things that make my process different to those I've seen online. These are:
- I perform the install with no drive present in the computer other than the target drive;
- I re-install the boot manager with a special command.
sudo grub-install --removable
The reasons for those two distinctions, are:
- there's no chance of anything being installed to anything but the target drive;
- the EFI boot that this new drive will enact, will not affect anything other than itself, even when doing later updates to the kernel and boot setup.
I learned the necessity of this the hard way, by building setups that would cause problems when the next kernel update happened. Back when I was doing MBR booting, achieving this was quite easy. It was EFI that changed everything, which is to say: complicated everything. I read a lot about EFI and tried a lot of things to no avail. Then I find out about the --removable option for installing grub.
It's now history, but I was doing this external USB full Linux for years, before computers changed from 32bit and MBR to 64bit and EFI.
Sequence in Detail
Here is my sequence in its full form. It's from my own notes, that I expect to re-use, years apart so I don't want to have to try remembering the little practical tips that I've learned are useful.
- Before I do anything, I look at my list of potential computer names and choose a new one.
- the USB drive always comes as NTFS so that has to go first. Boot an existing Linux, plug the new drive in, run GParted and have it create a new partition table (as GPT).
- while I'm there, download a current Xubuntu installer image and burn that to a DVD+RW and power down.
- Turn the laptop upside down, open a panel with a screwdriver then unplug the internal hard drive, put the panel back on, turn back over.
- Boot with the DVD, let it do an integrity check of itself - have a coffee and come back later. This integrity check would only be done the first time using that disc and is really only for the logic that it is better to check it once than later suffer consequences. The reality is that errors at this point are very unlikely nowadays. After the integrity check, let the disc boot into the "live" session.
- Plug in the new drive and start the installer process.
- As the only drive is the new USB drive I let the installer do what it wants. (It's really for this lack of complication + risk that I'd removed the internal drive.)
- When the installer asks, I put in my chosen new name for the "machine/computer", I give my standard name for an admin account. (I don't happen to use "admin" but that's an obvious good-enough choice.)
- The installation runs its course, and when done, prompts for a reboot which also ejects the DVD.
- I let it reboot.
- I log in to the admin account.
- I open a terminal and put in the command to re-install GRUB in "removable" mode (see below). This is vitally important for a USB booting drive as it will ensure it uses the EFI controls from the drive not the host hardware.
- Do a reboot to be sure nothing went wrong with GRUB.
- Shutdown and put the internal hard drive back in.
- Because I tend to be both cautious and pedantic, I boot the internal drive to be sure it's still ok.
- Reboot again, this time with the USB drive plugged in.
- Check the mounted partitions to confirm that the ESP (EFI System Partition) being used is the USB one, not the one on the internal hard drive.
The Special Command
With only one drive present, the special command is just
sudo grub-install --removable
Comments
While that might seem a lot, I have put in all the steps that experience has taught me to do - and it's all quite familiar by now. It also doesn't take very long. If it is slow that's because I usually use an optical disc in my process. When I've had to use a machine with no optical drive the installation step goes much faster - thumb drive beats a disc! But there is quite some more mucking about to prepare a boot thumb drive than a DVD, and I'm lazy.
The distro I install these days is the latest Xubuntu LTS 64bit Desktop. As far as I know the process is identical for any of the other 'buntu variants. How much it's the same for other main distros I can't say, but I'd be surprised that they'd be any different if they also use GRUB.
Most of us who use Linux will have a stock set of things we do after installation. There are many of these given as articles online. I also have such a set of things that I do, but that's really another topic.
The only part where those stock things-to-do intersect with this guide, is that I usually do my check of which ESP partition is in use by looking with the Gnome Disk Utility. That used to be trivial, but it’s not part of the default Xubuntu install, so I go through a few of my other usual post-install customisations to get it. There will of course be a command line method for doing that check, I just haven't needed to choose one.
Do you really need to unplug the internal drive?
Well no, but not doing so does introduce potential mishaps. I prefer a process that doesn't requite my careful attention to specifics.
There are two broad options should you leave the internal drive present.
- use firmware (aka BIOS) settings to have the drive seem to not be there;
- add a parameter to the
sudo grub-install --removablecommand to enforce which ESP it will direct to.
That second point is because if the internal drive is present, then there will be two ESP partitions present.
In all honesty, I haven't had the patience to experiment with the latter in order to gain certainty about it. Ditto for trusting the boundaries of firmware settings and booted drivers.
Indeed, I don't really trust the standard 'buntu installer to reliably ensure that the external drive will even get an ESP partition - at least, without being explicitly told by a custom partitioning step.
Even if/when I want to manually set the partitions, the process is still all clearer with the internal drive definitely absent. It seems a small hassle to unplug it.
Appendix
Because I cheated with some simplistic remarks, here are some caveats and extra details.
Live boot Linux with Persistence
Actually many of the methods used for Live boot Linux setups now allow for some kind of persistence. I had problems with the early attempts at these and have never revisited them. Hence I don't really have an opinion about them.
Firmware and Boot Order Variations
The issue here is that computers vary widely in how they allow for control of the booting order and setting of defaults. For this article, I considered that out of scope.
However I will say that for my own computer, I was able to use efibootmgr to reset its default EFI order so that I can just plug in my drive then power up to automatically boot into my roaming Linux. For the other computer in the house, that technique made no difference - and it needs a manual process for every boot to avoid going into Windows.
MBR vs EFI, 64bit vs 32bit
These aspects get complicated quickly. For this article, I've just assumed to only apply it to the current combination of 64bit computers, GPT partitioning and EFI booting.
But I have been using external USB Linux since 32 bit CPUs were the norm. There's been periods when I made setups that could be taken to and used on either 32bit or 64bit computers. It is very difficult to do a perfect setup, that will boot a 32bit Linux on a 32bit computer, but when used on a 64bit computer will boot into a 64bit Linux.
Instead if there is a need to use a system across all types, it's mainly better to just make it the old combination of 32bit computers, MBR partitioning and booting, with a 32bit Linux. Even with this, many modern computers will require some firmware setting changes to enable "Legacy" booting and/or turning of so-called "secure" boot.
p.s. I do want to credit the terrific comprehensive annotations about EFI at The rEFInd Boot Manager by Roderick W. Smith and while it wasn't there that I found the crucial GRUB installation trick, his notes meant that when I did, I understood the ESP puzzle.
Interaction with Windows
I've made no mention of having the booted Linux interact with the storage of Windows on a computer's internal drive. I simply have no need for that, and mainly see it as an anti-feature.
However, further in the past I would use these kinds of boots to backup/recover/repair the internal Windows system. When I noticed that Microsoft were having Windows shut down with the drive in some kind of locked state, I gave up dealing with it. Life is too short to be decoding Microsoft's machinations, so I just don't.
https://www.howtogeek.com/243901/the-pros-and-cons-of-windows-10s-fast-startup-mode/
See my other article
Except that I haven't posted it yet, I wrote a wiki page at my place of work so the text exists but will need adapting for a broader audience. When I've posted it, I'll return here to include a link.
Top comments (5)
Another Post Script
Doing the Setup Via VirtualBox
As this piece was a case of publishing the method that I actually use, the method it gives was based on the hardware that I happen to have. In particular:
But, I did wonder what better options there might be for the process.
A while ago, I chanced upon a video that provides a possible approach.
See:
removableflag for grubIn short, that method uses a virtualisation tool with a mapping of the external USB drive as being the only drive that a virtual machine is aware of. In the video he uses the proprietary VMware, so my question would be whether the same method can be used with the Open Source tool VirtualBox.
As it happens I do use VirtualBox myself, but by chance, the one instance I use is not on a setup that I'd be prepared to experiment with. Also, right now, I don't have any need to make another bootable USB setup. Still this might a method for me to trial the next time I make one/some.
It would also seem to open a possibility of making a VirtualBox-based tutorial for the whole process.
That said, the video left me a little concerned about how careful the attention to detail is required to not risk any damage to an existing system. More on this when/if I have a cause to trial this approach.
Try this, it worked pretty well for me.
Now fire it up and go thru the installer doing what you want.
If you have trouble booting and have some space left on the drive, try installing rEFInd afterwards -- rodsbooks.com/refind/getting.html -- found this useful for rescuing machines and bootable drives while figuring this out.
In reading this yet again, some time later, perhaps I'm not clear enough that this is all about the do-it-once process to make a fully portable boot-from-USB full installation of Linux.
Consequently, the machine on which this process is run, to make the portable system, is independent of the ones that system will then be used on for ever and a day. That's the whole point.
More Post Scripts
Some other points that came up while I responded to a related posting elsewhere, that I thought worth adding here.
Will USB2 be ok for this?
Yes. Will USB3 be noticeably better? Yes.
What size USB drive is required?
While my article is mainly about using hard drives, I do also use the same technique with USB "thumb" drives - i.e. memory sticks. I have a 16GB one with Ubuntu 20.04 on it, and have had to run a cleanup script a couple of times to make room for updates to work. FWIW the culprit is Snap. I also maintain a 64GB thumb drive with Ubuntu 20.04 without that problem. At a guess, a 32GB might be minimum size to recommend.
Watch out!
Lately, USB hard drives on the market are coming as Shingled Magnetic Recording (SMR) rather than the older Conventional Magnetic Recording (CMR).
I can vouch that SMR drives are NOT suitable for use as bases for a running system. The symptom you'll encounter is that your system will just pause for ten seconds at a time while the drive reshuffles the shingles on write.
Alas, like EFI, that becomes a whole other topic and is controversial as manufacturers are being neither open nor clear about which they are using.
Post Scripts
Some points added since the initial posting.
Booting the Installer disc in EFI or MBR mode
One of the quirks of booting from the DVD, is that this disc can boot either in EFI mode or in MBR mode.
Which of these actually happens may depend on the computer's firmware and/or the options it shows you and which you choose.
On the machine I normally use, the firmware is set to boot from a disc and shows me entries for both modes from which to choose. This really is a case of "your mileage may vary" and the variations of firmware settings and behaviours is beyond my (or anyone's?) ability to cover,
In short, you may need to be observant to be sure of what goes on when you boot the installer.
For booting from a thumb drive made from the disc ISO file - alas, by avoiding this I've not seen how/why/when it does which of the two modes.
Why did I have a step to remake the partition tables in GParted?
The short answer here is that I've never quite trusted whether the installer is running in EFI mode or MBR mode. But also on top of that, whether when it is told to use the whole drive for the install, will it reset the partition table type - i.e to GPT for the EFI install.
To be sure, the failing here is that the installer doesn't tell us exactly what it is doing.
Having done a lot of installs, I can sympathise about them having the right balance between selectable options and not being confusing.
I have found with the Ubuntu and Xubuntu installers, as long as they are running in EFI mode, and the target drive already has a GPT partition table, then the outcome is reliable.
What changes would I like to see made to the installer?