Hello, I want to share my Experiment of this 'Little Nightmare' about insufficient Storage Space in running live Server, in this case I use Google Cloud Services : Compute Engine and Ubuntu as the Operating System.
Note :
this post contain some Story, Trial & Error, Skip to the Solution Section if you want to see the Answer.
The Problem
I've got message from my friend, he sent me this Error Message :
and he also can't execute some ssh command on the Server, it always return this :
cd /va-bash: cannot create temp file for here-document: No space left on device
then, i told him to run this command for check directory file size :
du -h --max-depth=1
this is example of the output from command above, picture taken from Twitter
at this moment, he can running rm -rf to cleanup some Files / Directory and free up some space.
now the Server running well and the website up again.
The next day, he sent message to me, the disk on the server running out of space again, and he asked me to increase the size of the disk.
Everything getting worse here, and I've made my mistake, I'm in rush and ask him permission to shutdown the instance for a while, then I increase the size of the Disk from Google Cloud Console page.
after I boot up the instance again, we can't connect to the SSH (using Putty / WinSCP), the server refuse our public key and user profile :
the journey begin...
Attempt to Solve
1. Restart the Instance
The first thing I do is Restart the server, and hope with this little magic will fix the SSH Connect issue, but unfortunately everything still not working.
2. Connect to SSH in Google Cloud Console
Then I try to connect the SSH from Google Cloud Console Instance Page, and Still no luck, Connection Failed
3. Connect to Serial Console
So I start to Google my problem, and found some answer from Serverfault and Stackoverflow to turn on the Serial Console and try to login from there using this Startup Script :
#!/bin/bash
adduser USERNAME
echo USERNAME:PASSWORD | chpasswd
usermod -aG google-sudoers USERNAME
Unfortunately, the startup script Failed to Running, and I see this error from Serial Console :
GCEGuestAgent[1121]: 2022-02-28T14:40:50.0037+07:00 GCEGuestAgent Error non_windows_accounts.go:144: Error creating user: useradd: /etc/passwd.1520: No space left on device#012useradd: cannot lock /etc/passwd; try again later.#012.
Start to PANIC... then I think that we locked to the Server and almost can't do anything.
Moment of Silent...
4. Create new 'restore' disk with old disk snapshoot
After read some related answer on the forum that I've got from google, I try to create new Disk with bigger Storage Size (from 10GB to 20GB) and using old disk snapshoot as the source
I hope with this new Disk and bigger size everything will run smoothly when I attach this new disk as Boot disk to the Instance without losing any existing data. But Unfortunately, when the server run, we still Refused from the Server by using Putty / WinSCP and also Google Cloud Console SSH.
Solution
Almost gave up, but I will try my last Shoot. and YEAH!! it's working.
here is the step that I do to solve this problem
1. Create new Instance with new Disk and new OS Image
Creating new Instance with new Disk with Ubuntu and start it, make sure I can connect using SSH.
the idea is to Attach the old disk to this new Instance as additional disk
2. Attach old 'fully' disk as additional disk to the new Instance
go to Compute Engine Instance List, and choose the new instance then click the edit button.
find the Additional Disk Section and Click Attach Existing Disk, then choose the old disk.
Make sure to detach the old disk from Boot Disk at the main instance
3. Mount disk to filesystem folder
After attach the old disk, run this command to check if the disk Exist / loaded in the instance :
sudo lsblk
and it will show the list of the disk partition
in my case, the old disk identified by sdb and the partition that we need to fix is at sdb1 to filesystem, let's say we mount it to /myOldDisk in the new instance
- create myOldDisk folder by execute this command
sudo mkdir -p /myOldDisk
- mount sdb1 to /myOldDisk
sudo mount -o discard,defaults /dev/sdb1 /myOldDisk
- configure permission
sudo chmod a+w /myOldDisk
- run sudo lsblk again, now I can see my sdb1 mounted to /myOldDisk
4. Clean up or backup to local computer some files for free up more space in the mounted folder
since our old Disk mounted to /myOldDisk I can easily go to the directory then cleanup some files / folder to get more free space
cd /myOldDisk
then I can find some big files to delete or archive it as zip and download it to my local computer
I get 500MB+ free space and I thing it's save to go
5. Umount the old disk
unmount the sdb1 from filesystem by running this command :
umount /dev/sdb1
detach my old disk using Google Cloud Console at this new Instance Detail page, then we can shutdown and delete this new Instance & new Disk, we don't need that anymore.
6. Increase the old disk size at Google Cloud Console
Go to Google Cloud Console Disk Page, and select the old Disk, and increase size, in my case is from 10GB to 30GB, and save it.
7. Attach the old disk to the old Instance
I need to Attach my old Disk to my old Instance as Boot Disk, go to Compute Engine Instance Detail, edit the instance and add the Boot Disk, select my old Disk.
8. Run the instance and Growpart the Partition
Turn on the Instance, Finally now I can login SHH to the Server,
when I check the free storage by running df -h command, the storage space still 10GB.
my next step is to Update the disk partition by using Growpart from 10GB (existing) to the new size 30GB
I can check the new size by running :
sudo lsblk
It will show new dev/sda storage size, all i Need to do is Grow the Partition by running this command :
growpart /dev/sda 1
CHANGED: partition=1 start=2048 old: size=419428319 end=419430367 new: size=2097149919
,end=2097151967
then I need to Resize the Partition by execute this command :
sudo resize2fs /dev/sda1
it will return something like this :
resize2fs 1.42.13 (28-Feb-2022)
Filesystem at /dev/sda1 is mounted on /; on-line resizing required
old_desc_blocks = 13, new_desc_blocks = 63
The filesystem on /dev/sda1 is now 262143739 (4k) blocks long.
Finally Done ! now the Disk have extra space, I can verify the available / free storage by running df -h
It's time to rest and drink some Coffee..
Summary
from this Experience I Learn :
Never ever Shutdown your 'full storage' server, Free up some space before increasing the disk size, and make sure everything is fine before the shutdown.
You don't need to shutdown the server for increasing the storage Disk
I'm so sorry for my bad English, and I hope this post will be guide Me or Someone else when facing the same problem in the future.
if you have suggestion about this post feel free to write it down in the comment section below :)
Top comments (2)
Thank you so much, you saved my day!
You're welcome :D