DEV Community: Ethan Rodrigo

Teleporting Virtual Machines: Flipping the Script

Ethan Rodrigo — Fri, 26 Jun 2026 08:17:52 +0000

In the previous post in this series, Teleporting Servers, we examined how to move a live virtual machine between physical hosts using the pre-copy method. Pre-copy transfers the VM’s memory in multiple rounds while the source keeps running. When the number of dirtied pages falls below a configurable writable working set (WWS) threshold—or a preset maximum number of iterations is reached—the VM is suspended and its CPU state plus any remaining dirty pages are sent to the target host; that final transfer is the service downtime phase.

The Pre-Copy Bottleneck

We were good with pre-copy, right? Why do we need a new method? Well, pre-copy takes seconds and in virtual world, a second is an eternity. Let's see why pre-copy take time.

What is the cap to stop the iterations of transferring memory? Right, it's WWS floor. But what if the smallest WWS reached is too large? The downtime will be high and the processes that needed to be continued will take some time. This means that pre-copy is good only if the workload of the VM is read-intensive (not too much of page dirtying) and if the workload is write-intensive the downtime will be higher so is the applications' performance.

The Post-Copy Flip

Take the analogy from the previous post, the moving in example. What if you take your essentials in the first run and start living in the new apartment? Then you can move your other big items later and hand over the key to the owner.

Now apply that to the VM migration. First you transmit the VM's processor state to the target and start the VM there. Then actively push the VM's memory pages from the source to target. Meanwhile, if there's any page faults they are sent over the network from the source.

The Four Pillars of Post-Copy

Post-Copy under the hood uses 4 methods to make the migration process efficient.

Demand Paging - This is when a page fault occurs and it is requested from the source over the network.
Active Push - The pages are sent to the target from the source even without a page fault occurring to make sure the target will not be dependent on the source as soon as possible.
Prepaging - This is more like a forecasting technique used to identify the page access pattern and get the needed pages even before there's a page fault.
Dynamic Self-Ballooning (DSB) - Why are you sending the free pages over when you can just drop them? DSB takes care of that.

Predicting the Future with Bubbles

As mentioned above, pre-paging forecasts what pages will be faulted on the target. Let's see what happens behind the scenes.

As mentioned earlier, Prepaging is used to make pages available at the target before they are faulted on by the running VM. The effectiveness of the prepaging is measured by the percentage of the page faults that requires an explicit page request to be sent over the network to the source. Smaller the percentage, better the prepaging algorithm.

But how the pages needed in the future are decided? Computer programs don't usually access memory completely at random. If a program needs to read data at Memory Address 100, there is a very high probability that in the next millisecond, it's going to need Address 101, 102, and 103. This is called "spatial locality."

Now there are two methods in which the pages needed to sent are decided.

Bubbling with a Single Pivot

What happens when you throw a rock into a pond? There will be ripples spreading outwards in circles. Now compare it with a network fault. Initially the pages are sent over in one direction (0, 1, 2, ...), where the pivot is page 0. When a network page fault occurs, the pivot moves there. From there the pages are sent in both directions, front and back. Assume that page fault is 50. The pivot will be 50 and then the pages will be sent in backwards 49, 48, 47, ... and also frontwards 51, 52, 53, ... Whenever a pages which has been transferred is met, it is skipped, ensuring the pages are transmitted only once.

Bubbling with Multiple Pivots

Now imaging the VM having multiple processes running, the new VM would fault on page on multiple locations. Thus there need to be multiple pivots, causing multiple bubbles. Each bubble will expand around an independent pivot. If one edge of a bubble comes across a page that is already transmitted that edge will -be stopped. As for the efficiency, it has found limiting the number of pivots to around 7 is a good idea. Therefore whenever a new pivot is occurred and the limit is hit, the new pivot will replace the old one.

As for the direction of the bubble growth, it has found that forward expansion is essential, backwards-only expansion is counter productive and bi-directional expansion performs just right.

What to Send First?

Generally, Linux maintains two linked lists in which pages are accessed in Least Recently Used (LRU) order; one for active pages and one for inactive pages. There's a kernel daemon periodically swapping pages around the two lists. Later on, the inactive list is swapped out of RAM to the SWAP device. This is quite helpful in post-copy implementation, while deciding what pages to send first.

But Linux is lazy. If there's enough memory and no swap device, Linux just sits there and leave the lists unsorted. And the migration's pseudo-paging device is turned on last millisecond of the migration, there's no time to organize the list and migration algorithm has no idea which pages are active and which are inactive.

Therefore the developers have implemented a kernel thread which runs in the background long before the migration starts.

each time an application touches a page, it flips a tiny switch of the page called a "Referenced bit"
Then while the thread goes through memory and if it sees a page with Referenced bit turned on, it clears the bit and moves the page to the top of the Active List.
If a page sits there for a long time without its referenced bit being turned on it will be slides down in to the inactive list.

Setting the Trap: Catching Page Faults

There are three ways of trapping page faults by demand-paging component of post-copy.

Shadow Paging

Hyperviser have a read-only page table for each VM that matches its pseudo-physical pages to the physical page frames. If the VM tries to read or write a page that hasn't arrived yet, the hypervisor catches the violation, stops the vm, fetch the page and lets it continue.

Page Tracking

Page Tracking; During downtime, all the pages on target VM are marked as not present in their PTE. When the VM wakes up, it throws errors on everything, a custom software intercepts. This needs a lot of hacking into the guest OS.

Pseudo-paging

As soon as migration is started, the memory pages of the migrating VM at the source are swapped out to an in-memory pseudo-paging device, which resides on the guest kernel. Then the CPU state and non-pageable memory are transferred to the target during downtime. But there's a catch: if the OS detects that even a single ounce of its physical memory is missing, it panics (a kernel panic) and crashes. The solution? A heist.

If you found out there was a priceless diamond in a capital museum, how would you steal it without tripping the alarms? The easiest way is replacing the original with a replica of the exact same weight.

The same goes for pseudo-paging. If the OS won't let us take the memory, we just replace the original data pages with empty pages. This is called the MFN Exchange. Here's how it works:

Before the migration starts, the hypervisor goes to its reserves, gathers a massive pile of completely empty, useless physical memory (the sandbags), and temporarily doubles the VM's memory reserve. Next, all the running applications are temporarily frozen so they stop writing new data.

The guest OS is then instructed to swap out its memory. As it does this, the hypervisor intercepts the pointers. It takes the VM's internal addresses (PFNs) and reconnects them to those empty, useless physical chips (MFNs). The VM is satisfied. Meanwhile, the hypervisor takes the real physical chips holding the actual data (the diamonds) and quietly hands them over to Domain 0 to be beamed to the new server.

With the memory safely stolen, the whole VM is suspended for just a few milliseconds and its "brain" is sent over to the target. Once awake on the new server, if the OS hits one of those empty replica pages, it throws a "page fault." A third-party software driver (the MemX client) intercepts this error and immediately pulls the missing data across the network from the old server's Domain 0.

Handle the Free Memory

Transferring a large number of free pages is a waste of resources and would increase the total migration time regardless of the migration algorithm you use. Also, if the moved VM asked for a brand new empty page, there will be a page fault and an empty page will be fetched from the source wasting time as once arrived, that page is overwritten anyway.

A technique called ballooning is used for resizing the memory allocation of a VM. Usually there is a balloon driver in the guest kernel. It can either ask the guest for free memory and give them back to the hypervisor (inflate the balloon), or request pages from the hypervisor and return them to the guest (deflate the balloon).

Dynamic Self-Ballooning

Now that mechanism is used to avoid transmission of free pages during both pre and post copy migrations. The VM performs ballooning continuously over its execution lifetime - and its called Dynamic Self-Ballooning (DSB).

DSB has three components

Inflate the balloon - VM has a kernel-level DSB thread that allocates as much as free memory as possible and hand them over to the hypervisor.
Detect memory pressure - Memory pressure means some entity needs to access a free page.
Deflate the balloon - In response to a memory pressure the balloon must be partially deflated, i.e. reverse of inflating. DSB thread re-populates the free memory from the hypervisor and then release them to the guest kernel.

How to detect memory pressure?

Imaging you're a manager of a renowned restaurant. There are 100 tables in the restaurant, and the Reserved sign is put on 95 of them, and 5 are left as if anyone come those tables can be given. Now a massive VIP party walks in, you get panic and shouts "Get 20 tables and seats for the VIPs".

Same applies while detecting memory pressure. The DSB process (the manager) takes up (inflate the balloon) to 95% of the available free memory (if taken 100% there would be out-of-memory trigger). Then if an application (VIPs) asks for memory, and if there's no free memory a panic triggers. And that panic is a "Memory Pressure". In Linux if this panic occurs it sirens an alarm, a shrinker function. It's like yelling "Hey, if anyone has unused cache, throw them away". The DSB periodically do this process to make sure the OS run smoothly.

Now the developers of post-copy implemented the balloon driver such that it can listen to this alarm, so it can deflate as needed and the application run smoothly.

The Reality Check: Did it Actually Work?

The developers didn't just build this in theory; they put their Post-Copy prototype through the wringer with some heavy, real-world server applications. Here is the final scorecard:

The Triumphs (Where Post-Copy Wins):

The Bandwidth Savior: Because Post-Copy guarantees every page is transferred exactly once, it absolutely crushes Pre-Copy on network efficiency for write-heavy workloads. No more endless loops of re-sending dirtied data.
Bubbling is Magic: The multi-directional "Bubbling" algorithm, combined with the LRU (Least Recently Used) list sorting, worked beautifully. It predicted the VM's needs so well that it eliminated a massive chunk of the sluggish network page faults.
DSB is a Cheat Code: Dynamic Self-Ballooning drastically reduced the total amount of memory that needed to be transferred, speeding up the entire migration process from start to finish.

The Catch (Where it Stumbled):

The Downtime Spike: In a perfect world, Post-Copy downtime is near zero. However, because their specific "pseudo-paging" hack couldn't swap out the core, protected kernel memory, they had to pause the system to send that chunk over manually. This resulted in a slightly higher downtime than they wanted.
Read-Heavy Workloads: If your server is mostly just reading data (like a static web server), good old-fashioned Pre-Copy is still the reigning champion for overall speed.

The Final Verdict: Which should you choose?

There is no single silver bullet. They are different tools for different jobs.

If your server is a calm, read-intensive machine, use Pre-Copy. It's safe, reliable, and keeps downtime incredibly low.

But if your server is a chaotic, write-intensive beast—like a massive database actively changing gigabytes of memory every second—Post-Copy is the ultimate getaway vehicle.

Conclusion

We discussed how post-copy can be a complementary tool for pre-copy, and how it was implemented.

Note: This blog post is a summary of the research paper Post-Copy Live Migration of Virtual Machines, and the sole purpose of the post is to summarize what I have learnt by reading this paper.

Teleporting Virtual Machines: The Magic of Live Virtual Machine Migration

Ethan Rodrigo — Wed, 03 Jun 2026 04:52:10 +0000

Say you want to upgrade your laptop's RAM, what's the first thing you do if you have a new RAM stick in your hand? Turn off the laptop, install the new RAM, and restart. Pretty easy? Now imagine you're a system engineer at Amazon AWS. You are asked to upgrade the RAM of a physical machine, but it got some virtual machines running on it, what would you do? If you stop the server even for a second, the users will notice immediately. How to upgrade the RAM without shutting down the services?

The Zero-Downtime Dilemma

One thing you can do is to migrate the processes running on those virtual machines to a new ones. But the problem is the old virtual machine may still be needed for some network and system calls. Thus you can't shutdown the old machine easily.

What if you could take a virtual machine from one physical host and “teleport” it to another—without anyone noticing?

That’s not magic. It’s called live migration. With live migration, the users won't be having any issues (nor will they even know) if you replace the host of their virtual world. And on the other hand system engineers will have least time worrying about long migration time.

How Do We Live Migrate VMs?

But how is that possible? First let's consider two parties which we have to make sure are happy. The end user; there should not be a downtime. And the operator of the data center; They should do it in the least total migration time. If both those factors are achieved, everyone is happy.

To run the VM on a new machine, several things must be copied from the original. The most important one is memory, because it represents the VM's entire working state. Therefore first let's see three ways the memory can be transferred.

Push: Push the memory pages across the network while the source Virtual Machine (VM) is still running.
Stop-and-copy: Stop the source VM and copy the pages across the new VM. Then start the new VM.
Pull: Start the new VM while the old one is still running. And if the new VM access a page that hasn't copied yet, make a page fault and pull it across the network from the source VM.

None of these approaches work well on their own. For example if the old VM is completely stopped and start the new one, the downtime for the users is high, so is the total migration time. On the other hand if the new VM is started and pulled as needed, the total migration time is high.

The Winning Approach

So none of these approaches work well alone. The solution is to combine them; pre-copy, A combination of iterative push phases with a very short stop-and-copy phase. The memory pages are sent over the network in an iterative manner and after that the old host is suspended and sent the processor state to the new host so that it can be start.

The 5 Phases of Live Migration

Step 01: Reservation

A request is issued to migrate an OS from host A to host B. A calls B and says, "Hey, here's something for you". The host B confirms if the necessary resources are available and reserve a VM container of that size. "Yeah I got it, send over".

NOTE: Failure to secure resources means the VM simply continues to run on A.

Step 02: Iterative Pre-Copy

First iteration, all the pages are transferred to B. Subsequent iterations copy the dirtied pages (pages that have been modified) during the previous transfer.

Step 03: Stop-and-Copy

Suspend the running OS instance at A and redirect its network traffic to B. Then any remaining inconsistent memory pages are transferred to B along with the CPU state. Now both A and B have suspended copies of the VM. Still the A's copy is the primary and will resume in case of failure.

Step 04: Commitment

Host B indicates to A that the OS image is successfully received. Host A acknowledges this and it may now discard the original VM. Host B becomes the primary host now.

Step 05: Activation

The migrated VM on B is now activated. Post-migration code runs to reattach device drivers to the new machine and advertise the moved IP address.

The Bottleneck: WWS

Great, we can replace the VMs anywhere we want easily. Not really. It's too good to be perfect. There's a catch.

Writable Working Set

Consider this analogy. You're moving to a new apartment on the weekend. You have bought the apartment and all you have to do is move the items in your old apartment to the new one. But the problem is you can't move all at once, it may take few days, and you can't hinder your routine tasks, like brushing your teeth, washing yourself and eating.

Now in the virtual world, the routine tasks are the one that dirties the pages rapidly. Even after copying memory, the VM continues running—meaning new changes are constantly being made. And those are called writable working set.

The "Moving"

Let's say for the first round you moved the furniture and other big items you don't use daily into your new apartment. And for the second round you move your clothes and your sports items. Then for the last round you can take all the utensils you use daily and put them in your car and give the keys to the owner and move yourself to the new apartment. There you can unpack those utensils again and live normally.

Same goes for the VMs, there will be some rounds to send the memory that doesn't change rapidly. Then once the algorithm decides the WWS is small enough the system is frozen and moved (stop-and-copy) to the new VM.

Knowing When to Freeze: The Stop Triggers

In practice, systems rely on dynamic heuristics to decide when to stop.

The "good enough rule"; Once a round is completed, the software asks itself, 'how long will it take to send the remaining pages?'. If the answer is a couple of milliseconds, it just freezes and does the final copy
Hitting the WWS floor; Let's say the first round copied 413 MB, the second one did 112 MB, third 15 MB, fourth 14 MB and fifth 15 MB. The system knows it has hit the WWS floor as the dirtied memory isn't shrinking anymore. Thus it stops the iteration and initiate the final copy.
Setting a hard limit; What if there's a process which never stops writing to the memory. That's when the developers has put a cap on how many times the system will tolerate failure. It just aggressively slows down the rogue process and forces the migration process to finish.

Under the Hood: Two Ways to Build It

There are two ways migration can be done. In each method, managing the dirty pages and freeze are different.

Managed Migration

Managed migration is performed by migration daemons running in the management VMs (one special virtual machine in a physical host that is used for the administration and control of the other machines) of the source and destination hosts. '

Setting the Trap: Shadow Page Tables

As mentioned in migration steps, the copying is done in rounds. The first round copies the whole memory and the subsequent rounds only copies the dirtied pages during the previous round. To keep track of those pages dirtied during the rounds a dirty bitmap is used by the Virtual Machine Manager (the one who creates and runs the virtual machines) at the start of each round.

The VMM uses shadow page tables which is populated using the page table (a map to identify which memory page is used and which are not) of the guest OS, making all the page table entries there read-only. If the guest OS tries to write to a page, there will be a page fault. Then the VMM intercepts the fault, checks if the guest OS actually has permission to write, and if so, logs it in the dirty bitmap and allow the guest OS to write and continue working.

Once a round is over, the bitmap is cleared and the shadow page table is destroyed and recreated.

When it's determined that the pre-copy is no longer beneficial (using heuristics as mentioned above), a control message requesting to suspend itself is sent to the OS. Once the OS has done this, VMM informs the control software, the dirty bitmap is scanned one last time for inconsistent memory page, and these are transferred to the destination together with the VM's check-pointed CPU-register state.

When the final information is received at the destination, the VM state on the source machine can safely be discarded. The control software on the destination machine scans the memory map and rewrites the guest's page table. Execution is then resumed by starting the new VM at the point that the old VM checkpointed itself.

Self Migration

Self migration is done within the OS being migrated (migratee). It's a bit similar to managed migration. At each pre-copying round every PTE is write-protected. The OS maintains a dirty bitmap tracking dirtied pages. But the OS also manages other page faults like this. Thus to distinguish there is a reserved spare bit in each PTE to indicate that it's only for dirty-logging purposes.

In managed migration the migratee can be suspended and obtain a consistent checkpoint. But in self migration the OS must continue to run in order to transfer its final state. Thus the final stage is two stepped. First all the OS activities except for migration are disabled and a final scan is performed to find the dirty bitmap. Any pages that are dirtied during this final scan are copied to the shadow buffer. As second final stage, the shadow buffer is transferred as the OS checkpoint.

The Cheat Code: Paravirtualization

Paravirtualization is a virtualization technique where the guest operating system is modified to communicate directly with the hypervisor rather than using hardware emulations.

There are two ways it is beneficial for live migrations.

Stunning Rogue Process

There may be cases that some processes dirty memory at a rate which can't be transfer via Ethernet, and those are called 'rogue' applications.

We can mitigate this by forking a monitoring thread within the OS kernel to monitor the WWS of individual processes and take actions if required.

Freeing Page Cache Pages

There will be some part of memory that are free but have used to store cache pages. Those are irrelevant while migrating a VM and will reduce the performance. Thus we can just clear the cache and send. But if the contents of the pages be needed again, there will be a little time consumption to read them from the disk again.

The Invisible Backbone of the Modern Cloud

It sounds like magic, but the engineers who designed this system managed to get the downtime for a migrating server down to as little as 60 milliseconds. That is faster than the blink of an eye.

When you read through the mechanics—from shadow page tables trapping memory faults to the operating system dropping its own cache to speed up the move—you realize it isn't magic at all. It is just incredibly clever systems engineering.

Today, this foundational concept is the invisible backbone of the modern internet. It is the reason AWS, Google Cloud, and Azure can perform massive hardware upgrades, balance millions of terabytes of traffic, and maintain physical servers without ever dropping your Netflix stream or pausing your online game. The next time you are playing a multiplayer game without a single glitch, just remember: the physical server hosting your match might have just been teleported to a completely different rack miles away, and you didn't even notice.

Note: This post is a simplified breakdown and interpretation of the original research. If you want to dive into the deep technical math and architecture, I highly recommend reading the original 2005 paper by Christopher Clark and his team at Cambridge University.

Jetpack Compose 101; Setup the Environment

Ethan Rodrigo — Sat, 04 Feb 2023 17:57:20 +0000

Welcome to the second installment of the Android Development series! In our previous article, a brief overview of the Jetpack framework and Jetpack Compose was provided. In case you missed it, be sure to catch up by reading it here.

It's time to get hands-on and set up the environment for Jetpack Compose. So, grab a coffee, fire up your laptop, and let's dive in.

NOTE: This article series assumes that you have an understanding of native android development fundamentals. This series is here to take your skills to the next level. The main focus in this series is solely on the Jetpack framework and its capabilities.

Setup the Environment

Open up the Android Studio(you can download it from here). If this is your first time, you can see something as follows.

Now click the boxNew Project on the top left side.

In the following dialog, select Phone and Tablet(often it's selected by default), then Empty Compose Activity from the right side. Finally, click on Next button.

In the next dialog box, give your app a Name. I name it as My First Compose Application(I know that's too long).

The Package name is a unique identifier for your app on a device. Let it be the default. Also, consider changing the Minimum SDK, which is the minimum android version that your app can be installed on.

Keep the default values for the other settings.

Give it some time to create and initialize an environment for you. Your internet connection speed affects the initialization process as Android Studio downloads some packages from the internet.

Then you'll have something as follows

A quick glimpse at Android Studio

As mentioned earlier in the post, this series focuses more on the Jetpack framework, and assumes you have some experience with android development and Android Studio. The following tools will aid us in our journey of learning Jetpack Compose.

Design View

On the left side, you will find three panels labeled Code, Design, and Split. The Code panel displays the code that you are currently working on, the Design panel shows the visual representation of the UI in a composable function, and the Split panel combines both in a single panel.

Click on the Split box and you will see the UI from the function called Greeting. If you see nothing, just click on the build and refresh.

This is only for testing the UI. To ensure the proper functioning of the app, you will also need to test it using an emulator in Android Studio. The emulator allows you to run and test your app in a virtual environment that mimics a real device.

Emulator

You can run your app with the little green play button on the top. Additionally, the green hammer on the left side is to build the project.

Don't worry if you have no emulator selected as in the above screenshot. You can always create a device by going to the Device Manager on the left side.

And use Create device to create a new android device.

There you can download an iso image of the latest android version and then select the ram, CPU, etc.

Run on your phone

If your hardware is outdated and cannot run an emulator, consider testing the app on your android phone instead. You can use Pair Devices Using Wi-Fi option located below the selected emulator. Keep in mind that this option needs both of your devices(laptop and phone) connected to the same network.

You will be prompted with something as follows.

To connect your phone using this option, go to the Developer option(check the documentation on how) in your phone and select Wireless debugging. Then choose Pair using QR code. This will open your camera and you can use it to scan the QR code generated in the window. Then your phone will be connected.

You can also use Pair using pairing code. Click on it and choose your device, then select Pair. Type the pairing code in your phone and you'll be good to go.

However, this option works only for Android 11 and higher devices only. If your device doesn't have this option as well, you can use USB Debugging. Read the documentation to set it up on different platforms.

Dive into the code

It's coding time...

NOTE: To proceed further you need to have a solid understanding of Kotlin programming language. If you're not familiar with it, take some time and review the documentation and make yourself comfortable with the basics. Other complicated topics can be added on the fly.

Composable functions

The traditional way of designing the UI for an android app is XML. However, Jetpack Compose has thrown away XMLs.

And embraced Composable functions. Wait..., what is a composable function?

Composable functions are where you define your UI. It is just like a usual function in Kotlin but has the @Composable annotation above the function. Each composable function takes some inputs and returns some UI elements. These UI elements are then combined to form the final UI of the app.

Remember, Composable functions can be called from another Composable function only. You can't use them inside a usual function.

MainActivity.kt

Let's dive into the code in front of us. First, let's break down the class we have.

class MainActivity : ComponentActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContent {
            MyFirstComposeApplicationTheme {
                // A surface container using the 'background' color from the theme
                Surface(
                    modifier = Modifier.fillMaxSize(),
                    color = MaterialTheme.colors.background
                ) {
                    Greeting("Android")
                }
            }
        }
    }
}

The MainActivity class is derived from the base class ComponentActivity, which is used for activities that use the Compose UI toolkit. The ComponentActivity class provides a convenient entry point for integrating Compose into an existing Android app.

override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)
    //...
}

In the MainActivity, we have an overridden method; onCreate(). This is the function that is going to call first when the activity is launched. The savedInstanceState: Bundle? parameter in the function is used to pass in a Bundle object that contains the state of the activity. This can be used to restore the activity to its previous state after it has been destroyed and recreated.

setContent {
    MyFirstComposeApplicationTheme {
    // A surface container using the 'background' color from the theme
    //...
    }
}

setContent does what its name suggests; sets the content of an activity. This is a Composable function and it takes a single argument; another Composable function, which represents the root of the UI. In this case MyFirstComposeApplicationTheme.

MyFirstComposeApplicationTheme is also a composable function that sets the theme for the activity. Note the name originated from the application name. If you click on the function name while pressing the Ctrl, you may find that function is written on the Theme.kt file is located in the ui.theme directory.

MyFirstComposeApplicationTheme {
                // A surface container using the 'background' color from the theme
    Surface(
        modifier = Modifier.fillMaxSize(),
        color = MaterialTheme.colors.background
    ) {
        Greeting("Android")
    }
}

The Surface composable function inside the MyFirstComposeApplicationTheme is a visual container that can be used to group other composable together and apply a background color and shape to them.

Note that setContent is the only mandatory function that we need here. We can call Greeting inside the setContent without MyFirstComposeApplicationTheme and Surface. It would look like the following.

class MainActivity : ComponentActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContent {
            Greeting("Android")
        }
    }
}

Then we have a function call to Greeting and have passed Android as an argument. Greeting is a composable function. In the function, we have another function call for Text, which takes an argument text and displays it.

@Composable
fun Greeting(name: String) {
    Text(text = "Hello $name!")
}

Next, you can find another annotation Preview. In Compose, Previews are a feature that allows developers to see a live preview of their composable functions directly in the Android Studio editor. This allows developers to see how their UI will look and behave without having to run the whole app on a device or emulator.


@Preview(showBackground = true)
@Composable
fun DefaultPreview() {
    MyFirstComposeApplicationTheme {
        Greeting("Android")
    }
}

By default, the background of the preview is transparent, to show the background, you can use showBackground = true parameter.

You can have any number of previews you want.

Write your first composable function

With Jetpack Compose, you're now set up to develop apps. You know how to set up an environment for jetpack compose and what's in the Android Studio default. Let's create a composable function and display some text in the preview. If you don't understand the code yet, don't worry. We'll cover it in future lessons.

@Composable
fun MyFirstComposableFunction(){
    Text(
        "Hello World!",
        textAlign = TextAlign.Center
    )
}

In the MyFirstComposableFunction there is a call for a composable function; Text, to show some texts. The textAlign just aligns the text to the center of the screen.

You can call this from the MyPreview function that we have created to preview and see what it looks like.

@Preview(showSystemUi = true)
@Composable
fun MyPreview(){
    MyFirstComposableFunction()
}

Or if you want to run it in your emulator and see, you can put the function inside the Surface as follows.


class MainActivity : ComponentActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContent {
            MyFirstComposeApplicationTheme {
                // A surface container using the 'background' color from the theme
                Surface(
                    modifier = Modifier.fillMaxSize(),
                    color = MaterialTheme.colors.background
                ) {
                    MyFirstComposableFunction()
                }
            }
        }
    }
}

Conclusion

In conclusion, we have learned about setting up the environment for using Jetpack Compose in our Android Development series. Additionally, we explored the MainActivity class and its content, as well as created a Composable function to display some texts.

I would like to express my gratitude for taking the time to read this article. As we move forward, I will dive deeper into defining the UI with Jetpack Compose. Make sure to subscribe to my newsletter to stay up-to-date.

If you found this article informative and interesting, let's connect on LinkedIn, Twitter, and Instagram.

Jetpack Compose; A Modern UI ToolKit

Ethan Rodrigo — Thu, 26 Jan 2023 17:07:18 +0000

The world of programming changes rapidly and gets better day by day. The frameworks and tools that have been used ten years ago won't be able to fulfill the needs of our apps.

Today I'm with a new framework for Android development. In this article let's talk about what is Jetpack Compose and how it helps you to build better android apps. Whether you're a seasoned Android developer or just starting, this article will provide a great introduction to Android Jetpack Compose and its capabilities.

What is Jetpack?

This is what the Android Developers say;

Jetpack is a suite of libraries to help developers follow best practices, reduce boilerplate code, and write code that works consistently across Android versions and devices so that developers can focus on the code they care about.

In a nutshell, Jetpack is a collection of libraries that helps you write code with fewer bugs and in less time than traditional methods of developing Android apps.

Jetpack is made up of components in four categories

Foundation

This provides libraries for core functionality that is needed in most Android apps, such as the appCompat library, which provides backward compatibility for newer features on older Android versions, and the androidx.core library, which contains base classes for common Android components.
Architecture

This component has libraries that help developers to manage the app's architecture, such as navigation, which makes it easy to handle navigation between different screens in the app.
Behavior

Libraries in this component add behaviors to the app, such as notifications, permissions, and sharing. For example, the NotificationCompat library makes it easy to create and manage notifications.
UI

This component includes libraries for building the app's UI. Jetpack Compose belongs to this category.

By using Android Jetpack, developers can focus on building their app's unique features, while the Jetpack libraries take care of the common tasks, such as handling navigation and managing data. This can help to make the development process faster, more efficient, and more maintainable.

What is Jetpack Compose?

As previously stated, Jetpack Compose is a UI library that is part of the Android Jetpack library collection.

This is what Jetpack Compose according to the Android Developers team;

Jetpack Compose is Android’s recommended modern toolkit for building native UI. It simplifies and accelerates UI development on Android. Quickly bring your app to life with less code, powerful tools, and intuitive Kotlin APIs.

Why use Jetpack Compose?

You might be wondering, why to switch from using traditional XML layouts to Android Jetpack Compose. Here are a few reasons why you might consider using Compose for your next project.

Reactive programming

Compose uses a reactive programming model, which updates the UI automatically as the user interacts with the app, making it more dynamic. In contrast, XML is a declarative markup language, which means that the developer needs to explicitly specify the layout of the UI. Further, it does not have built-in support for reactive programming.
Simplicity

Jetpack Compose has a minimalistic and easy-to-read syntax(Compose uses Kotlin), which makes it simpler and faster for developers to create complex UI elements. In contrast, XML can be verbose and can require a lot of boilerplate code to achieve the same results.
Built-in support for animations

One of the features that I particularly appreciate in Jetpack Compose is its built-in support for animations. This makes it easy to add dynamic and engaging transitions to your app. However, adding animations in an XML layout can be more challenging and time-consuming.
Better performance

Jetpack Compose has a better algorithm for layout and rendering, resulting in improved performance compared to XML layouts. It also has built-in support for managing memory and app lifecycle.
Better data management

Jetpack Compose has a built-in mechanism for handling data and state management, making it easy for developers to organize and maintain their code.

Where can I learn Jetpack Compose?

Jetpack Compose is a relatively new toolkit, which means it may have a bit of a learning curve. Yet you can survive as you have the documentation.

Additionally, the Android Developer Team has some helpful tutorials available for learning Jetpack Compose.

Here are two YouTube channels that can help you with Compose. There can be other channels also, but for me, these are the best.

Android Developers - The official YouTube channel of Android Developers
Philipp Lackner
Florian Walther

I am also here to help you!

Conclusion

I hope this article provided a useful introduction to Jetpack Compose and its capabilities as a modern UI toolkit for Android development. As I continue to explore and learn about this technology, I plan to share my findings and insights through a series of articles. Be sure to subscribe to the newsletter of my blog to stay up-to-date with my latest posts and join me on my journey to mastering Jetpack Compose. Thank you for reading!

Additionally, If you find this interesting, let's connect on LinkedIn, Twitter, Instagram, and Hashnode.

Let's Destroy a Linux System

Ethan Rodrigo — Mon, 16 Jan 2023 19:17:20 +0000

Linux is a powerful and flexible operating system, but with great power comes great responsibility. As a Linux user, it's important to understand the potential dangers of certain commands and how to use them safely. In this blog post, we'll take a closer look at some of the most destructive commands in Linux and how to use them safely.

So let's dive in and learn how to protect yourself and your system from the destructive power of Linux commands.

1. `rm`

The rm command is used to delete files and directories(to find out other ways of deleting files in Linux, see this article), but it can be harmful if not used properly. By default, it does not prompt for confirmation before deleting, which means it's possible to accidentally delete important files or entire directories without realizing it.

rm can be even more hazardous when used with the -r and -f flags, which stand for recursive deletion and forceful deletion. These flags are mostly used for the deletion of a directory.

The most dangerous variation of the rm is sudo rm -rf / --no-preserve-root. This command says "Delete everything in the / directory and do not treat it as a special directory."

Preventing accidental file deletion

Use of the -i and -I flags: You can use -i to prompt before every removal and -I for prompt whether to proceed with the entire operation, before removing more than three files or when removing recursively.
Use a Trash: Try out trash-cli and prevent accidental file deletions.
Use a recovery tool: This article talks more about how to recover deleted files in Linux.

2. `:(){:|:&};:`

Looks like some emoticons, right? But these are dangerous. These symbols are called a fork bomb in Linux.

According to the Wikipedia;

In computing, a fork bomb (also called rabbit virus or wabbit) is a denial-of-service attack wherein a process continually replicates itself to deplete available system resources, slowing down or crashing the system due to resource starvation

In a nutshell, the above command creates a chain of processes.

If you're a programmer or student who is learning to program you may be familiar with recursion. This command utilizes the destructive power of recursive functions.

As the above command is a function, here is the breakdown version of it.

bomb(){ # creates a function
    bomb | bomb & # calling recursively and pipe its result to the background job of itself
}; 
bomb # calling of the function

As you can see. this is a function that calls itself (a recursive function). In the function, the function calls itself and the pipe its result to a background job of itself.

How to prevent a fork bomb?

Well, don't run that command. That's it.

And do the following also,

Limit the maximum number of processes per user: The ulimit command can be used to limit the number of processes that can be created by a user and prevent a fork bomb. Give a number to the -u flag and that will be the new limit.
Limit the maximum number of open file descriptors per process: The ulimit command can also be used here. ulimit -n [number], sets a hard limit of [number] file descriptors per process. For example, consider the following. It will restrict a process from opening more than 100 file descriptors at a time. Once a process reaches the limit of 100 file descriptors, the system will not allow it to open any new file descriptors.
Monitor system resources usage: This can be done by using a tool top, ps, htop or bpytop. These tools let you introspect the processes running on a Linux system and the resources they're consuming.

3. dd

The dd command in Linux is used to copy and convert files. Yet, it can also be deadly if not used correctly. The command is a very low-level command that operates on a device level, meaning that it doesn't care about file systems, partitions or any other abstraction layer, it will just write to the device whatever is given to it as input.

For instance, if you use /dev/random,/dev/urandom or /dev/zero as a value for if parameter and provide a hard drive as the value for of, the command will wipe all the data on that hard drive. (It's best to try this out in a virtual machine as a safe experiment.)

This makes it particularly dangerous because if the wrong device is specified, it can cause irreparable damage to the system, including the loss of important data.

Another reason why dd is dangerous is, if you specify only the input and output stream, the command will use up the available space on the system, eventually making the system out of storage.

Safe usage of `dd` command

Always double-check what you're writing: Make sure that the input and output files are correct.
Use of bs and count options: The bs and count options allow you to specify the block size and the number of blocks to be copied. By using smaller block sizes and fewer blocks, you can reduce the risk of overwriting important data. For example, here dd creates a file of 100MB with some random data.
Use status option: By using the status=progress option, you can see what's going on when you run a certain dd command.

4. /dev/null

/dev/null is a special file that discards any data written to it. It's often referred to as the null device or the bit bucket. This is not a physical device, but a virtual file located in the /dev directory. It acts as a placeholder for unwanted input or output.

In this example, I'm finding a file named 2022-12-03. However, as the command looks into the root(/) directory, there are several Permission denied errors.

And sometimes, it's not going to fix even if we use sudo .

To eliminate these errors, we can redirect them to the /dev/null device.

But, What makes /dev/null potentially so harmful? Well, it's the redirection. If you redirect the output of an important command, such as getting an API key, to /dev/null, it would be lost forever.

5. chmod

The chmod command in Linux is used to change the permissions on files and directories.

One way chmod can be destructive is by inadvertently removing the execute permission from a file that is required to run the system can cause issues. A shell script needs permission to be executed, and if that permission is removed from a script used during booting, the system can be halted. Similarly, adding executable permissions to an unknown file can also be malicious.

In the following example, we have a bash script named, hello.sh. It shows how to add and remove the executable permissions of a file.

Another way chmod can be destructive is, by granting too much access to a file or directory. This could allow malicious users or programs to access sensitive information or perform harmful actions.

For example, consider the following screenshot that shows the permissions of a file. It has granted read permission to everyone, and write permission only to the user.

But if we give read, write, and execute permissions to everyone, that would be a major security vulnerability.

Therefore chmod 777 can be considered one of the most destructive commands in Linux.

Preventing data loss with `chmod`

_Be careful with wildcard characters (\_, ?, etc.):* You can use wildcards with chmod command, as it can match multiple files and unintentionally change their permissions. However, let's say you run this command for changing the files in a dir called Dir0.
```
 chmod 755 Dir0/*
```
This looks fine until you realize you have given the same permission for the files and subdirectories as well.
Check the permission before changing: Check the current permissions of the file or directory using the "ls -l" command to avoid changing them in a way that can cause problems.
Be careful with -R flag: This allows you to change the permissions of a file or directory recursively, which means all the files and directories inside it will also have their permissions changed.

A bonus tip for preventing data loss

Well, this is an advice for the system admins, HAVE REGULAR BACKUPS. And also make sure to verify the backups.

Conclusion

Thank you for reading! If you haven't subscribed to the newsletter, DO IT IMMEDIATELY! Then you would find my latest article in your inbox. If you find this interesting, let's connect on LinkedIn, Twitter, Instagram, dev.to and Hashnode.

Now go and execute sudo rm -rf / --no-preserve-root and make tux happy. Until next time...

Let's Play with the Linux Kernel

Ethan Rodrigo — Thu, 08 Dec 2022 17:03:39 +0000

``In a world full of computers, there are hundreds of implementations of the same tool. When it comes to Linux, as it is an open-source project thousand of tools has been made just to do one thing.

Today the topic is the kernel. A mandatory concept of computers. In this article let me introduce what a kernel is and what kind of kernels you can use in Linux. Also, we will play with the kernel.

NOTE: Before you proceed further read my last article about an introduction to kernel and its types.

What is Linux?

For years there's been a debate on this question. However, this is what GNU says.

Linux is the kernel: the program in the system that allocates the machine's resources to the other programs that you run.

Well, then what Linux kernel you're currently using? To find that out the uname command can be used. uname -srm The following is my daily driver.

Alternatives to the Linux kernel

As I mentioned at the beginning of the article, there are numerous implementations of the same concept. In terms of kernel, there are an untold number of kernels. The following four are the officially supported kernels.

1. Stable

As the name suggests this kernel is stable. It is regularly being updated and new patches are released often. Most Linux users use the stable kernel.

You can install it with the package manager; pacman -S linux

Here is the directory tree in case u need it.

2. Zen

This can be introduced as the best kernel for the day-to-day system. It is included features that are not included in the mainline(stable) Linux kernel. Zen kernel also supports the latest hardware. If you use Linux as your daily driver, giving Zen kernel a try is worth it.

pacman -S linux-zen would install this kernel on your system.

Find the source here.

3. Longterm

The Long Term Support or LTS kernel can be considered more stable than the conventional Linux kernel. However, this Kernel is lack of latest features of the Linux kernel.

In the package manager, it's named as linux-lts. You can install it as follows pacman -S linux-lts

The source code for LTS kernel also resides at https://www.kernel.org/.

4. Hardened

A kernel that is focused on security. This has hardening patches and security-oriented configurations for the kernel. Yet, this is very hard to use as it doesn't allow you to run executables.

To install hardened kernel use pacman -S linux-hardened.

Here is the source code for the hardened Linux kernel.

To figure out what kernel you're currently using, try uname -r command. Following is my daily driver.

Install a Kernel

For testing purposes, I use Artix Linux(It's the only lightweight distribution with a fast installation process I could find, besides it's systemd-free). And you can use whatever you want.

Before doing anything you need to decide what kernel you need. Once you figured it out, you can proceed.

The kernel can be installed via the package manager. You can also compile the kernel and install it, but I don't want to talk about that here as it's way too complicated.

Artix also uses pacman, so...

Now you have the kernel installed. However as the grub (or whatever your bootloader is) need to know that you have a new kernel. Because it's the one who spawns the kernel, right? (more about the boot process can be found here).

To inform grub that we have installed a new kernel, we have to regenerate the grub config file. The following command will do it.

sudo grub-mkconfig -o /boot/grub/grub.cfg

See...It's that easy.

Then reboot the system to get the configuration working.

Once booted, in the GRUB go to the Advance options of the welcome menu.

NOTE: I haven't used another boot-manager, therefore can't tell how it works on others. Just google what you want.

There you will find your freshly installed kernel. Press Enter and start using the system with your new kernel.

Delete a Kernel

Well running a system without a kernel is far beyond the imagination. Yet, Linux let you do anything. Therefore I would like to teach you how to delete a kernel.

You get the kernel from the package manager, So delete it from there first.

ex: sudo pacman -Rs linux-lts

Regenerate the grub configuration file

sudo grub-mkconfig -o /boot/grub/grub.cfg

Then reboot

Conclusion

Thank you for reading! If you haven't subscribed to the newsletter, DO IT IMMEDIATELY! Then you would find my latest article in your inbox. If you have any problems, or doubts reach me on Twitter, Instagram, dev.to and Hashnode.

Now go and execute sudo rm -rf / --no-preserve-root and make tux happy. Until next time...

What is a Kernel?

Ethan Rodrigo — Sat, 19 Nov 2022 16:38:05 +0000

In the world full of computers, there are hundreds of implementations of the same tool. When it comes to Linux, as it is an open source project thousand of tools has been made just to do one thing.

Today, the topic is kernel. A mandatory concept while talking about computers. In this article let me introduce what really a kernel is and why it is important. Additionally few different types of kernels will be introduced.

So, why are you waiting? Let's goooooooooo...

What is a kernel?

Well, first of all, what the heck is a kernel?

According to the wikipedia;

The kernel is a computer program at the core of a computer's operating system and generally has complete control over everything in the system.

In a nutshell the kernel is a component of an operating system, but the one in control.

In the startup the kernel is spawned by the bootloader (to know how the Linux booting process works read this article) into the memory and it lives until you shutdown the system.

The kernel is loaded into a separate area of memory, called kernel space, protected from application software. It prevents the conflicts with the user space in the memory, which is used by the application software. But things get difference with different kernels. We will look into that later in the article.

Difference with an OS

Some gets confused with the OS and the kernel (mainly with Linux). In simple language the kernel interacts with hardware and makes an interface for the OS and other applications, OS on the other hand provides an interface between the user and the hardware.

What does kernel do?

Kernel is an important component of an OS. But, what does it really do? why is it so important?

Memory Management

Random Access Memory or RAM is one of the most vital devices in a computer. It’s where the instructions and data of the OS and other application are stored for immediate use. Programs load into memory, so does the kernel.

The RAM is a vital, yet limited device. Therefore it has to be managed. As programs store the data and instructions in the RAM there can be multiple requests for memory. The kernel is what takes these requests and respond.

Device Management

A computer uses devices to make it functional and more efficient. After all what's the usage of a computer without devices. Video cards, sound cards, printers, etc. can be few examples for devices we use day to day life.

Yet, these devices can’t be used if you don’t have device drivers, specifying each device. Device driver is a computer program that operates a particular devices that is attached to a computer. These drivers are managed by the kernel.

Devices drivers usually ships with the kernel. If you want, you can add or remove a specific device driver. Again it differs with the kernel type.

Input/Output devices

As other devices, IO(Input and Output) devices also has drivers in order to run successfully. While using the computer, different applications request to access IO devices. The kernel allocates these requests and provides a convenient method to use the device.

These are few functions of a kernel. There are more to this such as system calls, resource management, etc.

Types of kernel

As mentioned on the start of this article, there can be hundreds of implementations for the same tool. Therefore we have a number of kernels today. Each of them has their own pros and cons. Let's now see what are some major types of kernels.

1. Monolithic kernels

In a Monolithic kernel all OS services run along with the main kernel thread and resides in the same memory area. This provides a rich and powerful hardware access.

A Monolithic kernel has all the operating system core functions and the device drivers. In a nutshell it’s just one single program that contains all the code necessary to perform every kernel-related task. However one bug can crash the whole system.

The kernel and its components (device drivers, file systems, etc.) are resides in the kernel space. Applications use system calls to communicate with them.

2. Microkernels

Microkernels works with set of servers that communicate through a minimal kernel. This leaves a small amount in the kernel space and the rest lies in the user space.

The kernel, which runs in the kernel space, has a minimal OS services such as memory management, inter-process communication, etc. Other services such as networking, file systems are implemented in user space programs, called servers.

Servers allow the OS to be modified by simply starting and stopping programs. For example, a machine without a networking support is not started.

3. Hybrid(modular) kernels

It’s just the Microkernel but as an extension that has some Monolothic features. This is made of major advantages from both kernels.

Hybrid kernel runs some of the services (network stack, filesystem, etc) in the kernel space.

There are few other design approaches of kernel(Nanokernels, Exokernels, Multikernels). However it would be tedious to talk here.

Conclusion

In this article we discussed about what is a kernel and what does it do. Additionally few kernel design approaches are also introduced here.

In the next article let's play with one of the mostly used kernels, Linux. There I will walk you through how to add and remove kernels into your Linux system.

If you haven't subscribed to the newsletter, DO IT IMMEDIATELY! Then you would find my latest article on your inbox. Thank you for reading! Now go and execute sudo rm -rf / --no-preserve-root and make tux happy. Until next time...

If you find this useful let's connect on Twitter, Instagram, dev.to and Hashnode.

Recover Deleted Files on Linux

Ethan Rodrigo — Wed, 28 Sep 2022 12:16:43 +0000

We all are humans and we do make mistakes. Some can be recovered while some can't be. Same way accidentally deleting files can be happened even with an experienced Linux user. Therefore it would be great if the deleted files can be recovered.

NOTE: The following concept and tactics DO NOT apply if you've used shred(or any other data overwriting tool), as it destroys the content of the file rather than deleting.

Fortunately when a file is deleted it does not get removed right away. It just removes the link between the filename and the file, which makes it inaccessible to read (or write or do whatever by a user). Still, the file content is hidden in the file system. Therefore with the right tool you can access them and recover.

However the content will get removed from the file system as you create new files in the system, which makes it harder to find the file. Yet, if you have a crashed file system or a hard drive these tools can be used to recover the files.

Trash

Most of the Linux distributions provide a trash where you can put your unwanted files and recover them whenever you want.

And for cli users there's trash-cli which implements the GUI trashcan.

TestDisk

However trashcan doesn't work if you've use rm, unlink, etc. That's where TestDisk comes into play.

Here's what TestDisk is according to the Wikipedia;

TestDisk is a free and open-source data recovery utility that helps users recover lost partitions or repair corrupted filesystems. TestDisk can collect detailed information about a corrupted drive...

Yes, it can also be used to recover corrupted partitions, but let's focus on recovering deleted files as for this article.

First install TestDisk with a package manager. sudo pacman -S testdisk

Or get binaries from here. Note that TestDisk can be used on Windows and Mac OS also.

Recovery

I love cat memes, what about you? I have download some cat memes in a directory. Let me show you some.

OOPS! I got deleted everything. What now?

Fortunately I have TestDisk installed on my system. Therefore I'm going to fire it up.

Type testdisk on your terminal and press enter. You will see as below. Create a log file or not, then proceed further.

You may prompt for a password and for that let your cat walk on the keyboard. If TestDisk let you in select a device, if not type the password.

Now, the partition table. Wait... you don't know your partition table? Just use what is selected, TestDisk is smart enough to find it out.

Navigate to Advanced.

Select your partition type. It would be Linux most of the time. And then select List in the list below.

Here you have your directory tree. From there navigate to the dir where the files are deleted. I'm going to search my catMemes dir which is in the Pictures dir.

Woah! It's here. Can you see it? This is the whole list I've got deleted. Now as with the instruction below I select everything with a and copy them to a dir.

The following will be prompted. From here select what dir you want to put the recovered files.

And press C when the destination is reached and you will have your files copied into that dir.

PhotoRec

Then we have the cousin of TestDisk; PhotoRec. PhotoRec comes with testdisk package, which means if you have the TestDisk utility you should have photorec also.

The only difference between above two is, PhotoRec recovers more files. However the files are not in the original name and lot of unwanted files can be recovered while the process. Therefore if you want to look into a specific file I highly recommend using TestDisk. Still if you have a corrupted or bad hard drive that doesn't let you access files, you can use PhotoRec and recover most of the files.

Go ahead and fire up the command photorec and you'll get something as follows. As it suggests, select a disk to recover and then Proceed.

Now select a partition. If you want to recover the whole disk, select No partition, the first one.

Before selecting a partition, if you go to the File Opt you will see something like this. Here you can select what file type you want to recover. You can tick a file type with the left arrow and untick a one with the right arrow. To exit press Quit.

Then select the filesystem type of the partion.

Now you have the directory tree here. Select the directory your files have been stored. Once selected press C to continue.

The recovering process has started and you will be able to see the estimated time. If you have a big amount of files then you may have to wait for hours.

Once completed you'll get the following result. You can access your deleted files within the directory given.

PhotoRec recovers all, whether it is wanted or unwanted. After the recovery you can see a lot of unwelcoming files as below.

Conclusion

Thank you for reading! Now go and execute sudo rm -rf / --no-preserve-root and make tux happy. Until next time.

If you find this useful let's connect on Twitter, Instagram, dev.to and Hashnode.

Setup LVM for a Linux Installation

Ethan Rodrigo — Wed, 14 Sep 2022 08:37:15 +0000

In Linux there are hundred of ways to get the same work done, and the overwhelming amount of distros makes the number thousand. However it's life saving if we choose the efficient way.

The process of partitioning a hard drive in Linux is same as other stuff; we have got numerous approaches. Yet, there are more effective methods. The best I've met so far is LVM. LVM allows you to combine partitions, hard drives and use as one single partition (for more about LVM read the LVM introduction article).

In this article I'll walk you through the setup of LVM for a Linux installation. Before procedure you need to have followings;

A basic understanding of LVM (get it here)
A machine to install Linux on
A bootable USB stick that has a Linux iso

To get started, boot your Linux installation...

Partitioning

Though LVM is an alternative to conventional partition scheme, still we need to partition the hard drive we're going to install the OS on. The reason behind this is, Grub and some other bootloaders doesn't support on LVM partitions. However other secondary partitions can be added without partitioning into a created Volume Group of LVM.

Enough talking, let's get into the terminal.

But wait, you need to find what kind of firmware you have got. Yes, go and find whether it's BIOS or UEFI.

What, you don't know how to? Oh! My bad, here's a command. Run it and find out. ls /sys/firmware | grep efi Only the UEFI system has got efi directory inside /sys/firmware. Therefore if there's no output as follows, yours is BIOS.

Ok now, choose a hard drive. List all the hard drives on the system with fdisk --list and select the drive you want.

Or try lsblk.

UEFI partitioning

UEFI vs EFI

You may have heard UEFI as well as EFI. Is it entangling? let me break it down for you. UEFI is the replacement of BIOS, which is a firmware. EFI is the storage partition of UEFI, which contains the bootloader.

Let's get into partitioning with UEFI now.

Launch fdisk with your selected hard drive.

In fdisk run p command and see if there's any partitions. If you don't have any partitions it would look as follows.

Or else it would list the partitions withing the hard drives

If you see any partitions there delete them with d command.

However in case you have stuff in a partitions that you don't want to delete, leave it. Yet, don't create partitions to store different file systems as you do in Windows. We are doing it with LVM here.

It's time to create the EFI partition. Follow the next steps in order to achieve it.

Run command n to create a new partition. For the Partition type leave it as default (primary). Partition number leave the default number or choose a number within the given range as you wish. First sector, leave default value. Last sector is the actual size of the partition, therefore 500MB would be enough for the EFI partion.

The partition is labeled as Linux. But to make it support UEFI change the type into EFI with command t and for the Hex code, provide ef.

For the space left, create a normal partition. The type of the partition should be Linux LVM, whose Hex code is 8e.

Lastly write the changes to the disk and save with the w command.

BIOS partitioning

Partitioning for BIOS is really simple and we don't have to create multiple partitions. All we have to do is create a new partition and change its type to Linux LVM.

Following example illustrates how the BIOS partition is created.

Setting up LVM

Now, it's time to set up LVM on the system. If you remember from my LVM Introduction article, we have three main components in LVM.

Physical Volume (PV)
Volume Group (VG)
Logical Volume (LV)

First Physical Volumes are created with the existing hard drive or partitions and then a Volume Group needs to be created and all the Physical Volumes can be added there. Then the VG can be sliced into Logical Volumes and one of them can be used to install the Linux system.

Let's get into the terminal. Though I use Arch in following examples, it's general to all the other distributions.

As for the start, create a PV from the partitioned hard drive and put it into a VG. At this moment if you have other hard drives do the same with them as well.

Now the creation of the LVs comes in. Here you need to decide how many LVs you're going to use and what they are for. You can also leave some space for later usage, means without allocating them into any LV. After all, you just need a /root partition for Linux to work fine. Still it's a better approach if you create a /home partition for users.

NOTE: In the below example I give only 2Gigs to the root as I'm on a Virtual Machine, but it's better to use 10Gigs.

Then we need to load a kernel module for device mapping, dm_mod. For that execute modprobe dm_mod command. After that the VG can be activated with vgchange -ay command.

That's it! The LVM partitions are now ready to hold a Linux installation.

Post setup

Although the LVM setup is now done, there's some stuff left to do. These are general things we should do even we use conventional partition scheme.

All LVs need to have a file system (or else how can they open and read files?). Choose a file system as you wish and launch mkfs command with it. The basic syntax of mkfs is as follows. mkfs.$filesytem /dev/$vgName/$lvName

$filesystem - the file system you have chosen
$vgName - name of the VG
$lvName - name of the LV

I like to go with the popular ext4 file system.

Lastly, the LVs can be mounted into the system. The root partition has to mounted into the /mnt and the home partition needs to be mounted into the /mnt/home dir.

Linux Installation

The LVM and the partitions are ready to go for the installation. Here I won't show you the installation as it's just the classical way of the installations. But remember to install the lvm package as it doesn't come by default.

If you install the system manually, like in Arch, Gentoo you can now chroot into the /mnt and install the system there. And if you use GUI installation, select the root and home(if have created a one) Logical Volumes in the installation process.

NOTE: Also remember to add lvm into the mkinitcpio and other initramfs. Yet, this is not required in GUI installations.

A script to automate the process

As a programmer it's our responsibility to spend 10 hours on automating 5 minute task.

I did the same with above process. You can find it here.

You can use it on both BIOS and EFI systems. Multiple hard drives can also be setup at once and there are more options. Try using -h flag or analyze the source code. Pull requests are mostly welcomed.

Conclusion

Thank you for reading! Now go and execute sudo rm -rf / --no-preserve-root and make tux happy. Until next time.

If you find this useful let's connect on Twitter, Instagram, dev.to and Hashnode.

Remove files in Linux

Ethan Rodrigo — Tue, 02 Aug 2022 08:40:13 +0000

Hello folks! An alien here with another Linux blog.

Today let's talk about deleting files. No seriously, It may seem little bit trivial, yet I'm going to dive deep. Will you join with me?

HEADS-UP: Following commands are for deleting files from the terminal, means it's not going to store in the trash. Therefore execute them on some unwanted files or just create a Virtual Machine.

rm

Our first command is the great old classic rm. But did you know there's more to the rm command.

The rm command doesn't let you delete directories (let me call it dirs now on) without specifying. The -d flag specifies delete empty dirs. Similarly you can use rmdir command to delete an empty dir(and it's the only use of rmdir). However to delete a dir that has content needs a specific flag which is -r or -R.

Not sure about what's being deleted? This can be occur while deleting dirs. You can use -i flag to prompt before deleting each and every file.

Sometimes while deleting, specially a file that's belong to the root, can prompt you for a confirmation as you're using -i flag. It's bothering, isn't it? Just use -f and it will delete the dir(s)/file(s) without asking. Still remember to use -r if it's a dir.

If you're familiar with my articles you may have seen my conclusion encourages you to run sudo rm -rf /* --no-preserve-root. It's a really good command that teaches you a lot of things, not only about Linux but also about life, suchlike don't trust bloggers on the internet.

Well... what does --no-preserve-root means? It tells the rm command no to treat the / (root) dir as a special one and let the rm deletes what it contains. But if you do not use --no-preserve-root it's not going to delete the / as it calls --preserve-root which is the converse by default.

Yet, there's something called globbing in Linux, which prevents --preserve-root in rm. Frankly you don't have to use --no-preserve-root if you use globbing.

rm -rf /* means delete all the contents in the / dir, besides it uses globbing (the * character). Therefore it doesn't use --preserve-root, while rm -rf / means delete the /, which uses --preserve-root. Because of that if you do sudo rm -rf /* you don't have to use --no-preserve-root.

unlink

Speaking of deleting files in Linux we've also got unlink. unlink is not only a command but also a function in C (I know that PHP has that also, yet we don't do that here).

According to the GNU man page

You can delete a file with unlink...Deletion actually deletes a file name. If this is the file’s only name, then the file is deleted as well. If the file has other remaining names, it remains accessible under those names.

Let me explain it. Consider the meaning of the word unlink. Yes, it means removing a link from something that has a link. In Linux you can have multiple names for a single file. The creation of multiple names for a single file is called linking (Here I'm referring to Hard Links). This doesn't copy the file, it simply makes a new name (an alias) to the file.

In the following example I create a file in a dir named Test0 and create a link to that file from another dir called Test1. The file here in the dir Test1 is not really a file, but a link to the file hello in the dir Test0

When you do unlink (or even rm) on a file name which is a link to another file locates elsewhere will be removed, not the file. However if the file doesn't have a link, the file will be deleted, which is equal to the rm without frills.

find

Before seeing deleting files with our next command, find, let's have a look at what it really is.

find command

Here's what find is according to the Wikipedia;

In Unix-like and some other operating systems, find is a command-line utility that locates files based on some user-specified criteria and either prints the pathname of each matched object or, if another action is requested, performs that action on each matched object.

As the name suggest the command find is used to find a file/dir. Additionally we can perform actions (another commands) on those found file/dirs.

Syntax;

find [the path you want to look to into] -options [what to find]

The path - where the find command needs to exactly look into (use . to refer the current working directory)
What to find - the file/dir name or a regex pattern to get multiple results

Let me give you a brief idea about the find command with some examples.

Let's create 100 files that has some random values and name with another random value.

Now if I want to filter the files that has 10 in the name, I can do something as follow

find . -name "*10*"

And this time I want only the directories to be filtered,

find . -type d -name "*10*"

More about the find command can be found here.

delete with find

As this article is not about finding files, but deleting them, I'm stopping here with what else find can do. Instead let's see how the find command can be used to delete files.

NOTE: It's fine if you're seeking for one file and delete them. But be careful if you're using find with a regex pattern, as it's going to delete all the filtered files.

There are three ways to delete with the find command.

1. with -delete

Below I delete files and dirs that has 13 in the filename.

And order matters. I use -delete before the file name in the following example, which literally means find files in the current working dir and delete them, then filter the files that has 23 in the name.

2. with -exec The -exec flag of the find command is used to execute an action (a command) on the results. Therefore it's followed by another command, which in this case rm.

The syntax for rm with find's -exec flag as follows;

find [the path] -options [what to find] -exec rm {} \;

{} - refers to the current file that is being processed
\ - escape character for ;, it also can be quoted with double quotations (";")
; - to indicate the end of the command

3. with xargs As for the Wikipedia;

xargs (short for "eXtended ARGumentS" [1]) is a command on Unix and most Unix-like operating systems used to build and execute commands from standard input. It converts input from standard input into arguments to a command.

The command xargs just takes output from a command and pass it into another as arguments.

For the deletion with find, we can pipe it with xargs as following syntax;

find [the path] -options [what to find] | xargs -I {} rm "{}"

-I - replace {} with the find's results.
rm "{}" - execute rm on filtered files.

shred

Now I give you a bomb. A bomb that can blast your Linux system or make your day even worse than what it is so far.

This one is for destroying the file, or a hard drive. The shred command can be used when you need to destroy a file before deleting, so nobody can find what was there.

Try shredding a file and you will get something as following.

What the shred command really does is, it overwrites the content of a file. To make it even harder to recover with expensive recovering software, shred overwrites the file multiple times.

Not only files but hard drives also can be overwritten by the shred command. To overwrite a hard drive, just pass the path of the hard drive to the shred command. For example, execute following in order to overwrite the sda1 partition.

shred /dev/sda1

According to the man page, shred overwrites the file three times. To change that you can use -n flag to repeat the overwriting process for n times. See how shred without frills works and with the -n flag. (The flag -v stands for verbose)

In addition to the above there, are few more flags, such as -f to change the permissions to allow writing if necessary. Here is the man page.

Conclusion

In this article we discussed about removing files and dirs from a Linux system. However what if you deleted something accidentally? What if you want something that you didn't want and deleted?. The answer will be in a next article. If you haven't subscribed to the newsletter, DO IT IMMEDIATELY! Then you would find my latest article on your inbox.

Thank you for reading! Now go and execute sudo rm -rf / --no-preserve-root and make tux happy. Until next time.

If you find this useful let's connect on Twitter, Instagram, dev.to and Hashnode.

An introduction to LVM

Ethan Rodrigo — Wed, 27 Jul 2022 05:53:50 +0000

Hello guys! Your favorite Linux blogger here(Cough... Cough...)

Today I'm here with another interesting Linux tool. This one would help you with partitioning block devices in Linux. The tool is called LVM and this article would be an introduction for that. Without further delay, let's start.

Logical Volume Management

What is Logical Volume Management?

According to the Wikipedia;

In computer storage, logical volume management or LVM provides a method of allocating space on mass-storage devices that is more flexible than conventional partitioning schemes to store volumes. In particular, a volume manager can concatenate, stripe together or otherwise combine partitions (or block devices in general) into larger virtual partitions that administrators can re-size or move, potentially without interrupting system use

Confused? Don't worry, I've got your back. LVM does nothing but combine all the block devices into one or more virtual partition. Simple af, isn't it?

But how it really works? Let me explain it.

Working of LVM

Before dig into the functioning of the LVM, let's get familiar with few LVM terms.

Physical Volume

This is the fundamental part of LVM. A Physical Volume is nothing more than a block device. It is composed with a sequence of small chunks called Physical Extents(PE). PEs are the smallest storage blocks on a Physical Volume.

Volume Group

A Volume Group is a container that stores the Physical Volumes.

Logical Volume

A Volume Group can be sliced up into Logical Volumes. This is almost equal to normal partition, except that it sits upon the Volume Group instead of the raw disk. A Logical Volume is formed with small chunks called Logical Extents (LE).

Now, let's get into the working of LVM

First the typical hard disks should be converted into Physical Volumes. Then each PEs in those PVs map to LEs, and these LEs are pooled into a Volume Group. At the end the pooled LEs are concatenated into virtual disks called Logical Volumes.

A typical partitioning scheme slice a raw disks into partitions and then the filesystem can be installed on that device. Apart from that, LVM creates a Physical Volume from the disk and put it into a Volume Group. Then the Logical Volumes, which are almost similar to the conventional partitions are pooled into the Volume Groups.

Major advantage of using LVM over conventional partitioning scheme is that Logical Volumes can be resized, but partitions can not. Additionally a new hard disk can be added to the Volume group and a slice from that hard disk can be added to a existing Logical Volume.

Logical Volume Manager

Here's the Wikipedia's idea;

Logical Volume Manager is a device mapper that uses Logical Volume Management for the Linux kernel.

Simply Logical Volume Manager is the tool that uses Logical Volume Management concept for block device partitioning in Linux.

Managing block devices with LVM

Physical Volume

Creation

pvcreate is used for creation of physical volumes. It initializes a Physical Volume with a block device and writes a LVM disk label.

To create a physical volume just add the hard drive (or any other block device) after the pvcreate command.

The pvs command can be used to see how many pvs are there and how much free storage it has.

Shrinking

Physical Volumes can be shrink into a desired size. pvresize command is can be used for this.

To resize the Physical Volume use pvresize --setphysicalvolumesize [size] [PV], where size can be 1G, 100G or any size and PV is the Physical Volume to be resized.

Volume Groups

Creation

In order to create a Volume Group, vgcreate command can be used.

The vgcreate command takes a name for the Volume Group and the Physical Groups that has to be added into the Volume Group as mandatory arguments.

The block devices can be also assign to a Volume Group without making it a Physical Volume first. When this happens vgcreate automatically creates a Physical Volume with that device/s.

The vgs command can be used to display information about volume groups.

Renaming of Volume Groups

Once a Volume Group is created, the name can be changed. The command is vgrename and it takes /dev/oldName and /dev/newName.

Addition and removal of Physical Volumes from a Volume Group

To add a new Physical Volume the vgextend command can be used.

It's the usual way as the vgcreate command does. Just supply the Volume Group name and the new Physical Volume.

Or else a block device can be directly added to a Volume Group as vgcreate does.

The command vgreduce can be used to remove a Physical Volume from the Volume Group.

Note that vgreduce is not going to delete any Physical Volumes. It just remove it from the Volume Group.

Caution: Move data on the Physical Volume to another before removing a Physical Volume.

Use the command pvmove to move data from a one Physical Volume to another. pvmove takes the Physical Volume that you need to be emptied.

Additionally you can provide another Physical Volume where the data would be copied. This means you can also use pvmove to remove a hard disk and add a new one, if you're using LVM.

Logical Volumes

Creation

Creation of Logical Volumes are quiet simple. You guessed it right, the command lvcreate is the one that is used to create a Logical Volume.

The command lvcreate takes the size in two different ways.

-L - When the -L is used the size needs to be given with the Megabytes, Gigabytes, etc. Default is Megabytes.

-l - The l is quiet complicated and cool at the same time. Instead of using Bytes to give the size, l needs the size in extents.

The common way to provide this is using % sign. look into this man page for more info on l option with %.

Here is an example of using the 100% of the free storage in the Volume Group.

Note that lvs command is used to get the info on Logical Volumes.

Renaming

For renaming the lvrename command can be used. It can used in two ways.

First you can give the old and new names with /dev/[vgname] prefix. Or else just provide the Volume Group name followed by the old and new names.

Resizing

This is the best part of LVM and this is why you should use it. Yet, resizing of a Logical Volume is risky. If you do it in a wrong manner all the data can be corrupted and sometimes you won't be able to get them back.

Here's a real experience. It's 22 May as I remember and I finished this article until the preceding paragraph. And I was getting out of storage on root. I had 5Gigs left on the Volume Group. Therefore I did lvextend and BOOM the file system hadn't resized. Then I ran some commands and got a some kind of a kernel panic, and I realized some files had been deleted and I could not even run pacman to re-install the kernel. Then what? I re-install the whole OS.

NOTE: Once the LV is resized without resizing the file system can be restored sometimes. It's the commands that you run for resizing file system that makes the system corrupted.

WOAH! wait... Don't afraid that much. Let me help you with this. Ironic, huh? Anyway let me show how to do this properly.

The problem here is whenever a Logical Volume is resized, not only the LV is resized, but the filesystem would also be changed. Therefore the filesystem on it also should be resized according to the size of the LV in order to make it work.

There's two attempts you can try;

The resize2fs command

This command is used to resize the file system. It takes a unmounted device as a compulsory argument.

If you need to reduce the LV size run the resize2fs command with the -M flag, which means Magic. Just kidding! The -M flag shrinks the filesystem to the minimum size.

Then you can use lvresize for shrinking the LV. After that run resize2fs without frills.

If it gives you an error, run e2fsck -fy /dev/[VG_Name]/[LV_Name], which is going to fix the file system.

And If you want to extend the size, you just have to run lvresize followed by resize2fs.

resizefs in one go

The lvresize command also have an argument, resizefs. However it can be used only with ext2, ext3, ext4, ReiserFS and XFS file systems. Anything other than them should use the attempt 1.

`lvextend` and `lvreduce`

The lvextend and lvreduce commands are also used for resizing of Logical Volumes. The -l and -L are also used here and you can serve extents or size respectively.

CAUTION: Here also remember to use resize2fs or else you would loose the file system.

Remove LVs

Be cautious with the removal of LVs also. First make sure you have backed up all your data. Then cheAs this article is not about finding files, but deleting them, let's see the find command can be used to delete files. ck whether the LV you want to remove is unmounted. If it's not unmount it first.

The removal process is done with the command lvremove, which takes VG_Name/LV_Name as mandatory arguments.

Conclusion

In this article we have gone through the basics of LVM. We have discussed how LVM works and creation, deletion and other action done with LVs, PVs and VGs.

In the next article let's see how to partition devices with LVM for installation of Linux. If you haven't subscribed to the newsletter, DO IT IMMEDIATELY! Then you would find my latest article on your inbox.

Thank you for reading! 😊😊 Now go and execute sudo rm -rf /* --no-preserve-root and make tux happy 🐧. Until next time 👋👋👋.

If you find this useful let's connect on Twitter, Instagram, dev.to and Hashnode.

Analyze Linux Logs

Ethan Rodrigo — Mon, 25 Apr 2022 12:21:49 +0000

Hello everyone! I'm back after an eternity.

Then, let's dive into the topic.

Errors in Linux can be so hard to find sometime, as you can't see lot of issues with GUI. However Linux has few tools for gathering logs, and with that logs you can find what is going on.

Today I'm here with two tools that is used by Linux for logging.

dmesg
journalctl (Yeah... You are right, there is a connection with systemd.)

But before we start, please refer to my last posts about systemd, If you want to know what systemd is and how it works. And get systemd installed on your system to use journalctl. You can find them here.

journalctl

journalctl is used to query the content of systemd journals.

journalctl? or journald?

journald is the daemon spawned by systemd to collects logs from various log sources. It is also responsible for creation and maintenance of journals that are received from various resources.

journalctl is the cli-tool that lets you interact with journals created by journald. With journalctl you can read, monitor and filter logs in real time.

Setting up the System Time

Let me interrupt for a second. The logs are recorded in UTC or local time, and because of that you need to setup your timezone correctly.

systemd suite comes with a handy tool called timedatectl that can help with this.

Follow the steps below to set it up.

timedatectl list-timezones -- This will list the timezones available on your system. Choose a one.
sudo timedatectl set-timezone [zone] -- Once you find your timezone, set it up with this command, where zone is the timezone you have chosen.
timedatectl status -- Execute this (or timedatectl alone) to ensure that your machine is using the correct time.

journalctl without frills

journalctlcommnad would give the journal logs in chronological order, i.e. they are arranged according to the time. It uses less underneath which means you can use the same keys to move around the logs as you do with less.

Yet, if you don’t want to see the logs in a less-like order, you can use --no-pager flag.Which display the all logs in the terminal. Trust me though, that would be a mess. Therefore use it with grep.

The raw journalctl command gives you the old logs first. However if you want the recent logs to be displayed first you can use --reverse or -r flags to view the logs in reverse order.

filtering the journal logs?

Use -n flag to define the number of the logs you want.

according to a time stamp

The -f flag would give you the logs in real time, that is it shows the logs that are currently being written.

If you want logs from a specific time period you can use -S and -U flags, which indicates time since and until.

ex: journalctl -S 2022-01-22 -U 2022-01-25 gives you the logs since 22nd of January until 25th.

The relative time is also allowed in journalctl. The h, d, w and m flags specify hours, days, weeks and months respectively.

You can also define the time with words using yesterday, today and tomorrow.

with data fields

In order to filter the output with data fields use THEFILED=field-you-want

You’ll find list of data fields here.

If you want to list out all the values that have been used for a data field you can use -F flag.

change the output format

Because the journal is a binary file, the data need to be translated into plain text. Thus, we can present the output in different output format.

The -o flag is used define the output format.

Following are some output formats you can use,

short
verbose
export
json
cat
with-unit

PS: you can find a list of formats in the man page also

boot message

If you have got an error relating to the booting process and want to check for that, journalctl have you hold.

The -b flag lists out entries related to each boot. -b without extra filters list the entries of the last boot.

In order to get logs of a specific boot, the boot ID should be prescribed. It can be retrieved using --list-boots flag.
The boot ID is the second one as depicted below.

Now just add the identifier, which is the longs number from list-boots after the -b to get the logs of a specific boot.

kernel message

The -k flag would give you the kernel messages. This is same as using dmesg.

manage storage

The storage that is used by journalctl can be managed. The --disk-usage command gives you how much storage that’s been used by journals.

You can remove journals that you don’t want with --vacuum flag. --vacuum has three types.

--vacuum-size - deletes until the size provided. This flag is like saying "reduce the journals to this size".

In the below image the size of the journals is almost 800Mb and here journalctl says reduce it to 700Mb.

--vacuum-time = deletes logs that are older than the time provided. The time argument can be same as options for -S and -U flags.
--vaccum-files = deletes journal files until the provided number.

show error, warning logs

The raw journalctl command gives all the logs, means everything with errors, warnings, etc. Yet, you can filter them.

The -p flag in order to list out errors and warnings. You can pass following values for the -p flag.

name

emerg

alert

crit

err

warning

notice

info

debug

You can use either the number or the name as follow.

Since the log data is stored in a binary format and the data can be displayed in arbitrary output formats, the -o flag can be combined.

dmesg

According to the man page,

dmesg - print or control the kernel ring buffer

Now what is ring buffer?

In a nutshell a ring buffer is a message store, which has messages from the kernel.

The booting processes (BIOS/UEFI and GRUB) start and load the kernel into the memory and then the kernel starts the systemd(or any other init system). Then the startup processes takes the control and initializes the system. In the early stage, logging daemons are not yet started. To overcome the losing notable errors and warnings, kernel uses the ring buffer to store those messages. And dmesg is used to read those messages.

A ring buffer can be thinks as a Queue data structure as it follows the FIFO rule. Since the size of the buffer is fixed, the older messages are deleted.

`dmesg` command

The dmesg command needs to be run with the sudo privileges as we are communicating with the kernel.

dmesg with no flags gives you a long list of messages, and you can pipe it with less in order to make it smaller.

Human readable

By default the dmesg uses seconds and nanoseconds as the time, since the kernel is started. You can make it a human readable output with -H flag. Note that the -H calls the less by default.

If the seconds are bugging you, you can make it into a full day format with -T flag.

Live Events

You can use --follow or -w to get the kernel messages in real time. And if you only want the new messages you can use -W flag. You can plug a USB, add a kernel module, etc. to see the new kernel messages.

Filtering the output

There are lot of ways to filter the output and following are few.

Get specific number of logs

For this there's no built in flag, yet you can use the commands tail and head.

Search for a specific term

The grep can be used for this.

Log levels

Every message logged to the ring buffer has a level, which represents the importance of the information in the message. Those levels can be used to filter the output.

The -l and --level flag is used to provide the level. You can use one or more.

Following are the supported levels;

emerg - system is unusable
alert - action must be taken immediately
crit - critical conditions
err - error conditions
warn - warning conditions
notice - normal but significant condition
info - informational
debug - debug-level messages

Log facilities

The logs are grouped into some categories and you can use --follow or -f to list logs according to the category.

Following are the supported facilities;

kern - kernel messages
user - random user-level messages
mail - mail system
daemon - system daemons
auth - security/authorization messages
syslog - messages generated internally by syslogd
lpr - line printer subsystem
news - network news subsystem

Display the log facility and level

The flag -x list out the corresponding log level and facility before the message.

Conclusion

Thank you for reading! 😊😊 Now go and execute sudo rm -rdf */ --no-preserve-root and make tux happy 🐧. Until next time 👋👋👋.

If you find this useful let's connect on Twitter, Instagram, dev.to and Hashnode.

DEV Community: Ethan Rodrigo

Teleporting Virtual Machines: Flipping the Script

The Pre-Copy Bottleneck

The Post-Copy Flip

The Four Pillars of Post-Copy

Predicting the Future with Bubbles

Bubbling with a Single Pivot

Bubbling with Multiple Pivots

What to Send First?

Setting the Trap: Catching Page Faults

Shadow Paging

Page Tracking

Pseudo-paging

Handle the Free Memory

Dynamic Self-Ballooning

How to detect memory pressure?

The Reality Check: Did it Actually Work?

The Triumphs (Where Post-Copy Wins):

The Catch (Where it Stumbled):

The Final Verdict: Which should you choose?

Conclusion

Teleporting Virtual Machines: The Magic of Live Virtual Machine Migration

The Zero-Downtime Dilemma

How Do We Live Migrate VMs?

The Winning Approach

The 5 Phases of Live Migration

The Bottleneck: WWS

Writable Working Set

The "Moving"

Knowing When to Freeze: The Stop Triggers

Under the Hood: Two Ways to Build It

Managed Migration

Setting the Trap: Shadow Page Tables

Self Migration

The Cheat Code: Paravirtualization

Stunning Rogue Process

Freeing Page Cache Pages

The Invisible Backbone of the Modern Cloud

Jetpack Compose 101; Setup the Environment

Setup the Environment

A quick glimpse at Android Studio

Design View

Emulator

Run on your phone

Dive into the code

Composable functions

MainActivity.kt

Write your first composable function

Conclusion

Jetpack Compose; A Modern UI ToolKit

What is Jetpack?

Jetpack is made up of components in four categories

What is Jetpack Compose?

Why use Jetpack Compose?

Where can I learn Jetpack Compose?

Conclusion

Let's Destroy a Linux System

1. rm

Preventing accidental file deletion

2. :(){:|:&};:

How to prevent a fork bomb?

3. dd

Safe usage of dd command

4. /dev/null

5. chmod

Preventing data loss with chmod

A bonus tip for preventing data loss

Conclusion

Let's Play with the Linux Kernel

What is Linux?

Alternatives to the Linux kernel

1. Stable

2. Zen

3. Longterm

4. Hardened

Install a Kernel

Delete a Kernel

Conclusion

What is a Kernel?

What is a kernel?

1. `rm`

2. `:(){:|:&};:`

Safe usage of `dd` command

Preventing data loss with `chmod`

`lvextend` and `lvreduce`

`dmesg` command