The Anatomy of a PostgreSQL Hack: How It Happened and What We Did About It.

Jay Tailor — Fri, 10 Feb 2023 11:20:26 +0000

A database hack can have serious consequences for an organization, including financial loss, damage to reputation, and loss of sensitive information. PostgreSQL is a popular open-source relational database management system, but it is not immune to hacking attempts. In this article, we will discuss the very basic way in which a PostgreSQL database can be hacked, as well as best practices for preventing and responding to such a breach. By understanding the risks and taking steps to secure your PostgreSQL database, you can help protect your organization and its data.

The beginning

I have been using PostgreSQL in my web application using a Docker container. I noticed that suddenly my server usage spiked up to 400%! At FULL CAPACITY! (more than 100% as I had 4 virtual CPUs)

The quick fix

I fixed it by restarting Postgres docker container. It fixed the issue but it happened again. I assumed I got hacked. So I decided to move to a new server instance. I go through all the processes to deploy a full-stack web application from scratch.
To me, that fixed the issue only for a week or so. Then I received an email from the hosting provider saying "A network scan has been performed from the IP address which is under your server"

The realization

I was not sure what exactly was going on. On initial research, I suspect that Postgres runs a service called vacuum. The Postgres docs say,

VACUUM reclaims storage occupied by dead tuples. In normal PostgreSQL operation, tuples that are deleted or obsoleted by an update are not physically removed from their table; they remain present until a VACUUM is done. Therefore it's necessary to do VACUUM periodically, especially on frequently-updated tables.

I believed that the vacuum is the culprit for all of these. Because high CPU usage was getting fixed as soon as I restart the Postgres docker container.

But I was wrong.

The final fix

After a week of restarting the server and same time searching for a potential solution, I found that my server was hacked indeed.
I tried a lot of things to fix the issue. Finally, I came up with a solution that worked.
Long story short, I fixed the server by recreating a new Postgres container with a strong password.

Technical overview

There are different kinds of hackers and different kinds of hacks they perform. One I was the victim of was the PostgresSQL server hack.
The hacker got into my server and was mining cryptocurrency using my server resources at full capacity.

The hacker got into my server by guessing the supper user password of the database server. They used the lo_export utility to upload malicious executables for a user-defined-function which executes shell commands. Function with random name fun451813001 is created, the SQL function which was created to wrap the executable. Now hackers have access to my database server and server resources.

Things don't end here. They use nmap to find more publicly exposed PostgreSQL databases on the same network to find the next victims. This is the reason why I got the following notice from the server provider.
"A network scan has been performed from the IP address which is under your server"

Conclusion

In this article, we learn about how hackers can get into our system and use servers to do potentially anything.
Also, we learn that simple things like choosing strong can help you from a serious data breach or heavy financial loss.

If you like the post please share your thoughts in the comments. Feel free to follow me @jaytailor45 for more tech content. Have a fantastic day.

TLDR; DON'T USE SIMPLE PASSWORD

PS - One of my friend @slk5611 has also written a blog and mongodb attack. Here's the interesting read on Avoid a Malicious Attack MongoDB: How It Happened and What We Did About It.