DEV Community

Bobby Iliev
Bobby Iliev

Posted on • Edited on • Originally published at devdojo.com

How to keep a process running even after closing SSH connection?

There are many reasons why you would like to keep a process running even if you close your SSH session.

Here are a few examples:

  • Your working day is going to be over soon and you are running a huge database import that's taking a long time to complete, you would not want to stay and wait for the import to complete but just hand over the task to the next person on shift
  • You are downloading a huge file and it's going to take a good few hours, you would not want to leave your terminal open and wait, or even worse, you don't want to start all over again in case that your internet connection drops

What I usually do in such cases is to use the screen command and run the processes in a screen session. That way I could detach from the screen session and close my SSH connection and the process would still run allowing me or other people to attach to the session later on and follow up on the process.

Here's a quick introduction to how to use screen!

According to the official documentation screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells. When screen is called, it creates a single window with a shell in it where you could run commands as normal.

I've noticed that in most cases screen is installed by default, so to check if you have screen installed you could run the following:

screen --version
Enter fullscreen mode Exit fullscreen mode

If you get command not found, then you could install screen by running:

  • For Ubuntu and Debian:
sudo apt update -y
sudo apt install screen
Enter fullscreen mode Exit fullscreen mode
  • On CentOS:
yum update -y
sudo yum install screen
Enter fullscreen mode Exit fullscreen mode

Once you have screen installed, to start a new screen session run:

screen -S SOME_NAME_HERE
Enter fullscreen mode Exit fullscreen mode

This would spin up a new screen and you would be attached to it automatically, inside the screen session run your script.

After that to detach from the screen session press CTRL+a+d.

If you need to attach back to the screen session run and check on your process just run:

screen -R SOME_NAME_HERE
Enter fullscreen mode Exit fullscreen mode

If you've forgotten the name of your screen session or if you've not set a name, you could list all available screen sessions by running:

screen -ls
Enter fullscreen mode Exit fullscreen mode

This is just a really brief introduction on how to use screen, I would recommend checking out the official documentation as well:

https://www.gnu.org/software/screen/manual/screen.html

Hope that this helps!

Top comments (44)

Collapse
 
eggbean profile image
eggbean • Edited

I would recommend looking at tmux as an alternative.

Collapse
 
pklapperich profile image
pklapperich

Indeed tmux has simpler keyboard shortcuts, maintains window arrangement while connecting/disconnecting sessions, and uses a unix socket to control the session so giving another user access to the session is as simple as doing chown or chmod on the socket 'file'. Letting two users share a screen session is a bit of a nightmare.

Collapse
 
bobbyiliev profile image
Bobby Iliev

That definitely sounds interesting! Thanks for sharing! I will be checking tmux and possibly switch to it if it works better than screen.

Thread Thread
 
bobbyiliev profile image
Bobby Iliev

Forgot to upload the image:

Screen

Collapse
 
bayindirh profile image
Hakan Bayındır

Screen also uses sockets. You can see your sockets with screen -ls. IIRC, the sockets should be under /var/run/screen.

While tmux is thrown around a lot, I've found screen to be more stable and mature. Also it can be configured pretty extensively. Just see the customization done by GRML distribution. It runs screen on every TTY by default. See an example here.

Thread Thread
 
pklapperich profile image
pklapperich • Edited

Screen does use sockets, but AFAIK there's not an option to "attach to this socket". With tmux it's just 'tmux -S /path/to/socket'. I've found this very useful for sharing a terminal while doing remote support with family/friends. I had done this via screen previously and it was significantly more effort (and involved screen's internal permissions system: wiki.networksecuritytoolkit.org/in...)

Compare the "easy 6 step" process described on the screen wiki with the following in tmux:
$ tmux -S /tmp/shared_socket # start the session using the named socket file
$ chmod 777 /tmp/shared_socket # grant any user on the system access to the socket
$ # that's it. Now the other user can use tmux -S /tmp/shared_socket to connect with full read/write access. Want to give only read access? then give them permission 6 instead of 7. Want to limit to less than all users? Then use chown, setfacl, etc.


Both can be configured extensively and launched/controlled via scripts. Both can be set as default on a tty. I find the tmux config file to be a lot easier/straight forward.


How exactly are you defining "more stable and mature"?


While I think tmux is easier to use (the config file structure and default key bindings make more sense to me), the #1 selling point for me with tmux is that if I split the screen 6 ways and detach, when I re-connect (from any device), the screen is still split 6 ways. With screen all 6 windows are still running, but I have to re-arrange them into a 6-way split again.

Collapse
 
dkoczka profile image
David Koczka

I use tmux myself on every server I'm working with. It's a simple, yet effective tool.

Collapse
 
rizkysyazuli profile image
Rizky Syazuli

+1 tmux

Collapse
 
loki profile image
Loki Le DEV

There is also the nohup command

Collapse
 
drewmullen profile image
drewmullen

This is my go-to

nohup command ... &

Collapse
 
loki profile image
Loki Le DEV

Just be careful because by default it opens a nohup.out file and appends all output of the process you launched into this file. It can grow a lot.

Collapse
 
bayindirh profile image
Hakan Bayındır

If you're afraid of filling your disk, just toss the output to /dev/null during the command invocation ;)

Collapse
 
email2vimalraj profile image
Vimalraj Selvam

I highly recommend this command, we use this on daily basis instead of 3rd party tools.

Collapse
 
krishnanunnir profile image
Krishnanunni R

If I may ask how do you handle killing the process?

Collapse
 
aderchox profile image
aderchox

Here's my note on this for you:

===== Listing Processes =====
List processes owned by the current user (sorted by PID, from PID 1 to higher/user-level PIDs)
ps
List all processes, including system processes
ps ax
List all active processes, including system processes, with extra information like RAM/CPU usage, owner user, etc.
ps aux
Human-readable shiny process list, with features like search, filtering, tree structure, realtime resources usage monitor, etc.
htop

===== Killing Processes (kill vs. killall vs. pkill) =====
Kill a process by its id
kill <process_id>
Or use killall with the process name (Can be risky if multiple processes share the same name)
killall node
Or use pkill with the process name or other attributes (The most versatile of the three commands, allows terminating processes by name, exact match, or pattern matching using regular expressions) (Can be risky if the pattern matches unintended processes)
pkill

Killing without mercy
kill -9 <process_id>
Killing by giving the stuck process a chance to clean up before dying (RECOMMENDED)
kill -15 <process_id>

Collapse
 
bobbyiliev profile image
Bobby Iliev

Thanks for sharing Loik! I’ll definitely check it out!

Collapse
 
setzke profile image
setzke • Edited

I do:

screen -d -m $script
multitail $log1 -I $log2 -I $log3 -I $log4

My script writes everything to those logs as it works, and I get what essentially would be the output of the script as it's working. That way any CTRL+C or disconnect just quits the viewing, and not my script.

Collapse
 
bobbyiliev profile image
Bobby Iliev

That's super cool! Thanks for sharing!

Collapse
 
ianwijma profile image
Ian Wijma

For simplicity sake, you can also just run screen to get a random session. And screen -r to reconnect to your last session.

Collapse
 
bobbyiliev profile image
Bobby Iliev

That is a really good remark! Thanks Ian!

Collapse
 
ianwijma profile image
Ian Wijma

No problem! :)

Collapse
 
jnareb profile image
Jakub Narębski • Edited

Screen or tmux is very helpful tool if you use either some text-based tool (like e.g. text-based mail reader like alpine or mutt), or use an interactive session like e.g. ipython, and you are on not entirely reliable SSH connection. Loosing email you have been composing for 15 minutes because network connection dropped is... irritating...

For example screen -S alpine -d -RR alpine reconnect to session (forcefully closing it if it did not timed out yet), or start alpine from start if there is no screen session to go back to - for example if the host was rebooted.

The nohup command is better for batch jobs, like large wget / curl download.

Collapse
 
mihaylov profile image
Petar Petrov

I like tmux better because it allows me to also split the terminal.

Collapse
 
bobbyiliev profile image
Bobby Iliev

Hi Petar, you could do the same with screen actually.

If you press CTRL+a then SHIFT+S (note the capital S), this would create a second screen window like this.

Then you can press CTRL+a and then TAB to go to the second screen and after that press CTRL+a+c to create the second terminal.

But I'll definitely check xmux as well!

Collapse
 
mihaylov profile image
Petar Petrov

It feels clunkier than tmux, which can split horizontal and vertical.

But its good to know for systems that don't have tmux, thanks!

Thread Thread
 
moopet profile image
Ben Sinclair

Newer versions of screen let you split vertically and there's been a patch around for older versions for quite a few years.

Thread Thread
 
pklapperich profile image
pklapperich

Yeah, I was doing vertical splits in screen back on Ubuntu Dapper. It took a surprisingly long time for upstream screen to accept the patches, but distros have commonly included the patches for vertical splits since before I first used screen.

Collapse
 
nexus_software_systems profile image
Nexus Software Systems

Screen is one way to keep a process running even after closing SSH connection. Like most things in Linux there are more ways to keep a process running even after closing SSH connection:

“Ctrl+a” immediately followed by “d” and you will be back to the terminal seeing the message that the Screen is detached. Now you can safely logout and your session will be left alive.

Tmux is another piece of software which is created to be a replacement for screen. It has most of the capabilities of screen, with few additional capabilities which make it more powerful than screen.

$ tmux detach
$ tmux attach

Executing command using nohup in background

Disown, removes the job from the process job list of the system, so the process is shielded from being killed during session disconnection as it won’t receive SIGHUP by the shell when you logout.

You can also use execute any command using setsid.
setsid allocates a new process group to the process being executed and hence, the process created is a newly allocated process group and can execute safely without fear of being killed even after session logout.🐧👍

Collapse
 
radulle profile image
Nikola Radulaški

Why not just: CTRL+Z -> bg -> disown? Sure, it suspends your job for a moment but in most cases that is not a problem.

Collapse
 
asharlohmar profile image
Lohmar ASHAR

I also use this a lot for jobs that are already started and don't want to lose because of some disconnection.
On a day-to-day basis it's a fair fight between screen, mostly for when I didn't had time to take too much care of the logging, and nohup + disown otherwise, because I like to avoid the hassle of cleaning old screen sessions.

Collapse
 
bayindirh profile image
Hakan Bayındır

Why not add & to the command to start detached, get the pid and disown in a single command. Your job doesn't get stopped, you don't lose anything (if you're capturing packets or doing something timing-critical).

Collapse
 
bobbyiliev profile image
Bobby Iliev

This is a valid point, but I'm afraid that if you do that the jobs that you have in your background would be stopped once you close your SSH session.

Also you would not want to interrupt a critical database import by suspending the job.

Collapse
 
radulle profile image
Nikola Radulaški

From my experience it doesn't get stopped. One more downside is that you can not own it again. It's like fire and forget kind of solution. Upside is that you do not have to have any preparation for it like in other cases, you run commands as usual.

Thread Thread
 
bobbyiliev profile image
Bobby Iliev

Ah yes, I've just tested that and indeed you are right!

Collapse
 
ghgore profile image
Griffin Gore 🤔

I've heard that screen is insecure (not sure why, but Red Hat no longer recommends it because of the issue), and I'd recommend trying Tmux or Byobu, being that they're safer and possibly easier to work with depending on your tastes. :)

Collapse
 
bobbyiliev profile image
Bobby Iliev

That's quite interesting! Thanks for sharing, I'll do some research on the vulnerability that you've mentioned! Thank you :)

Collapse
 
rsajdok profile image
rsajdok🇪🇺

try byobu

Collapse
 
bobbyiliev profile image
Bobby Iliev

I've not used this one either, but I've just watched a quick introduction and it looks pretty cool! Thanks for sharing!