Are you a freshser feeling lost in the world of servers, devboxes, debugging realtime systems etc?. We've been sailing in the same boat. Here are a few handy stuff that will be your lifesaver till you are a fresher.
We can all do a man now and understand what the command actually does, but here's a cheatsheet to get you started quick.
Returning complete Exceptions/Trace from logs using grep
You can use the -B and -A to print lines before and after the match.
grep -i -A 10 'some.java.Exception' my-server.log
This will print 10 lines after the Exception, including the line with exception. similarly -A will print lines before the exception
Print all files , with ports, parent processes etc
list the files/processes: sudo lsof -i -P -n
[For Debugging] LISTENING to: sudo lsof -i -P -n | grep LISTEN
Want more detail?: ps aux
a:- This option prints the running processes from all users.
u:- This option shows user or owner column in output.
x:- This option prints the processes those have not been executed from the terminal.
When you know the PID of the process, you can use kill PID_OF_THAT_PROCESS to kill it
Tracking/ Debugging Network connections
netstat pntl
Digging deeper into networking | fixing 502s, 504s
Did you get a Gateway Timeout or a Bad Gateway response? Do you know if the packet is even reaching that server ? Try to do a tcpdump to see what packets are going in and coming into your system.
tcpdump -nnvvS src 172.31.1.237 and dst port 80
this command will track all packets coming to port 80 of your system from 172.31.1.237. If we do the same on 172.31.1.237 and use our system's public IP on that device, we can see the packets arriving to that system from our server.
If you want to analyse network dum on private cloud cluster you can generate a pcap file and analyse it on Wireshark
`sudo ip -s -s link show
sudo nstat `
Use curl instead of POSTMAN for request/response debugging
curl -k -i -X POST your_url_of_concern.com -d"{your data in json or urlencoded text}" -H -H 'set cookie: some_cookie_expected_in_request=someid'
The -k is for avoiding checks for ssl, which might be unnecessary inside your own dev-box or organization's internal APIs.
the -i option will print the headers, cookies information, which is highly useful while debugging server failures, session tracking
replace edit delete occurence of text in file
this one replaces unix with linux in geekfile.txt
sed 's/unix/linux/' geekfile.txt
Allowing an app through firewall
First list all your apps requiring the feature
sudo ufw app list
Now allow the app through firewall
sudo ufw allow 'APP_NAME'
Transferring files to-from remote host over SCTP
This can come in handy when you want to get some downloaded or config files from/to servers to run some stuff locally
without keys: scp -r source_host_full_dir_path destination_host_full_dir_path
In this example, we transfer s.py from local to 172.31.1.237 remote machine
if using ssh key: scp -i your_ssh_key_path -r source_host_full_dir_path destination_host_full_dir_path
The -r is for including subfolders and files inside that folder. you wont need that for single files
If ever you get permission denied during transfer to or from a server, go to the source's directory and change ownership of the file to the current logged in user
Trouble sending certificates in json body ?*
certificates are in PEM format and line breaks are a hassle to pass through in json format. If you have to pass the plaintext of a cert in json, better convert those linebreaks into '\n'
awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' your-cert.pem
Shortcut links to long directory paths
Do you have very long directorypaths, which you frequently need to access from current directory? why not have a shortcut to that file in your current directory. A symlink will create a pointer to that actual file from the current directory. All changes will be reflected on the original link.
ln sourcefilepath destinationfilepath
Executing commands directly in remote box
example : ssh -t username@host 'pwd'
*The -t is required if you are executing multiple commands or if you want the shell to be interactive. Not required if you are not doing any of that. However if you want to execute a shell script from your local to your remote machine , you have to scp file transfer the script to that machine and then execute it. credits to this *link.
Follwing up .logs using tail
tail -10f /location/of/your/.log
the "10" denotes that it will tail/print the last 10 lines, in that file. You can set it to whatever value you want. This will update in real-time as your log file updates.
You can use *.log instead of specific filename to track all log files in that directory
Setting up ENVIRONMENT VARIABLES :
export variablename=dont_add_spaces_here_whateveryouwant
use printenv variablename to print them
Comparing Floats in a shell script
Yes, it's a pain to compare floating point numbers in shell. This is not an issue in bash though ( yes bash and sh are different, in case you didn't know that). Here is an example to do so
num1=18.04
num2=18.04
OUTPUT=$(echo "$num1 == $num2" |bc -l)
will produce output 1 since they are equal.
Now, to use this OUTPUT, you again have to compare it using
[ $OUTPUT -ne 0]
this checks if the value of OUTPUT is not equal to zero.
Debugging | Logging stdouts from shell scripts
There might be occassions where you have to debug stuff inside a shell script, but there is no elegant way of logging inside of shell ... or is it ?
We can dynamically open, close, and copy stdout to achieve logging operations. Let’s redirect stdout (FD 1) to the log file:
#! /bin/bashdate +%F
script_log="/tmp/log_.log"
exec 1>>$script_log
echo "This will be written into log file rather than terminal.."
echo "This too.."
Learn more
commits between 2 git hashes for better gitblame
git log <older commit hash>..<newer commit hash> --oneline | cut -d " " -f 1
easy git update between branches without rebase
Is rebasing causing conflicts for you even though you know your diff is perfectly fine and conflicts have already been resolved in upcoming commits ? In this case forget rebasing and do patching. take a diff between your current branch and the branch you wanna update (say master)
git diff origin/master --no-color > ~/Documents/your_patch_file.patch
now checkout to a fresh master branch and
git apply ~/Documents/your_patch_file.patch
you will have all the changes of your current branch, in this one now, without any conflicts.
Local git branch messed up? fix to latest working remote branch
sudo git clean -f -d; sudo git reset HEAD —hard
remove untracked files from git commits
git status | grep 'modified:' | cut -d' ' -f4|xargs sudo git rm -f
if you want to untrack deleted files too:
git status | grep 'deleted:' | cut -d' ' -f5|xargs sudo git rm -f
that cut -d' ' -f_number means you are splitting the current line by spaces and picking up the string corresponding to your file. so that f_number should be the position your file occurs in. It might vary in your case
clean unused old libraries from .m2 repository
cleans all files more than an hour of last access from m2. They will be accessed if you had built within the past hour
find ~/.m2 -amin +60 -iname '*.pom' | while read pom; do parent=`dirname "$pom"`; rm -Rf "$parent"; done
Using anaconda? Do you clear the unused files?
if you are thinking where all my disk space is going, go and check your anaconda package directory. Mine was 17.6GB. its crazy. Anaconda does not automatically clean files after installation.
conda clean -a will remove all tarballs, and setup packages of already installed packages. My directory size is now 10GB.
A few stuff for precautions:
Here are a few things to avoid when you are using sudo.
1.NEVER sudo in the root folder. This will deadlock your system.
2.To avoid using sudo in every place else, all the time, use sudo -s. and from there onwards, you need not type sudo again.
3.To avoid sudo in some situations,( especially when you are just testing some stuff), maybe consider changing ownership of files in that folder to your name so you can liberally access them.
chown -R your_username /foldername will change ownership of all files subfolders, in that directory to you
Learning regex and VIM
This will save you hours of typing time and your reputation for faster prod-issue debugging, once you get used to it. I recommend the below places to learn these skills in the fastest way possible:
Regexone
thoughtbot
Working with JAVA ? Read about tweaking JVM with these
Java Options
Oldest comments (2)
Another recommendation:
Get ZSH/FISH/better bash completions. Auto complete make things a hell lot better
yes ! great suggestion. I have been using zsh :) forgot to add it