Sed It Right: Mastering the Stream Editor for Text Magic

Welcome back to the Textual Healing series! In the first part, we dove into the basics of text manipulation using cat, grep, and pipes. Now, we’re stepping things up a notch with one of the most powerful tools in your shell arsenal: sed, the stream editor.

sed is all about transformation. It’s designed to take in streams of text (or data), edit that text on the fly, and spit it back out—either to the terminal, a file, or even piped into another command. Think of sed as a scalpel for your text, performing precise operations to search, replace, delete, or modify patterns with ease. It’s especially handy for automating repetitive edits, which makes it an essential tool for any shell wizard.

---

1. The Basics of sed

At its core, sed follows a simple structure:

sed 's/search_pattern/replacement/' filename

• s stands for substitute, telling sed to find the search_pattern and replace it with replacement. The command operates on the file line by line.

For example, if you want to replace "apple" with "orange" in a file called fruits.txt:

sed 's/apple/orange/' fruits.txt

This will find the first occurrence of "apple" on each line and replace it with "orange." Simple enough, but sed has a lot more depth when you need it.

---

2. Global Substitution with g Flag

By default, sed only replaces the first occurrence of the pattern on each line. But what if you want to replace every instance of the pattern? This is where the global flag (g) comes in:

sed 's/apple/orange/g' fruits.txt

Now, every "apple" on every line will be replaced with "orange."

---

3. Editing Files In-Place with -i Flag

One of the coolest things about sed is that you can make changes directly to your files without needing to create a new output file. To edit a file in-place, you’ll use the -i flag:

• On GNU sed (common on Linux):

  sed -i 's/apple/orange/g' fruits.txt

This command modifies fruits.txt directly, saving the changes back to the original file.

• On BSD sed (like macOS): You’ll need to specify a backup file extension. If you don’t want to keep a backup, pass an empty string:

  sed -i '' 's/apple/orange/g' fruits.txt

Warning: Use the -i flag with caution. Once the file is edited in-place, you can’t undo the changes easily unless you’ve made a backup.

Edit: Thanks to the Ben Sinclair's comment, I've updated this part to include GNU and BSD versions of the -i flag since they work slightly differently. Really appreciate the comment!

---

4. Deleting Lines with d

sed isn’t just about finding and replacing text—it’s also super handy for deleting lines that match a specific pattern. Let’s say you want to delete every line in a file that contains the word "banana":

sed '/banana/d' fruits.txt

This command tells sed to delete all lines containing "banana" and output the result. If you want to delete a specific line number, you can also do:

sed '5d' fruits.txt

This would delete the 5th line from the file.

---

5. Substituting Multiple Patterns

Let’s say you want to replace multiple words in a file in one go. You don’t need to run sed multiple times—you can chain multiple substitution commands together:

sed -e 's/apple/orange/g' -e 's/grape/mango/g' fruits.txt

Here, we’re replacing "apple" with "orange" and "grape" with "mango" in a single command.

---

6. Using sed with Regular Expressions

We all feel the pain of regex's weird syntax, but hey, we're not going to let that stop us from achieving greatness! Despite its quirks, regex is a powerhouse when used in combination with sed. Let’s harness its strength!

sed becomes a real powerhouse when you start using regex to match patterns more intelligently. For example, suppose you want to replace any sequence of digits with the word "[number]":

sed 's/[0-9]\+/[number]/g' text.txt

This will replace any group of digits ([0-9]+) with "[number]" throughout the file. Regex allows you to create powerful and flexible searches, letting you target specific patterns of text, not just exact strings.

---

7. Inserting and Appending Text

Need to insert or append lines into a file? sed can do that too! To insert text before a specific line, use the i command:

sed '3i\This is inserted before line 3' file.txt

This will insert the text "This is inserted before line 3" just before the 3rd line. To append text after a line, use a:

sed '3a\This is appended after line 3' file.txt

---

8. Using Line Addresses with sed

To make your substitutions more targeted, you can use addresses to limit sed commands to specific lines. For example, to substitute "apple" with "orange" only on the first four lines of the file:

sed '1,4s/apple/orange/' file.txt

This substitutes "apple" with "orange" only on lines 1 through 4. You can use patterns and regular expressions as well, giving you precise control over where substitutions occur.

Additionally, there’s a tip about the info command, which provides more detailed pages for many GNU tools than their standard man pages. Tools like grep, sed, and find often have rich examples and details that can make learning them even easier. To view an info page, just run:

info sed | more

Note: This command is using more because it's installed by default on most Linux machines, but I do prefer less because it makes navigation and search capabilities a breeze with vim-like commands. Less is more!

---

9. Combining sed with Other Commands

Like all powerful shell commands, sed works beautifully when combined with other tools like cat, grep, or even find. Let’s say you want to find all text files in a directory and replace "apple" with "orange" in all of them:

find . -name "*.txt" -exec sed -i 's/apple/orange/g' {} \;

This command finds every .txt file in the current directory and applies the sed substitution to replace "apple" with "orange" directly within each file.

---

10. Real-World Use Cases

sed is a lifesaver in real-world scenarios, like quickly making bulk text changes in configuration files, automating logs cleanup, or prepping data for further processing. For example:

• Update URLs in HTML files: Replace old URLs with new ones in all .html files:

  sed -i 's/oldsite.com/newsite.com/g' *.html

• Clean up logs: Remove unnecessary log entries (e.g., removing "INFO" lines from log files):

  sed '/INFO/d' log.txt

---

Wrapping Up

sed is a game-changer when it comes to stream editing and text manipulation. It’s fast, flexible, and powerful enough to handle everything from quick find-and-replace tasks to complex text transformations. Once you get comfortable with the basics, you’ll find sed to be an indispensable tool in your shell toolkit.

Next up in the Textual Healing series, we’re diving deep into awk, another heavy hitter for text processing that’ll help you analyze and manipulate data with ease.

---

Want to hang out with other Linux lovers and coding enthusiasts? Come join our community on Discord! We’re a group of friendly folks who love to code, share tips, and help each other grow. Click here to join the conversation!

Comments

[Ben Sinclair]

Note that this use of -i is for GNU sed, others have a different syntax which requires you to specify a backup pattern.

[Jimmy McBride]

I have updated the blog to reflect this. Thanks for pointing it out!

[Joe]

Not sure if I did anything special, but my info command uses a pager by default.

This looks helpful.

[Joe]

Nice. To round it out a little, I would include an example using addresses.

sed -re '1,4s/apple/orange/' file

substitutes the first occurrence of apple with orange, but only on the first four lines of the input file. Patterns and regexes work as well.

One thing I never see mentioned is that most GNU tools have an info page that may be far more detailed than their man pages. The ones for grep, sed, and find are quite detailed with lots of examples and help with things like regexes. It's really to bad that this didn't catch on for the rest of Linux commands.
To see one, use the info command instead of man.

[Jimmy McBride]

Great point! I've updated my blog with your suggestion. :) I've never used info before now!