Hi there! I'm Shrijith Venkatrama, founder of Hexmos. Right now, I’m building LiveAPI, a tool that makes generating API docs from your code ridiculously easy.
A while back, I was working on a Go project that needed to run multiple system commands in parallel.
I thought, "No big deal, I’ll just fire up some goroutines and let them rip."
Big mistake—my outputs turned into a garbled mess, and I spent hours debugging.
That’s when I realized concurrent command execution needs careful handling.
Let’s walk through the problem and solve it with safecmd.go, a handy utility I built to make this easier for all of us.
The Problem
Running commands in parallel with goroutines sounds great, but it’s tricky.
Go’s bytes.Buffer isn’t thread-safe, so data gets corrupted.
Outputs to stdout and stderr can overlap across goroutines if not handled correctly.
It’s easy to mess up without proper synchronization.
Here’s an example of what goes wrong:
package main
import (
"bytes"
"fmt"
"os/exec"
"sync"
)
func main() {
var buf bytes.Buffer
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
cmd := exec.Command("echo", fmt.Sprintf("Hello %d", id))
cmd.Stdout = &buf // Shared buffer, no protection
cmd.Run()
}(i)
}
wg.Wait()
fmt.Print(buf.String())
}
Output (varies, often broken):
Hello 0Hello 1Hello 2
The outputs smash together because the buffer isn’t safe for concurrent writes.
Here’s another messy case:
package main
import (
"fmt"
"os/exec"
"sync"
)
func main() {
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
cmd := exec.Command("echo", fmt.Sprintf("Thread %d", id))
out, _ := cmd.CombinedOutput()
fmt.Print(string(out))
}(i)
}
wg.Wait()
}
Output (unpredictable):
Thread 2Thread 0Thread 1
No control over order, and it’s still risky with shared resources.
Step 1: Thread-Safe Buffers
Let’s fix the buffer issue first.
We’ll wrap bytes.Buffer with a mutex to make it safe.
Here’s SafeBuffer:
package main
import (
"bytes"
"fmt"
"sync"
)
type SafeBuffer struct {
b bytes.Buffer // Underlying buffer for data
m sync.Mutex // Mutex to lock access
}
func (b *SafeBuffer) Write(p []byte) (n int, err error) {
b.m.Lock() // Lock before writing
defer b.m.Unlock() // Unlock when done
return b.b.Write(p)
}
func (b *SafeBuffer) String() string {
b.m.Lock() // Lock to read safely
defer b.m.Unlock() // Unlock after
return b.b.String()
}
func main() {
var buf SafeBuffer
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
buf.Write([]byte(fmt.Sprintf("Data %d\n", id)))
}(i)
}
wg.Wait()
fmt.Print(buf.String())
}
Output:
Data 0
Data 1
Data 2
Now each write is protected, and the data stays clean.
Step 2: Safe Writers
Next, we need to handle stdout and stderr safely.
Here’s safeWriter to serialize writes:
package main
import (
"bytes"
"fmt"
"sync"
)
// SafeBuffer as defined above
type SafeBuffer struct {
b bytes.Buffer
m sync.Mutex
}
func (b *SafeBuffer) Write(p []byte) (n int, err error) {
b.m.Lock()
defer b.m.Unlock()
return b.b.Write(p)
}
type safeWriter struct {
buf *SafeBuffer // Buffer to write to
lock sync.Mutex // Extra lock for writer
}
func (w *safeWriter) Write(p []byte) (n int, err error) {
w.lock.Lock() // Lock the writer
defer w.lock.Unlock()
return w.buf.Write(p) // Write to safe buffer
}
func main() {
var buf SafeBuffer
writer := &safeWriter{buf: &buf}
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
writer.Write([]byte(fmt.Sprintf("Line %d\n", id)))
}(i)
}
wg.Wait()
fmt.Print(buf.String())
}
Output:
Line 0
Line 1
Line 2
This keeps each command’s output intact.
Step 3: Command Results
We want a clean way to return results.
Here’s CommandResult:
package main
import "fmt"
type CommandResult struct {
ExitCode int // Command’s exit status
Stdout string // Output from stdout
Stderr string // Output from stderr
}
func main() {
result := CommandResult{
ExitCode: 0,
Stdout: "All good",
Stderr: "",
}
fmt.Printf("Exit: %d, Out: %s, Err: %s\n", result.ExitCode, result.Stdout, result.Stderr)
}
Output:
Exit: 0, Out: All good, Err:
It’s straightforward and gives us everything we need.
Step 4: Building safecmd.go
Let’s put it together in safecmd.go.
Here’s the core function:
package main
import (
"bytes"
"fmt"
"os/exec"
"sync"
)
// SafeBuffer and safeWriter as above
type SafeBuffer struct {
b bytes.Buffer
m sync.Mutex
}
func (b *SafeBuffer) Write(p []byte) (n int, err error) {
b.m.Lock()
defer b.m.Unlock()
return b.b.Write(p)
}
func (b *SafeBuffer) String() string {
b.m.Lock()
defer b.m.Unlock()
return b.b.String()
}
type safeWriter struct {
buf *SafeBuffer
lock sync.Mutex
}
func (w *safeWriter) Write(p []byte) (n int, err error) {
w.lock.Lock()
defer w.lock.Unlock()
return w.buf.Write(p)
}
type CommandResult struct {
ExitCode int
Stdout string
Stderr string
}
func ExecCommand(name string, args ...string) (CommandResult, error) {
var result CommandResult
var stdout, stderr SafeBuffer // Buffers for output
// Set up safe writers
stdoutWriter := &safeWriter{buf: &stdout}
stderrWriter := &safeWriter{buf: &stderr}
cmd := exec.Command(name, args...) // Create command
cmd.Stdout = stdoutWriter // Assign safe stdout
cmd.Stderr = stderrWriter // Assign safe stderr
err := cmd.Run() // Execute the command
if err != nil {
if exitErr, ok := err.(*exec.ExitError); ok {
result.ExitCode = exitErr.ExitCode() // Capture exit code on error
}
}
result.Stdout = stdout.String() // Store stdout
result.Stderr = stderr.String() // Store stderr
return result, err
}
func main() {
result, err := ExecCommand("echo", "Hello, safecmd!")
if err != nil {
fmt.Printf("Error: %v\n", err)
}
fmt.Printf("Exit: %d, Out: %s, Err: %s\n", result.ExitCode, result.Stdout, result.Stderr)
}
Output:
Exit: 0, Out: Hello, safecmd!
, Err:
It’s simple and works reliably.
Step 5: Testing with Goroutines
Let’s test it with concurrency:
package main
import (
"bytes"
"fmt"
"os/exec"
"sync"
)
// SafeBuffer, safeWriter, CommandResult, ExecCommand as above
type SafeBuffer struct {
b bytes.Buffer
m sync.Mutex
}
func (b *SafeBuffer) Write(p []byte) (n int, err error) {
b.m.Lock()
defer b.m.Unlock()
return b.b.Write(p)
}
func (b *SafeBuffer) String() string {
b.m.Lock()
defer b.m.Unlock()
return b.b.String()
}
type safeWriter struct {
buf *SafeBuffer
lock sync.Mutex
}
func (w *safeWriter) Write(p []byte) (n int, err error) {
w.lock.Lock()
defer w.lock.Unlock()
return w.buf.Write(p)
}
type CommandResult struct {
ExitCode int
Stdout string
Stderr string
}
func ExecCommand(name string, args ...string) (CommandResult, error) {
var result CommandResult
var stdout, stderr SafeBuffer
stdoutWriter := &safeWriter{buf: &stdout}
stderrWriter := &safeWriter{buf: &stderr}
cmd := exec.Command(name, args...)
cmd.Stdout = stdoutWriter
cmd.Stderr = stderrWriter
err := cmd.Run()
if err != nil {
if exitErr, ok := err.(*exec.ExitError); ok {
result.ExitCode = exitErr.ExitCode()
}
}
result.Stdout = stdout.String()
result.Stderr = stderr.String()
return result, err
}
func main() {
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
result, err := ExecCommand("echo", fmt.Sprintf("Hello from %d", id))
if err != nil {
fmt.Printf("Error %d: %v\n", id, err)
}
fmt.Printf("Goroutine %d: %s", id, result.Stdout)
}(i)
}
wg.Wait()
}
Output:
Goroutine 0: Hello from 0
Goroutine 1: Hello from 1
Goroutine 2: Hello from 2
Each goroutine gets its own clean output.
Why safecmd.go Helps
safecmd.go takes the pain out of concurrent commands.
It stops buffer corruption in its tracks.
It keeps outputs separate and readable.
You can drop it into your project and use it right away.
It’s a small tool that saves big headaches.
Top comments (2)
Hi, wouldn't it make more sense to structure the routines as workers in a worker pool. Each routine/Struct would have it's own buffered reader and writer and maybe a Chan to deliver results?
Sharing buffers was bound to be problematic
I'd think of that as another layer of convenience on top of
safecmd. Many solutions possible, this is just one way of getting there.