Text Processing: grep, sed, and awk

Introduction

Log analysis and quick data fixes rely on three classic tools: grep finds lines, sed edits streams, awk splits columns. Together they replace opening huge files in Excel. This chapter focuses on patterns you will reuse when debugging Java/Node services on Linux.

Prerequisites

grep — Search Lines

Regular expressions with extended regex (-E):

bash
# Lines starting with ERROR or WARN
grep -E "^(ERROR|WARN)" application.log
 
# IP-like pattern (simple example)
grep -E "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" access.log | head

Common options:

OptionMeaning
-iIgnore case
-rRecursive
-nLine number
-vInvert
-cCount matches
-EExtended regex

Count errors:

bash
grep -c "ERROR" application.log

sed — Stream Editor

Print-only substitution (safe):

bash
# Replace first foo per line
sed 's/foo/bar/' file.txt
 
# Replace all on each line
sed 's/foo/bar/g' file.txt

Delete lines matching pattern:

bash
# Delete blank lines
sed '/^$/d' file.txt

Warning

sed -i edits files in place. Always backup first:

bash
cp config.properties config.properties.bak
sed -i 's/debug=false/debug=true/' config.properties

Wrong sed -i can break production configs irreversibly without backups.

awk — Columns and Simple Reports

Default whitespace-separated fields: $1, $2, … $0 is whole line.

bash
# Print usernames from /etc/passwd (colon separated — set FS)
awk -F: '{print $1}' /etc/passwd | head
 
# Print third field
awk '{print $3}' data.txt

Simple stats:

bash
# Sum numbers in first column
awk '{sum += $1} END {print sum}' numbers.txt

HTTP access log (format varies)—illustrative:

bash
# Print 7th field (often request path) — verify format first
awk '{print $7}' /var/log/nginx/access.log | head

Always head the log first to learn column positions.

sort, uniq, cut, tr — Helpers

bash
# Extract column 1 with colon delimiter
cut -d: -f1 /etc/passwd | head
 
# Sort and unique
sort names.txt | uniq
 
# Count duplicate occurrences
sort names.txt | uniq -c | sort -nr | head
bash
# Replace colon with newline
echo "a:b:c" | tr ':' '\n'

Pipeline Recipes

Count errors per hour (if log timestamp in first two fields):

bash
# Example — adjust field indexes to your log format
grep ERROR application.log | awk '{print $1,$2}' | sort | uniq -c | sort -nr | head

Find Java stack traces (lines with at com.):

bash
grep -n " at com\." application.log | head

Search codebase for TODO on server:

bash
grep -rn "TODO" /opt/myapp/src 2>/dev/null | head

Mini Example: Sanitize a CSV Export

bash
# Create sample data
printf "name,score\nAda,90\nBob,85\n" > scores.csv
 
# Print names with score >= 88
awk -F, 'NR>1 && $2>=88 {print $1, $2}' scores.csv

FAQ

grep vs ripgrep (rg)?

rg is faster on huge trees—optional install. grep is always available on servers.

sed on macOS vs Linux?

macOS uses BSD sed; syntax differs slightly (sed -i '' on Mac). This course targets GNU sed on Ubuntu.

awk field numbers wrong?

Log formats differ—inspect with head -n 1 and count fields manually.

When to use a real language?

Large JSON parsing: use jq. Heavy analytics: Python. grep/sed/awk stay best for quick terminal work.

What comes next?

Package management with apt and dnf (upcoming).