The single most common silent error in shell work. grep -c reports how many lines matched, not how many matches there were. A line with six matches counts once. It bit me twice while building this site — both times the number looked plausible, which is exactly the problem.

Print just the matched text rather than the whole line, one match per line. Turns grep into a crude extractor, and combined with wc -l it is how you actually count occurrences.

Print nothing, exit 0 if found. The right form for a test — it stops at the first hit instead of reading the whole file. if grep -q pattern file; then

Treat the pattern as a literal, not a regex. Faster, and it means you stop escaping dots and slashes in things like IP addresses and file paths. Badly underused.

-E gives you +, ?, | and () without backslashes. -P gives real PCRE — lookarounds, lazy quantifiers, \d — but it is a GNU extension and is missing on macOS and BSD, so it is a portability landmine in anything you ship.

Search a tree, filtered by filename: grep -rn --include="*.ts" TODO src/. Add --exclude-dir=node_modules or the results are worthless.

Lines after, before, or around each match. -C3 is the one to reach for when reading logs — a match without context is usually just a taunt.

grep exits 0 when it found something, 1 when it did not, and 2 on error. That "1" is not a failure, but set -e cannot tell the difference and will kill your script on a perfectly normal no-match. Guard it: grep -q x f || true.

Faster, sane defaults, and it respects .gitignore — which is the feature and the trap. It silently skips ignored files, so if you are grepping build output or vendored code you will get a confident empty result. rg -u (or -uu) turns that off.

The whole language, for most people. g replaces every match on the line rather than the first, i ignores case, p prints, and a bare number replaces only the Nth match. Without g you get one per line — the cause of a lot of "it only half worked."

The delimiter after s is whatever you want. Substituting a file path with / produces a thicket of \/; using s|/usr/local|/opt| is the same command, legible.

GNU takes sed -i. BSD and macOS require an argument: sed -i ''. Get it wrong and macOS silently eats your next argument as a backup suffix, or GNU creates a file literally named ''. Use -i.bak on both until you trust the expression — it works everywhere and leaves you a way back.

By default sed prints every line. -n suppresses that, so sed -n '/pattern/p' behaves like grep, and sed -n '5,10p' prints a line range — handy on a file too big to open.

Every command can be scoped. /error/d deletes matching lines; 2,5d deletes a range; $d deletes the last line; /start/,/end/ spans between two patterns. The address is where sed stops being search-and-replace and starts being an editor.

POSIX sed has no .*?. .* takes the longest match it can, which is why your regex ate the rest of the line. The workaround is a negated character class — [^"]* instead of .* — or a different tool.

sed sees one line at a time and has no idea what nesting is. HTML, JSON, YAML and XML are not line-oriented, and a sed expression that appears to work on them is working by luck on your particular file. Reach for jq, yq, or a parser the moment structure matters.

awk is pattern { action } repeated. No pattern means every line; no action means print the line. So awk '/error/' is grep, awk '{print $2}' is cut, and awk '/error/ {print $2}' is both — which is the moment it clicks.

$1…$n are fields, $0 the whole line, NF the field count (so $NF is the last field, and $(NF-1) the one before), NR the record number. Printing the last field regardless of width is where awk starts beating cut.

awk treats any run of whitespace as one separator by default. cut -d' ' treats every single space as a separator, so aligned output turns into a mess of empty fields. For ls -l, ps, or anything column-aligned, awk simply works.

awk arrays are hash maps, and they are why a one-liner can replace a pivot table:
awk '{sum[$1] += $3} END {for (k in sum) print k, sum[k]}' file
That is "total column 3, grouped by column 1" in nine tokens.

BEGIN runs before input — set separators, print a header. END runs after the last line, which is where totals, counts and averages live. awk 'END {print NR}' is wc -l.

-F, sets the input separator (-F'\t' for tabs). Output uses OFS, and the classic surprise: setting OFS alone changes nothing, because awk only rebuilds $0 if you modify a field. The idiom is awk -F, '{$1=$1; print}' OFS='\t' — assigning a field to itself forces the rebuild.

Same formatting as C, and the reason your numbers stop looking like 1.0000000001e+09. awk '{printf "%-20s %8.2f\n", $1, $2}' gives you aligned columns without a spreadsheet.

Unquoted, the shell splits the value on whitespace and expands globs in it. So rm $file on a file named my report.txt deletes two things that do not exist, and a value of * means something much worse. Quote every expansion unless you have a specific reason not to.

"$@" expands to each argument as its own word — the only correct way to forward arguments. "$*" joins them into one string, and bare $@ re-splits them. If a wrapper script mangles filenames with spaces, this is why.

[[ ]] is a bash keyword rather than a command: it does not word-split its operands, supports &&, ||, and gives you =~ for regex. [ ] is old test and needs paranoid quoting. Use [[ ]] in bash; use [ ] only when POSIX sh is genuinely a requirement.

Nests cleanly, does not mangle backslashes, and is readable. Backticks inside backticks require escaping that nobody enjoys. There is no case where backticks are better.

Defaults and trimming without calling out to another process. ${v:-default} if unset, ${v:?message} to fail loudly, ${f%.txt} strips a suffix, ${f##*/} strips everything up to the last slash — basename without the fork.

Real arrays exist; use them instead of a space-separated string. "${arr[@]}" expands element-per-word, ${#arr[@]} is the length. Building command arguments in a string and hoping the splitting works out is how spaces in paths become an outage.

Without -r, read treats backslashes as escapes and quietly eats them. There is essentially no situation where you want that. while IFS= read -r line; do …; done < file is the loop — IFS= also preserves leading and trailing whitespace.

It breaks on spaces, newlines and glob characters in filenames, and it is unnecessary. Use a glob (for f in ./*.txt) or, for a tree, find … -print0 | xargs -0, which is safe against every filename a filesystem allows.

-e exit on error, -u error on undefined variable, -o pipefail make a pipeline fail if any stage failed. Worth having in every script. It is not a safety net so much as a smoke alarm with known blind spots — listed below.

set -e is ignored for any command in a condition — the whole left side of && or ||, anything in if, while, or after !. So if my_broken_function; then will not exit no matter what happens inside. This is the single biggest gap between what people think -e does and what it does.

Without it, a pipeline reports only the last command's status. curl bad-url | tee out.txt exits 0 because tee was fine, and you happily process an empty file. With pipefail, the failure surfaces.

Under set -e, a grep that legitimately finds nothing returns 1 and terminates everything. Same for diff when files differ. Neutralise deliberately: count=$(grep -c x f || true).

trap 'rm -rf "$tmp"' EXIT runs on normal exit and on error, so your temp directory disappears even when the script dies at line 40. Pair with tmp=$(mktemp -d) and never invent your own /tmp/myscript.$$ path again.

A linter that catches unquoted expansions, useless cat, set -e misunderstandings and a hundred portability bugs. It is the highest-value tool in this entire guide, and it takes one command.

find . -name "*.log" -print0 | xargs -0 rm. The -print0/-0 pair uses NUL as the separator — the one byte that cannot appear in a filename — so spaces and newlines stop mattering. -P4 also gives you free parallelism.

Feeds a command's output where a filename is expected: diff <(sort a) <(sort b) compares two things that were never files. Bash only — it will not work under sh.

Unquoted <<EOF expands variables and command substitutions inside the body; quoted <<'EOF' passes it through literally. If you are writing a script that contains $ or backticks, you almost certainly want the quoted form.
They do not nest. A heredoc inside another heredoc ends at the first terminator it sees, and the outer script runs truncated — silently. I hit this writing this very site.

Split a stream: write to a file and keep it flowing down the pipe. … | tee build.log | grep -i error keeps the full record while you watch only the bad news. tee -a appends; sudo tee is the standard way to write a root-owned file from a non-root pipeline.

Locale changes how sort orders and how character classes match, so the same script produces different output on two machines. For anything mechanical — deduping, comparing, hashing sorted output — pin it: LC_ALL=C sort. Also noticeably faster.

A pipeline should become a script when you need a second look at it. A script should become a program when it grows a data structure, needs tests, or handles errors in more than one way.
The tell is editing: if changing the behaviour means rebuilding the pipe from scratch rather than editing a line, it stopped being a pipeline a while ago.

Parsing an HTML glossary, I matched class="entry" — and silently missed class="entry buzz" and class="entry faq". It returned 43 of 52 entries with no error at all. The only reason I caught it was checking the count against a number I already knew.

Verifying that same job, grep -c told me 62 articles existed where there were 52 — because minified HTML put several matches on one line. Both of my verification numbers were wrong in different directions on the same afternoon.

A <<'PY' python heredoc inside a <<'EOF' shell heredoc: the outer one terminated early, python received a truncated script, and the error pointed at a line that was fine.

On Windows, Git Bash rewrites arguments that look like Unix paths into Windows paths — so curl .../posts became C:/Program Files/Git/posts. Set MSYS_NO_PATHCONV=1. Then remember to unset it for native Windows binaries that want Windows paths, which is its own small hell.

cmd | head then echo $? reports head's status, not cmd's. I read an exit code of 0 as "the clone worked" when the clone had failed and head had merely succeeded at reading nothing. Use PIPESTATUS, or set -o pipefail.

None of these threw an error. Each returned a plausible number or an empty result, and every one was caught by checking against a figure known independently — a stated total, a count from another method, a file that should have existed.
Verify the effect, not the exit status.