Reading and Searching Files

Back to the practice folder

cd ~/linux-essentials

Every command below assumes you are sitting inside this folder. Before you search for anything, you need something to search. Create a small sample log file now:

cat > sample.log << 'EOF'
2024-01-10 08:12:01 INFO Starting service
2024-01-10 08:14:33 WARN Disk usage at 85%
2024-01-10 08:17:02 Failed password for root from 203.0.113.9
2024-01-10 08:17:05 Failed password for root from 203.0.113.9
2024-01-10 08:17:09 Failed password for admin from 203.0.113.9
2024-01-10 08:22:14 INFO Backup completed
2024-01-10 08:30:00 ERROR Disk write failure on /dev/sdb1
EOF

That file now sits in ~/linux-essentials/sample.log. It is small enough to read in one glance, but it is enough to make every command below runnable against something real, instead of something imaginary.

cat, less, and more: reading a file

For a short file, cat dumps the whole thing straight to your terminal:

cat sample.log

That works fine for seven lines. It stops working the moment a file is thousands of lines long, a real /var/log/auth.log on a busy server, for instance, scrolling past faster than you can read it. For anything that long, page through it instead with less:

less sample.log

less opens the file one screen at a time. Use the arrow keys or Page Up/Page Down to move around, and press q to quit back to your prompt. You may also see more used the same way; it is an older, more limited paging tool, and less is the one worth defaulting to.

head and tail: just the edges

Often you only care about one end of a file. head shows the first lines:

head sample.log

tail shows the last lines:

tail sample.log

Both default to ten lines, and both accept -n to change that:

head -n 3 sample.log
tail -n 3 sample.log

There is one more form of tail worth knowing even though you will only read about it here rather than run it live: tail -f sample.log follows a file, printing new lines as they are appended, instead of exiting once it reaches the end. That is the exact pattern for watching a log file update in real time, for example keeping an eye on an authentication log while an incident is actively unfolding, without having to keep re-running tail by hand.

grep: finding things inside files

This is the command that does the actual searching, and it is the one most worth being fluent in. In its plainest form, grep prints every line containing a given substring:

grep "Failed password" sample.log

That returns the three lines where a login attempt failed, and nothing else. This is, almost word for word, what a defender does with a real system: auth.log and syslog are exactly the kind of files you search this way for signs of a brute-force attack, a string like “Failed password” repeated many times in a short window, or for a known indicator of compromise, a filename, hash fragment, or IP address you were told to look for across a whole directory of logs.

A few flags make grep considerably more useful:

-i makes the search case-insensitive, so it matches ERROR, Error, and error alike:

grep -i "error" sample.log

-n prints the line number alongside each match, so you know exactly where in the file it sits:

grep -n "Failed password" sample.log

-r searches recursively through every file in a directory, which is what turns grep from a single-file tool into something you point at an entire folder of logs at once:

mkdir -p logs
cp sample.log logs/auth.log
grep -r "Failed password" logs

-v inverts the match, printing every line that does not contain the pattern, useful for filtering out noise you already understand so only the unfamiliar lines remain:

grep -v "INFO" sample.log

Pipes and redirection: chaining commands together

The pipe character | takes the output of one command and feeds it directly into the next, instead of printing it to your screen first. You can chain cat and grep together, for instance:

cat sample.log | grep ERROR

That particular example is a little redundant, since grep ERROR sample.log does the same thing on its own, but the pattern matters more than this one case: pipes let you compose small, single-purpose commands into a longer pipeline. A common defender move is counting matches rather than reading them all, by piping into wc -l (word count, with -l for lines):

grep "Failed password" sample.log | wc -l

That single pipeline answers “how many failed login attempts are in this file” with one number, which scales to a multi-megabyte auth.log just as well as it does to your seven-line sample.

Redirection sends output to a file instead of the screen. > creates or overwrites a file:

grep "Failed password" sample.log > failed-logins.txt
cat failed-logins.txt

>> appends to a file instead of overwriting it:

echo "Reviewed by analyst on duty" >> failed-logins.txt
cat failed-logins.txt

Both commands stay inside ~/linux-essentials, so running them again, or getting the pattern wrong, costs you nothing.

What to notice

• cat is for short files you want to see all at once; less is for anything long enough to need paging, and q is how you get back out.
• grep, with -i, -n, -r, and -v, is the single most useful search tool in this lesson, and it is the literal command you reach for when searching auth.log or syslog for brute-force attempts or a known indicator of compromise.
• Pipes and redirection let you chain commands like grep and wc -l together, turning a pile of log lines into a single, specific answer.