How to apply a filter to real time output of `tail -f `?

Solution 1:

With Unix you can pipe the output of one program into another.

So to filter tail, you can use grep:

tail -f path | grep your-search-filter

Solution 2:

Short answer: tail -f somefile | grep somepattern

However, this tends to fall short. Let's say you're tailing a file that gets rotated often (if its a debug log, it might be rotated multiple times). In that case tail -F is your friend. I'll let you look up the difference.

But tail -f and tail -F print out a bunch of lines first, which is often undesirable in this use-case, so in this case add -n0

tail -F -n0 somefile | grep somepattern

That will be fine, up until you want to do some other filtering, and then you need to beware of buffering. stdout is line-buffered by default when writing to a terminal but when fully-buffered when writing to a pipe. So the following will emit lines as soon as they are found, because tail is explicitly line-buffered (or it flushes its output at the end of each line), and grep is also line-buffered because its output is going to your terminal:

tail -F -n0 somefile | grep somepattern

But then you decide to use something like awk or cut to further process the output.

tail -F -n0 somefile | grep somepattern | awk '{print $3}'

And now you wonder where your output has gone... depending on the volume of logs, you may find that you do get output, but it will be a page at a time because now the stdout of grep is operating in fully-buffered fashion, and so awk receives it input 4kB at a time (by default).

In this case, you can tell grep to always make stdout line buffered by using the --line-buffered option.

tail -F -n0 somefile | grep --line-buffered somepattern | ...

However, most commands don't have an analogue of --line-buffered. In the case of more scriptable tools, you can use a function to flush the output (eg. in awk, the function is fflush(), which shares the same name as its C counterpart, tools like Perl and Python have something similar).

With the likes of cut you're likely out of luck; ... but you might try searching for unbuffer, which is I think something provided by the expect toolchain (I've never used it).

I hope you've found this useful.

Cheers, Cameron

---

Solution 3:

and you can use multiple pipes and greps, and exclude things with grep -v, get case insensitivity with grep -i, etc.

i.e.: tail -100f /var/log/messages | grep -V ACPI | grep -i ata

start tailing 100 lines from the end, and keep tailing, first exclude any lines with ACPI, then show lines with ata, ATA, or any mix of those.

Another handy one is the ABC options, for the lines After, Before, and Context (lines before and after).