Tailing log files on multiple remote servers

Recently, our EC2 infrastructure went down which aggregates log from our multiple servers. I wanted to investigate a production issue which involved looking up logs of multiple servers simultaneously.

I tried different methods such as tmux, multitail and ssh command such as

ssh HOST_1 "tail -f /path/to/file" | tee -a /path/to/merged/contents

ssh HOST_2 "tail -f /path/to/file" | tee -a /path/to/merged/contents

tail -f /path/to/merged/contents

All these are nice solutions, but none of them is elegant. It involves setting up multiple terminal sessions just to start the monitoring.

There is also this open source project which tries to solves this problem but don’t think does a good job at it.

However, the best solution comes in the form of using Fabric

Fabric is a high level Python (2.7, 3.4+) library designed to execute shell commands remotely over SSH, yielding useful Python objects in return. It builds on top of Invoke (subprocess command execution and command-line features) and Paramiko (SSH protocol implementation), extending their APIs to complement one another and provide additional functionality.

Once, Fab is installed using pip install fabric , the multiple logs can be tailed as follows:

fab -P -u username  --linewise -H x.x.x.x,y.y.y.y  -- tail -f /path/file.log
fab command -i /path/to/key.pem [-H [user@]host[:port]] -- tail -f /path/file.log

Also, what I like to do is instead of typing this log command, just alias it into my ~/.zshrc

alias catalinadev='fab -P -u username  --linewise -H x.x.x.x,y.y.y.y  -- tail -f /path/file.log'

Thus, I can just type catalinadev anytime and start tailing my Tomcat catalina log files from dev environment.

Here is the output in action, notice how the IP address on the left handside tells us the machine from which the log line was fetched:

Open in fullscreen