Lens is the photojournalism blog of The New York Times, presenting the finest and most interesting visual and multimedia reporting — photographs, videos and slide shows.

Things were so simple in the old days, when we had a single server acting as web-/application-server at the same time. In order to see what was going on, one would just “tail” the servers log-file (for example Apache’s access_log and/or error_log) to get live information about the servers activity.

Things are not as easy these days. For one of my work projects only god knows how many individual servers are involved: web-server, application server, rendering server, conversion server, etc. To see activity on all systems involved, developers usually have multiple Terminal windows open, are logged in to multiple servers and “tail -f” logfiles in those windows. While this allows people to get a live view into the system, it makes it very difficult to correlate log-events on one system to log-events on another system. The logs are tailed independently and the information is not “interleaved”.

There are existing utilities out there that allow you to look at multiple files at the same time (most notably http://www.vanheusden.com/multitail/), but as far as I know, there is no platform-independent one.

Solution

I created an AIR application that allows connecting to multiple servers at the same time, listening on multiple log-files (local or remote) at the same time, color-coding information from individual log-files and interleaving the information into a single log-display.

The solution consists of two pieces: the multitail-server and the AIR client. You can skip the portion about the “Multitail server” if you only want to listen on local log files

Multitail server

The first step is to identify all hosts that have valuable log-information and to expose all those logs via a tcp-socket. On a typical modern Linux system, I made the following modifications:

1.) /etc/services

I added a new service called “multitail” to the /etc/services file using a non-privileged, unused port number:

multitail     20000/tcp

2.) xinetd

Our Linux servers run xinetd and I added the following file as /etc/xinted.d/multitail:

service multitail
{

        socket_type     = stream
        wait            = no
        user            = root
        server          = /usr/local/bin/multitail

}

Unix systems running the traditional inetd can also be modified by adding a line to /etc/inetd.conf (not shown here).

3.) multitail server

After that I created the server-program (/usr/local/bin/multitail in this case) that would be executed whenever a client makes a connection to port 20000 on the particular system:

#!/bin/sh
echo "logfile=app:/var/log/jboss/server.log"
echo "logfile=web:/var/log/apache/access_log"
echo "logfile=err:/var/log/apache/error_log"
tail -f /var/log/jboss/server.log /var/log/apache/access_log /var/log/apache/error_log

The “logfile” lines announce which log-files are being served by this instance of multitail. “app”, “web” and “err” are tags that allow the AIR application to classify the log-information that’s coming from this server instance.

After creating the /usr/local/bin/multitail file, make sure that it is executable by using “chmod +x” on it.

4.) SIGHUP

Once the above steps are complete, make sure you send the xinetd (or inetd) process a SIGHUP (“kill -HUP …”) signal to make it reload it’s configuration and recognize the new server process.

AIR client

The AIR client will establish a connection to the above multitail server and will parse information from the server process. In order to know which systems to connect to, an XML configuration file is used. The XML file is called “airlog.xml” and is located in the same directory where you installed the airlog-application. Let’s look at a simple example:

<?xml version="1.0" encoding="UTF-8" ?>
<connections>
 <connection name="staging">
 	<host name="staging.host.com" port="20000"/>
 </connection>
</connections>

This XML-code creates a single connection called “staging”. In the AIR application UI you will find a Combo-box that lists all the defined connections and allows you to connect to one of them at a time. Based on the above XML file, you would find one entry in the Combo-box called “staging”. Upon selecting the entry and clicking on “connect”, the AIR application will make a Socket connection to name:port. It will read the first few “logfile=…” lines from the server and create an internal list of all log-files exposed by the server.

A second configuration file called “colors.xml” associates tags (lines from a certain logfile) with specific colors. Here’s an example “colors.xml” file:

<?xml version="1.0" encoding="UTF-8" ?>
 <colors>
 	<color name="app" value="#880000"/>
 	<color name="web" value="#000088"/>
 	<color name="err" value="#008888"/>
 </colors>

As log-data arrives from the server, information is color-coded (using the colors-tag) and output in the Text-area that represents the combined log.
If multiple log-servers are involved for “staging”, then multiple host-entries can be given:

<connection name="staging">
 	<host name="staging1.host.com" port="20000"/>
 	<host name="staging2.host.com" port="20000"/>
 	<host name="staging3.host.com" port="20000"/>
 </connection>

And of course, one can have multiple connection entries in the XML file as well:

<connection name="local">
 	<host name="localhost" port="20000"/>
 </connection>
 <connection name="staging">
 	<host name="staging1.host.com" port="20000"/>
 	<host name="staging2.host.com" port="20000"/>
 	<host name="staging3.host.com" port="20000"/>
 </connection>
 <connection name="deployment">
 	<host name="deploy1.host.com" port="20000"/>
 	<host name="deploy2.host.com" port="20000"/>
 	<host name="deploy3.host.com" port="20000"/>
 	<host name="deploy4.host.com" port="20000"/>
 	<host name="deploy5.host.com" port="20000"/>
 </connection>

The above XML file would allow people to connect to three different environment called “local”, “staging” and “deployment”.

Especially for development systems, those log-files are usually not on remote systems, but are located on the developers machine directly. To avoid having to install an xinetd/inetd environment, the AIR application also knows how to listen on local filesystem resources. Consider this case:

<connection name="development">
 	<host name="authentication.server.com" port="20000"/>
 	<file name="app" location="c:tmpserver.log"/>
 	<file name="web" location="c:apachelogsaccess_log"/>
 </connection>

This XML configuration declares one remote server for authentication logging information and two local files (in this case a Windows system) which are color-coded with the “app” and “web” tags.
Again, information from all three log-sources would be interleaved into one log-display.

History and Follow

In the AIR UI you will find a history-popup. This popup allows you to specify how many log-lines you want to keep in memory. The more lines, the higher the applications memory requirements. A value of 1000 is a good tradeoff between memory-consumption and time-span covered by the history items (that is unless one of your servers creates 200-lines stacktraces every few seconds).

The “follow” option allows you to always scroll to the bottom in the Text-area when new log-lines arrive (very much like “tail -f” in a Terminal window).

Enabling/Disabling “tags”

As soon as log-data arrives in your AIR client, you will see a number of checkboxes on the right-hand side with all the tags announced via “logfile=xxxx:” or specified in name for file-entries. Sometimes you may want to concentrate only on a subset of the log-information that’s available to you. By deselecting a checkbox, you can remove the items associated with that tag from the log display. Information is still recorded, however it is suppressed as long as the checkbox is deselected.

Next to the checkbox for each tag, you’ll also find a color-picker that allows you to change colors for tags on the fly. The same applies to the background color of the Text-area. If the application detects a color change you’ll also get option to save out the new color values.

Filtering

The “filter” textinput on the top allows you to narrow down information from all log files. Entering text in that textinput-field will only show log-lines from all log-files that contain the case-sensitive text entered there. Entering for example “Exception” in the textinput-field will only show all lines that contain the word “Exception”.
The AIR client also supports connection specific auto-filtering. Consider the following XML configuration fragment:

<connection name="test">
 	<host name="a.host.com" port="20000"/>
 	<filters>
 		<filter name="file_not_found" expression="File not found: (.*)"/>
 	</filters>
 </connection>

If the AIR client encounters one or more connection specific filters, it will test the regular expression “expression” against all log-lines and will turn all items in capture-groups (those within parenthesis) into hyper-links. Clicking on a hyperlink in the log display will only show lines that contain that specific search term.

Download

The AIRlog application requires the latest AIR-runtime from Adobe Systems. Click on the links below to download the requires pieces and the AIRLog application as well.

I would give my right leg to get an opportunity to stand behind the camera in this shot … (click on the image above to see the full panorama on NASA’s site).

I’m not surprised to hear that the photo below won: