Things were so simple in the old days, when we had a single server acting as web-/application-server at the same time. In order to see what was going on, one would just “tail” the servers log-file (for example Apache’s access_log and/or error_log) to get live information about the servers activity.

Things are not as easy these days. For one of my work projects only god knows how many individual servers are involved: web-server, application server, rendering server, conversion server, etc. To see activity on all systems involved, developers usually have multiple Terminal windows open, are logged in to multiple servers and “tail -f” logfiles in those windows. While this allows people to get a live view into the system, it makes it very difficult to correlate log-events on one system to log-events on another system. The logs are tailed independently and the information is not “interleaved”.

There are existing utilities out there that allow you to look at multiple files at the same time (most notably http://www.vanheusden.com/multitail/), but as far as I know, there is no platform-independent one.

Solution

I created an AIR application that allows connecting to multiple servers at the same time, listening on multiple log-files (local or remote) at the same time, color-coding information from individual log-files and interleaving the information into a single log-display.

The solution consists of two pieces: the multitail-server and the AIR client. You can skip the portion about the “Multitail server” if you only want to listen on local log files

Multitail server

The first step is to identify all hosts that have valuable log-information and to expose all those logs via a tcp-socket. On a typical modern Linux system, I made the following modifications:

1.) /etc/services

I added a new service called “multitail” to the /etc/services file using a non-privileged, unused port number:

multitail     20000/tcp

2.) xinetd

Our Linux servers run xinetd and I added the following file as /etc/xinted.d/multitail:

service multitail
{
        socket_type     = stream
        wait            = no
        user            = root
        server          = /usr/local/bin/multitail
}

Unix systems running the traditional inetd can also be modified by adding a line to /etc/inetd.conf (not shown here).

3.) multitail server

After that I created the server-program (/usr/local/bin/multitail in this case) that would be executed whenever a client makes a connection to port 20000 on the particular system:

#!/bin/sh
echo "logfile=app:/var/log/jboss/server.log"
echo "logfile=web:/var/log/apache/access_log"
echo "logfile=err:/var/log/apache/error_log"
tail -f /var/log/jboss/server.log /var/log/apache/access_log /var/log/apache/error_log

The “logfile” lines announce which log-files are being served by this instance of multitail. “app”, “web” and “err” are tags that allow the AIR application to classify the log-information that’s coming from this server instance.

After creating the /usr/local/bin/multitail file, make sure that it is executable by using “chmod +x” on it.

4.) SIGHUP

Once the above steps are complete, make sure you send the xinetd (or inetd) process a SIGHUP (“kill -HUP …”) signal to make it reload it’s configuration and recognize the new server process.

AIR client

The AIR client will establish a connection to the above multitail server and will parse information from the server process. In order to know which systems to connect to, an XML configuration file is used. The XML file is called “airlog.xml” and is located in the same directory where you installed the airlog-application. Let’s look at a simple example:

<?xml version="1.0" encoding="UTF-8" ?>
<connections>
 <connection name="staging">
 	<host name="staging.host.com" port="20000"/>
 </connection>
</connections>

This XML-code creates a single connection called “staging”. In the AIR application UI you will find a Combo-box that lists all the defined connections and allows you to connect to one of them at a time. Based on the above XML file, you would find one entry in the Combo-box called “staging”. Upon selecting the entry and clicking on “connect”, the AIR application will make a Socket connection to name:port. It will read the first few “logfile=…” lines from the server and create an internal list of all log-files exposed by the server.

A second configuration file called “colors.xml” associates tags (lines from a certain logfile) with specific colors. Here’s an example “colors.xml” file:

<?xml version="1.0" encoding="UTF-8" ?>
 <colors>
 	<color name="app" value="#880000"/>
 	<color name="web" value="#000088"/>
 	<color name="err" value="#008888"/>
 </colors>

As log-data arrives from the server, information is color-coded (using the colors-tag) and output in the Text-area that represents the combined log.
If multiple log-servers are involved for “staging”, then multiple host-entries can be given:

<connection name="staging">
 	<host name="staging1.host.com" port="20000"/>
 	<host name="staging2.host.com" port="20000"/>
 	<host name="staging3.host.com" port="20000"/>
 </connection>

And of course, one can have multiple connection entries in the XML file as well:

<connection name="local">
 	<host name="localhost" port="20000"/>
 </connection>
 <connection name="staging">
 	<host name="staging1.host.com" port="20000"/>
 	<host name="staging2.host.com" port="20000"/>
 	<host name="staging3.host.com" port="20000"/>
 </connection>
 <connection name="deployment">
 	<host name="deploy1.host.com" port="20000"/>
 	<host name="deploy2.host.com" port="20000"/>
 	<host name="deploy3.host.com" port="20000"/>
 	<host name="deploy4.host.com" port="20000"/>
 	<host name="deploy5.host.com" port="20000"/>
 </connection>

The above XML file would allow people to connect to three different environment called “local”, “staging” and “deployment”.

Especially for development systems, those log-files are usually not on remote systems, but are located on the developers machine directly. To avoid having to install an xinetd/inetd environment, the AIR application also knows how to listen on local filesystem resources. Consider this case:

<connection name="development">
 	<host name="authentication.server.com" port="20000"/>
 	<file name="app" location="c:tmpserver.log"/>
 	<file name="web" location="c:apachelogsaccess_log"/>
 </connection>

This XML configuration declares one remote server for authentication logging information and two local files (in this case a Windows system) which are color-coded with the “app” and “web” tags.
Again, information from all three log-sources would be interleaved into one log-display.

History and Follow

In the AIR UI you will find a history-popup. This popup allows you to specify how many log-lines you want to keep in memory. The more lines, the higher the applications memory requirements. A value of 1000 is a good tradeoff between memory-consumption and time-span covered by the history items (that is unless one of your servers creates 200-lines stacktraces every few seconds).

The “follow” option allows you to always scroll to the bottom in the Text-area when new log-lines arrive (very much like “tail -f” in a Terminal window).

Enabling/Disabling “tags”

As soon as log-data arrives in your AIR client, you will see a number of checkboxes on the right-hand side with all the tags announced via “logfile=xxxx:” or specified in name for file-entries. Sometimes you may want to concentrate only on a subset of the log-information that’s available to you. By deselecting a checkbox, you can remove the items associated with that tag from the log display. Information is still recorded, however it is suppressed as long as the checkbox is deselected.

Next to the checkbox for each tag, you’ll also find a color-picker that allows you to change colors for tags on the fly. The same applies to the background color of the Text-area. If the application detects a color change you’ll also get option to save out the new color values.

Filtering

The “filter” textinput on the top allows you to narrow down information from all log files. Entering text in that textinput-field will only show log-lines from all log-files that contain the case-sensitive text entered there. Entering for example “Exception” in the textinput-field will only show all lines that contain the word “Exception”.
The AIR client also supports connection specific auto-filtering. Consider the following XML configuration fragment:

<connection name="test">
 	<host name="a.host.com" port="20000"/>
 	<filters>
 		<filter name="file_not_found" expression="File not found: (.*)"/>
 	</filters>
 </connection>

If the AIR client encounters one or more connection specific filters, it will test the regular expression “expression” against all log-lines and will turn all items in capture-groups (those within parenthesis) into hyper-links. Clicking on a hyperlink in the log display will only show lines that contain that specific search term.

Download

The AIRlog application requires the latest AIR-runtime from Adobe Systems. Click on the links below to download the requires pieces and the AIRLog application as well.

To enable it, you would either modify your Flex projects’ compiler settings:

or you would use “-link-report output.xml” as an option on the mxmlc command line. In either case, an XML file with all definitions will be created after the compiler successfully built the SWF-file.

The contents of the link report file look like this:

Hmmm – not very easy to digest by just reading the file in a text-editor. Theo Hultberg over at iconara.net created an XSL-script that transforms link-reports into human-readable HTML. While this is interesting, it did not give me the “big picture” for all the pieces that go into my applications.

I created an AIR application which can read, digest and visualize link-report files.

After installing the LinkReportAIR application, you can use the “Browse …” button to select a link-report XML file on your system or alternatively just drag and drop an XML-file on the AIR application itself.

The DataGrid on the left lists all symbols included in the report (under “id”), the modification date for the symbol (if available), the package it is included in, the size of the symbol and the optimized size (if available). You can use the DataGrid headers to sort the symbols using any of the columns.
All symbols that do not have a package associated with them are automatically assigned to a package called “global”. Any symbol that ends in “_properties” is assigned to a package called “properties”.

If you click on any of the symbols in the DataGrid, the bottom portion will show you where this symbol comes from (this can be a SWC or any of your sources) and whether is has “prerequisites” or “dependencies” associated with it (the livedocs link above has more details about these):

The neatest feature of LinkReportAIR is the little treemap (using ILOG Elixir) that is automatically created from the data in the link report. It tries to visualize the space consumption per package. In the screenshot below, I’m hovering over the “mx.controls” package (and all packages included in it) and LinkReportAIR tells me that the (unoptimized) size of everything contained under mx.controls.* is roughly 500 KBytes and that this equals about 25% of my applications total size.

Double-clicking on a package in the tree-map will “drill down” and allows you to focus on certain portions of the package tree. While “drilled down”, the DataGrid on the left will also be filtered to only show symbols in the current drilled down subtree. To get back to the entire view, you would double-click again on the subtree-symbol that is currently in focus (sounds complicated, but you’ll get it once you use it).

And finally the “External Symbols” tab will show all symbols that are externally required (like the stuff that’s exported from the Flash Player itself). That list will grow if you decide to link the Flex Framework externally via Runtime Shared Libraries (RSLs):

You can download and install the AIR application using the badge below. If you need a sample link-report file to play with, then use the one that I generated from the application itself: link-report.xml (use “Save Link As …” to save the report locally)