Retrospective 6.0 introduces the Retrospective Query Language RQL

Wading your way through massive volumes of log data just became a lot easier in Retrospective 6. The new Retrospective Query Language RQL offers a precise and easy way to find what you’re looking for. RQL is a query language where a query consists of one or multiple search terms and groups. Terms and groups can be combined with Boolean operators to form a more complex query.

RQL Queries explained

Retrospective offers a two staged approach to filter your data. For the first stage you enter your rough filter criteria using simple text or regex expressions. In this first stage, Retrospective collects data via SSH from remote hosts and via command line tools from remote docker containers and eventually stores large data on the local disk. In cases of fetching data via SSH, Retrospective optimizes your search, if possible on the server side, so that only data matching your rough filter criteria is travelling over the wire to your machine.

In this example we use a Retrospective profile using log data from a bunch of test docker containers, whilst applying three rough filter criteria to the data stream:

In Retrospective there are two types of fetch processes, the “Search” for one time data fetching, and the “Monitor” that fetches continuously as data is produced by the source. To get data for the RQL queries, we let the “Monitor” run for a while:

The data emitted from our test docker containers is invented just for testing Retrospective features, as we cannot show real life log data from ourselves or our customers for instructional purposes.

When using Retrospective you will end up with large amounts of data, which Retrospective stores in memory and on a local H2 database on your hard disk. At this point, in the the second stage of local filtering most users want to further refine their view on this data using Retrospective’s local filter and this is where the new Retrospective Query Language RQL comes into play.

In the preferences, the default setting for the local filter is now RQL:

You can choose whether the local filter acts immediately when typing or, in a more controlled fashion, only when the enter button is hit.

As usual in Retrospective application the round information buttons provide online documentation for this feature:

Additionally, the default mode for hiding or justing coloring out the locally filtered items can be set:

In the following first RQL example, I changed the setting in the preferences to not hide the filtered-out entries, so my matches are highlighted in green. The hiding of non-matching entries can be toggled with the filter icon on the right of the local filter box. The icon changes when clicked and now shows a red circle when hiding out non-matching is off.

After having fetched data for 16 minutes, I stopped the “Monitor” and Retrospective shows in the left of the bottom status area that is has collected 341 kilobytes of data from 2 docker containers, which will be enough for our RQL examples.

The RQL term in the local filter box filters for log entries with transaction dates starting with the year 2020:

After clicking the filter icon again, only the matching entries are visible in the result pane and the red circle disappears from the icon to show that hiding out non-matching is on:

You may wonder why I enclosed my RQL search term above with double quotes. I had to do this because the colon is a reserved character in RQL. The colon can be used to combine RQL with the custom columns feature of Retrospective.

There are the three custom columns “Index”, “Name” and “IP” configured in the Retrospective profile for these docker containers. Also, the default columns “Host” and “Path” can be used for RQL terms. In the next RQL term, I have filtered for entries that contain a “Name” and were produced by the “whale” docker container as indicated in the “Path” column:

For an easy access to using the column in the Retrospective profile as RQL search terms, when typing the colon “:”, then the column can be selected from a drop-down list. Here I choose to filter for entries having a “Name” set and the “Index” column value containing “118”:

A refined search result:

If no column prefix with a “:” character is given, then Retrospective applies any RQL search term to the default column “Data”, that always contains all data of the log entry. In the last example to show the grouping capabilities in RQL, I added some RQL search terms that match in the “Data” column. The screenshot shows the result of the filtering and also the configurable highlighting the result details pane of a single log entry:

The Retrospective Query Language can help you to create a snapshot of your data that contains precisely what you are looking for. All of its features are described in detail in chapter 6 of the Retrospective user manual.

Log Time Synchronization

If you give our updated user manual a look, then maybe also check the new chapter 7 about Log Time Synchronization. For our customers with log data stemming from SSH hosts, VM’s and containers located in different time zones, we have built new features for time zone fallback and time offset adjustments into Retrospective 6.0.

Updated JRE and Eclipse target platform

Last but not least, Retrospective was already upgraded from Java 11 to Java 17 in the previous release, so for release 6.0 we have packaged a newer Java 17 JRE to ensure a bug free and secure operation. Also, the Eclipse target platform was upgraded to the current one, to continue with a flawless GUI experience on Windows, Mac and Linux machines.

Give it a try

We hope that our newest Retrospective release will proof valuable to all our current users. If you have never tried the Retrospective app, why not find out if ad-hoc data mangling on your laptop might be practical solution for you - instead of going down the road of setting up or extending a large-scale collector-agent and indexing infrastructure.