Monitoring camel routes using camel-bam

Last week I got a question from an Apache Camel user about ways to get statistics on the activity on a route, including errors. There are quite a few ways to achieve that. First there are obviously the logs, but they are mostly useful to investigate a particular problem. Then there are the performance counters (defined in the org.apache.camel.api.management package) that could be obtained either via the JMX support or even exposed as a RESTful service. If one wants to be more fancy she could use a custom ErrorHandler.

There are however situations when message processing is faulty even when all individual messages are processed correctly. As an example, messages not processed in time can lead to service level agreement violations, or messages may be missing, or other business process execution use cases. While not pretending to compete with more specialized tools for business processes (such as BPEL or BPM engines) and while being inherently stateless, camel does offer support for scenarios like the ones described above, via the camel-bam component.

The BAM component is actually one of the rare exceptions that implements a stateful engine using JPA and a database of your choice. It also offers a very neat way to correlate messages by using Expressions (typical camel style) that evaluates against a message to produce a value used for correlation. As such, messages don't necessarily need to have an explicit correlation identifier as long as one can be computed from the content of the message, which means that it can normally be retrofitted to support systems not designed with correlation in mind. At its core camel-bam provides a set of temporal rules that trigger events. These events are processed by camel via processes (called activities) that are nothing else but specialized routes. BAM also offers a dsl for building activities via a ProcessBuilder (which is actually a specialization of the RouteBuilder you may be more familiar with).

That said, I put together a small example in my github account. It illustrates some very basic banking operations (like debits and credits from an account). The relevant part of the process is in ActivityMonitoringBuilder.configure() and looks something like this:

  ActivityBuilder teller = activity("direct:monitor-teller")
    .name("teller").correlate(xpath("/operation/@account"));
  ActivityBuilder office = activity("direct:monitor-office")
    .name("office").correlate(xpath("/operation/@account"));
  ActivityBuilder errors = activity("direct:monitor-errors")
    .name("errors").correlate(header("SampleRef"));
        
  office
    .starts().after(teller.completes())
    .expectWithin(Time.seconds(1))
    .errorIfOver(Time.seconds(1)).to("mock:overdue");
        
  errors
    .starts().after(teller.completes())
    .expectWithin(Time.seconds(1))
    .errorIfOver(Time.seconds(5)).to("mock:error");



The trigger events are generated when regular message flow down the routes defined in the BusinessProcessBuilder.

If you have more questions about how this example works (or bam in general) drop me a note or ask on the camel users@ lists. Enjoy!



Apache Rave a new ASF Top Level Project

After one year of hard work, Apache Rave graduated from incubation. Although the resolution was approved by the ASF board last week, the official announcement came only this morning (also featured on NASDAQ GlobeNewswire).

First initiated by Ate Douma and discussed at ApacheCon 2010 in Atlanta, the idea quickly captured the imagination of a group of developers and the proposal was submitted in February 2011. After a bit more then a year and a few releases, Rave is a promising top level project.

Dubbed a "web and social mashup engine", Rave is a powerful yet lightweight, secure, customizable platform for services and content delivery also supporting a number of social network technologies. Due to its content aggregation capabilities via specs like OpenSocial or W3C Widgets, Rave is already adopted by a number of portal projects. As content aggregation happens in the browser, many of the issues of traditional portlet based technologies are avoided.

For those those interested in a versatile platform for content delivery, check out Rave. Congrats to the Rave team for reaching this milestone and good luck going forward!

How to protect the release GPG key

Recently I have been asked about how I handle the gpg key I use for Apache releases. For what is worth, the question popped up in the context of a few other community members taking on the release manager role. As those of you that follow Apache Camel already know the Camel PMC decided to actively support and issue patch releases for the two latest minor branches.

But I digress. As I mentioned, I don't keep my private key on my laptop, but on an encrypted usb flash disk. The main reason is security, as the probability of someone getting access to my box greater than zero. In particular the key used for Apache releases is trusted by other ASF members and making sure it doesn't get compromised is one of the duties of the release manager. Of course one could revoke a key, but then verifying the integrity of a release becomes complicated at best.

My setup works on Ubuntu 11.10 and the idea behind it was using something similar to 2-factor authentication (something I have and something I know). I use a relatively cheap 4G usb flash memory (a sturdier, metal one). My usb flash uses a FAT32 file system and is not encrypted, but on it I created a 256M file (ringo.img) as an encrypted ext3 disk partition. My usb flash is mounted as '/media/APACHE', not much to do about the uppercase, a perk of FAT32. The encryption uses 256-bit aes and sha512. Below are the commands I used to create my encrypted partition (inspired from this article).

sudo apt-get install cryptsetup
cd /media/APACHE
export usbdisk=$PWD
dd if=/dev/zero of=$usbdisk/ringo.img bs=1M count=256

sudo modprobe cryptoloop
sudo modprobe dm-crypt
sudo modprobe aes_generic
export loopdev=$(sudo losetup -f)
sudo losetup $loopdev $usbdisk/ringo.img
sudo cryptsetup -c aes -s 256 -h sha512 -y create usbkey $loopdev

sudo mkfs.ext3 /dev/mapper/usbkey
sudo mkdir -p /media/encrypted
sudo mount -t ext3 /dev/mapper/usbkey /media/encrypted

After that, the encrypted media should be working an mounted as '/media/encrypted'. The article explains how to write mount.sh and umount.sh scripts to automate the mounting and unmounting of the encrypted media. You need to make sure you don't forget to unmount your encrypted partition to avoid data loss, although the risk is in good part mitigated by the use of a journaled file system.

Next is to setup gpg to use the keys from the encrypted file system. If you already use gpg, it's very simple. It stores the keys in a .gnupg directory in a user's home. The simplest way is to move the .gnupg directory on the encrypted media and create a softlink to it. As I want to be able to perform public key operations (encrypt/verify) even when my usb flash is not mounted, this is what I did:

mkdir ~/.gnupg-local
cp ~/.gnupg/pubring.gpg ~/.gnupg-local
cp ~/.gnupg/trustdb.gpg ~/.gnupg-local
sudo mv ~/.gnupg /media/encrypted
ln -sf /media/encrypted/.gnupg ~/.gnupg
# unmount ringo
sudo ln -sf ~/gnupg-local /media/encrypted/.gnupg

Note that the last command was executed after unmounting the encrypted partition. That way, if the encrypted partition is mounted the .gnupg softlink in my home will point to it, otherwise it will point to the softlink in the /media/encrypted which points back to ~/.gnupg-local.

What I did is something slightly different, I wanted to avoid issuing a manual command every time I plug the usb flash in. The system that manages dynamic devices is udev, so I had to write a udev rule. I made (renamed) copies of the mount and umount scripts in /usr/local/sbin and got rid of sudo in the scripts

-rwxr-xr-x  1 root root  449 2012-01-08 22:29 mount-ringo
-rwxr-xr-x  1 root root  167 2012-01-08 22:29 umount-ringo

I added a udev rule:

hadrian@rem:~$ cat /etc/udev/rules.d/100-mount-ringo.rules 
ACTION=="add", SUBSYSTEMS=="usb", ATTRS{idVendor}=="1307", ATTRS{idProduct}=="0165", RUN+="/usr/local/sbin/mount-ringo"

The idVendor and idProduct values you can get from `lsusb`. They make sure the rule only fires when you mount the right usb flash memory. You can use other attributes to identify your flash if you prefer, like the label for instance.

hadrian@rem:~$ lsusb
[...] 
Bus 003 Device 018: ID 1307:0165 Transcend Information, Inc. 2GB/4GB Flash Drive

You can test your rules using `udevadm`. For that you need to know the mounted device which you can get if you know the label ('sde' in my case). If everything is ok, you should see a 'run: <your-script>' message near the end.

hadrian@rem:~$ ls -al /dev/disk/by-label/APACHE 
lrwxrwxrwx 1 root root 9 2012-01-09 11:13 /dev/disk/by-label/APACHE -> ../../sde
hadrian@rem:~$ sudo udevadm test /sys/block/sde
run_command: calling: test
[...]
ID_VENDOR_ID=1307
ID_MODEL_ENC=USB2FlashStorage
ID_MODEL_ID=0165
ID_FS_LABEL=APACHE
ID_FS_LABEL_ENC=APACHE
ID_FS_UUID=6F8A-7C90
ID_FS_UUID_ENC=6F8A-7C90
ID_FS_VERSION=FAT32
ID_FS_TYPE=vfat
ID_FS_USAGE=filesystem
[...]
run: '/usr/local/sbin/mount-ringo'
run: '/lib/udev/hdparm'
run: 'socket:@/org/freedesktop/hal/udev_event'

Once you are happy with your test you can restart udev to use your new rule:

hadrian@rem ~$ sudo udevadm control --reload-rules

As a backup, you may want to put your private key on a non-encrypted usb flash that you don't use and is kept in a secure location. Now you should be all set, just pull your usb flash out and put it in your pocket and remember to always umount first. Enjoy!

The Case for Patch Releases

A new patch release of Apache Camel, version 2.8.1 was released last week. I didn't announce releases in a while. Why write about camel-2.8.1 then?

The reason is that the Apache Camel community didn't produce patch releases in the past (we only did it for the 1.6.x version before discontinuing support for the 1.x versions). We actually started producing patch releases back in April for camel-2.7.x. At the time, I wasn't sure if and how this will continue because you need enough support within the community and some had the view that at Apache we should only focus on innovation.

The untold truth about that though has to do with the business models of open source. ZDNet had an excellent article a while ago, which makes for interesting reading. The author left out another business model, let's name it FUD ware, that relies on bundling known, successful open source projects and claim that only your distribution is production ready and offers what the market needs. That works better if you have some influence over the community that produces the original open source project. It is not enough, that's for sure, but combined with other business models (such as 1. Support Ware or 4. Project Ware) it may make the needed difference.

A few of us however, want the original ASF distribution to be stable, secure and production ready. After a bit of a struggle I am happy we managed to get the Apache Camel community used to the idea of producing patch releases more frequently. Between Dan Kulp and myself we issued since mid April three patch releases on camel-2.7.x, camel-2.8.1 last week and camel-2.8.2 is only a few weeks away.

This way you, our users, won't have to wait at least a quarter to have your issues resolved. At the end of the day, you the users, are the Apache Camel community.

Congrats Master Melegrito!

Personal things don't usually belong here, but this is a bit special. I am also still sore after this weekend.

Once a year, me and Ama have the opportunity to go with our instructor, 4th Dan Tae Kwon Do Master Josh Geeson, to train in self defense against stick and knife with Master Julius Melegrito, event organized in Charlotte, NC by Master Evins.

The seminar was a real treat as expected. Master Melegrito's speed and technique are amazing. After drills with two sticks, we got to one stick, repeating the same techniques. After the shoulders started burning we dropped the stick altogether and went mano-mano, repeating again, the same techniques only to realize how similar they are to Hapkido and Tae Kwon Do. Too bad this year we ran out of time and didn't get to practice defense against knife attacks. I am not complaining much though because last year my partner was Master Robert Shin who's not very forgiving and the hardwood floor was, well, hard.

Yes, Ama, you are the best!

What makes this year's event special is that Guro Melegrito is now a Black Belt Magazine 2011 Hall of Famer, the Weapons Instructor of the Year. Congrats Master Melegrito, many thanks for visiting and see you again next year!

Annotation for the Claimcheck Pattern

The claimcheck pattern is described in the EIP book Camel is based on. The way it is described, the pattern uses a data store which means that it applies mostly to local processing. However, in most of the cases, at least in my experience, the processing is not local so one needs to retrieve the original message at a different location where access to the data store is not necessarily possible.

As a bit of background, the commonly used analogy for the claimcheck pattern is the process used to check-in baggage while traveling and claim it at the destination. The rationale is that cabin space is a scarce resource (as is cpu, memory and bandwidth) and different parts of the in-flight entity have different SLA requirements: humans need oxygen, leg room, pressurized cabin space, food, entertainment, etc. whereas baggages can be stacked in the cargo hold. In some cases they may take different routes to destination as well.

So how should this be implemented? Let's take a look at the elements that define the pattern. First we have a message that via some logic will be transformed into two messages (1).  Let's call the two Messages the "initial message" and the "claim message". That is the Content Filter part described in the pattern, not to be confused with the filter dsl in camel that is actually a Message Filter, very different thing.

Our two messages will travel separately between departure and arrival, to borrow the terminology from the travel analogy via two different Message Channels (2), which in Camel we know as routes. The first message channel is the initial route we setup, let's call it the "main channel". The second channel is a one way one, let's call it the "baggage channel". Between departure and arrival, the initial message will go on the bagage channel and the claim message will go on the main channel.

We also need to generate a Correlation Identifier (3) to preserve the association (the equivalent of the bar coded tag on the baggage and the boarding pass). The fact that one could have multiple baggages is outside the scope of this pattern, it can be handled by a splitter/aggregator. The correlation id is both attached to the initial message and supplied somewhere in the claim message.

The claim message will replace the initial message on the main channel and may undergo further processing. It may contain partial content from the initial message required for processing, it may need to be of a specific type, we cannot make many assumptions. Two things are clear though: it is produced from the initial message, so we need a Processor (4) for that, and it contains the correlation id, so we need an Expression (5) to extract it from there at destination. While we need to attach the correlation id to the initial message too, we have a bit more freedom on the baggage channel, so let's simplify things a bit and use a custom header/property instead of customizing too much and require another Expression.

Putting together the five elements above, it starts to look like in the general case we need a separate, configurable one-way (in-only) route with some processing in the departure and arrival endpoints. I already looks like our implementation will actually require a different kind of Endpoint which also means a Component. It should support multiple queues at departure and arrival (i.e. both for check-in and claim). Since processing takes place on the main channel and at arrival during baggage claim the initial message is restored, it must be retrieved from the arrival claim queue, which means that the arrival queue must support random access. This is a key aspect, sometimes overlooked, that gives a lot of grief when implementing this pattern.

As far as implementation goes, I started to play with a bit of code so you could see a solution in one of my next posts and probably in the Apache Camel trunk soon.

Apache Camel 2.5.0 Released

The 2.5.0 release of Apache Camel is finally out. It is the result of some two and a half months of improvements since the last release in July. You can check the release notes for more details and give it a try.

The PMC is planning to focus now on releasing the 3.0 version of Camel in Q1 of next year. The 2.x version will continue to be actively improved for the foreseeable future (at least through 2011) and we plan to have another camel 2.x release this year.

To help planning the new 3.0 release the Camel PMC conducted a survey during the month of October. Many thanks to all who participated in the survey and made their voice heard. The results of the survey will be made public this week during ApacheCon. If you want to see the goodies coming in the 3.0 version, keep an eye on the roadmap.