I recently wrote about how to easily get all your logs from Amazon Web Services’ EC2. The point with that article was to show how easy it is to instrument your EC2 application for logging and forwarding to Logentries. But the whole point of using AWS EC2 is the ability to scale quickly and near infinitely, right? So what about the scaling aspect? Sure, you can rerun the setup script every time that you spin up an instance. When scaling is achieved by launching new instances from existing snapshots though, log forwarding becomes even easier!
Why Clone An Instance?
With EC2 instances, you can create snapshots of an instance and then launch new instances from these snapshots. Doing so, you end up with an exact replica of the first instance. For instance, you can clone your web server in order have two of them sharing the traffic coming to your website. As anybody well versed in cloud infrastructure knows, this is common enough practice when you need to scale up in the cloud.
How Does This Help with Logging as We Scale?
Logentries makes it possible for you to use the same configuration on as many servers as you wish. Moreover EC2 instances launched from snapshots inherit installed software as well as configuration files from the original instance. Combining both provides with a way to automatically and transparently get your EC2 logs forwarded to Logentries.
To use dynamic pick up, just use token-based logging through RSyslog. Having several sources for a single log in Logentries is no problem. Configurations files will get cloned along with the instance. Because cloned instances logically represent a single concept (e.g. your web server), it often makes sense to have log entries of those instances in one single log within Logentries. Plus if the host name is set to be part of the log entries, then it is still possible to isolate data coming from a single instance, using the Logentries filtering/search feature.
So, to wrap it all up, between our easy log set up for AWS EC2 and scaling your application by cloning instances, you should have no problem with getting all the log data from your application regardless of how big or how quickly your environment has to grow.