Click here to Skip to main content
15,350,008 members
Articles / Web Development / Apache
Posted 6 Mar 2013


2 bookmarked

Lightening Fast Troubleshooting for Your Cloud Applications

6 Mar 2013CPOL4 min read
Lightening Fast Troubleshooting for Your Cloud Applications

This article is in the Product Showcase section for our sponsors at CodeProject. These articles are intended to provide you with information on products and services that we consider useful and of value to developers.

Image 1


Cloud service providers like Microsoft® Windows® Azure, Amazon® Web Services, Heroku®, Rackspace® and Google® App Engine make it fast and easy to deploy and run applications in the cloud. However, running applications in the cloud comes with a potential loss of visibility and control. While logs and metrics can provide some insight into how your application is doing, this data is often in different formats scattered in different places, making it difficult to gain full visibility into your operations.

How can you best troubleshoot, monitor and proactively manage your applications in the cloud?

The Solution

Splunk Storm provides a technology-agnostic approach to monitoring and managing cloud applications. Harness data from every tier of your application and trace transactions across multiple hops, correlate application events with infrastructure or user experience issues and proactively prevent outages from impacting the business. Splunk Storm, delivering the industry-leading Splunk software as a service, indexes and stores machine data in real time from virtually any source, format, platform, or cloud provider without the need for custom parsers or connectors. Whether your application is written in Ruby, Java, Python, PHP, .NET, Node.js or any other language or framework, send data to Splunk Storm using network streams such as syslog, using a REST API, or via the universal forwarder for indexing and searching.

Let’s Walk Through the Example

We’ll put some sample Apache access and MySQL logs into Splunk Storm and see how quickly we can troubleshoot issues. First, a little setup:

  • Sign-up for Splunk Storm at

    Image 2

  • Create a new project
  • Image 3

    Image 4

    Image 5

    Image 6

  • Now add data to your new project. While you can also send data to Splunk Storm projects over TCP/UPD (including syslog), via a REST API or via a Splunk Universal Forwarder, we’ll manually upload some sample data for this exercise. First thing we’ll do is download the sample data (Apache Web server logs and MySQL database logs from a hypothetical online flower shop) from
  • Once you download and unzip, you'll see three folders each with an Apache "access_combined" log file inside and one Mysql folder with a MySQL log file inside it.
  • No back in the browser, you’ll see the Inputs page

    Image 7

  • Click on Files

    Image 8

  • Click the "Upload" button

    Image 9

  • Browse to the access_combined.log file in and choose it.
  • For each of the log files, choose a source type. Specifying a source type tells Storm how to parse your data, and allows you to group all the data of a certain type together when searching. When you add your own data to Storm, you'll want to specify the right source type so that Storm extracts timestamps and linebreaks your data correctly.
  • For the Apache access logs, choose Apache web access logs and click the "Upload" button.

    Image 10

  • For the MySQL log file, choose Generic single-line data and click the "Upload" button.

    Image 11

  • Repeat the upload process until all of the sample data is in your project
  • Once the data is added to your project, the "Explore data" button will become enabled. click it!
  • Now that you have the data in a Splunk Storm project, see how quickly you can troubleshoot your applications. Let’s say you receive a call from a customer who keeps hitting a server error when he tries to complete a purchase on your company’s online flower shop. He gives you his IP address –
  • Everything in Splunk Storm is searchable, so you just type "" into the search bar, hit enter, and you will see all of this customer's traffic to your shop.

    Image 12

  • So, you see a lot of 200 response codes, but you're only interested in errors. Filter out any event that's not a 200 success response by typing "NOT 200", narrowing down the list of events.

    Image 13

  • Notice that each of the events appears on the timeline below the search bar. And double-clicking on the bars re-runs the search over a smaller time-range. So if you drill down a bit you can get to the one-minute window when one of the errors occurred.

    Image 14

  • Knowing the time-window of one of the errors, you broaden the search to include everything that was happening in the application around the same time... and you get a handful of database errors, which are pretty good leads to the root cause of the server error our customer is seeing.
  • Finding the root cause of issues using production logs saves long hours of trying to reproduce bugs in a development environment so you can fix issues quickly and keep your users happy.


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


About the Author

Jon Rooney
United States United States
Jon Rooney is the Director of Developer Marketing at Splunk, focused on making it easy for developers to work with big data using the skills and tools they know and love.

Comments and Discussions

-- There are no messages in this forum --