A revolution with Business Activity Monitor (BAM) 2.0

 

• Performance – We plug in to our ESBs and App Servers and all metrics were perfect. It nicely showed request counts, response times, etc. It was perfect as long as the load is low. If one server starts sending 1000 events/sec, things started getting ugly. Even worse, if we plug in to a few servers and start getting 1 billion events / day, well, that would have been a nightmare from the word go. We couldn’t even fathom what would happen at that type of scale.
• Scalability – We need to store events and process them. Sadly, we discovered the hard waye this would mean is we need to scale in many different ways.
  • Event load – We need to scale in terms oh handling large amounts of events. We didn’t have a high performance server, but no matter how good our performance would be, there is still a breaking point. Afterwards, you need to scale.
  • Storage – If you store 1000 events a day, your data will grow. And, all of us hate to delete off old email, to get more inbox space. So naturally, everyone wants to keep their events.
  • Processing power – When you want to analyze events that you collect, a single server can only give you that much of processing power. You need to scale out your analytics. Another, ‘oh, so obvious’ thing that we learnt eventually.
• Customizability – We provided a lovely set of dashboards that showed all you wanted to know about your server and API metrics. But, no one is ever satisfied with what we they have. They want more. They want to monitor their metrics and analyze their data and put up their own graphs. And, of course, they want to do it now, not in 2 months.

• Divide and conquer – We divided the problem. We have to aggregate, analyze and present data. So we built separate components for each, keeping in mind that we need to scale each individually. We mapped these into the event receiver, analyzer framework and a presentation layer. Data agents are the link between anyone who wants to send events and the BAM server. The WSO2 Carbon platform, allows us to easily uninstall a component from any server. This means we can take the BAM distro, uninstall other components just to make an Event Receiver BAM server. Or to make an Analyzer BAM server. It’s just a click of a button.

• Scalable and fast storage – We chose to use Apache Cassandra as our storage solution. I do not want to argue that it’s the best data store ever. But, it works for us well. It allows us to do fast writes to store a large amount of data, quickly. Also, it’s built to scale. Scaling up Cassandra, takes minutes, not weeks. And scaling up doesn’t mean it’s going to cost you. Also, it’s written in Java, and being a Java house, it allows us to hack around the code.
• Fast protocol – We chose to use Apache Thrift as our default protocol. There are many arguments against it, but it holds up well for us. It’s fast and it does it’s job. It allows us to maintain sessions, supports a bunch of languages. One key thing was Cassandra uses it as well, allowing us to gain more performance in streaming data into Cassandra without deserializing.
• Scalable analytics – We chose to write our own analytics language. But, if it doesn’t suit you, you can plugin your own java code. Hadoop is unavoidable when it comes to scaling analytics. So, we decided to have a Hadoop mode for large amounts of data and a non-Hadoop mode, so that anyone can just use BAM without worrying about any Hadoop cluster.

 

• Gadget based dashboards/reports – Drag and drop visualizations are very attractive when you don’t want to spend weeks writing code to visualize. We developed a gadget generator so you can quickly visualize your analyzed data easily.

Reference: A revolution with Business Activity Monitor (BAM) 2.0 from our JCG partner Mackie Mathew at the dev_religion blog.