THE INVITATION:
For this Rolling Review, we require either synthetic transaction monitoring, network probe application monitoring, or application agents. We'll allow other software components that provide additional visibility, such as dashboards or correlation engines. Client and OS agents may also be installed.
THE TEST BED:
We'll test APM products in our Real-World Labs at Windward Consulting Group, judging each product on how broadly it supports existing applications, how well it detects and reports performance problems, how well the architecture supports distributed app performance monitoring, and whether it supports a tiered architecture with native high availability and failover capabilities. We'll explore how well an offering detects the true performance issue, how capably it determines the root cause, and how seamlessly it integrates with the surrounding environment.
For our test application, we'll use EMC's Documentum knowledge management system, eRoom 7. Our app infrastructure includes a Microsoft SQL database server for document storage. Hardware platforms are Windows 2003 servers with 1 Gbyte of RAM and 1 Tbyte of physical space using a RAID array. The Web server is running Internet Information Services Manager v6 with Web services extensions and SMTP virtual server. The application is connected to a Cisco switch, and traffic is routed via a Cisco WAN to the ISP.
THE VENDORS:
We plan to test products from BMC, CA/Wily, Compuware, HP/Mercury, EMC/Smarts, IBM, Indicative, InfoVista, NetIQ, NetQoS, NetScout, Network General, Nimsoft, Oracle, ProactiveNet, Quest Software, and Symantec.
THE PREMISE: InformationWeek Labs' Rolling Reviews present a comprehensive look at a hot tech category, from a market analysis to a synopsis of our findings.
If you've ever gotten an earful from an irate business user complaining about sluggish response times, you know that good application performance management is worth its weight in log files. The complexity of today's applications is such that APM is required for all but the smallest businesses, yet choosing the right system is a complex undertaking.
There's no shortage of vendors looking to help--we easily came up with more than 100 products--but given the sheer number of ways to collect performance data, the utopian vision of holistic APM can still turn into a nightmare of high support costs, visibility gaps, and unwieldy complexity. Even if you solve the collection challenge, you may still struggle to consolidate and correlate diverse data sets.
Those who've dabbled in performance management with products like CA's eHealth, Hewlett-Packard's OpenView Performance Insight, and BMC's Performance Manager are left with a dilemma: Start fresh with a dedicated APM suite or cobble together a system with existing tools? If you decide to start anew, get ready to justify that decision and crack open the piggy bank--APM implementations start around $80,000 and can scale well over $1 million. To help you choose, we decided to put APM offerings to the test and determine if one is superior for collecting and reporting on application performance issues.
Get Down To Basics
Most APM products gather data either by actively inserting additional traffic into the environment or by passively collecting real user data. Products comprise a few specialized elements, though not all support all types of monitoring:
Synthetic transaction software, found in suites from BMC, HP Mercury, IBM Tivoli, and Symantec, resides in key locations in the infrastructure and generates traffic that simulates use of the application, then reports performance results.
Network probes, provided in CA's Wily and Quest Software's Foglight, reside as physical devices in the network and are typically connected to a Switched Port Analyzer port on a switch. They track and classify application traffic by monitoring Web app sessions or TCP traffic.
Application server agents are widely supported; they reside on the app server and report on specific metrics to determine the cause of a performance problem.
Client agents, offered only by Compuware and HP Mercury, live on the user's machine and monitor application performance using APIs or TCP sockets.
System agents look deeply into the performance of operating systems and server hardware and report on CPU, disk performance, and other critical components. BMC, Compuware, Quest, and others use them.
Application performance integration is universal and relies on existing hardware, system, application, and network monitoring data to create application service levels that help IT pinpoint a problem's cause.
For deep background on this class of technology, see our APM primer.
To Buy Or Not To Buy
To determine if you can justify the cost of APM, weigh how poor application performance is affecting the bottom line against the added management effort, deployment costs, and maintenance fees. Decide whether you can stomach application server agents, which are necessary to gain deep application performance metrics.
Note that too often, intermittent problems are never reported, even though they have a real impact on productivity. If you suspect underlying issues, consider engaging a vendor, such as BMC, Keynote, or Mercury, that offers synthetic user monitoring as a service.
Solving problems requires an understanding of application complexity and dependent components, such as system-level CPU utilization, network performance, database latency, and client problems, all of which can have adverse effects on an application. Many organizations monitor some of these, but few have the visibility into complex interrelationships needed to decipher the impact a problem in one area has on the whole system.
"Mainstream BI" continues to be more vision than reality, but how about mainstream analytics? This week, visualization and analytics vendor TIBCO Spotfire took one step closer to making that a reality with its 3.1 release.
Cassandra and the NoSQL Scalable OLTP Argument Todd Hoff put up a provocative post on High Scalability called "MySQL and Memcached: End of an Era?"... It seems as if the super-scalable Web site biz has moved beyond MySQL/Memcached...
I've certainly had to sort through a lot of dubious competitive claims in the last week. As a journalist, I have many years of experience hearing ill-informed assertions, half truths and occasional bald-faced lies. I usually know BS when I hear it. Sometimes I'm still taken off guard.
