I’ve heard a joke that if a Gartner analyst stepped in a pile of dog poop with an IBM or Cisco logo on it, that they would place it in the MQ for fine Swiss chocolate.
A customer recently forwarded a Magic Quadrant (MQ) graphic to me and asked, paraphrasing, “’X’, ‘Y’ & ‘Z’ all seem to be doing better than ‘A’!?” I responded with a point-by-point critique of that MQ report. The first line in my reply was, “The devil’s in the details.”
Over the years, I’ve dissected Gartner Magic Quadrant reports. I’ve been passed a couple of recent reports, ones where I don’t have a dog in the fight, and too often I’m astounded at how incongruent the MQ chart compares to the details in the report. It seems to me to be grossly negligent for an organization with so much influence to allow one greatly simplified graphic to be used/misused/abused in a way that results in uncertainty, confusion and/or obfuscation in the marketplace.
Here are my suppositions about Gartner’s Magic Quadrant:
– There is a great deal of analysis that goes into each report that is not exposed to the general audience. (I don’t have a problem with this.)
– Many people use the graphic alone to overly influence their purchase decisions. If Gartner is not aware of this, then many would consider it irresponsible to fail to understand their audience and how their tool is used. I don’t know. It might amount to conveniently sticking one’s head in the sand. It might be naivety.
– Gartner profits from the “reprint” of MQ reports. Would it be surprising if most licensing fees come from vendors that are in the “upper right” quadrant? I’m a curious cat, so I would so be interested to see the ratio of those revenues that do or do not come from the vendors up there where everyone wants to be.
The claims I will not make, but that I hear more than occasionally:
– There is an unwritten “pay-to-play” influence that goes into the results in the MQ. I rely on Gartner analysis. Outside the MQ, I can’t say that their work isn’t world-class. I’ve met & interacted with a number of Gartner people over the years, and they have all exuded a sense of integrity & commitment to their craft. And when I dig into the details they publish in their reports, even the MQ, I generally find the analysis very good.
– End-user executives allow technology decisions to be disproportionately influenced by the MQ so that they can later point to Gartner’s “endorsement” in the event a technology doesn’t perform. But I have personally witnessed instances when an MQ graphic alone had a major impact on a short list or the final decision. To Gartner’s credit, they caution about this, but let’s just concede that it’s in the fine print.
I was taught, “Never complain without offering a recommendation to solve that which I am complaining about”. So, I have come up with a very simple three-point plan that would serve Gartner and our industry greatly. I would like to submit that the premise behind Gartner’s very existence is predicated on making technology decisions more transparent. I would hope that they would understand the market’s desire that Gartner should be held to a similar degree of transparency. In that spirit, I propose that in every MQ, that Gartner…
1) Disclose the credentials of each analyst specific to the technology that they have analyzed. The most important aspect is their real world experience in designing, implementing and/or supporting the technology they have evaluated.
2) Disclose their previous 12 months’ revenues generated to Gartner by each of the vendors covered in the report. (This doesn’t have to be a $$$ figure, just a ratio. I’m ok with that.)
3) Provide a historical analysis of their own accuracy in determining “ability to execute”. No doubt that Gartner has extraordinary resources with analytics skills. Why not quantify how well their past predictions have held up? (I’ve done my own qualitative analysis in a couple of technologies where I have extensive knowledge and there were more than five years of results, and I can tell you, it wasn’t pretty. Not even close.)
Doing just the three things above will greatly increase Gartner’s reputation, especially since transparency is so important in business today. I would like nothing more than to have those three simple things remove the mist of uncertainty about the potential conflicts of interest that swirl around the Gartner brand.
I’m afraid that some here may want to lash out at Gartner, but hey, let’s keep this professional and moving in a positive direction. I’ll be disappointed if this becomes a “bash Gartner over the head with a blunt object” stream of consciousness.