In recent months, Nucleus Research has grabbed headlines and embarrassed large software vendors with a series of reports called "The Real ROI."
In interviews following the release of the reports, Rebecca Wettemann, vice president of research at the Wellesley, Mass.-based firm, said that customers stood the same chance of achieving a positive ROI from companies such as Siebel Systems Inc. and SAP as they did winning at a Las Vegas slot machine.
It's a real gamble, though, to make IT decisions based on the Nucleus ROI reports, according to a nationally known survey methodologist who reviewed the results.
Based on the Nucleus research, there is no way of judging the real rate of ROI at the companies surveyed by Nucleus, said Jim Lepkowski, a senior research scientist at the University of Michigan whose most recent project was dedicated to telephone survey methodology.
"I'm a conservative investor," Lepkowski said. "And if my financial adviser was giving advice based on a report like this -- I wouldn't be too happy."
Lepkowski, who advises international groups such as the World Health Organization on survey methods, based his assessment on the size of the sample population, the response rate and a number of other variables.
Wettemann defended the series of ROI reports, saying: "We know what we're doing when it comes to numbers and statistics."
Nucleus has included in its Siebel ROI report an opening statement from Ian Campbell,
The interest in Nucleus reports can be traced as much to the company's reporting strategies as it can to their results. The research firm telephones customers whose success stories are featured on company Web sites, ones that can be expected to speak in favor of a company.
"Shouldn't those customers be the best ones?'' Wettemann said. "I only put the people who like me as references on my resume."
Siebel closed the customer section of its Web site after it was the subject of a Nucleus ROI report. The company said it needed time to reassess the names of companies featured on the site. SAP chose not to make any changes, though they loudly protested the report, from its headline, "Nucleus finds 57% of SAP customers have not achieved positive ROI", to its bottom-line analysis.
"Why should we make changes to our Web site?" SAP spokesman Bill Wohl said. "We have never said that the people who appear on SAP Web sites are those who reach ROI the fastest. They appear on our Web site, because, in some fashion, they are happy to be SAP customers."
In SAP's case, the results of the ROI report were based on interviews with 21 clients, 12 of whom had not reached a positive ROI after an average of 2.8 years.
"What we don't know about this report is telling," Wohl said. "We don't know where the customers are -- in their deployment and life cycle. We don't even know if they've finished their projects.
"What we know is that 12 customers allegedly told Nucleus that they had not yet reached ROI -- as Nucleus defines it -- within this mysterious, arbitrary average of 2.8 years."
Asked whether she felt the Nucleus Web site headline, referring to 57% of SAP customers, was a fair representation of the facts, Wettemann said: "It's a headline. It's a press release."Headlines and bottom lines
The real point of the ROI studies is being lost amid the protestations of software vendors, Wettemann said. Some of the Nucleus ROI reports have yielded positive results for companies, she said.
"The response from SAP is that 'your methodology might be wrong,'" Wettemann said. "We can argue: 'We talked with a significant number of their reference customers who are getting ripped off.'"
The Nucleus reports do include compelling, anonymous quotes from customers who say they were disappointed with their software projects, for reasons ranging from unreasonable expectations at the outset to promises made to them during sales pitches.
Nucleus contacted 93 SAP reference customers. However, only 21 agreed to interviews.
"You can say that 57% out of 21 is not statistically significant," Wettemann argued. "I say that 21 out of 93 is significant."
Not in this case, Lepkowski said.
"In this case, even if they wanted to say that the 93 people on the their list are the only 93 people they care about, and they only have gotten 21 of them to respond, that's less than a 30% response rate," he said. "Telephone surveys done by well-respected firms typically have about a 55-60% response rate."
Even if a consumer of the report was comfortable with a low response rate, the analysis still doesn't hold up, Lepkowski said. The Nucleus bottom-line analysis doesn't factor in the influence that non-respondents might have had on the results. The margin of error, or confidence interval, is enormous, he said.
In the SAP study, more than half the customers never returned the initial telephone call from Nucleus, according to the report. In the Siebel ROI report, Nucleus reported that half of the survey respondents acknowledged they didn't have methods in place for measuring ROI.
"If they are making an argument about the rigor of this methodology, it's a weak one," Lepkowski said.
Contacted for the purposes of this article, Lepkowski made clear that he could provide an opinion based only the results that were available on the company's Web site, and that he was not in a position to judge the value of the firm's research or opinions.
The reports, which Nucleus plans to release every six weeks on various technology companies, are available for free on the company's Web site and are not funded by any outside source, according to Wettemann. They appear under the trademark title "ROI Analysis You Can Trust."