"Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted"
- Albert Einstein
This quote certainly applies to governments, where things that "count" are often abstract and difficult to quantifiably measure, such as political stewardship, citizen service efficiency and quality of life. How do governments measure these intangible yet meaningful aspects of their missions?
Governments have unique challenges not faced in the commercial sector. Governments must preserve fiscal stability while encouraging citizen involvement; deliver social programs; enhance the health and safety of constituents; promote sustainable development; build consensus; manage natural, cultural, and historic resources; and plan for the future. Furthermore, governments deliver economic, social and political value to a wide variety of stakeholders, including citizens, businesses, elected officials, regulatory agencies and other governments, to name a few.
More than ever, governments are leveraging information technology (IT) to help them address these varied challenges and meet the needs of their diverse stakeholders. Concurrently, they are seeking ways to justify and prioritize these IT expenditures. They also want to measure IT performance as a means to build public trust and maximize public value. In the commercial sector, this corresponds to measuring "return on investment", or ROI.
The Need for Multi-Dimensional ROI Analysis in the Public Sector
Historical attempts at applying ROI measurement to IT expenditures in the public sector involved financial based quantitative methods from the commercial sector that calculate cost-avoidance or cost-benefit analyses. Since governments are non-profit organizations, applying only financial ROI measurements is limiting -- and does not present the full picture of value that governments deliver. ROI measurements in the public sector must include both the tangible and intangible impact of government IT spending.
In traditional cost-benefit analysis, tangible financial benefits might be easily calculated, but some intangible costs might be overlooked. For example, centralized government procurement across multiple agencies in a single marketplace generates productivity benefits for government and some vendors. However, local and small suppliers incur additional technology costs to join these marketplaces, which may make it difficult for them to join in the bidding process, thus impacting their business. In the end, this has a negative value for these business stakeholders in the community.
Governments worldwide recognize the limitations of traditional cost-benefit analysis, as evidenced by recent social value measurement initiatives within the federal government of Australia and the General Services Administration in the United States. Furthermore, according to a recent Economist study, nearly 70 percent of almost 800 public sector executives worldwide will consider both financial and social criterion in assessing IT investments in the next five years -- and make that ROI transparent to citizens and stakeholders. (1)
This confirms a global trend of governments to show effectiveness across multiple dimensions of ROI -- beyond traditional, quantifiable financial metrics -- to include measurements of how IT impacts the social and political landscape of government programs. An initial challenge facing the public sector is reaching consensus on the definition of social and political ROI. Simply put, these are qualitative, indirect and intangible benefits (or consequences) of IT expenditures. Often, social and political ROI are intertwined. In many cases, social and political ROI supersede financial ROI, particularly during election cycles. Examples of positive social ROI include improved education, increased adoption of technology, lowered unemployment, and reduced crime. Examples of positive political ROI include improved public trust, positive media coverage, or any political "capital" which can be leveraged to influence policy and legislative action.
Given these market characteristics, SAP recognized that governments need a multi-dimensional methodology to help them track the true value of IT investments and subsequently created the phrase "Public ROI". This refers to an approach which measures not only the financial, but also the social and political value of IT investments.
Examples of Public ROI in Government
An example of government achieving ROI from IT beyond simple financial metrics includes the Brandenburg Police Force in Germany. This central police service for the state of Brandenburg is responsible for equipping all of the state's 25 police departments, across 6,000 employees. In 2000, the regional audit office examined the logistics operations and found warehouse stock levels were too high, and the procurement ordering process overly time consuming and inefficient. Employee satisfaction with the existing processes was low, and time spent on logistics administration was time not spent focused on the core mission of the police force -- namely, public safety.
Brandenburg subsequently implemented a variety of reforms to reorganize uniform supply and warehousing processes. By leveraging a Supplier Relationship Management (SRM) solution, Brandenburg removed outdated warehouse stock, simplified the ordering process for employees, and sped up delivery cycles. An independent investigation by Gartner Group has shown classic financial and operational ROI. Today, Brandenburg's 6,000 employees each save 8 hours of processing yearly by leveraging the eProcurement capabilities of SRM, resulting in an annual savings of approximately 970,000 (~ $1.2 M USD). Secondly, warehouse stock costs have been reduced 47% from 6 million to 2.8 million. (2)
While these numbers show a clear financial return, Brandenburg also achieved a "social ROI" from the SRM implementation. Specifically, Brandenburg freed up 11 employees previously working on logistics administration to focus on mission-critical tasks – providing improved public safety and prison services to the community. This reallocation of resources, enabled by IT, helps Brandenburg focus its efforts on delivering improved social value.
An example of "political ROI" was realized by London Borough of Waltham Forest, United Kingdom. Before implementing an ERP solution, Waltham Forest had highly fragmented procurement and finance operations and was running almost 20 different IT systems. The Council used a general ledger system that did not address the strategic or operational management needs of the organization. The operational issues in procurement and finance were constraining the performance of the Council. Waltham Forest had no overall visibility of its budgets, payment commitments and suppliers. Critically, management information was spread across multiple systems, and it took an average of eight weeks to provide key management information.
The Council therefore decided to update and consolidate all corporate information systems into a single integrated system that would support human resources, payroll, procurement and finance. An independent investigation by IDC (3) has shown that after successfully implementing an Enterprise Resource Planning (ERP) solution, Waltham realized benefits in several key areas:
- Waltham Forest now has an integrated system for all human resource, payroll, procurement and finance operations. This provides real-time access to business information, which had not been possible before. The time to produce an end-of-month report has been reduced from three days to five minutes and the budget reporting elapsed time has been halved. Payroll costs have been reduced by 30% and eProcurement targets of the national eGovernment strategy for local government have been met.
- Cash flow has improved significantly through the enforcement of 30-day payment terms and better debt collection. As a result of better cash flow management, Waltham Forest has reduced its borrowing by some £20 million over a period of 24 months and has reduced cash owed by its debtors by £6 million in the last 12 months alone. In itself, the reduction in borrowing has resulted in significant savings through reduced interest payments.
- The number of suppliers has been drastically reduced, and the remaining suppliers are better managed through the system and a set of preferred suppliers has been established. Better working relationships between the council and suppliers have ensued through a clearer and consistent procurement process. Invoice matching has improved, due mainly to the centralization and integration of procurement and accounts payable functions. This is a key best value performance indicator for local authorities in the United Kingdom.
- The overall integrity of the budget and accuracy of the finance function have improved, with better internal cost controls, real-time access to information, and security and traceability being introduced by the new capability. There is now full control and transparency with regard to Waltham Forest's spending.
Perhaps the most tangible benefit delivered by IT to the council was the improvement in its Comprehensive Performance Assessment (CPA) rating. Waltham Forest was rated as 137th in the UK for use of resources. It is now rated 27th in the country, and 4th in London. As with Brandenburg Police Force, Waltham Forest achieved quantifiable, financial benefits from implementing IT solutions. However, many of these benefits also correspond to "political" ROI, particularly Waltham's advancement from 137 to 27 on the CPA rating, its enhanced working relationship with suppliers, improved cash flow, and lower debt obligations. Together, these exhibit improved management control, maximum leverage of public funds for procuring goods and services, and improved operational transparency. Waltham Forest's employees and elected officials leverage and promote these improvements to the community at large, bolstering public trust and improving the public perception of Waltham's stewardship.
Launching the Public ROI Initiative
Historically the public sector has lacked a defined methodology for calculating ROI. Although there are various ROI models available in the market, developed by both governments and the commercial sector, they often have limitations, are too focused on financial metrics, or have limited geographic applicability.
Recently, enterprise software maker SAP partnered with the Center for Technology in Government (CTG), an applied research center devoted to improving government and public services through policy, to develop a universal, non-proprietary methodology to help government track financial, social and political value of IT investments. The goal of this initiative was to provide government leaders with a framework to identify and show a linkage between IT and the impact it has on real-life issues.
The project goals were straightforward - to develop a new framework that is:
- Relevant to federal, state and local governments
- Based on best practice, case studies, and expert workshops
- Geographically "agnostic"
- Comprehensive in risk assessment
- Flexible for all types of government programs
- Balanced across a set of measurements from research/statistical methods with practical application
Five case studies -- SAP customers and non-SAP customers alike -- were conducted to help CTG formulate this new framework. These case studies included the Integrated Enterprise System in the Commonwealth of Pennsylvania and the Washington State Digital Archives in the United States; the Merkava Project in Israel; the Austrian Federal Budgeting and Bookkeeping System; and Service New Brunswick, Canada. This project culminated in September, 2006, with the publication of a White Paper documenting the new Public ROI framework (full white paper can be accessed at http://www.ctg.albany.edu/publications/reports/advancing_roi). Delivering a New ROI Framework
The new Public ROI framework is broad in scope so that it can be applied to virtually any government IT investment – from simple Web sites to government-wide information systems and architectures. The framework provides an analysis process that starts with a high level view of the IT investment and then drills down through successive steps to identify the specific measures and methods that will reveal and document public value. This allows the assessment to be tailored to the size and nature of a particular investment decision. The framework provides a comprehensive and universal method to help governments measure public value -- and the assessment framework offers both a systematic approach to thinking about public value as well as how to apply that approach to a particular IT decision.
The framework's strategy is simple in concept -- connect what happens in the government (left side of Figure 1), with stakeholder impacts in the public domain (right side of Figure 1), then report and apply the results.
The activities on the left identify the potential value mechanisms and outcome goals of an IT investment. Those are linked to stakeholder interests, impacts, and risks to the right. Next, specific variables are selected for collecting assessment data, whether by leveraging existing schemes or creating new ones. Then risk analysis is conducted to identify and evaluate threats that yield information for decision making, such as whether to move ahead or cancel an IT investment. The framework provides illustrative examples of how to summarize and present the results of the analysis, particularly in the language of each of the appropriate stakeholders.
The curved arrows indicate that in practice the process would seldom be linear, requiring reflection and backtracking to adjust for learning and new information. The White Paper presents a detailed version of this schematic, showing the links among these activities. The public value proposition takes center stage. This value proposition must be broadly conceived to do justice to the scope of government and how it affects individuals, groups, and both public and private organizations. This framework presents a new and more comprehensive way of describing public value, based on different kinds of impacts government IT can have on the interests of public stakeholders:
Financial - impacts on current or anticipated income, asset values, liabilities, entitlements, and other aspects of wealth or risks to any of the above.
Political - impacts on personal or corporate influence on government actions or policy, role in political affairs, or influence in political parties or prospects for current of future public office.
Social - impacts on family or community relationships, social mobility, status, and identity.
Expanding the view of stakeholder interests in this way brings into focus two distinct but equally important types of public value: the delivery of benefits directly to citizens and enhancing the value of government itself as a public asset. An IT investment that makes government more transparent, more just, and a better steward has added public value, a non-financial but nonetheless important return. This framework describes how to include both in public value assessments.
The framework identifies the ways government IT investments link to public value. The simplest link results from an IT investment embedded directly in a service delivery process in a way that enhances its value to citizens or officials. The total value may be a composite of several separate measures: cost savings, quality, and enhanced trust. An indirect link can result from infrastructure improvements to business processes.
Identifying these links is just part of the story. Each link can involve several value generating mechanisms. The framework describes how to trace these mechanisms working through specific business processes to produce different kinds of value. The value-generating mechanisms are:
Increases in efficiency - obtaining increased outputs or goal attainment with the same resources, or obtaining the same outputs or goals with lower resource consumption. In the Austrian and Pennsylvania case studies, for example, new ERP systems helped achieve substantial efficiencies in financial management.
Increases in effectiveness - increasing the quality and/or quantity of the desirable thing. The case study of Service New Brunswick, for example, reports how an online registry for land data can contribute to improvements in property tax administration.
Enablement - providing means or allowing otherwise infeasible or prohibited desirable activity, or preventing or reducing undesirable events or outcomes. In the Washington State Digital Archives case study, for example, putting birth and marriage records online enabled research by local historians and genealogists.
Intrinsic enhancements - changing the environment or circumstances of a stakeholder in ways that are valued for their own sake. For example, the Israel case showed how enhanced financial accounting and reporting in the Merkava ERP system opened government financial decision making to greater transparency.
An IT investment project can deliver public value through any or all of these mechanisms. Every government IT project will have its own unique goals, value propositions, and stakeholders. So this framework can be used to plan and guide a public value assessment, in combination with measurements, analysis tools, and reporting techniques chosen for the specific situation. To aid in this regard, the White Paper presents an overview of more detailed and highly specified assessment methods that can be used in conjunction with this framework. This includes a more in-depth flow chart for the assessment process, a summary of several ROI models and methods, and suggestions for analysis and reporting of results.
Case Studies Yield Real-World Examples & Reinforce Need for New Framework
The new Public ROI framework is based in part on the results of the five case studies identified above that examined how government IT investment projects came to deliver value to the public. Each case study examined how the IT investments were conceived and developed, with particular attention to the role of public value in the process. This was accomplished by considering the links between IT goals, implemented systems, government performance, and public returns in terms of where they represent value in the chain below (Figure 2).
The public returns from the investment can flow from the internal improvements in the agency resulting in returns to individual citizens and the public at large (the main flow through the center). Other returns can flow to the political system and the economic environment (below the center), or through effects on other agencies (secondary performance gains). This general view of public returns informed the case studies and helped summarize the results.
Throughout this project, it was evident that measuring ROI from an IT expenditure – whether by a traditional financial metric, or expanded to include social and political dimensions - isn't a cultural standard and rarely mandated in the public sector. Common reasons for not conducting such measurements were limited resources (people, data, time, funding); lack of incentives to assess ROI; limited baseline data and overly complex measurement models. Yet, governments worldwide are contending with continually rising service demands and increasing stakeholder expectations for measurable outcomes.
As exhibited in the five case studies in this project, and in ongoing dialogues with public sector experts, the need for connecting IT investments to public value is a growing concern within governments around the world. More and more, elected officials are insisting on more comprehensive cost and return analyses for IT investment proposals. Government agencies are now developing business cases with targeted outcomes before submitting budget requests. Citizens are demanding accountability and transparency of government operations. Simply put, the tide is shifting, and governments may have no choice but to implement assessment methods showing the public value of IT investments.
Conventional approaches to ROI analysis may simply be inadequate for these increased demands. Most traditional ROI models are too narrow in scope and do not consider the context and culture of government. The motivation for a more comprehensive justification for IT investments is tied to its integrative and transformative potential. Large IT investments require a means to assess public value that matches their scope. In this light, using this new, non-proprietary Public ROI framework is justified. Moreover, the new knowledge about public value possible from such an assessment can help guide other forms of investment and contribute to long-term government improvement and improve societal credibility, build political support, and ultimately improve public trust in government.
Using such a Public ROI framework, perhaps everything that counts in government can be counted, after all.
For more information on SAP's Public ROI Initiative, contact Russ LeFevre. Or visit http://www.sap.com/industries/publicsector/roi.epx or http://www.ctg.albany.edu/projects/proi.
1. The Economist, Economist Intelligence Unit, "Business 2010: The Public Sector". 2005. Page 10.
2. SAP Case Study, in conjunction with Gartner Consulting. "Logistics Optimization in the Brandenburg Police Force", October 2005.
3. IDC Case Study. "London Borough Achieves 72% ROI on ERP Managed Service Implementation", July 2005.
"Advancing Return on Investment Analysis for Government IT: A Public Value Framework". Anthony M. Cresswell, G. Brian Burke, Theresa A. Pardo, Center for Technology in Government, University at Albany, SUNY. September 2006.
Dig Deeper on SAP ROI and TCO