State Agencies and “Legacy” Systems… The Cost of Aging Government Technology

Almost two-thirds of “mission-critical” business applications — those considered essential to the agency’s basic responsibilities — relied on unsupported legacy components, as did more than half of other business applications.

Courtney King and Bruce Wright

Information technologies (IT), including personal computers, the web, email, social networks, smartphone apps and more, have changed the way millions of people shop, bank and keep up with friends. And, increasingly, they expect to interact with governments in the same way, paying taxes or applying for benefits through a phone or laptop quickly and efficiently.

Yet Texas’ state government, like many others, has an aging IT infrastructure, and ours must serve a population growing by nearly half a million new residents each year. As the population — and its expectations — continue to grow, modernizing the state’s IT capabilities will become imperative to ensure public services keep pace with the speed and convenience of the private sector.

At present, however, much of the state’s IT infrastructure is outmoded, often so much so that the personnel who maintain it must be trained in archaic computer languages no longer taught in universities. Each year, the state spends hundreds of millions of dollars on maintaining such outdated “legacy” systems. Updating or replacing them would cost even more.

IT Spending at Texas State Agencies

Despite its importance, IT accounts for a relatively small share of the state budget. In fiscal 2016, Texas state agencies (excluding higher education institutions) spent nearly $1.8 billion on IT — only 1.6 percent of total state spending (Exhibit 1).

Note that contract services accounted for the largest share of state agency IT expenditures — a sign of outdated systems.

IT spending has fluctuated in recent years but in all rose by 6 percent between fiscal 2010 and 2016; Exhibit 2 shows this growth as percentage variations from the index year of 2010. IT as a share of total state spending, however, fell slightly during this period (Exhibit 3).

The Legacy Portfolio

The Texas Department of Information Resources (DIR) provides state agencies with a variety of IT-related services including purchasing, data-center and telecom services and information security products. In 2013, the Legislature required DIR to evaluate the state’s use of legacy business applications, combining software and hardware products, which support many users at once. The resulting 2014 study found state agencies using about 4,130 applications, of which about 58 percent included either legacy software or hardware, and in most cases, both.

Almost two-thirds of “mission-critical” business applications — those considered essential to the agency’s basic responsibilities — relied on unsupported legacy components, as did more than half of other business applications.

DIR estimates the one-time cost of either replacing or modernizing the state’s entire legacy application portfolio to be at least $450 million.

The DIR study thus found the state’s most important applications were more likely to rely on outdated components. Business applications used for licensing, business automation, systems management and reporting most commonly involve unsupported hardware and software.

DIR estimates the ongoing maintenance of the state’s legacy application portfolio costs more than $300 million each year.

Despite obsolescence, legacy business applications often can be kept running reliably for years and even decades, although at a steadily increasing price. They persist largely due to organizational inertia and the high cost of replacement, as well as the inherent risks involved in adopting new systems.

Over time, however, legacy applications become more difficult and expensive to support. Workers familiar with older systems inevitably become scarcer as the years roll on. Legacy systems are more difficult to adapt to new business needs and new file and data formats. They’re also more prone to cyberattack. The nature and type of threats in the digital arena evolve constantly, and the fixes and “patches” needed to counter them may not be available for older systems.

And when vendors go out of business or stop supporting older applications, agencies are forced to contract for support or develop their own in-house expertise in outmoded technologies.

Ironically, given state budget constraints, the increasing cost of supporting outdated systems inevitably reduces the funding available for newer, more efficient technologies. DIR estimates the one-time cost of either replacing or modernizing the state’s entire legacy application portfolio to be at least $450 million.

Growing Skills Gap

Texas’ ability to maintain and protect its legacy systems will become more difficult as veteran personnel who understand the technologies retire or resign.

A 2015 study by the National Association of State Chief Information Officers found nearly a third of government IT staff members would be eligible for retirement within five years. Survey respondents also noted a significant legacy skills gap between those close to retirement and younger workers. In Texas, DIR estimates half the state’s IT workforce is 50 or older, and about 14 percent is 60 or older.

Recruiting, training and retaining younger employees with the right skills to support legacy systems poses a unique set of challenges for Texas state agencies, according to Jay Waldo, director of Information Technology at the Comptroller’s office.

“There’s high competition for qualified IT staff in general, since most state agencies are based in the city of Austin where there are so many high-paying tech companies,” Waldo says.

Many legacy applications were coded using outmoded languages such as Common Business-Oriented Language (COBOL), first developed in 1959 and often unknown to younger recruits. In 2013, Computerworld magazine reported that 73 percent of universities no longer included COBOL in their computing curriculum.

“There are limited external training options for some older languages, so a lot has to be done internally using textbooks and exams to make sure the material is mastered,” Waldo says. “New employees also are trained on the job by their coworkers. But it’s a circular thing — once they’re trained up, they eventually want to move on to work with newer tech, even if they’d really prefer to stay with the agency. We try to give opportunities to work with newer systems and keep growing marketable skills, while also maintaining our legacy systems.”

Many legacy applications were coded using outmoded languages such as Common Business-Oriented Language (COBOL), first developed in 1959 and often unknown to younger recruits.

And again, inadequate vendor and staff support often force state agencies to contract for IT workers to maintain legacy systems.

“A technical contractor is twice as expensive as an FTE [full-time equivalent employee],” Waldo says. “Since contractors tend to get paid by the hour, unplanned work on evenings and weekends costs the agency additional money. And it’s preferable to put critical business or tech knowledge in the hands of people with a vested interest in the organization, rather than contractors.”

Improving Cost-Effectiveness

A lot can, and often does, go wrong in major IT changes. According to a 2012 study by consultants McKinsey & Co. and the University of Oxford, large software projects tend to run an average of 66 percent over budget and 33 percent behind schedule, and deliver 17 percent less value than originally predicted. More recently, cloud computing consultant Innotas reported more than half of all IT projects fail altogether due to poor planning and execution.

To avoid these pitfalls, DIR develops cost-effective, collaborative approaches to modernizing the IT infrastructure.

DIR’s Data Center Services (DCS) program, for instance, created by the Legislature in 2005, consolidated and standardized certain IT products and services across agencies, employing a consumption-based pricing model so that customers pay only for what they use. By 2016, DCS had fully consolidated mainframe, print-mail and service-desk services across a number of the largest state agencies.

DIR’s statewide project delivery team consults with agencies in planning major IT projects and provides oversight through the project lifecycle. The agency reports the average cost of major state IT projects has fallen from $28 million in 2012 to $18 million in 2016.

DIR also is working with state agencies to introduce an Application Development Decision Framework (ADDF), a strategy for modernization designed to guide its users toward best practices in areas such as the identification of user needs, purchasing and development, deployment of cloud technologies, staffing models, changes to organizational culture and more. Each ADDF topic area includes recommendations to improve agencies’ current approaches.

In 2016, as agencies prepared to submit their legislative appropriations requests for fiscal 2018 and 2019, DIR provided the Legislative Budget Board (LBB) with a report suggesting priorities for legacy modernization or replacement.

“The report was used by the LBB and state leadership to clarify funding allocations based on impact, risk and legacy characteristics,” says John Van Hoorn, DIR’s director of Enterprise Solution Services. “The 85th Texas Legislature used it to fund agencies that significantly needed to modernize or replace costly legacy systems.”

DIR will provide the report again for the next legislative session.

IT modernization efforts must address agency business needs while maintaining or improving customer service. Thus, it’s important to understand that the effects of these changes go far beyond agency IT divisions; they affect the organization as a whole, as well as its customers.

“Organizational culture, change readiness and change management are key concepts agencies address as part of modernization,” Van Hoorn says. “Agency leaders must promote vision for the change, build support and trust and show value throughout the change.”

This article originally published by Texas Comptroller’s Office.