On Feb. 16, less than two weeks after a mysterious attacker made headlines around the world by hacking a water treatment plant in Oldsmar, Florida, and nearly generating a mass poisoning, the city's mayor declared victory.

“This is a success story," Mayor Eric Seidel told the City Council in Oldsmar, a Tampa suburb of 15,000, after acknowledging “some deficiencies." As he put it, “our protocols, monitoring protocols, worked. Our staff executed them to perfection. And as the city manager said, there were other backups. ... We were breached, there's no question. And we'll make sure that doesn't happen again. But it's a success story." Two council members congratulated the mayor, noting his turn at the press conference where the hack was disclosed. “Even on TV, you were fantastic," said one.

“Success" is not the word that cybersecurity experts use to describe the Oldsmar episode. They view the breach as a case study in digital ineptitude, a frightening near-miss and an example of how the managers of water systems continue to downplay or ignore years of increasingly dire warnings.

The experts say the sorts of rudimentary vulnerabilities revealed in the breach — including the lack of an internet firewall and the use of shared passwords and outdated software — are common among America's 151,000 public water systems.

“Frankly, they got very lucky," said retired Adm. Mark Montgomery, executive director of the federal Cyberspace Solarium Commission, which Congress established in 2018 to upgrade the nation's defenses against major cyberattacks. Montgomery likened the Oldsmar outcome to a pilot landing a plane after an engine caught fire during a flight. “They shouldn't celebrate like Tom Brady winning the Super Bowl," he said. “They didn't win a game. They averted a disaster through a lot of good fortune."

The motive and identity of the hackers, foreign or domestic, remain unknown. But Montgomery and other experts say a more sophisticated hacker than the one in Oldsmar, who attempted to boost the quantity of lye in the drinking water to dangerous levels, could have wreaked havoc. They're skeptical of the city's assurances that “redundant" electronic monitors at the plant protected citizens from any possible harm. “If the attackers could break into the lye controls," Montgomery said, “don't you think they could break into the alarm system and alter the checkpoints? It's a mistake to think a hacker could not introduce contaminated water into our water systems." Oldsmar officials, citing the ongoing investigation, declined ProPublica's requests for an interview or to address emailed questions about the city's cybersecurity practices.

The consequences of a major water system breach could be calamitous: thousands sickened from poisoned drinking water; panic over interrupted supplies; widespread flooding; burst pipes and streams of overflowing sewage. (This is not merely theoretical. In 2000, a former municipal wastewater contractor in Australia, rejected for a city job, remotely manipulated computer control systems to release 264,000 gallons of raw sewage, which poured into public parks, turned creek water black, spilled onto the grounds of a Hyatt Regency Hotel and generated a stench that investigators called “unbearable." The man was sentenced to two years in prison.)

In congressional testimony on March 10, Eric Goldstein, cybersecurity chief for the federal Cybersecurity and Infrastructure Security Agency, described the Oldsmar incident as illustrating “the gravest risk that CISA sees from a national standpoint." He said it should be “a clarion call for this country for the risk that we face from cyberintrusions into these critical systems."

Grave warnings have sounded for years. As far back as 2011, a Department of Homeland Security alert advised that hackers could gain access to American water systems using “readily available and generally free" internet search tools. Such admonitions have abounded in recent years. Booz Allen Hamilton's 2019 “Cyber Threat Outlook" called America's water utilities “a perfect target" for cyberattacks; a 2020 Journal of Environmental Engineering review found “an increase in the frequency, diversity, and complexity of cyberthreats to the water sector"; and the Cyberspace Solarium Commission's March 2020 report warned that America's water systems “remain largely ill-prepared to defend their networks from cyber-enabled disruption."

Despite the warnings, and some high-profile breaches dating back a decade, the federal government has largely left cyberdefense to the water utilities. For years, it relied on voluntary industry measures, dismissing any need for new regulation. Then, in 2018, Congress included a provision addressing cybersecurity in a 129-page water bill that covered everything from river levee repairs to grants for school water fountains.

The requirements were less than demanding. Every U.S. water system serving more than 3,300 customers was obliged to conduct a self-assessment of the risks and resilience of its physical and electronic systems and prepare an emergency-response plan. Different-sized utilities got different deadlines; for the smallest covered by the law, such as Oldsmar, the self-assessment must be done by June 30, 2021, more than two and a half years after the law was signed. (Oldsmar had completed its cybersecurity review by early November but hadn't yet incorporated its recommendations in the city's emergency response plan before the February hack, according to a statement provided by the city manager.) Tens of thousands of U.S. water systems with fewer than 3,300 customers were exempted entirely from the law's requirements.

Those utilities required to perform a self-assessment were not obliged to submit a report to any government agencies. The utilities merely had to attest to the Environmental Protection Agency that they had conducted the assessment. The 2018 legislation also provided $30 million for grants to help water districts deal with “risk and resilience" problems, including cyberattacks. But Congress never appropriated that money.

The water provisions fall far short of federal requirements (including penalties for violating those rules) and funding aimed at protecting electricity infrastructure, according to Montgomery. “An assessment's a good thing," he said. “But this is well short of what we require from energy companies. We have developed a tool for self-identification of problems. But if you're really bad at cybersecurity, I'm not sure your self-identification is going to solve the problem."

He also pointed to low staffing at the EPA's Water Security Division. “The water security office is a handful of people, probably three," Montgomery said. “It historically has not done much, if any, cybersecurity work. This is the product of 20 years of low prioritization." The agency's most recent report to Congress on “Drinking Water Infrastructure Needs," submitted in 2018, identified $472.6 billion in long-term priorities, but it didn't mention the word “cybersecurity" once in its 75 pages.

An EPA official, speaking on the condition of anonymity, agreed that the agency had only “a small team" devoted to water cybersecurity but said Oldsmar “and other recent incidents have highlighted the importance of the priority and the investments we need to make."

The origins of the problem are clear. The vast majority of the nation's water systems are small and publicly owned, with limited resources and aging infrastructure. As they turned to digital systems and monitors to boost efficiency while saving money and staff, they failed to install the safeguards and carry out employee training needed to secure the resulting vulnerabilities. “Every one of them had one guiding principle over the last 50 years: increased automation to lower the size of the workforce to keep costs down," Montgomery said. “Along with that, there should have been an investment in the cybersecurity of the infrastructure. But that did not happen."

Traditionally focused on physical risks, such as natural hazards, burst pipes and on-site intruders, most water systems also have little or no in-house IT staff. The pandemic, which encouraged remote management, has only made the problem worse. In testimony last month to the House Homeland Security Committee, former CISA Director Chris Krebs called Oldsmar's vulnerability “probably the rule rather than the exception. ... These are municipal facilities that do not have sufficient resources to have robust security programs. That's just the way it goes."

The industrial control systems that water districts use to manage valves, pipes and other infrastructure are notoriously open to attack. A 2018 study by IBM and a private security company found 17 major vulnerabilities in equipment widely deployed in “smart cities," a term that refers to municipalities that manage a wide array of their systems — anything from water treatment plants to parking meters and street lamps — via the Internet. Among the security problems: Every product the group examined was still using the default passwords (such as “admin") they came with in the box, allowing “even the most novice hacker to easily gain access to these devices." A 2018 study by the firm Positive Technologies reported that it was able to penetrate nearly three-fourths of industrial organizations it investigated, revealing gaps offering hackers “plenty of opportunity to access critical equipment." The most common vulnerabilities: remote-access networks, obvious passwords and software so old that the manufacturer had stopped making fixes to protect against intruders. The report found that vulnerabilities known for years often “remain untouched, because organizations are afraid to make any changes that might cause downtime."

These industrial control systems are considered such obvious targets that hacking contests use them as quarry. At the DEFCON computer security conference, an “ICS Village" let curious programmers try to break into devices set up inside a Las Vegas hotel room — demos not connected to real-life systems — in an effort to expose weaknesses. At the event in 2018, one water pipe control system, likely used for a commercial building, had its computer screen defaced with graffiti-type messages.

The exact number of attacks on water utilities remains unknown. Many go undetected or unreported, and no federal law requires disclosure, even to regulators or law enforcement. Michael Arceneaux, managing director of the Water Information Sharing and Analysis Center, an industry group promoting cybersecurity, said water systems often refuse to reveal breaches, even to his group, out of fear that they will somehow reveal their vulnerabilities to other hackers. “It's not something members wanted potentially floating around in some database."

The episodes that have been made public reveal a growing array of threats, from random vandalism and disgruntled employees to identity theft and ransomware.

In Oldsmar, for example, the FBI and the Pinellas County Sheriff's Office, which are jointly investigating, have already revealed multiple lapses. The attack took place at the city's water treatment plant, which purifies groundwater for drinking using filters and chemicals, including small amounts of sodium hydroxide. Commonly known as lye, it is used to reduce the water's acidity. (In considerably stronger concentrations, sodium hydroxide is also a chief ingredient in drain cleaner.)

The hack began around 8 a.m. on Feb. 5, when a plant operator noticed someone had remotely accessed the computer system that monitors and controls the chemical levels added to the water. The hackers entered through a remote access software program called TeamViewer. The city had actually replaced TeamViewer six months earlier, but it never disconnected the program, according to county Sheriff Bob Gualtieri. Logging into the system remotely was a breeze: The water plant's computers all used a single shared password, required no two-factor verification and had no firewall in place protecting the controls from the internet, according to FBI findings described in a Massachusetts state advisory. A final vulnerability: All the computers were still running on Windows 7, a decade-old, discontinued operating system; Microsoft had stopped issuing regular software updates to plug its security vulnerabilities in January 2020.

After noticing the hacker's morning log-in, Gualtieri later said at the press conference, the plant operator “didn't think much of it" and didn't contact anyone since other city employees routinely accessed the system remotely. (It's not clear why the attacker's use of the replaced TeamViewer software didn't immediately raise concern.)

The hacker reappeared about 1:30 p.m., this time visibly taking over the computer, mousing around for three to five minutes and opening the plant's control system software. After ratcheting up the water's sodium hydroxide level from 100 parts per million to 1,100 parts per million, the intruder departed.

After watching all this, the Oldsmar plant operator quickly lowered the sodium hydroxide level and called his boss. The city contacted the county sheriff's office nearly three hours later, at 4:17 p.m., according to an incident report on the event.

Oldsmar officials maintained that the public was never in danger. They noted that it would have taken at least 24 hours for poisoned water to start flowing out of kitchen taps, and that even if the onsite operator hadn't intervened, the plant had backup systems monitoring the water's chemical balance that would have sounded alarms long before then.

A small number of other incidents present the nightmarish “what-if" scenarios that scare experts, particularly from so-called state actors. Both Russia and Iran have been implicated in such accounts, according to government reports and legal actions. One such episode occurred in 2013, when a state-backed hacker sitting at his keyboard in Iran breached the computer controls at the Bowman Dam in suburban Rye, New York, with a presumed plan to open the sluice gates. The gates happened to have been manually disconnected at the time for maintenance, and the dam was actually just a narrow, 20-foot-high structure holding back a babbling brook. Federal intelligence officials speculated that the Iranians had actually intended to seize controls at the massive Arthur R. Bowman Dam in Oregon, where similar actions would have flooded thousands of homes. A federal indictment later charged that the Bowman Dam hacker worked for Iran's Revolutionary Guard and was part of a seven-man team that successfully breached America's biggest banks, paralyzing their computer servers and blocking customers from accessing their accounts online. The hacker remains at large, and on the FBI's “most wanted" list. In 2019, Revolutionary Guard hackers struck again, deploying malware to launch an ultimately unsuccessful attack on a municipal water system in Israel.

In recent years, three U.S. states — New York, New Jersey and Connecticut — decided to go beyond the federal rules and adopted tougher cybersecurity measures for the water utilities within their borders. After passing new legislation, New Jersey required all public water systems with internet-connected controls to develop a cybersecurity risk-mitigation plan within 120 days, submit it to the state, create a process for reporting all cyberattacks and join a special state-government clearinghouse promoting strong cybersecurity practices. Connecticut launched a “Cybersecurity Action Plan" and began holding private annual meetings with each of the state's largest water (and other) utilities to scrutinize the adequacy of their cyberdefenses.

For its part, New York amended its public health law to require water systems to conduct assessments of their susceptibility to cyberattacks and submit them to the state within a year. A team at the state comptroller's office has also conducted seven cybersecurity audits of municipal water systems, in each case posting the audit publicly while reserving some findings for confidential briefings to avoid offering hackers a road map of vulnerabilities. Its audit of the city of Syracuse's water system, for example, found shared user passwords and accounts that hadn't been disabled long after employees left the city. The Binghamton audit discovered a video on the water department's own webpage showcasing the treatment plant's controls.

“There's a tremendous amount of work that needs to be done to shore up the systems," said assistant New York state comptroller Randy Partridge, who oversees the water system audits. Since January 2019, he said, his auditors have issued 239 findings at various municipal facilities (including water systems) regarding weak password security alone. “It's a health and safety risk for any resident that lives in our local government. No community can really survive for any length of time without access to potable water."

Arthur House, who served as Connecticut's chief cybersecurity risk officer, said: “I hope it doesn't take the poisoning of a lot of people or a catastrophic shutdown for people to say, 'Omigosh, this is serious.' The federal government has to have a role on this. You cannot leave something that would cripple us as a country solely in the hands of 50 different states."

As America struggles to assess the damage from the devastating SolarWinds cyberattack discovered in December, ProPublica has learned of a promising defense that could shore up the vulnerability the hackers exploited: a system the federal government funded but has never required its vendors to use.

The massive breach, which U.S. intelligence agencies say was “likely Russian in origin," penetrated the computer systems of critical federal agencies, including the Department of Homeland Security, the Treasury Department, the National Institutes of Health and the Department of Justice, as well as a number of Fortune 500 corporations. The hackers remained undetected, free to forage, for months.

The hackers infiltrated the systems by inserting malware into routine software updates that SolarWinds sent to customers to install on its products, which are used to monitor internal computer networks. Software updates customarily add new features, remove bugs and boost security. But in this instance, the hackers commandeered the process by slipping in malicious code, creating secret portals (called “back doors") that granted them access to an untold bounty of government and company secrets.

The incursion became the latest — and, it appears, by far the worst — in a string of hacks targeting the software supply chain. Cybersecurity experts have voiced concern for years that existing defenses, which focus on attacks against individual end users, fail to spot malware planted in downloads from trusted software suppliers. Such attacks are especially worrisome because of their ability to rapidly distribute malicious computer code to tens of thousands of unwitting customers.

This problem spurred development of a new approach, backed by $2.2 million in federal grants and available for free, aimed at providing end-to-end protection for the entire software supply pipeline. Named in-toto (Latin for “as a whole"), it is the work of a team of academics led by Justin Cappos, an associate computer science and engineering professor at New York University. Cappos, 43, has made securing the software supply chain his life's work. In 2013, Popular Science named him as one of its “Brilliant Ten" scientists under 40.

Cappos and his colleagues believe that the in-toto system, if widely deployed, could have blocked or minimized the damage from the SolarWinds attack. But that didn't happen: The federal government has taken no steps to require its software vendors, such as SolarWinds, to adopt it. Indeed, no government agency has even inquired about it, according to Cappos.

“In security, you almost never go from making something possible to impossible," Cappos told ProPublica, during two video interviews from Shanghai, where he is teaching. “You go from making it easy to making it hard. We would have made it much harder for the [SolarWinds] attackers, and most likely would have stopped the attack." Although the SolarWinds breach was a “really sneaky" approach, Cappos said, “in-toto definitely can protect against this. It's very possible to catch it."

In-toto's system has supporters among experts in the government and corporations. When ProPublica asked Robert Beverly, who oversees in-toto's federal grant as a program director at the National Science Foundation, whether using in-toto could have saved the government from the hack, he replied, “Absolutely. There seems to be some strong evidence that had some of the, or all of the, in-toto technologies been in place, this would have been mitigated to some extent." Beverly, whose NSF responsibilities include “cybersecurity innovation for cyberinfrastructure" and who is on leave from his post as a computer science professor at the Naval Postgraduate School, added that it's impossible to know for sure what impact in-toto would have had, and that the system remains at an early stage of adoption. “Unfortunately," said Beverly, “it often takes some of these kinds of events to convince people to use these kinds of technologies."

Some companies have embraced in-toto, and others, like Microsoft, have expressed interest. “I am a big fan of in-toto," Kay Williams, head of Microsoft's initiatives in open source and supply-chain security, said in an email to ProPublica. A second Microsoft program manager, Ralph Squillace, praised in-toto in a recent NYU press release for applying “precisely to the problems of supply chain confidence the community expects distributed applications to have in the real world." (After Williams' initial response, Microsoft declined to comment further.)

One senator blasted the government's failure to use a system it paid for. “The U.S. government invested millions of dollars in developing technology that can protect against this threat, and while several large technology companies have already adopted it, they are the exception," said Sen. Ron Wyden, D-Ore., a member of the Senate Intelligence Committee. “The government can speed up industry adoption of this best practice by requiring every government contractor to implement the best available technology to protect their supply chains."

The in-toto system requires software vendors to map out their process for assembling computer code that will be sent to customers, and it records what's done at each step along the way. It then verifies electronically that no hacker has inserted something in between steps. Immediately before installation, a pre-installed tool automatically runs a final check to make sure that what the customer received matches the final product the software vendor generated for delivery, confirming that it wasn't tampered with in transit.

Cappos and a team of colleagues have worked to develop the in-toto approach for years. It's been up and running since 2018. The project received a three-year grant from the National Science Foundation that year, aimed at promoting “widespread practical use" of in-toto. (Later in 2018, President Donald Trump signed the Federal Acquisition Supply Chain Security Act, aimed at protecting government secrets from software supply-chain threats.)

In-toto could block and reveal countless cyberattacks that currently go undetected, according to Cappos, whose team includes Santiago Torres-Arias, an assistant electrical and computer engineering professor at Purdue University, and Reza Curtmola, co-director of the New Jersey Institute of Technology's Cybersecurity Research Center. In an August 2019 paper and presentation to the USENIX computer conference, titled “in-toto: Providing farm-to-table guarantees for bits and bytes," Cappos' team reported studying 30 major supply-chain breaches dating back to 2010. In-toto, they concluded, would have prevented between 83% and 100% of those attacks.

“It's available to everyone for free, paid for by the government, and should be used by everyone," said Cappos. “People may still be able to break in and try to hack around it. But this is a necessary first step and will catch a ton of these things." The slow pace of adoption is “really disappointing," Cappos added. “In the long game, we'll win. I just don't know that we want to go through the pain that it'll take for everyone to wise up."

One of in-toto's earliest adopters, starting in 2018, was Datadog, a SolarWinds competitor that provides monitoring software for internet cloud applications. Now a publicly traded company with 2020 revenues of nearly $600 million, its customers include Nasdaq, Whole Foods and Samsung. Datadog uses in-toto to protect the security of its software updates. In an NYU press release, Datadog staff security engineer Trishank Kuppusamy, who worked on the program's design and implementation, said that what distinguishes in-toto is that it “has been designed against a very strong threat model that includes nation-state attackers." (Datadog did not reply to ProPublica's requests for comment.)

The General Services Administration, which provides access to software for federal government agencies, still lists SolarWinds products available for purchase. But it said in a statement that “compromised versions" of SolarWinds programs identified by DHS are no longer available.

SolarWinds itself declined to weigh in on whether its hack could have been prevented. “We are not going to speculate on in-toto and its capabilities," a spokesman said in an emailed statement. “We are focused on protecting our customers, hardening our security and collaborating with the industry to understand the attack and prevent similar attacks in the future."

Previously little known to the general public, SolarWinds is a public company based in Austin, Texas, with projected 2020 revenues of just over $1 billion. It boasts of providing software to 320,000 customers in 199 countries, including 499 of the Fortune 500 companies. In a recent SEC filing, the company said its flagship Orion products, the vehicle for the cyberattack, provide about 45% of its revenues. A SolarWinds slogan: “We make IT look easy."

After the hack was discovered, SolarWinds' stock plunged, and it is now facing shareholder lawsuits. The company has shifted aggressively into damage-control mode, hiring CrowdStrike, a top cybersecurity firm; elite Washington lobbyists; a crisis-communications advisor; and the newly formed consulting team of Christopher Krebs, the former director of the Cybersecurity and Infrastructure Security Agency (who was famously fired for contradicting Trump's claims of mass voting fraud) and Alex Stamos, former security chief at Facebook.

News of what's now known as the SolarWinds attack first came on Dec. 8. That's when FireEye, perhaps the nation's preeminent hack-hunter, announced that it had itself fallen victim to a “highly sophisticated state-sponsored adversary" that had broken into its servers and stolen its “Red Team tools," which FireEye uses to try to hack into the computer networks of its clients as a test of their cyber-defenses. FireEye soon discovered the attackers had gained access through corrupted updates to the SolarWinds Orion network-monitoring software that it used.

On the evening of Dec. 13, CISA issued an emergency directive, identifying SolarWinds as ground zero for the hack and alerting federal agencies using Orion products to disconnect them immediately. Over the following weeks, investigators discovered that SolarWinds had been targeted back in early September 2019, when hackers started testing their ability to inject code into its software updates. After remaining undetected for months, they inserted malware in new updates between February and June 2020. SolarWinds estimated these infected updates affected “fewer than 18,000 of its customers."

Precisely what the hackers saw, and stole, has yet to be determined and is under investigation. But the full impact of the breach is becoming clearer, as we now know it touches several tech companies, including Microsoft. The software giant has also labored to limit the damage by helping seize an internet domain in the U.S. that the hackers used to siphon data from some SolarWinds customers.

Stamos told the Financial Times, in an interview after being hired to help SolarWinds, that he believed the attackers had embedded hidden code that would continue to give them access to companies and government agencies for years. He compared the situation to Belgian and French farmers going out into their fields where two world wars were fought and discovering an “iron harvest" of unexploded ordnance each spring.

Dmitri Alperovitch, who co-founded CrowdStrike (the cybersecurity firm SolarWinds has hired to investigate the hack) before leaving last year to start a nonprofit policy group, said he thinks that, in theory, the in-toto system could work. But he warned that software is so complex, with many products and companies in the supply chain, that no one defense is a panacea. Still, he agrees that in-toto could provide protection, and said “it's always a good thing to have more protection for supply chains."

Russian intelligence services have clearly identified supply-chain attacks “as a much better way to get in," offering “a much bigger set of targets," Alperovitch said. “This is an indictment of the entire cybersecurity industry, as well as the intelligence community, that they were able to orchestrate such a broad, sweeping attack right under our noses."