03/31/2023 Update:
On March 20, 2023, former Oldsmar City Manager Al Braithwaite explained during a panel discussion at the American Society for Public Administration's Annual Conference, that the 2021 incident was not caused by a cyber attack. According to him, it was just a case of an employee mistakenly clicking on the wrong buttons, before alerting his superiors to his error. The FBI declined to comment on the investigation or on any conclusions it drew. The cyber attack scenario described at the time by the city sheriff (see below) is still considered as one of the most common attack tactic and calls for strong security measures such as managed remote access and multi factor authentication. But there are reasonable doubts that unsecure remote access was the cause of this particular incident. More details in this article.

On February 8, 2021, the City of Oldsmar, Florida gave a press conference to disclose "an unlawful intrusion to the city's water treatment system." Someone on the Internet successfully accessed the computer controlling the chemicals used to treat drinking water for the city and changed the level of sodium hydroxide to 11,100 parts per million (ppm), a significant increase from the normal amount of 100 ppm.

Sodium hydroxide (NaOH), also known as lye or caustic soda, is a corrosive chemical used in low concentrations to regulate pH level of drinking water and protect water pipes. In higher levels, it is toxic and can damage human tissues. Luckily, the attack was immediately detected, and normal operating parameters were restored before any harm could be done.

While security controls such as automated pH testing would have prevented the poisoned water from being distributed, this story highlights how much critical infrastructures, such as water utilities, have become vulnerable to cyber attacks. The City of Oldsmar should be credited for its transparency and candid explanation of what happened.

What happened?

According to the city sheriff, someone on the Internet managed to connect to the TeamViewer software that was installed on the workstation used to control the water treatment process. TeamViewer is a popular tool used by technicians and support personnel to gain remote access to a computer and use it as if they were physically in front of it.

An employee of the water treatment plant saw the mouse cursor moving on the workstation screen performing unauthorized tasks. He raised the alarm and thwarted the attack before anyone was harmed. The sheriff had no information on how the hacker gained access to the plant's IT network or how it was able to log into the TeamViewer application.

Most IT professionals would be very surprised. How did this industrial workstation become accessible from the Internet? How could remote access capabilities be installed without proper security policies and strong authentication being enforced? How could someone set process parameters to dangerous levels without specific authorizations and controls?

Unfortunately, this type of situation is not uncommon. Operations staff and equipment vendors need remote access into industrial machines and sometimes install unapproved solutions by themselves. Most industrial equipment has no cybersecurity feature. Default passwords are widely used for technicians to gain easier access to machines. Many industrial organizations have not built a demilitarized zone (DMZ) to isolate industrial networks, leaving these vulnerable devices accessible from IT networks. And the list could go on.

Making cybersecurity top of mind

As for all industry verticals, digitization brings tremendous benefits to the water industry. But it also increases the threat landscape. The solution is not about refraining from modernizing operations. It's about building strong security protocols into industry digitization projects from day 1. It's also about gaining factual information on the organization's security posture so that IT and operations teams can start working together securing industrial networks.

In the United States, the America's Water Infrastructure Act (AWIA) requires water utilities serving more than 3,300 people to develop or update risk assessments and Emergency Response Plans (ERPs). In the European Union, the NIS Directive classifies water utilities as critical infrastructures that must identify their cybersecurity risks, train their personnel, and build measures for response and recovery. This, and other recommendations, are well described in the whitepaper Cisco recently published on cybersecurity for water utilities.

What can water utilities do next?

Public safety, digital operations, and regulations demand water utilities to deploy reliable and robust cybersecurity. In the short term, installing a firewall to isolate the industrial network and blocking unauthorized traffic (such as TeamViewer) is the mandatory first step.

In parallel, two main efforts must be made to implement security best practices and enable modern connected industrial operations:

1. Gaining clear visibility on all the industrial devices, systems and software that have been deployed over the years. Long story short: you cannot protect what you don't know.
2. Enabling effective collaboration between IT and OT teams to design and enforce security policies that will help operations staff do their work effectively without disrupting production.

Gaining visibility into your industrial infrastructure can be done manually. You can also use OT security software such as Cisco Cyber Vision to automatically build a detailed asset inventory listing all devices, software, vulnerabilities, remote accesses, network relationships, process behaviors, etc. Such a tool provides precise information to build a plan for improvement. It can also monitor operations to ensure that any malicious activity is immediately detected by something other than luck.

Now that IT and OT have a shared understanding of the situation, they can work together in building a security framework that suits operational constraints. Both teams have everything to win. As explained in this white paper, when it comes to OT security, united we stand, divided we fall.

To learn more about how you can secure your IoT/OT infrastructure,

visit our IoT Security page or contact us.

Subscribe to the Cisco IoT Security Newsletter to get the latest

industry news on IoT Security delivered straight to your inbox.