SANS Industrial Control Systems Security Blog

ICS Cross-Industry Learning: Water Distribution Systems and How to Earn Trust of Operations

SANS ICS

In this third installment of the ICS Cross-Industry Learning series (Part 1 on Solution Polymer Chemical Process may be found here and Part 2 on Electric Transmission and Distribution here) I am very excited to have Jake Brodsky write on water distribution systems and more importantly key points to know which can help earn trust in operations. Jake is one of my trusted friends and a true subject matter expert in the industry. He's also the co-editor and co-author of the Handbook of SCADA/Control Systems Security with Robert (Bob) Radvanovsky which has it's second edition publishing soon (they allowed me to write a chapter in this version as well which was a lot of fun).

The following was written by Jake Brodsky:

 

___________________________________________________________________________________

Not all integrity cues for SCADA are or can be network or
controller-oriented. There are some basic methods one can use to detect
problems early. This is a discussion of those methods.

To illustrate them, I will cite some simple examples of how some water
distribution systems work. It is not authoritative, there are
significantly different methods that others use to distribute water.
They all have a purpose. Some are optimized for saving energy, some for
reliability, some for ease of maintenance or low cost.

In any of these cases, water distribution systems are usually operated
according to altitude. The water source from a filtration plant or a
well is typically pumped up to elevated storage of some sort and then
allowed to drain during periods of higher demand. Some water
distribution systems may need to cover a wider range of altitudes. The
problem is that if one were to serve water from an altitude 300 feet
above the zone of interest, the pressure would be about 130 PSI. That's
more than most common household plumbing and appliances such as clothes
and dish washers can handle reliably.

That is why water distribution is divided up in to service zones of
altitude. In any given zone, there are usually one or more elevated
water storage tanks of some sort. It could be a reservoir on a hill
side, an elevated water storage tank (there are many shapes and
disguises for this), or what is called a Standpipe (a tall cylindrical
tank rising more than 100 feet above the surrounding terrain). In all
these cases, the water pressure is measured at the base of each tank
from a carefully surveyed altitude.

With each tank for a single zone, the water levels will be at
approximately the same altitude, plus or minus some nominal amount of
friction loss in the pipe, known as head loss. That difference is
typically around a foot or less. The tanks closest to the source of
water for that zone, whether it is a pressure reducing valve from a
higher altitude zone or a pumping station from a lower altitude zone,
will usually have a slightly higher altitude than the storage tanks with
more pipe head loss while they are filling.

In a well planned system, all water tanks will have the same maximum
altitude for the water. However, cities tend to grow organically and
those plans are often very different 30 and 40 years later. As a result,
it is not unusual to have some smaller tanks that may not have the same
maximum fill altitude as other tanks in the zone. In cases like that,
there are usually two pressure gauges. The first is at the tank itself.
It is used to measure the altitude of the water. The other is typically
underground on the distribution system side of the valve that fills the
tank. That valve is known to the water distribution operators as an
"Altitude valve." The purpose of an Altitude valve is to cut off flow to
the tank to prevent it from being overfilled while filling the other
tanks in the zone to higher altitudes. In that situation, the tank water
altitude reaches a certain level and then the valve is configured to
close. It won't open until the distribution pressure altitude drops
below a level of where the water in the tank is (plus some margin for
instrument error).

The Altitude valve closing setpoint is almost always configured locally.
There is little practical value in changing it remotely. The tank
overflow point is not likely to change. The methods may be a simple
electrically operated pilot valve that closes the Altitude valve, or it
may be a completely hydraulic system where the pilot pressure has to
reach a certain absolute level and that triggers the valve to close. The
control system does not use the PLC or the RTU features.

Most of the time, however, water utilities tend not to fill tanks that
aggressively. It is usually more common for the altitude valve to be
fully open. In that case, the system and tank altitude gauges should
read nearly the same when compensating for altitude differences.
However, due to head loss through the pipe and the valve, if the tank is
draining, the tank altitude gauge will read a very slightly higher
altitude than the distribution gauge. If it is filling from the
distribution side, the system gauge will read a very slightly higher
altitude (an inch or less).

These are the sorts of cues that an experienced operator may look for
when one or more of the gauge readings may be suspect. Gauges at
different tanks in the same zone should read very similarly. If the
altitude valve is fully open, the system and tank pressure gauges should
read very similarly. Thus an RTU can stop all communication, and it is
still not an emergency because there are other tanks on the system that
can give some indication of the water altitude.

The reason for explaining all this is to illustrate why attacking one
reading from one gauge is not enough. A successful attack will need a
full performance replay of that particular distribution system. If the
zone reports to more than one SCADA master, it means that both masters
will have to be attacked simultaneously.

Furthermore, a sudden replay attack will manifest itself in all the
tanks on the zone by simultaneously causing transients to show up on the
tank level graphs. It is never normal to see transients in a water
storage tank. Water does not suddenly appear or disappear from a tank.
This will rapidly manifest itself in to alarms when calculating
consumption (how much water flowed in to the zone, subtracting how much
went in to storage). At the very least, the operators will be looking
for a pipeline break. The reaction to this sort of activity is to
increase flow in to the zone. If there is no reaction to the increased
flow, operators will start suspecting the SCADA system.

And THAT is when Operations Technology (OT) staff get the call to find
out what's going on. It also means by the time OT receives a call, the
problem has been noticed for a few hours already, and OT staff have to
be able to come up to speed fast. It is important to understand at least
the basics of what the operators have to deal with every day. That way,
when an operator comes says "The 425 zone readings aren't acting right"
someone in OT should know what questions to ask and where to start looking.

This is no time for a help desk and a "your call is very important to
us" response. OT staff should head for the field and the operations
center first, and document what happened after the fact, just like all
other field staff.

In other words, by teaching operators to look for clues like this, and
by maintaining good survey data and instrumentation calibration, a
replay attack will be noticed even if none of these events trigger
alarms in a Network Operations Center. The operators may not be identify
the causes right away as a hack because they have to eliminate other
possibilities such as a pipe break. However, someone will be going out
to the site to have a look. If they find nothing wrong, that's a big red
flag that things aren't adding up.

OT is not just about computers or even about industrial controllers. The
successful OT group is a hard-hat wearing, steel toed, safety aware
engineer too. OT staff need to understand what the routine operation
looks like, and what effect various failure modes can have on a working
system. And when operators start talking, the OT staff need to
understand the concepts behind what they're saying. Only then is OT
going to gain the trust and respect from Operations that they deserve.

Bio: Jake Brodsky is a Professional Engineer of Control Systems with 30 years experience in the water and waste-water utility business. He is chair of the DNP Users Group, and co-editor and co-author of the Handbook of SCADA/Control Systems Security, published by CRC Press.