SANS Industrial Control Systems Security Blog

Pictures and Theories May Help, but Data Will Set Us Free

This blog post was written by Tim Conway SANS Technical Director - ICS/SCADA Programs.

In reviewing the open source information available on the most recent Ukraine activity, I have seen numerous references to either a device failure or a cyber-attack as being the leading theories behind the recent electric system event. As the asset owner conducts an ongoing investigation as to what may have occurred, I found myself thinking about some of the cyber-attack vector possibilities that might be considered by an organization investigating a similar unexplained operation.

For now, this is just a collection of thoughts intended for comment and discussion. I have assembled some concepts taught in the ICS410 course, the ICS456 course, and a couple of links that I think may be useful from our STH awareness products.

First, the STH awareness products that I thought may be relevant for your viewing and listening pleasure is a short video walking through the technology and tools utilized by control center operators in the routine operation of the electric system:

And a short video highlighting a fictitious targeted adversary campaign that starts as a Stage 1 campaign and continues on to a Stage 2 campaign with operational impacts:

Now for the Pictures...

First, consider a big picture view of some possible cyber-attack vectors that could be leveraged by an adversary with a goal of misoperating a component or multiple components of an electric system. This is not intended to be an exhaustive representation of all possible attack vectors and I am not covering the physical attack vectors in any detail. The general image being used in each discussion is an overly simplistic drawing quickly made to represent three primary environments:

1) a control center with an operator workstation or HMI, a combo data and display server identified as a control server and a Communications front end for all SCADA communications;

2) the communications infrastructure that enables communications from the control center to the field locations;

3) and the substation environment that contains numerous types of devices with the capability to provide data and system values to the operator as well as perform actions either autonomously based on programming or as directed by an operator.

The following image contains the representation of the three environments, along with the basic attack vectors identified.


Now with the basics out of the way, let's consider some specifics. I will walk through 5 perspectives to consider if you were investigating unexplained field operations and you have reason to believe it was possibly caused by a targeted cyber-attack.

First, as the world learned from the events that occurred on Dec 23, 2015 in Ukraine. We must consider the possibility of our own tools and technology being misused against us through direct operator HMI hijacking. In addition, consider the potential for shell level operator workstation compromise with commands being scripted or manually sent from the operator workstation to the control servers with no visible indication. Access logs to the environment including the workstation, event logs from the control servers and the front ends, and operator application level events would help in determining if this approach was attempted. The figure below highlights the attack vector, however if a rogue client was utilized, the ability to identify and detect would include the server event logs and network activity logs, but would not include the client or application level logging reviews as those data sets would likely not be available to system defenders.


Next, we will consider server attack vectors which might be utilized if protections or controls have been put in place or if an adversary determined it would be easier to manipulate the data server or display servers directly through the applications, by sending commands directly to the front end server, or even by going directly to the front end server and issuing commands from there to the field devices. As your investigation attempts to identify this condition you would be looking for sequence of event mismatches. Meaning you would likely be able to locate corresponding event records as you walk the event back from the field device and piece together the artifacts you may find. These might include a record from a field device, an artifact within the communications environment, a record in the front ends, or a record in the control servers. However the key point of information in this process is the information that does not exist, i.e. if you cannot discover any records, or event data on the operator HMI then perhaps the adversary did not use it in the achieving the objective, likewise if there is no record within the control servers, perhaps the adversary manipulated the front end directly, and so on down the line. This may be an effective way to locate assets that were used and to identify an attack vector, however it should be anticipated that records, logs, events, etc... may not exist as the adversary may have taken steps to remove those records. Also, a very real problem is that many field components do not perform adequate levels of cyber event logging to be helpful in this scenario.


Moving on to attack vector 3, the network based attack vectors where adversaries are performing reconnaissance, data collection or manipulating network switching architecture, VLANs, or routing perimeter controls. They might pivot from business environments or leverage remote connections to gain a position to perform network based attacks like a MitM attack, which while a very effective attack vector, would also be very noisy. However, this brings the age old question to mind - if a tree fell in the forest and nobody is there to hear it, does it make a sound? When I say these network attack vector activities are noisy, it is from the perspective of someone who is listening and can further distinguish what abnormal "sounds" are. I must at this point call upon my Padawan learnings from Robert M. Lee and the ICS 515 course he authored. There are a number of items to list here in regards to what you would look for to identify this attack vector, but I will wait patiently for my friend to respond to this blog with a list of artifacts and evidence that would be detected, and identified, if a sound active defense approach to managing and responding to network based attacks were in place.


We are almost at the end of this blog, and as far as attack vectors go, we are now outside the control center environment. We are now considering the other end of the communication and the attack vectors located in the field environments. Substation environments have some unique attack vectors associated with physical attack elements because they are typically unstaffed, lightly monitored, and have delayed response times due to their distance from response teams as well as being in fairly remote locations. For these environments, consider performing physical site surveys to determine if any unauthorized or undocumented communications paths are identified, as well as conducting a review of the running configurations and running firmware versions on field devices in the environment. If any direct access paths exist for field personnel, conduct access log reviews and compare to operational event information.


Lastly, we will look at the communications path between these two environments. The approach and method used to investigate this environment will vary depending on communications infrastructure technology, available logging, and cooperation from any 3rd party service providers. Items to consider within this attack vector include the ability for an adversary to misuse the trusted communications path to send control signals from a field site to another field site, or possibly from a field site up to the front end and then back down to additional field sites. Also, consider the possibility of an adversary with appropriate test sets and tools to send operations commands directly from within this communication path at a protocol level. If this attack vector was used, you may be investigating sequence of events, and be unable to identify any control center assets with visibility to the operations performed. This type of investigation and attribution will drive organizations to adopt technologies with the capabilities needed to capture at this level of the communications path or within the field environment to allow correlation actions.


As we wait for investigation details from the latest Ukrainian event to emerge, I thought it would be interesting to start a conversation on what a combined operations and cybersecurity investigation might need to consider when identifying an adversary attack vector. With an attack vector identified, many steps will follow - if public information emerges confirming a cyber-enable attack, stay tuned here as there will likely be additional blogs and possibly a SANS ICS Defense Use Case (DUC) on the event.

And now we wait for more details and possibly some additional discussion on attack vectors not discussed here.

Tim Conway

If you're interest in attending ICS456: Essentials for NERC Critical Information Protection, I'm teaching at these upcoming events.

To view all our upcoming courses and events, click here

Free Stuff Reminder