Many organizations, including portions of our national critical infrastructure, rely on industrial control systems (ICS) and supervisory control and data acquisition systems (SCADA) to automate critical processes. This includes manufacturing, water treatment, power generation and distribution, and transportation. These systems present unique challenges to teams that perform penetration testing on them. While it is clear that these systems should be tested for security flaws, public and personnel safety must take priority, and continuity of service must be considered for critical systems.
A report from Sandia National Labs provides the public with some insight into challenges faced by penetration testers seeking to effectively test control systems safely. Though the report is from 2005, it is a rare glimpse at the kinds of incidents that occur, to this day, when automated testing is carelessly performed against SCADA/ICS systems:
- An automated scanning tool causing a 9-foot robotic arm to activate and swing 180 degrees, luckily with no one in striking distance.
- The same tool causing the destruction of $50,000 worth of material on a manufacturing line.
- A penetration test halting a gas pipeline’s operations for four hours.
Improper testing can recklessly lead to downtime, destruction, or injury. When I see industrial accidents and natural disasters in the news, such as the recent flooding in Texas, I worry about the potential for security testing of devices (such as those controlling dams and floodgates) becoming the catalyst for a major incident. The primary goal of any tester of this should be to do no harm.
When we perform advanced penetration testing for a client, our overall goal is to identify as many vulnerabilities as possible, and to determine the impact of those vulnerabilities being exploited by a nation-state level threat. While we use tools, tactics, and procedures that emulate those of advanced threat groups, we have some self-imposed restrictions and special procedures we employ when we perform tests for clients that rely on SCADA/ICS systems. Real-world testing of the entirety of an organization’s network is important, but not as important as ensuring safety of the public, personnel, and systems that are critical to continued operations.
When we scope a test with an organization, we have a frank conversation about the potential impact automated testing can have on a SCADA, ICS, or other automation system. At this point, more often than not, we are told by the client of previous tests (in-house and third-party) that resulted in down-time or damaged hardware. We ask that the client provide IP addresses and subnets that contain automation hardware and software.
While testing SCADA/ICS systems shouldn’t be avoided, it must be done with care. If a test environment that replicates the production environment with high fidelity exists, then it is much easier to conduct testing there. Even then, some examination of the production environment should be undertaken to ensure that devices and software are configured as they are in the test environment.
Often, a test environment does not exist. When this is the case, the production environment must be treated with care. Depending on the organization’s needs and the importance of devices to operations, this testing may have to occur on a schedule that allows for in-person observation of the device and the physical system it controls. A plan should be put into place for resetting and recovering devices that could fail during testing.
A key part of avoiding problems is avoiding automated vulnerability scanners and other tools that “throw the book” of attacks at a system. Many products in this field have not been thoroughly tested for resilience against “bad” input, and fail quickly when network traffic varies too far from the norm. Human-driven testing by a team experienced in testing control systems can slow the pace, identify potential problems as they arise, and provide a sanity check for potentially dangerous operations. This prevents clearly-avoidable problems, such as identifying and avoiding protocol/menu options that have the potential to cause harm, and makes it easier to identify the root cause when an incident does occur.
Penetration testing of SCADA, ICS, and other automation devices should be carried out by a team that has experience with control systems, embedded systems, and that has the capability to find new and unique vulnerabilities. Simply running an automated tool, going through a list of publicly-known vulnerabilities, and “checking the box” on having done a test does not help to identify serious problems in control system security. Ultimately, for stakeholders, defenses must be layered around vulnerable systems, in order to reduce the reliance on the security software and firmware that see infrequent patches and security review.
For weekly insights into cybersecurity, please sign up here: