By Fred Gordy Director of Cybersecurity at Intelligent Buildings, LLC, (CS)²AI Fellow
Threats to building control systems (BCS’s) have grown exponentially in the past two years. More importantly, the attacks on BCS has grown at an even higher rate. How can that be? Easy—There have always been threats to control systems. However, successful attacks are becoming more frequent, growing in intensity, and wreaking more havoc. In 2019, there was a 400% increase in attacks and 2020 is shaping up to a 600% increase(1).
Attacks are not the only thing causing disruption of service to BCS’s. Informational technology (IT) is becoming more involved in securing control systems and their networks. This is a good thing; however, IT software, processes, and procedures are also causing interruptions in operations and damage. These systems and their devices require a different approach.
The National Institute of Standards and Technology (NIST) realized that control systems cannot be managed like IT systems. The NIST report IR 8228(2) summarizes three key statements:
Many Internet of Things (IoT) devices interact with the physical world in ways conventional IT devices usually do not.
Many IoT devices cannot be accessed, managed, or monitored in the same ways conventional IT devices can.
The availability, efficiency, and effectiveness of cybersecurity and privacy capabilities are often different for IoT devices than conventional IT devices.
Over the past few years, Intelligent Buildings, LLC (IB) has completed cybersecurity risk assessments across the U.S., Canada, and overseas. The results of these assessments showcase that attacks and self-inflicted wounds are typically caused by well-meaning facility personnel not following basic best practices.
The most common attack is ransomware. Ransomware is malicious software that locks all the files on a PC or server until either a ransom is paid, or the PC or server is wiped clean and reloaded. The delivery of ransomware is typically through email, but it can come from social media sites. Ransomware makes up 80% of the attacks to control systems(1).
In all the cases IB has seen, ransomware was 100% avoidable. Historically, the BCS is located in the engineer’s office and is used like a workstation. Facility management (FM) staff tend to use it to check their company and personal email and to look at social media, all of which have been the ransomware delivery mechanism. Had operational technology (OT) policies been in place and enforced, these attacks could have been avoided.
BASIC OT POLICY AND PROCEDURE
Policies are important because they address issues, which reduces risk to the control systems. Procedures are the steps necessary to follow the policy guidelines. Policy and procedure are often missing from the building control space when it comes to OT cybersecurity. Ransomware can be avoided if policy is implemented that outlines the proper use and physical security of the application server/front-end server and compliance is enforced.
Some basic policies that should be implemented are shown below:
Some of these policies are not what you would find in a typical IT policy library. This list is not a full list of OT policies, but by enacting just these few, an organization can reduce risk to the building systems that supply comfort and safety and reduce financial and brand damage impact.
CONSEQUENCES OF NOT HAVING AN OT POLICY AND PROCEDURE
MISUSE OF THE APPLICATION HOST
As stated earlier, using the application host as a personal PC/workstation puts it at high risk, especially for a ransomware attack. Ransomware is continually evolving and the consequences to an organization are ballooning. In recent history, two types of ransomware have emerged that can cause major reputation and financial damage, as well as negatively impact operations: 1) Nuclear and 2) Snake (Ekans).
Nuclear ransomware has elevated ransomware to a lose-lose situation. Prior to nuclear ransomware, your choices were either to pay the ransom (not recommended) and get your files unlocked, or—if you had a good backup—ignore it and wipe your system clean, starting with a fresh copy. With this latest iteration, you can still pay the ransom; however, if you have good backups and refuse to pay, the threat actors will release your sensitive, private data to the web. This happened to Visser Precision, a third-party vendor for Tesla, SpaceX, Lockheed Martin, and Boeing—all of whom had non-disclosure agreements with Visser(3).
The other ransomware type is Snake, also known as Ekans (snake spelled backward), Ransomware. Unlike typical ransomware that targets Windows and Linux operating systems, Snake targets industrial control systems (ICS’s). What makes it really nasty is its ability to climb outside of one system and spread throughout the network, infecting other devices. (4)(5)
As nasty as these variations of ransomware are, the fix/prevention is easy. Remove the application host from the engineer’s office, place it in a locked location, and remove the keyboard, mouse, and monitor. Just like with IT application servers, no one should ever physically touch the machine unless they are performing maintenance or repair. You wouldn’t check your email on a SharePoint server, so why would do it on machine that is running your building?
It is easy to find exposed systems and no special tools are required. There are several free device search engines that anyone can use to search for publicly exposed systems. What does exposed mean? It means that the control system can be accessed by anyone, anywhere in the world. The only thing that stands between the attacker and access to the system is the application’s credential management. What makes this even more risky is the fact that the majority of BCS’s are not configured to withstand password cracking tools and shared user accounts are rampant. There is typically no access monitoring to indicate if the system is being attacked. Additionally, there is a large number of systems that are running outdated/unsupported operating systems and applications with well documented vulnerabilities that the threat actor can exploit.
These search engines continually scour the Internet 24/7/365. A simple search at the time of this article revealed that are 37,622 exposed BCS’s. Don’t assume none of these are yours. In over 60% of the assessments, IB was told that there was nothing exposed; however, in every case, we found at least one system or one part of the system that was exposed.
What is the consequence? Wasted time and money? In the past 18 months across eight IB-led assessments, there have been approximately 18,816 manhours lost for an approximate total of $1,505,280 for an average cost of $188,160 per incident. This does not include self-inflicted wounds, which will be discussed in the next section.
To drive this point home, IB performed an assessment for a company that believed they had zero exposed devices. They had invested significant time and money on revamping their network architecture—IT was monitoring the networks using the tools they were accustomed to and would not believe they had any holes. Although IT was monitoring the switches, PC, and servers, they were not watching beyond these devices. Once we began hunting using the same free search tools that anyone could use, we found a single BACnet broadcast management device (BBMD), which is essentially a router. From this, IB discovered over 1,000 BACnet devices in locations throughout the U.S. No username and password is required for legacy BACnet. From here, we would have been able to fully control any or all of the devices. This is a good example of how IT cannot fully manage OT with the tools at their disposal.
IT is beginning to engage in securing OT systems, which can be a good thing or a bad thing. IT and OT have to become partners in protecting systems, but only after OT policy is developed and IT has been educated on the dos and don’ts when interacting with OT systems.
We have seen numerous examples of IT’s zealous attempts to manage control systems, such as:
Patching the front-end application caused the local staff at 50 hospitals to be locked out of their control systems. Surgeries had to be cancelled for two days at these 50 hospitals.
An employee was fired and IT began removing their user from central application server. This user was also the user that allowed communication and control between the application server and the supervisory control at numerous locations throughout the U.S. Communication and control was lost to over 100 locations before it was realized there was a problem. It over six weeks to fully recover.
IT scanned the OT networks in a location with thousands of controllers. Over 60% of the controllers were locked up and required each individual device to be power cycled in order to bring them back online. After they were back online, each had to be checked to ensure functionality was fully restored. Over 96,000 manhours were lost.
What is the consequence? Wasted time and money. In the past 18 months over 10 IT-induced events, there have been approximately 107,924 manhours lost for an approximate total of $8,633,920 for an average cost of $863,392 per incident. This does not include external attacks from threat actors.
VENDORS ARE IN CONTROL
Without policy, there are only verbal understandings. It’s great to have an understanding; however, these understandings have to backed up with written guidelines. We see time and again that the vendor is the primary knowledge keeper, user administrator, controller of remote access, controls data, and may or may not be backing up systems. Even if they are backing up systems, these backups are not usually readily accessible to onsite staff. The other risk a vendor usually introduces is a single, admin user in your system for all of their employees at every location they service, which hasn’t been changed in years. This means any of their employees, past or present, can get into your system anytime they choose without your knowledge. Some very basic policies you can enact now for vendor management are:
Vendor training for staff to increase understanding regarding vendors and the abilities of systems. This will allow your staff to self-perform for better response to system events.
Remote vendor access controlled by owner and not vendor.
Unique users for each vendor employee.
Vendor must provide a list of employees that need access to the system and no longer leave it open for any employee to access your system.
Vendor must notify immediately when an employee either no longer needs access to your system or is no longer employed by the vendor.
If the vendor is contracted to maintain backups, the vendor must provide you a copy of backups to be stored under your control.
Create a vendor separation agreement to establish such things as turning over data, intellectual property, etc.
By no means should this be considered a full list of policies. It will get you started and help you minimize risk.
NO LONGER A POSSIBILITY, IT IS REALITY
In 2010, Stuxet (6) was the first major attack on a control system. It was an attack on an Iranian nuclear facility (7). Most people didn’t see this as the beginning of attacks on control systems, regardless of their function. They believed that these types of attacks would only happen to large industrial systems and not to BCS’s. This opened a new attack vector for other threat actors to take advantage of, showcasing that control systems are vulnerable and easy to exploit in several different ways. The industry is seeing an increase in these types of attacks, occurring at an exponential rate, with no sign of slowing.
If you haven’t taken a serious, unbiased look at your OT risk profile, you should. You can start by seeing if you have exposure to the world, if your application server is being used for anything other than its designed function(s), and who is in control of your system. Are you in control or is your vendor? The policies listed earlier can help you take control of your system(s), preventing external threats and internal self-inflicted wounds.