By Dan Clark, Director of OT Cybersecurity Architecture for Verve Industrial Solutions
July 11, 2023
We hosted a (CS)²AI Online™ Seminar on June 21, 2023 that focused on Applying Network Segmentation to Secure OT Environments. The event was sponsored and led by subject matter experts from our Strategic Alliance Partner, Verve Industrial Solutions.
Here is a bit about the event:
As threats continue to rise and target industrial organizations, one of CISA’s consistent recommendations for effective OT security is network segmentation. But between unique challenges in industrial environments, aligning IT & OT teams, and understanding where to start to secure critical networks, many organizations struggle to put it into practice.
How should IT and OT work together? How do we segment with the least impact on operational uptime?
In this webinar, Verve CEO, John Livingston, and Director of OT Cybersecurity Architecture, Dan Clark, will share over 30 years of OT networking experience and discuss how to:
• Implement network segmentation in OT systems
• Effectively bridge IT & OT systems security
• Achieve effective visibility of segmentation
• Gain buy-in from your team and investors
Do you want to have access to more content like this? Register for our next (CS)²AI Online™ seminar or symposium here, or consider becoming a member of (CS)²AI to access our entire library of recorded sessions.
The Q&A portion below represents selected overflow questions and commentary from the event, and has been answered in detail by Dan Clark.
******************************
QUESTION:
What’s your opinion on SDN vs VLAN?
ANSWER:
We are generally brought in to separate an IT (Corporate, Business) network from an OT (Process Control, Manufacturing) network. Typically, the networking equipment between the IT and OT networks are different and some of the routing, switching and firewalling equipment can be quite old, specially in the OT environment. It is not unusual to find switches and routers over ten years old. As I understand SDN, this is typically used in cloud computing for Internet Service Providers and is used to improve network performance and monitoring. The OT networks that we have been asked to segment have not matured to that point yet. However, overtime, I could see that SDN could be used for the OT environment of the future if SDN is still viable.
QUESTION:
How many engineer hours does it take to implement a new protocol-specific deep-packet-inspection engine?
ANSWER:
I don't know. How do you define implement (proof of concept, tested on any device that uses this protocol, tested on every device)? Does the protocol already exist and is it well defined? Does the engineer have all of the equipment that can be used to test the implementation of the deep-packet-inspection engine? Does the vendor of the device used to do this inspection support new protocols? Which vendor or vendors does this implementation need to support?
QUESTION:
Were you aware that by using a Tripwire-like firewall with a protocol-specific deep-packet inspection engine, we were able to mask existing vulnerabilities in devices on both the human interface side and the device-interface side? By doing this there were many patches which did not need to be applied.
ANSWER:
Many of the new NGFWs (Cisco, Fortigate, Palo Alto, e.g.) have the ability to look into the packets for ICS protocoIs (MODBUS, DNP3, e.g.) to make decisions on whether or not the commands meet the requirements for the protocol. Decisions can then be made to on allowing reads, writes, etc. We have had mostly successes when implementing but a few failures as well, but they are really nice when they work. The failures sometimes get fixed with patches and firmware upgrades as well. However, not everyone wants to implement these features. It may sound shocking, but this is all too common.
QUESTION:
Everyone is aware of DARPA's definition of the four pillars of security: Physical (spatial), Logical, Temporal, and Cryptographic Threats. What are you doing about the fifth pillar, Perceptive Threats?
ANSWER:
I was not aware of "DARPA's ... four pillars of security", and could never find on the DARPA website ... but I didn't make an exhaustive search. However, from the context of your question we do deal with physical, logical, temporal and cryptographic threats in various modes with our clients … network segmentation, training personnel, physical room and cabinet security, and others. Other than the training that we do for personnel and specifying what and details of the segmentation efforts like configuring firewalls, routers, and switches which directly addresses real not perceived threats, I am not aware of other any other methods that we use to address perceived threats. For what it is worth, our clients don't always follow all of our recommendations either. There are many practical business and operational requirements that impact what is actually implemented during a segmentation project and why. Many industries do not have governmental regulations for cybersecurity; so, compliance is optional and decisions are made on a risk/reward basis.
QUESTION:
Did you know there is a way to wrap zero trust around something like DOS or Windows NT? It is called Virtualization. My PhD advisor wrote the book on Virtual Machines. I helped with the book. If you would like to talk to me about it, contact me: https://cs.wisc.edu/~bezenek. The TRUST in this system comes from the newer technology VM running on a contemporary processor with a root of trust built into it.
ANSWER:
We are very familiar with Virtualization. OT people are very reluctant to change out systems that work and they know. Mostly they don't want to upgrade because it costs them money in production downtime, equipment cost, and training for production personnel. This is why we see pieces of a production system still running DOS6, Windows NT, Windows XP, Windows Server 2003, and many others that are obsolete. Many, but not all older systems, have upgrade paths to newer platforms including VMs. When moving an existing older system to a Virtual Machine the costs mentioned earlier are likely even higher. For VMs, you must have a host hardware and software which costs, development costs, and testing costs. After that you still have to add the other costs as well. We have also had issues with drivers on some of this old hardware specially systems that use serial communications. So typically, we don't have Virtualization as a practical option. By the way, last year I was involved with replacing a 1990's vintage Sun Unix computer with a Moxa Industrial PC. Serial communication was involved so we ported the software from Unix to Linux and used the serial port on that Moxa computer for the communication ... so it is doable.
QUESTION:
When discussing VLAN hopping, is hopping between VLANS on the same switch (intra-switch hopping) essentially the same vulnerability-wise as inter-switch VLAN hopping with homogeneous switches? Note: Define homogeneous switches as being the same model made by the same manufacturer running the same version of the management software on the same OS with all the same patches to all software.
ANSWER:
As I understand your question, yes. However, VLAN hopping (inter- or intra-) primarily occurs as a result of poor configuration and can be mitigated with a minimal amount of quality configuration standards … not using VLAN 1, defining a VLAN for each access port, filtering trunk ports based on VLAN IDs, and SHUTDOWN each unused port.
QUESTION:
DOS 6 that's hard-core (but probably simple too simple)
ANSWER:
There is nothing simple about making older systems work, because if it was, it would have already been replaced. We have had issues with motherboards dying when rebooting an old computer and getting the system back up and running is difficult, and it isn't always possible. However, there hasn't been a system so far, that we couldn't get running again ... but it never is easy. Sometimes we have to upgrade a system.
QUESTION:
Is it best to prevent SVI (switch virtual interface) completely or is there use cases for inter-VLAN comms....?
ANSWER:
Absolutely there are cases for inter-VLAN communication. Segmentation wouldn't be very effective we couldn't route between the VLANs. Ideally, there is a firewall device to negotiate when and how this communication is allowed.
QUESTION:
How do you get your customers to update their OT equipment to the latest version of firmware? Many of our customers are at very old and vulnerable releases of firmware.
ANSWER:
Mostly you don't until they have some type of cybersecurity event. It is easy to pick on the OT guys, but most of their issues are related to production and cost. They don't want to upgrade because it costs them money in production downtime, equipment cost, and training production personnel. This is why we see pieces of a production system still running DOS6 or Windows NT. One of the other questions suggested moving these old systems to a Virtual Machine. That still costs money for equipment, development costs, and testing costs. After that you still have to add the other costs as well. We have also had issues with drivers on some of this old hardware specially systems that use serial communications.
QUESTION:
Would an additional advantage in segmenting network be able to more readily isolate the targeted network if/when there's an impacted attack?
ANSWER:
Yes.
QUESTION:
How can segmentation effectively be implemented when OT environment is connected to cloud?
ANSWER:
Directly connecting from an OT network to the "cloud" has its challenges, but we do this in some manner in all segmentation projects. The key is to only make those connections that are required, limit the number of TCP/UDP ports being used, authenticate the traffic and do IDS/IPS filtering where possible. The reality for OT systems and networks requires that the OT(Manufacturing) network have access to the IT (Business) network. The Business sells something and the Manufacturing has to build what was sold. Without this connection, the entire Business would not likely exist.
QUESTION:
What is the best practice in segmentation when it comes to deploying Managed SW Vlans for different IT/ OT environment?
ANSWER:
The best practice would be to have IT manage the IT network and the OT manage the OT network and provide visibility between the networks to both management groups. Whenever this is not feasible to have two different management groups, use a single management group but still give visibility to both groups. Significant problems arise when with lack of transparency.
QUESTION:
Which are the main or widely used network segmentation techniques used (for example: VLAN ID tag was mentioned). Thanks
ANSWER:
I presented in the webinar the techniques that we typically use for segmentation. However, we are mostly led by what our clients desire. Some clients are looking for assistance on the segmentation how, but most want support in implementation of what they know they want. So for larger organizations that have 10, 20, 50, 100, or more sites, they may do a proof of concept for a few sites, then we are hired to implement their existing solution to the remaining sites.
QUESTION:
What percentage of environments that Verve sees follow the "Company Network A" model?(estimated are okay)
ANSWER:
I don't have an actual number. This percentage keeps changing as we do more segmentation projects, but I will throw out 10% as an estimate for the systems that we see in the industrial space that the IT/OT is combined into a single network without a firewall. However, most of the time, there is least some access control rules on the incoming router which typically include multiple VLANS so there is some protection. Nevertheless, there is no physical separation between IT and OT. I really want to say that I am exaggerating, but unfortunately I am not. Hopefully, others doing this work in the OT space see something better.
QUESTION:
Leasing from telecom carriers for ICS/OT? This is nearly the biggest achilles heal of anything mentioned this far. An owner can secure assets to n-th degree, but has very limited or no control over the infrastructure on which the information is being carried by a telecom carrier from a system configuration, physical protection and cyber-security aspect. Can a user control the telecoms patching procedures? Haven’t seen it other than penalties for not meeting uptime guarantees.
ANSWER:
I have not seen a situation where a user can control an ISP. Many of our clients are in remote areas and getting access to any internet connection can be a challenge.
QUESTION:
I’m from a newer world of things, and while being able to see the risk management and monitoring side of things; how would serverless cloud hosting procedures withstand the long tail of vulnerabilities?
ANSWER:
Directly connecting from an OT network to the "cloud" has its challenges, but we do this in some manner in all segmentation projects. The key is to only make those connections that are required, limit the number of TCP/UDP ports being used, authenticate the traffic and do IDS/IPS filtering where possible. The reality for OT systems and networks requires that the OT(Manufacturing) network have access to the IT (Business) network. The Business sells something and the Manufacturing has to build what was sold. Without this connection, the entire Business would not likely exist.
QUESTION:
In a situation like where you plugged in that cable, how did you resolve it? Was there like a spare system that was used to replace it? How did this downtime impact the company? Was there a need to update the OS with another system?
ANSWER:
The example I gave was during a planned outage. So, when the server failed, it did not impact operations at all. This is one of the reasons we are really uncomfortable doing implementation while the site is making product, or what we deem as "hot". If we were not already in an outage situation, the plant would have stopped making product. So, no downtime. Also, we had a about 40 hours before we were to bring this site back up to operation. It was also on Saturday, so we did not have access to many plant personnel. Many, but not all, process control systems have some form of plan to resolve equipment failures... backup or spare equipment. This particular site had neither. The first reaction to these situations is to panic, but don't, unless you are doing this hot. We asked if they had a spare computer for this process and the answer was no. This was a 2015 Dell computer running Windows XP Server. There was a BestBuy about 45 minutes away from the site. We knew that we wouldn't be able to get a computer to run Windows XP, but we "hoped" that we could get a new computer and run a newer version, Windows 10/11, version of the software. We are OT people so we know about many of the applications that run in the OT space. We also know about PLCs, DCSs, SCADA Systems, and many more. Luckily, I had multiple people that I could call for help. So, we sent one of our guys doing this implementation to Best Buy. We had a site support guy that thought they might have a spare XP machine at a sister facility about 20 miles away, so we sent him to check that option out. We did try to fix the computer, hoping that it was just a video card, but that failed. Unbelievably about three hours later, we got a spare computer from the sister facility that didn't throw that old computer away, swapped out the hard drive for the machine that failed and magically it worked. Success. We got lucky. It did not impact production because we did this during an outage. Most of our client's personnel didn't know how close we were to causing an outage.
QUESTION:
With the new movement to have IT and OT feeds on a single pane of glass, are there any differences in segmentation rules you are presenting?
ANSWER:
For sure the rules will have to change to allow IT (Business) network devices to communicate with OT devices. Many times this can be a single feed and technically you could put a data diode in the network for this feed as well to protect the OT network from the IT network. We have used firewall rules and data diodes. It just depends on what the client needs.
QUESTION:
Is the recommendation for complete physical segregation with a single point of contact, or a logically segregated OT and Enterprise network?
ANSWER:
We would suggest physical segregation of the IT and OT network zones and logical separation among the subnetworks in each zone. The only issue is budget. We have done both.
QUESTION:
How to achieve segmentation of assets which are inside the target network
ANSWER:
Sub-segmentation of the target network. We are working on another presentation to discuss the details and how to subsegment a process control network.
QUESTION:
At what point the process identifies vulnerability of assets in consideration & its possible mitigation?
ANSWER:
If we have a client that has a flat IT (business) network, we would want to start with segmenting and separating the OT (process control, manufacturing) network for the existing IT (business) network. Once separated, we would discover the assets for both networks using our Verve Software, identify asset vulnerabilities, and then use our software to mitigate those vulnerabilities. We also periodically do a survey at the beginning to determine the extent of existing segmentation and the types of IT/OT assets that exists. This survey can sometimes identify specific issues that may be mitigate prior to other work.
QUESTION:
What are the best practices in segmenting SIS from BPCS?
ANSWER:
Add another segment to the network for SIS and separate this network from the IT (business) and BPCS networks. I have seen fire systems isolated (as islands) from the network.
QUESTION:
Could you speak a little bit about the adoption of Zero Trust in the field of ICS/OT? Have you seen any adoption? and could you give us an example of full Zero Trust adoption in OT/ICS?
ANSWER:
We have seen some adoption of Zero Trust in ICS/OT, but so far it is not typical. I personally have not seen an example of full Zero Trust adoption.
QUESTION:
When it comes to endpoint protection, such as HMI and Engineering workstations, have you observed any technologies used other than antivirus and application whitelisting? Have you seen adoption to technologies like EDR (Endpoint Detection & Response) or orchestration?
ANSWER:
Mostly no, but we have seen some adoption of other technologies.
QUESTION:
What kind of tools you use for discovery? Passive or active? Nmap or Nazomi ?
ANSWER:
It depends upon the project and what is available, but we have used multiple tools for discovery. We have used our Verve product, which uses both and active and passive functions. When not directly using our product, we will query routers, switches and firewalls (active) through SSH to get configurations, status, mac addresses, routing information, interfaces, hit counts, etc. Once we get that information, we analyze that data to find process control equipment that lives on the IT network. Then we take that information and track it down once at the site for physical discovery. We also ask questions of site personnel to confirm information we find and get additional information that is not readily available through data analysis.
QUESTION:
How does segmentation give you and ROI?
ANSWER:
Segmentation helps prevent bad actors from getting into your network. Segmentation minimizes the ability of bad actors pivoting with the network to reduce damage from an intrusion. All of the "returns" are cost avoidances through process downtime and ransomware or protection of intellectual property. Our clients understand clearly the cost of process downtime and typically can quote a $/hour number. The client has to evaluate these "benefits" relative to the costs of segmentation.
QUESTION:
Statistically. using different vendors for the same type of device increases the likelihood of an exploitable vulnerability. Also you need to stay aware of more security announcements and staff must know more systems (increasing the likelihood of misconfiguration).Compare this to using just one vendor.
ANSWER:
I am not sure what specific statistics that you are referring to, but you make a very valid point. However, if IT manages the set of firewalls separating the IT (Business) systems from the DMZ and OT manages the set of firewalls separating the OT (Process Control, Manufacturing) systems then their should be no statistical difference from IT managing both IT and OT firewalls. Also, by separating management of the different zones (IT/OT) and giving visibility to the other management group, this gives the organization an additional set of checks and balances.
QUESTION:
What is your recommendation on implementing network segmentation on legacy systems?
ANSWER:
So far, we have done this on every segmentation project that we have been involved with, so we recommend it. Ultimately, network segmentation costs money and rarely, if ever, there is an unlimited budget for doing this work. We will normally put pieces of a legacy system on a VLAN by itself, or subsegment that system into multiple VLANs depending upon client requirements.
QUESTION:
What is your recommendation on implementing network segmentation on legacy systems and cost?
ANSWER:
All projects are based upon risks and costs. We work with our clients on finding the best network segmentation solution.