Search Results

By Dan Clark, Director of OT Cybersecurity Architecture for Verve Industrial Solutions July 11, 2023 We hosted a (CS)²AI Online™ Seminar on June 21, 2023 that focused on Applying Network Segmentation to Secure OT Environments. The event was sponsored and led by subject matter experts from our Strategic Alliance Partner, Verve Industrial Solutions. Here is a bit about the event: As threats continue to rise and target industrial organizations, one of CISA’s consistent recommendations for effective OT security is network segmentation. But between unique challenges in industrial environments, aligning IT & OT teams, and understanding where to start to secure critical networks, many organizations struggle to put it into practice. How should IT and OT work together? How do we segment with the least impact on operational uptime? In this webinar, Verve CEO, John Livingston, and Director of OT Cybersecurity Architecture, Dan Clark, will share over 30 years of OT networking experience and discuss how to: • Implement network segmentation in OT systems • Effectively bridge IT & OT systems security • Achieve effective visibility of segmentation • Gain buy-in from your team and investors Do you want to have access to more content like this? Register for our next (CS)²AI Online™ seminar or symposium here, or consider becoming a member of (CS)²AI to access our entire library of recorded sessions. The Q&A portion below represents selected overflow questions and commentary from the event, and has been answered in detail by Dan Clark. ****************************** QUESTION: What’s your opinion on SDN vs VLAN? ANSWER: We are generally brought in to separate an IT (Corporate, Business) network from an OT (Process Control, Manufacturing) network. Typically, the networking equipment between the IT and OT networks are different and some of the routing, switching and firewalling equipment can be quite old, specially in the OT environment. It is not unusual to find switches and routers over ten years old. As I understand SDN, this is typically used in cloud computing for Internet Service Providers and is used to improve network performance and monitoring. The OT networks that we have been asked to segment have not matured to that point yet. However, overtime, I could see that SDN could be used for the OT environment of the future if SDN is still viable. QUESTION: How many engineer hours does it take to implement a new protocol-specific deep-packet-inspection engine? ANSWER: I don't know. How do you define implement (proof of concept, tested on any device that uses this protocol, tested on every device)? Does the protocol already exist and is it well defined? Does the engineer have all of the equipment that can be used to test the implementation of the deep-packet-inspection engine? Does the vendor of the device used to do this inspection support new protocols? Which vendor or vendors does this implementation need to support? QUESTION: Were you aware that by using a Tripwire-like firewall with a protocol-specific deep-packet inspection engine, we were able to mask existing vulnerabilities in devices on both the human interface side and the device-interface side? By doing this there were many patches which did not need to be applied. ANSWER: Many of the new NGFWs (Cisco, Fortigate, Palo Alto, e.g.) have the ability to look into the packets for ICS protocoIs (MODBUS, DNP3, e.g.) to make decisions on whether or not the commands meet the requirements for the protocol. Decisions can then be made to on allowing reads, writes, etc. We have had mostly successes when implementing but a few failures as well, but they are really nice when they work. The failures sometimes get fixed with patches and firmware upgrades as well. However, not everyone wants to implement these features. It may sound shocking, but this is all too common. QUESTION: Everyone is aware of DARPA's definition of the four pillars of security: Physical (spatial), Logical, Temporal, and Cryptographic Threats. What are you doing about the fifth pillar, Perceptive Threats? ANSWER: I was not aware of "DARPA's ... four pillars of security", and could never find on the DARPA website ... but I didn't make an exhaustive search. However, from the context of your question we do deal with physical, logical, temporal and cryptographic threats in various modes with our clients … network segmentation, training personnel, physical room and cabinet security, and others. Other than the training that we do for personnel and specifying what and details of the segmentation efforts like configuring firewalls, routers, and switches which directly addresses real not perceived threats, I am not aware of other any other methods that we use to address perceived threats. For what it is worth, our clients don't always follow all of our recommendations either. There are many practical business and operational requirements that impact what is actually implemented during a segmentation project and why. Many industries do not have governmental regulations for cybersecurity; so, compliance is optional and decisions are made on a risk/reward basis. QUESTION: Did you know there is a way to wrap zero trust around something like DOS or Windows NT? It is called Virtualization. My PhD advisor wrote the book on Virtual Machines. I helped with the book. If you would like to talk to me about it, contact me: https://cs.wisc.edu/~bezenek. The TRUST in this system comes from the newer technology VM running on a contemporary processor with a root of trust built into it. ANSWER: We are very familiar with Virtualization. OT people are very reluctant to change out systems that work and they know. Mostly they don't want to upgrade because it costs them money in production downtime, equipment cost, and training for production personnel. This is why we see pieces of a production system still running DOS6, Windows NT, Windows XP, Windows Server 2003, and many others that are obsolete. Many, but not all older systems, have upgrade paths to newer platforms including VMs. When moving an existing older system to a Virtual Machine the costs mentioned earlier are likely even higher. For VMs, you must have a host hardware and software which costs, development costs, and testing costs. After that you still have to add the other costs as well. We have also had issues with drivers on some of this old hardware specially systems that use serial communications. So typically, we don't have Virtualization as a practical option. By the way, last year I was involved with replacing a 1990's vintage Sun Unix computer with a Moxa Industrial PC. Serial communication was involved so we ported the software from Unix to Linux and used the serial port on that Moxa computer for the communication ... so it is doable. QUESTION: When discussing VLAN hopping, is hopping between VLANS on the same switch (intra-switch hopping) essentially the same vulnerability-wise as inter-switch VLAN hopping with homogeneous switches? Note: Define homogeneous switches as being the same model made by the same manufacturer running the same version of the management software on the same OS with all the same patches to all software. ANSWER: As I understand your question, yes. However, VLAN hopping (inter- or intra-) primarily occurs as a result of poor configuration and can be mitigated with a minimal amount of quality configuration standards … not using VLAN 1, defining a VLAN for each access port, filtering trunk ports based on VLAN IDs, and SHUTDOWN each unused port. QUESTION: DOS 6 that's hard-core (but probably simple too simple) ANSWER: There is nothing simple about making older systems work, because if it was, it would have already been replaced. We have had issues with motherboards dying when rebooting an old computer and getting the system back up and running is difficult, and it isn't always possible. However, there hasn't been a system so far, that we couldn't get running again ... but it never is easy. Sometimes we have to upgrade a system. QUESTION: Is it best to prevent SVI (switch virtual interface) completely or is there use cases for inter-VLAN comms....? ANSWER: Absolutely there are cases for inter-VLAN communication. Segmentation wouldn't be very effective we couldn't route between the VLANs. Ideally, there is a firewall device to negotiate when and how this communication is allowed. QUESTION: How do you get your customers to update their OT equipment to the latest version of firmware? Many of our customers are at very old and vulnerable releases of firmware. ANSWER: Mostly you don't until they have some type of cybersecurity event. It is easy to pick on the OT guys, but most of their issues are related to production and cost. They don't want to upgrade because it costs them money in production downtime, equipment cost, and training production personnel. This is why we see pieces of a production system still running DOS6 or Windows NT. One of the other questions suggested moving these old systems to a Virtual Machine. That still costs money for equipment, development costs, and testing costs. After that you still have to add the other costs as well. We have also had issues with drivers on some of this old hardware specially systems that use serial communications. QUESTION: Would an additional advantage in segmenting network be able to more readily isolate the targeted network if/when there's an impacted attack? ANSWER: Yes. QUESTION: How can segmentation effectively be implemented when OT environment is connected to cloud? ANSWER: Directly connecting from an OT network to the "cloud" has its challenges, but we do this in some manner in all segmentation projects. The key is to only make those connections that are required, limit the number of TCP/UDP ports being used, authenticate the traffic and do IDS/IPS filtering where possible. The reality for OT systems and networks requires that the OT(Manufacturing) network have access to the IT (Business) network. The Business sells something and the Manufacturing has to build what was sold. Without this connection, the entire Business would not likely exist. QUESTION: What is the best practice in segmentation when it comes to deploying Managed SW Vlans for different IT/ OT environment? ANSWER: The best practice would be to have IT manage the IT network and the OT manage the OT network and provide visibility between the networks to both management groups. Whenever this is not feasible to have two different management groups, use a single management group but still give visibility to both groups. Significant problems arise when with lack of transparency. QUESTION: Which are the main or widely used network segmentation techniques used (for example: VLAN ID tag was mentioned). Thanks ANSWER: I presented in the webinar the techniques that we typically use for segmentation. However, we are mostly led by what our clients desire. Some clients are looking for assistance on the segmentation how, but most want support in implementation of what they know they want. So for larger organizations that have 10, 20, 50, 100, or more sites, they may do a proof of concept for a few sites, then we are hired to implement their existing solution to the remaining sites. QUESTION: What percentage of environments that Verve sees follow the "Company Network A" model?(estimated are okay) ANSWER: I don't have an actual number. This percentage keeps changing as we do more segmentation projects, but I will throw out 10% as an estimate for the systems that we see in the industrial space that the IT/OT is combined into a single network without a firewall. However, most of the time, there is least some access control rules on the incoming router which typically include multiple VLANS so there is some protection. Nevertheless, there is no physical separation between IT and OT. I really want to say that I am exaggerating, but unfortunately I am not. Hopefully, others doing this work in the OT space see something better. QUESTION: Leasing from telecom carriers for ICS/OT? This is nearly the biggest achilles heal of anything mentioned this far. An owner can secure assets to n-th degree, but has very limited or no control over the infrastructure on which the information is being carried by a telecom carrier from a system configuration, physical protection and cyber-security aspect. Can a user control the telecoms patching procedures? Haven’t seen it other than penalties for not meeting uptime guarantees. ANSWER: I have not seen a situation where a user can control an ISP. Many of our clients are in remote areas and getting access to any internet connection can be a challenge. QUESTION: I’m from a newer world of things, and while being able to see the risk management and monitoring side of things; how would serverless cloud hosting procedures withstand the long tail of vulnerabilities? ANSWER: Directly connecting from an OT network to the "cloud" has its challenges, but we do this in some manner in all segmentation projects. The key is to only make those connections that are required, limit the number of TCP/UDP ports being used, authenticate the traffic and do IDS/IPS filtering where possible. The reality for OT systems and networks requires that the OT(Manufacturing) network have access to the IT (Business) network. The Business sells something and the Manufacturing has to build what was sold. Without this connection, the entire Business would not likely exist. QUESTION: In a situation like where you plugged in that cable, how did you resolve it? Was there like a spare system that was used to replace it? How did this downtime impact the company? Was there a need to update the OS with another system? ANSWER: The example I gave was during a planned outage. So, when the server failed, it did not impact operations at all. This is one of the reasons we are really uncomfortable doing implementation while the site is making product, or what we deem as "hot". If we were not already in an outage situation, the plant would have stopped making product. So, no downtime. Also, we had a about 40 hours before we were to bring this site back up to operation. It was also on Saturday, so we did not have access to many plant personnel. Many, but not all, process control systems have some form of plan to resolve equipment failures... backup or spare equipment. This particular site had neither. The first reaction to these situations is to panic, but don't, unless you are doing this hot. We asked if they had a spare computer for this process and the answer was no. This was a 2015 Dell computer running Windows XP Server. There was a BestBuy about 45 minutes away from the site. We knew that we wouldn't be able to get a computer to run Windows XP, but we "hoped" that we could get a new computer and run a newer version, Windows 10/11, version of the software. We are OT people so we know about many of the applications that run in the OT space. We also know about PLCs, DCSs, SCADA Systems, and many more. Luckily, I had multiple people that I could call for help. So, we sent one of our guys doing this implementation to Best Buy. We had a site support guy that thought they might have a spare XP machine at a sister facility about 20 miles away, so we sent him to check that option out. We did try to fix the computer, hoping that it was just a video card, but that failed. Unbelievably about three hours later, we got a spare computer from the sister facility that didn't throw that old computer away, swapped out the hard drive for the machine that failed and magically it worked. Success. We got lucky. It did not impact production because we did this during an outage. Most of our client's personnel didn't know how close we were to causing an outage. QUESTION: With the new movement to have IT and OT feeds on a single pane of glass, are there any differences in segmentation rules you are presenting? ANSWER: For sure the rules will have to change to allow IT (Business) network devices to communicate with OT devices. Many times this can be a single feed and technically you could put a data diode in the network for this feed as well to protect the OT network from the IT network. We have used firewall rules and data diodes. It just depends on what the client needs. QUESTION: Is the recommendation for complete physical segregation with a single point of contact, or a logically segregated OT and Enterprise network? ANSWER: We would suggest physical segregation of the IT and OT network zones and logical separation among the subnetworks in each zone. The only issue is budget. We have done both. QUESTION: How to achieve segmentation of assets which are inside the target network ANSWER: Sub-segmentation of the target network. We are working on another presentation to discuss the details and how to subsegment a process control network. QUESTION: At what point the process identifies vulnerability of assets in consideration & its possible mitigation? ANSWER: If we have a client that has a flat IT (business) network, we would want to start with segmenting and separating the OT (process control, manufacturing) network for the existing IT (business) network. Once separated, we would discover the assets for both networks using our Verve Software, identify asset vulnerabilities, and then use our software to mitigate those vulnerabilities. We also periodically do a survey at the beginning to determine the extent of existing segmentation and the types of IT/OT assets that exists. This survey can sometimes identify specific issues that may be mitigate prior to other work. QUESTION: What are the best practices in segmenting SIS from BPCS? ANSWER: Add another segment to the network for SIS and separate this network from the IT (business) and BPCS networks. I have seen fire systems isolated (as islands) from the network. QUESTION: Could you speak a little bit about the adoption of Zero Trust in the field of ICS/OT? Have you seen any adoption? and could you give us an example of full Zero Trust adoption in OT/ICS? ANSWER: We have seen some adoption of Zero Trust in ICS/OT, but so far it is not typical. I personally have not seen an example of full Zero Trust adoption. QUESTION: When it comes to endpoint protection, such as HMI and Engineering workstations, have you observed any technologies used other than antivirus and application whitelisting? Have you seen adoption to technologies like EDR (Endpoint Detection & Response) or orchestration? ANSWER: Mostly no, but we have seen some adoption of other technologies. QUESTION: What kind of tools you use for discovery? Passive or active? Nmap or Nazomi ? ANSWER: It depends upon the project and what is available, but we have used multiple tools for discovery. We have used our Verve product, which uses both and active and passive functions. When not directly using our product, we will query routers, switches and firewalls (active) through SSH to get configurations, status, mac addresses, routing information, interfaces, hit counts, etc. Once we get that information, we analyze that data to find process control equipment that lives on the IT network. Then we take that information and track it down once at the site for physical discovery. We also ask questions of site personnel to confirm information we find and get additional information that is not readily available through data analysis. QUESTION: How does segmentation give you and ROI? ANSWER: Segmentation helps prevent bad actors from getting into your network. Segmentation minimizes the ability of bad actors pivoting with the network to reduce damage from an intrusion. All of the "returns" are cost avoidances through process downtime and ransomware or protection of intellectual property. Our clients understand clearly the cost of process downtime and typically can quote a $/hour number. The client has to evaluate these "benefits" relative to the costs of segmentation. QUESTION: Statistically. using different vendors for the same type of device increases the likelihood of an exploitable vulnerability. Also you need to stay aware of more security announcements and staff must know more systems (increasing the likelihood of misconfiguration).Compare this to using just one vendor. ANSWER: I am not sure what specific statistics that you are referring to, but you make a very valid point. However, if IT manages the set of firewalls separating the IT (Business) systems from the DMZ and OT manages the set of firewalls separating the OT (Process Control, Manufacturing) systems then their should be no statistical difference from IT managing both IT and OT firewalls. Also, by separating management of the different zones (IT/OT) and giving visibility to the other management group, this gives the organization an additional set of checks and balances. QUESTION: What is your recommendation on implementing network segmentation on legacy systems? ANSWER: So far, we have done this on every segmentation project that we have been involved with, so we recommend it. Ultimately, network segmentation costs money and rarely, if ever, there is an unlimited budget for doing this work. We will normally put pieces of a legacy system on a VLAN by itself, or subsegment that system into multiple VLANs depending upon client requirements. QUESTION: What is your recommendation on implementing network segmentation on legacy systems and cost? ANSWER: All projects are based upon risks and costs. We work with our clients on finding the best network segmentation solution.

By Vivek Ponnada, Engineer, MBA, GICSP, Regional Sales Director at Nozomi Networks, (CS)²AI Fellow January 2, 2023 While attending another well-organized BSides in Edmonton last month (closing out the triumvirate in the Western part of Canada after Calgary and Vancouver earlier in the year), a student inquired how she might get started in OT Cybersecurity. This is a common question that is often asked in other forums including Social media, online webinars etc. While I answered her a bit in our conversation, I figured that publishing a more detailed article is helpful since the answer is slightly complicated. Quick background: OT is the term used for technology used in Industrial control applications, and includes purpose-built systems & protocols (e.g., Programmable Logic Controllers, Distributed Control Systems), general IT systems repurposed with specific software to configure or view data from those control systems (e.g., Windows based Human Machine Interface, or Engineering Workstations), or a combination of technologies (Firewalls that might be ruggedized or available with ability to parse industrial protocols). Definition here from NIST. A few things about OT Cybersecurity that trip up a lot of folks are: 1) OT domain knowledge: Some contend that no one can be in OT cybersecurity unless you’ve already had experience in industrial control systems, having worked in power plants, refineries etc. While that might be a valid expectation in several contexts, especially if you are billing a customer as a ‘experienced’ consultant, that stance is often is overplayed. Don’t get me wrong, a sure way for a consultant to be kicked out of a plant is to make a basic error in safety, such as not wearing the appropriate safety boots, or removing your hearing/eye protection where they are mandated. And if you go in with a swagger that you are better than the ICS personnel as you ‘know’ security better, you’ll burn bridges really quick. However, various IT Security skills are much needed in the OT world e.g., configuring firewalls & routers as part of network segmentation projects, evaluating secure remote access solutions, helping figure out patching options etc. While not every IT best practice is applicable to OT Security, a strong collaboration between those that have ICS experience and those that have IT skills can improve the overall security posture of the organization. People with IT security experience are essential for the improving overall OT Security. 2) Certs: Frankly, it’s a related topic to above, but certifications are almost never a starting point in OT security, or even the best way to be successful. Reputed organizations now offer ICS Security certs but they are fairly expensive and ideally paid by organizations, not individuals on their own. Your credibility as an OT Security practitioner is more established based on your experience, projects completed etc. rather than the most recent cert you spent time, money and effort on. Lots of communities exist where you can get involved – see below ‘Content, Connections and how you can get started!’ section - while you are building either your ICS knowledge or Cybersecurity skills. It never hurts to get real-world experience including plant visits whenever you can (that surely means safety orientations, general awareness of what’s important to plant personnel etc.). 3) Pace of technological change: If you are a keyboard ninja that’s excited about constantly updating your tools with docker, k8s etc., this industry, even the rare penetration-testing roles, might not be for you. OT technologies rarely update that quickly though learning about them might take a lifetime. Suffice to say that OT technology will be like your home iPad/tablet that’s been around for 5 years even while newer versions are available. Though you might typically update other devices like a Smartphone every year, you use the iPad/tablet longer because it works really well for the purpose you bought it for, and unless it fails, you find it hard to justify upgrading. OT has a lot of older equipment because the controllers and systems were engineered for a purpose they work for very well even after 15+ years, replacement is CapEx driven & resource intensive etc., so upgrading them based on security alone is rarely justifiable due to the outage time required that impacts business revenue. I sincerely hope you consider a career in OT Cybersecurity because it’s challenging, fun and rewarding. From a dispassionate point of view, the industry is fairly new, clearly needs significant additional resources, and the added public scrutiny leading to regulatory pressure is adding more jobs. For organizations, it’s not easy to find anyone with the ideal combination that is a mix of IT Security skills and operational knowledge. So, the more people cross-pollinate their skillsets and focus on OT Security, the better. Being in OT Security invariably leads to working towards protecting Critical Infrastructure, such as Power, Oil & Gas, Transportation and so many more verticals, which besides being economically attractive can be something you can be extremely proud of, a rare combination in a civilian job! Content, Connections and how you can get started! There are quite a few resources available these days that you can learn in your preferred audio/visual/in-person methods! 1) Podcasts (search in your favorite podcast store) a. CS2AI Podcast b. The Industrial Security Podcast c. Unsolicited Response Podcast d. Hack the Plant Podcast e. Many other vendor-sponsored ones (search for Industrial Security) 2) Videos/Webinars a. On-Ramp, Highway and Autobahn playlists in S4xEvents YouTube - https://www.youtube.com/@S4Events b. https://www.cs2ai.org/ - anyone can join the free webinars, membership required to watch recordings c. Sans Webcasts - https://www.sans.org/webcasts/ - several ICS/OT focused webinars including recordings from past years d. https://www.brighttalk.com/ has several ICS focused videos e. https://www.cisa.gov/uscert/ics/Industrial-Control-Systems-Joint-Working-Group-ICSJWG has several video recordings and content including training 3) Cybersecurity groups – whether it’s a local DefCon chapter, a BSides Committee, or Security meetup, explore the different options in your region. Obviously bigger cities have more options than in the rural areas, but you’ll be surprised how welcoming and close-knit the security members if you just reach out. While most might not have a significant OT content/focus, chances are they have someone or something that is in ICS/OT a. https://forum.defcon.org/social-groups b. http://www.securitybsides.com/w/page/12194156/FrontPage 4) Projects of special interest a. MITRE ATT&CK for ICS - https://attack.mitre.org/techniques/ics/ b. Secure PLC Coding Practices – https://www.plc-security.com/ c. Incident Command System for Industrial Control Systems - https://www.ics4ics.org/ d. Sign up to the mailing list at ISA https://www.isa.org/connectivity-and-cybersecurity - they regularly blog on ISA/IEC 62443 implementations, Risk management in OT etc.; if you want to get further involved, join an ISA chapter in your area, and you can get view access to the standards as well, or participate in the committees 5) Conferences – while ICS/OT focused conferences are few but famous, there are many others that are critical infrastructure focused, so naturally will have a cybersecurity topic or two that would be relevant and are much more affordable for individuals, and might even be free for students. While building your interest/value prop towards being able to attend the later ones of the list, start with your regional conferences that might have some OT content a. BSides (same link as above) b. Houston Security Conference - http://houstonseccon.org/ c. Cyber Security for Critical Assets https://www.cs4ca.com/ d. Cyber Senate - https://www.cybersenate.com/control-systems-cybersecurity-usa/ e. Sans ICS Security Summit – Orlando - https://www.sans.org f. API Cybersecurity Conference - https://events.api.org/ g. ICS Cyber Security Conference – Atlanta - https://www.icscybersecurityconference.com/ h. Industrial Security Conference – Copenhagen - https://insightevents.dk/isc-cph/ i. S4 Conference – Miami - https://s4xevents.com/ 6) Vendors – use a throw-away email to sign up to all major OT vendors’ mailing lists. They provide regular content on various topics including threat reports, recent vulnerabilities and exploits etc., and you can unsubscribe later to those that don’t align with your interests 7) Social Media a. There are some fantastic content creators and influencers; a recommended list is here https://mobile.twitter.com/i/lists/1549766676392165377 but be aware that due to recent shifts, many of these sources have moved to the federated universe. You can find many of them on LinkedIn as well, and they might be hosting content on YouTube, Github etc. b. CISA (US-CERT) have social media presence. Pick your platform to follow them (or email list https://www.cisa.gov/uscert/ics/advisories) 8) CTFs and Gamified learning a. https://store.steampowered.com/app/994670/ThreatGEN_Red_vs_Blue/ b. https://tryhackme.com/paths - not ICS focused but still relevant c. https://www.sans.org/mlp/holiday-hack-challenge/ - not ICS focused but still relevant 9) Books/Reading Material a. NIST - 800-82R2 - Guide to Industrial Control Systems Security https://csrc.nist.gov/publications/detail/sp/800-82/rev-2/final b. Industrial Network Security: Securing Critical Infrastructure Networks for Smart Grid, SCADA, and Other Industrial Control Systems - by Eric D. Knapp , Joel Thomas Langill c. Handbook of SCADA/Control Systems Security – Illustrated, by Burt G. Look , Robert Radvanovsky , Jacob Brodsky d. Hacking Exposed Industrial Control Systems: ICS and SCADA Security Secrets & Solutions –by Clint Bodungen, Bryan Singer, Aaron Shbeeb , Kyle Wilhoit , Stephen Hilt e. Industrial Cybersecurity: Efficiently monitor the cybersecurity posture of your ICS environment, 2nd Edition – by Pascal Ackerman f. Pentesting Industrial Control Systems: An ethical hacker's guide to analyzing, compromising, mitigating, and securing industrial processes Paperback – by Paul Smith g. Countering Cyber Sabotage: Introducing Consequence-Driven, Cyber-Informed Engineering (CCE)– by Andrew A. Bochman , Sarah Freeman h. Industrial Cybersecurity: Case Studies and Best Practices by Steve Mustard 10) Formal training a. https://www.abhisam.com/industrial-control-system-cybersecurity/ b. https://icscsi.org/training.html c. https://www.isa.org/certification/certificate-programs/cybersecurity d. https://www.sans.org/job-roles-roadmap/industrial-control-systems/?msc=job-roles-page

By Bayron Lopez, Director of Operational Technology at Kilroy Realty Corporation (CS)²AI Fellow September 22, 2022 With the growing landscape of intelligent building systems being deployed into commercial real estate, asset owners must develop a cyber-physical strategy to meet the ever-changing threats. As we continue integrating access controls, cameras, smart lighting, and even intelligent irrigation at scale, we must ensure that we do not sacrifice security for accessibility. With the addition of new technologies into the space, we must be able to vet a solution's software components and hardware integration. As the Director of Operational Technology, I work with all verticals of our organization to deploy technology that meets site requirements and protects them. One of the biggest threats I see is the lack of seriousness surrounding these systems. Don't get me wrong, as an industry, we spend millions on the usage and development of technology to meet the needs of the properties. We have elevators that can track your phone and take you to the correct floor, a turnstile that can recognize your face and allow you into a building, and sensors that can tell you how busy the cafeteria is, so you don't miss that hot cup of coffee. The comforts are there, yet we still lack enough understanding of the hardware security that operates those systems. Many are still under the impression that because these are not your traditional "IT systems," they don't require as many security policies around them. Yet if someone hacks into the system that stores all the faces and names to that turnstile access control system, that would be a significant breach. We have seen threat actors expose camera vulnerabilities due to lax security policies. The industry keeps deploying technology to make the lives of both the occupants and operators easier, but it also opens the sites up to potential harm from others. For years, individuals have been screaming at the top of their lungs regarding these threats, yet we decided to focus our attention elsewhere. The buzz words of digital twinning, fault detection, and many others that I call bells and whistles overtook the industry. We became infatuated with having more tech, more systems, and more shiny things that we hoped would distract threat actors from really looking under the hood. There was an explosion of solutions, and if you had an idea, there was a vendor that would promise you that it was possible. As the layers of systems and data became deeper and deeper, cracks began to show on the foundation. We lacked the seriousness of deploying a cyber-physical foundation for these systems. Many believed that they were not as complex as the corporate side of the house and that there was no way that they could produce large amounts of data. Some deployed 4K cameras without thinking that those unmanaged 10/100 switches would never be able to handle the traffic. Some gave access to vendors to access their data via open internet connections, not understanding the potential threats they had created. Even I was unaware that we could produce more than 15,000 data points daily from a single occupancy system. It wasn't until someone got hacked that most of us started to pay attention to those individuals screaming for seriousness in control systems. Fortunately, those individuals never gave up, and now we are starting to understand the foundational implementations they have been pushing. I've been fortunate to chat with my fellow CS2AI Fellows on some of these topics and have learned the importance of re-sealing our foundations.

By Jaco Benadie, Partner, Technology Consulting at Ernst & Young Consulting (CS)²AI Fellow July 4, 2022 Operational Technology Cybersecurity – United and Strong We are living in a time where speed and connectivity is everything. The pace of digital transformation continues to accelerate, and the complexity of these technologies makes it extremely difficult to fully understand the vulnerabilities and risks it brings until it is sometimes too late. “Countering Cyber Sabotage” by Andew A. Bochman and Sarah Freeman opens a whole new perspective of combining traditional safeguards with modern day digital controls to protect our critical systems. Please check out my original article on LinkedIn to read about my view on how to implement Consequence-based, Cyber-informed Engineering. https://www.linkedin.com/pulse/operational-technology-cybersecurity-united-strong-jaco-benadie/

By Fred Gordy Director of Cybersecurity at Intelligent Buildings, LLC, (CS)²AI Fellow March, 2022 Building Control System WhisperGate Attack Post on LinkedIn https://www.linkedin.com/posts/intelligent-buildings_intelligentbuildings-smartbuildings-cre-activity-6912767514456305665-hEZY?utm_source=linkedin_share&utm_medium=member_desktop_web I have periodically monitored several Russian aligned ransomware groups’ dark websites, primarily focusing on Conti. If you are not familiar, Conti is by far the most successful ransomware group in operation today, routinely pulling in multi-million-dollar payments from victim organizations, and they publicly announced their support for Russia when they invaded Ukraine. They are not the only ransomware groups to announce support for Russia. Others include UNC1151, Zatoichi, Killnet, Stormous Ransomware, Digital Cobra Gang (DCG), Freecivillian, SandWorm, The Red Bandits, and Coomingproject. I have noticed an upwards spike in U.S. companies showing up on Conti’s site. As recent as today, March 24th, 2022, a U.S.-based mechanical engineering and construction firm that, according to their website, is a leader in the Washington, D.C. market. Their website says they work on complex commercial, government, and institutional design-build projects. Two days ago, a U.S.-based cancer diagnostics laboratory was ransomed. Data is up for sale for both these companies and others, and they have been locked out of their systems. These two examples are centered around data, but the building controls community is not exempt. We recently were able to stop an attack on several building systems from what we believe to be Russia. WhisperGate malware was found and contained before it could do any damage. WhisperGate is a sophisticated malware known for targeting multiple organizations in Ukraine. It has two stages that corrupt a system’s master boot record, displays a fake ransomware note, and encrypts files based on certain file extensions. Conti Post on LinkedIn https://www.linkedin.com/posts/fredgordy_cybersecurity-conti-ransomware-activity-6910592084202655744-Wn8P?utm_source=linkedin_share&utm_medium=member_desktop_web

By Derek Harp, (CS)²AI Founder, Chairman and Fellow April, 2021 I am proud to announce today that the work to produce the 2021 (CS)²AI-KPMG Control System Cyber Security Report has begun! I would like to ask you to join our Members helping Members effort by doing two things today: Contribute to the body of knowledge and be one of the first to participate in the 2021 CS2AI annual survey right now today! Share this article and/or the survey link with your network. As the (CS)²AI organization continues to rapidly grow and evolve, foremost in my mind is the alignment of the diverse interests of key stakeholders among the people who make up our community. We are committed to increasing the range of respondents in every way to ensure the insights drawn from the data represent as many different stakeholders as possible. In studying the data for our 2020 report, we did find some things we want to understand more and one of those is coloration of answers from very different respondents. If we assume typically that leadership sets the goals and provides the resources needed to achieve those, it follows that operations focuses on using the supplied resources to accomplish the mission. Yet when looking at top priorities reported by these two groups, we find that executives and non-executives do not always hold the same set of targets. One area in which these two groups were in relatively close agreement is the low prioritization of cloud-based services in their control system environments. This caught my eye because the use of those same services is a major component of the technology trends variously referred to in terms such as the IT-OT Convergence, Industry/Industrie4.0, Digital Transformation, and Smart Factories (Cities, Grids, etc). Few if any control systems remain without multiple connections to one or more clouds, each creating potential exposures for attackers to exploit. So why isn’t the security of these connections and the services running over them a top priority? We have some clues and are working on further research to dig deeper into this question. Probably our greatest area of success in this research project has been identifying some clear differences between organizations at opposite ends of the cyber security program maturity scale. Areas like the use of managed security services, frequency and thoroughness of cyber security assessments, may seem self-evident places to find these differences, but we found the groups diverged in other important ways as well, such as what security technologies they had implemented and which attack vectors were used in cybersecurity incidents in their environments. It is findings like these, ones that help organizational leadership identify the gaps their teams can target for the greatest potential security ROI, that we search for most diligently. The range of threats, exposures and vulnerabilities, and the array of methods and tools to protect our people and assets against them, are dynamic and vast, while the resources which can be brought to bear are always going to be limited. The need to work smarter, to maximize the effectiveness of the people, skills and funds, is what drives demand for key decision-making tools, and I’m very glad to say that feedback for (CS)²AI first annual control system cyber security report has confirmed we achieved our goal in creating such a tool. If you did not get a chance to review the 2020 (CS)²AI-KPMG Control Systems Cyber Security Annual Report, A free copy can be downloaded here: https://www.cs2ai.org/reports I would like to thank our title sponsor and Platinum Strategic Alliance Partner, KPMG for continuing to underwrite and contribute resources to this project and decision support tool for the community. I also would like to thank Waterfall Security Solutions, Fortinet, Tempered, Industrial Defender, Verve, Applied Risk, Bedrock Automation, Fend and GBQ for joining the effort to make the research and annual report better each year.

George Kalavantis, Industrial Defender COO August 23, 2021 Getting budget approval is clearly a challenge for many in our community. Getting for OT cyber security can be even more challenging in some companies, depending largely on executive awareness and “buy in”. Your success most often comes from how you engage the conversation. So first let’s consider why this is even an issue at all. It seems like not too long ago, there was a time when OT was not even on the security roadmap, and CISOs couldn’t spell OT if you spotted them the “T”. But then STUXNET, UKRAINE, NOTPETYA, and most recently the ransomware attack on Colonial Pipeline happened. These events have accelerated the learning curve, and many CISOs have had a crash course on the criticality of OT to their business and the lack of visibility into OT environments within the enterprise cybersecurity stack. It is now clear that siloed, uncoordinated teams across the same enterprise is not a recipe for success. Though there are a number of preparatory steps one should consider, I want to share a few of my own tips on how to make the case for OT cybersecurity budget to your CISO. Read them here: https://www.industrialdefender.com/how-to-ask-your-ciso-for-ot-cybersecurity-budget/ Keep charging, George

By Peter Lund, Vice President Of Product Management at Industrial Defender and Chris Duffey, OT/ICS Specialist at Splunk We hosted a (CS)²AI Online™ seminar on August 18, 2021 that focused on Why Hasn’t SOAR Taken Off in ICS?. Here is a bit about the event: Besides the typical reluctance to embrace new technology in the ICS world, security orchestration, automation and response (SOAR) tools haven’t been as widely adopted as they probably should be because of the contextual data deficiency found in most security alerts. To create an appropriate automated response, you need to know exactly which devices are compromised and whether you can/should isolate them, which up until recently has been extremely difficult to do for industrial control systems. Let’s say you’re alerted that an HMI has a banking Trojan. That’s not great, but not likely something you’d feel compelled to take offline. However, if there was ransomware in an HMI, you have a serious problem. So, what should you do? Well, if you have 7 HMIs, it’s likely fine to just disconnect the infected one to stop the spread, but if that’s your only one, then it’s definitely not ok. This is a prime example of why having access to contextual data about both the threat AND the affected asset is so critical to informing automated security management. In this seminar, you’ll learn: • Why security orchestration and automation reduce the risk of operational downtime from a cyberattack • What type of contextual security information is critical to powering a next-gen program • How feeding the right ICS asset data into your SIEM + SOAR helps demonstrate ROI across your security ecosystem As always, we encourage our audience to participate throughout the event by contributing feedback and questions for our speakers. We weren't able to answer all of your questions, so we have asked some of our speakers to answer a few of them. Below, are some answers to a few questions posed during the event. Do you want to have access to more content like this? Register for our next (CS)²AI Online™ seminar or symposium here, or consider becoming a member of (CS)²AI to access our entire library of recorded sessions. ******************* To view the full recording of their talk, please visit https://www.cs2ai.org/sponsored-seminars. QUESTION: Is it like SNMP Management/ Sniffer s/w ? Doesn’t interrogate the network devices to get the devices info ? Could it interrogate RTU that talks Modbus or other industrial protocol?RESPONDENT: Pete ANSWER: We have numerous methods of active and passive data collection methods which do include SNMP, network traffic analyis, and interrogating via Modbus. QUESTION: How to make sure IDS system wouldn’t flag it as intrusion? RESPONDENT: Pete ANSWER: The SOAR/SoC team should be working in conjunction with the OT Secuirty team to ensure that the network path and traffic are allowed/whitelisted by the IPS Sensor. A test and check approach is already a good idea, espically if you have a QA/Sandbox enviroment. QUESTION: How does an organisation ensure that all devices are registered? Especially with a mobile (e.g. vehicles) or Work-From-Home workforce where things may be vulnerable and diffuse? RESPONDENT: Pete ANSWER: There are ways to monitor/instrument these types of devices and if for some reason that is not possible - monitoring the ingress point to your company network for all these devices. QUESTION: Would IT and OT incident response protocol come into play if a pen test session thats knocked equipment off line...? RESPONDENT: Chris ANSWER: It would depend on the scenario whether this should be involved. Typically in this use case you would want additional contextual data. For example, maintenance records so you could determine whether a device going off line is expected. In most cases, you would want to detect something like a network scan and if that combined with a device going offline happened, then it might be suspicious. In most use cases, it'll involve someone going "onsite" to fix the device. QUESTION: You've mentioned a little about this but I would image integrating SOAR in a live environment is difficult. How much does Sandboxing, Simulation and Virtual help the process work. RESPONDENT: Chris ANSWER: Implementing SOAR in live environments can be challenging, which is typically why there needs to be some kind of use case development and incident response plans in place first. This gives you a good place to start, but it has typically been validated and so there is minimal impact to operations. Sandboxing, Simulations, or Virtual Process could help during the testing scenario, but in OT environments that may be difficult to perform due to a lack of a true testing environment. QUESTION: We're using a tool that focuses on identifying devices by listening to traffic. Do you have any ideas for merging that with the CMDB that the IT guys maintain? RESPONDENT: Pete ANSWER: Yes, we at Industrial Defender do this for many of our customers. It's as simple as exposing the asset data and relevent changes via a method support by the CMDB. For modern systems its normally an API, older systems usually have file-based import methods. QUESTION: How does SOAR address securing physically separate legacy systems? RESPONDENT: Chris ANSWER: How much can be done in a legacy environment will often depend on what is possible with OS, equipment interfaces, etc. A good example is the logging of USB devices is much easier and verbose in newer OS'es. Making sure you have some way to get that data in and make it usuable is typically the larger challenge as not all products may support older operating systems and equipment. What you automate may only be limited to information gathering to help an analyst make a decision faster. QUESTION: In distributed environments (pipelines, water treatment, power distribution) the bandwidth to the facility may be very limited. In some cases older equipment may have limited communication throughput capacity before device operability starts to be affected. What "levers" are available to be able to control traffic? ANSWER: This question was answered during the seminar. Please watch the recording to learn more.. QUESTION: Any best practise and expierence that you can share on implementing SOAR on legacy systems? RESPONDENT: Chris ANSWER: There are several best practices: 1) Make sure you can actually gather data from those systems, 2) Have well-defined use cases that you know on those legacy systems, and 3) Try to leverage existing tools or scripts you may have. Gathering data from legacy systems is often a challenge, but teams will usually have existing scripts to handle problems beforehand. For example, one customer had a service from a legacy system that would simply stop logging which was the only indication there was a problem (service would continue running). The customer has already deployed a script that would run when this even occured, but it was very ad-hoc and cludgey and they were never sure whether it succeeded. Running this same kind of action in Phantom allowed them to get feedback from the script so they could validate the script succeeded and then perform an additional step to make sure the service was actually functioning correctly. In this case, it made an after-hours call out only necessary if things did not restart appropriately. QUESTION: Is it safe to say that the 'safety' concern with SOAR has to do more 'what' you automate vs. whether or not you deploy a SOAR solution at all? In other words, the implementation dictates the safety 'risk' more than the technology itself? RESPONDENT: Chris ANSWER: Yes, in most cases it depends on what is the potential impact or effect of the action that SOAR platform takes. One exception though would be a SOAR workflow could not be properly implemented and then cause an issue. For example, say a second action should only occur if the first succeeded and this was never validated which then resulted in a second command causing a problem. However, validating that an automation won't impact safety should be part of building workflows and playbooks in the SOAR solution. QUESTION: Can there be a problem where the IOT is vendor provided on a proprietatary basis insofar as they may "drag their feet" on providing some of the asset inventory information because they consider that their proprietary business information? RESPONDENT: Pete ANSWER: Yes, this is a scenario that can happen when dealing with an OEM. Don't be afraid to push back, as the equipment owner and operator you are the one that is resposbile for the risks related to not knowing your asset's inventory and vulnerability posture. We have helped customers through these types of conversations, often ending in mutal benefits of all parties. QUESTION: Are procurement contracts now including provisions that third-party vendors be required to participate actively in their customers/ SOAR efforts? ANSWER: This question was answered during the seminar. Please watch the recording to learn more. QUESTION: Isn't it a problem to start with SOAR given Chris said you need to know everything on the network and Paul said often that ppl had retired and left systems running on the network nobody understands... seems it'd be difficult to get started? RESPONDENT: Chris ANSWER: This partly depends on the purpose of the SOAR. In most cases, SOAR is implemented with known scenarios and use cases; hence you need to know what systems you want to use SOAR on. Using SOAR on an unknown asset or system could be risky and affect safety and operations. However, many platforms like Splunk might be able to do some detection of unknown systems so you can at least identify their presence (especially if they are networked). In almost all the deployments I have been involved with, we found systems and interfaces that were not properly documented. So making sure you know about an asset's characteristics should be considered when implementing the workflow and be adjusted based on that information. For example, your SOAR platform could do something as simple as create a ticket about an unknown asset and then not perform any additional actions. QUESTION: For the asset owners currently using SOAR, can we get an idea of what industries they are in (e.g. electric utilites or O&G) and what they are throwing into Splunk currently?RESPONDENT: Chris ANSWER: Personally I have seen SOAR most often deployed in Power Utility and Oil and Gas industries. I think those industries have been high profile and a heavy focus in the past (either regulation or in the media) so their security practices tend to be more mature, and as a result, they are looking at how to extend their capabilities. Manufacturing has a heavy interest but I would say are generally less mature. QUESTION: Please pardon my rather mundane question, but coming from an academician's perspective, do you foresee a time at which we might transfer/transition this concept to home-based IoT (security automation) applicability? ANSWER: This question was answered during the seminar. Please watch the recording to learn more..

By Derek Harp, (CS)²AI Founder, Chairman and Fellow September, 2021 Of course, we all want to mitigate risk in our environments. It goes without saying. However, HOW we do that does not. There are so many different products, services, approaches, guidance, regulations and frameworks. Some are broad and some tailored to specific types of asset owners and operators. And then we have to ask ourselves, “Is what we have been doing working?” Are we effectively mitigating or “Knocking Down” the risk to our OT systems? It is believed that Albert Einstein said “Insanity is doing the same thing over and over and expecting different results”. Not to paint that broad of a brush stroke against all that we are doing as clearly there is far more new work to be done than just repeating old. However, there are old methodologies and thought processes that plague our consciousness and leak into our plans for improving cyber security. As we prepare for our next Symposium focused on Cyber Security Risk to Operating Technology - an idea emerged to bring together authors who are writing about consequence-based cybersecurity methodologies that we all can learn from. These are methodologies unique to OT networks and physical operations – approaches that don’t make sense on enterprise networks or in the cloud, and approaches that are robust, even in the face of a constantly-evolving threat landscape. I personally am fired up to learn from Andrew Ginter author of Secure Operating Technology, Andy Bochman and Sarah Freeman authors of Countering Cyber Sabotage (Introducing Consequence Driven Cyber informed engineering) and Jim McGlone co-author of Security PHA Review (for consequence-based Cyber Security) Each of these authors are collaborating to make this (CS)2AI Symposium a valuable education opportunity by opening our minds to new ways of thinking about HOW we address our collective OT cyber security challenges. For me, adding even more industry veterans and true pioneers, Dr. William (Art) Conklin, Bryan Owen, and Mark Fabro to an event closeout panel at the end of the day is icing on an already great cake. I think about the years that some of these very people have been working on the unique challenges to cybersecurity in operating technology systems and am in awe of the persistence I know it required of them. We are only just now entering a time where a broader segment of industry and business leadership is taking the threat to OT systems seriously. Now that this is occurring, HOW we go about mitigating risks or “Taking them down” is everything. Per our mandate and commitment to support the entire control system cyber security workforce everywhere we can, this event has no cost and due the generous support of our Symposium title Sponsor, Waterfall Security Solutions, we are able to give away a copy of each of these authors’ books to 12 winners who participate in our Quality Question submission raffle the day of the event. In addition, this time we are also able to give each of the first 400 attendees to register a copy of Andrew Ginter's book Secure Operating Technology, a super useful pen and a practical gift that I think everyone will find useful instead of taking up space we don’t have on our desks 😊 Stay safe and be well my friends and colleagues, Derek Harp

By Derek Harp, (CS)²AI Founder, Chairman and Fellow November, 2021 During a recent presentation about the Key Findings of the (CS)2AI-KPMG Control System Cyber Security Annual Report 2021 during the SecurityWeek's 2021 ICS Cyber Security Conference & virtual expo, I mentioned a few books that are on my nightstand right now. Some of you reached out for a list, so I have included links below to a few I am currently reading: SECURE OPERATIONS TECHNOLOGY by @Andrew Ginter SECURITY PHA REVIEW FOR CONSEQUENCE-BASED CYBERSECURITY by @Jim McGlone and @Edward M. Marszal COUNTERING CYBER SABOTAGE: INTRODUCING CONSEQUENCE-DRIVEN, CYBER-INFORMED ENGINEERING (CCE) by @Andrew Bochman and @Sarah Freeman CRITICAL INFRASTRUCTURE RISK ASSESSMENT: THE DEFINITIVE THREAT IDENTIFICATION AND THREAT REDUCTION HANDBOOK By @Ernie Hayden

By Robin Berthier, CEO & Co-Founder at Network Perception February 9, 2022 We hosted a (CS)²AI Online™ symposium on January 19, 2022 that focused on Cyber Security for Energy: Part 2 - Electric Sector. Here is a bit about the event: Part 2 of the Symposium on Control System Cyber Security for Energy will provide tangible recommendations and best practices for electric utilities to address current and upcoming compliance and cybersecurity challenges. First, attendees will gain a detailed understanding of the latest government regulations that have been pushed by recent changes in the threat landscape. Second, industry practitioners will share their experience on technology solutions and process improvements to mitigate risk faster and build a strong culture of cyber resiliency. The symposium will provide ample opportunities throughout the event to interact, ask questions, and leverage the shared expertise of the (CS)²AI community. Speakers: • Melissa Hathaway (President, Hathaway Global Strategies) - Keynote • Marc Rogers (VP of Cybersecurity at Okta): Hands-on experience on exploit • Ben Sooter (Principal Project Manager EPRI: Responding to High Impact Cyber Security events in Operations • Branko Terzic (Former FERC Commissioner): Challenges for electric utilities • Philip Huff (Univ. of Arkansas): Vulnerability Management for electric utilities • Todd Chwialkowski (EDF-RE): Implementing Electronic Security Controls • Saman Zonouz, Threats to Programmable Logic Controllers (PLCs) • Robin Berthier (Network Perception): NERC CIP Firewall Change Review Workflow As always, we encourage our audience to participate throughout the event by contributing feedback and questions for our speakers. We weren't able to answer all of your questions, so we have asked some of our speakers to answer a few of them. Below, are some answers to a few questions posed during the event. Do you want to have access to more content like this? Register for our next (CS)²AI Online™ seminar or symposium here, or consider becoming a member of (CS)²AI to access our entire library of recorded sessions. ****************************** QUESTION: Haven't we learned our lesson about companies claiming they're 100% hack-proof? ANSWER: Exactly, it is now well-established that 100% security is unrealistic. This is why organizations have to invest in cyber resiliency: designing systems, processes, and training to be able to keep operating despite being under attack. QUESTION: How are firewalls and DMZ’s validated for built-in backdoors? ANSWER: Through a combination of configuration verification and network traffic monitoring. Independent verification of firewall and router configuration files enables security team to validate that no backdoor access has been inserted. Network traffic monitoring enables security team to ensure that no process is subverting the access control implemented. QUESTION: Is this NP approach IT first, then proceeding to OT? not many firewalls in OT that need to be constantly tweaked or tuned. ANSWER: This depends on organizations. Some have many firewalls in their OT environments. Even if network changes are less frequent in OT compared to IT, we recommend starting with OT verification since that's where the most critical cyber assets are located, and then expanded into IT. QUESTION: What are the most electric companies getting incorrect witht their configurations? ANSWER: 1. Lack of egress access control 2. Lack of documentation 3. Overly permissive rules 4. Insecure services 5. Access list complexity QUESTION: Would all the pieces of the firewall analysis and monitoring done from inside the ESP, cloud, corporate side? ANSWER: We recommend to deploy the firewall analysis platform in the DMZ next to the ESP with unidirectional data retrieval. QUESTION: What is the major differences between traditional firewall system and your suggested firewall system that we can consider? ANSWER: Traditional systems rely on a single firewall management solution. We recommend to separate monitoring from management. Monitoring should be done independently from the management platform so it can be done read-only and reduce the risk of human error. QUESTION: What is best security way during OT connectivity with IT ? data diode or firewall ANSWER: Misconfigured data-diodes can be less secure than correctly configured firewalls, so the key to best security isn't one or the other, but the correctness and continuous verification of the device configurations.

By Branko Terzic, Former FERC Commissioner February 1, 2022 We hosted a (CS)²AI Online™ symposium on January 19, 2022 that focused on Cyber Security for Energy: Part 2 - Electric Sector. Here is a bit about the event: Part 2 of the Symposium on Control System Cyber Security for Energy will provide tangible recommendations and best practices for electric utilities to address current and upcoming compliance and cybersecurity challenges. First, attendees will gain a detailed understanding of the latest government regulations that have been pushed by recent changes in the threat landscape. Second, industry practitioners will share their experience on technology solutions and process improvements to mitigate risk faster and build a strong culture of cyber resiliency. The symposium will provide ample opportunities throughout the event to interact, ask questions, and leverage the shared expertise of the (CS)²AI community. Speakers: • Melissa Hathaway (President, Hathaway Global Strategies) - Keynote • Marc Rogers (VP of Cybersecurity at Okta): Hands-on experience on exploit • Ben Sooter (Principal Project Manager EPRI: Responding to High Impact Cyber Security events in Operations • Branko Terzic (Former FERC Commissioner): Challenges for electric utilities • Philip Huff (Univ. of Arkansas): Vulnerability Management for electric utilities • Todd Chwialkowski (EDF-RE): Implementing Electronic Security Controls • Robin Berthier (Network Perception): NERC CIP Firewall Change Review Workflow • Saman Zonouz, Threats to Programmable Logic Controllers (PLCs) As always, we encourage our audience to participate throughout the event by contributing feedback and questions for our speakers. We weren't able to answer all of your questions, so we have asked some of our speakers to answer a few of them. Below, are some answers to a few questions posed during the event. Do you want to have access to more content like this? Register for our next (CS)²AI Online™ seminar or symposium here, or consider becoming a member of (CS)²AI to access our entire library of recorded sessions. ****************************** QUESTION: What additional challenges will green technologies bring to the operators? ANSWER: Challenges will come, not from the fact that the technologies will be “green”, but that many of the solar units will be small, distributed on customer premises and customer owned rather than at large utility owned facilities. California utilities a looking to address the problem of secure communications with residential solar by application of HardSec. QUESTION: What are your thoughts on improving transmission capability between Eastern and Western grids to aid resilience, and ERCOT's unreliability in winter (as demonstrated in Feb 2020!)? ANSWER: I am all for it. The problem is that transmission siting and expansion is under state regulation and not under the FERC. It is very difficult to obtain licensing for new transmission from those states between the power sources and the remote market region. Congress has to address the problem but is reluctant to remove authority from the states. QUESTION: For self-regulated rural utilities, what have you seen to be the best framework to follow for cybersecurity? ANSWER: I guess I would follow whatever guidelines are set out by the National Rural Electric Cooperative Association (NRECA) and other similar organizations. The NRECA has, for example, filed joint comments at the FERC with the EEI the trade organization of investor-owned utilities. QUESTION: Grid operators with high penetration of intermittent resources, such as Ireland, have shifted capacity acquisition to specific "essential grid services". How important is it for the US Grid operators to also change from acquiring "plain old capacity" to the acquisition of specific grid services, like ramping? ANSWER: The question goes to the point that electricity is an instantaneous “service” and not a bulk commodity to be stored, repackaged and delivered when convenient to the marketer. The various US wholesale power markets have already moved to identifying specific electricity “ancillary services” which need to recognized, measured and priced to insure adequate and reliable service. The Texas ERCOT ignored this fact by not having a capacity market and only relying on an “energy market”. QUESTION: Your comment about a status quo in terms of vulnerabilities suggests that something new needs to enter the picture. Does that include federal funding of capabilities and capacity in "secure" microelectronics manufacture within the boundaries of the United States? ANSWER: Its always nice to get federal funding rather than spend your own funds, I suppose. The microelectronics problem is a slightly different one from the problem of vulnerability of electric utilities to hacking. My suggestion was that utilities look at the new HardSec claims and capabilities, especially for OT systems. QUESTION: Are operators just accepting they will not be able to block attackers and so focusing on how to manage the risk and minimize the blast radius? ANSWER: That seems like the current standard for cybersecurity services, a recognition that either the computer systems are already infected or that an intrusion can only be identified, not blocked. The job of the cybersecurity firms is then is one of rapid identification and recovery. QUESTION: With more security services being offered on the cloud, is FERC/NERC moving towards allowing cloud services for the energy sector? And what are the major concerns of using a cloud service for the energy sector? ANSWER: FERC and NERC regulation is somewhat limited as state Public Service Commissions have significant authority over electric utility budgets, for example, among their ratemaking powers. If use of “cloud services” is demonstrably cost effective versus alternatives then its likely state PSC’s would approve “cloud services’. I do not know about FERC’s position. QUESTION: When a new safety mechanism is introduced, like NERC CIP 013,......is that a FERC lead and NERC follow or vice versa? ANSWER: The NERC can lead but it s under the authority of the FERC, which means that the FERC can approve, modify or supersede NERC regulations. QUESTION: How do you recommend mitigating the security issues of the legacy systems in utilities sector? ANSWER: That sector is perfect for the capabilities of the new HardSec option which is indifferent to software type or age. QUESTION: Is it a problem to implement compliance for power companies because of the different sizes of the power companies? ANSWER: I think the problem has more to do with the management priorities of power companies than the actual size of the utility. Even the smaller Investor-Owned Utilities are large enough to have significant budgets to address cyber security issues. QUESTION: If this sector is heavily regulated, why not force the Supply chain Vendors to adhere for regular upgrade cycles? ANSWER: That can be done by the utilities themselves in their purchasing practices. QUESTION: How should/could a regulator incentivize good cybersecurity practices? ANSWER: The regulator can make cybersecurity performance an explicit indicator of management performance and of utility service quality. Then both penalties and rewards in the forms of financial incentives and disincentives can be adopted after the necessary regulatory procedures. Of course, the regulator has to approve electric budgets commensurate wit the new cyber security requirements.