Defense in Depth: Version Control, Backups, and Disaster Recovery
Author’s Note: Our Defense In Depth series will demonstrate how over the past several years Corso Systems has helped companies implement security. The series includes relatively easy best practices like ensuring your systems are secured with user accounts, network segmentation with Firewalls to reduce the flow of traffic to only what is necessary to move between each part of your operation. We will also discuss more complex technologies such as threat detection, and backups/disaster recovery so you can get back up and running even in the worst case scenarios.
Most manufacturing and critical infrastructure systems (utilities, wastewater treatment, communications, dams, and many more) use complex systems to automate and run their operations. This includes SCADA systems, PLC programs, databases, and business system integrations.
Many attacks aim to install malware on Operational Technology (OT) systems, which in many cases can then expand easily into the IT side because OT has typically not been set up with security in mind. Sometimes this is because technological advancement in OT hasn’t kept up with IT, though complacency and bad habits after decades of OT having no connection the rest of the business (or the internet).
Now, with many systems getting more connected to the internet, it is urgently important to make sure your SCADA, PLC, databases, and other systems are backed up. You’ll also need disaster recovery documentation to get back up and running as quickly as possible after a cybersecurity incident.
This post focuses on Version Control, Backups, and Disaster Recovery. We’ll share a number of steps any company can take to reduce the amount of pain caused by a cybersecurity incident.
Version Control
Version control is already a well solved problem in IT. Tools like Git and Subversion let companies keep their code stored along with any changes that have taken place between releases. But, in the OT world, technology to make version control easy has been lagging, even though great strides have taken place in the last few years.
In most version control systems, a changelog shows what changed in each version, and you can also make forks and branches of the codebase if you need to develop a version especially for a specific customer, facility, or use case.
Version control systems also help implement the concept of checking code in and out of the system. This allows you to deploy code that has been checked in and essentially “locked down”. It will also require users to check out the code to make changes before checking it back in again. This will enable a continuous integration and deployment pipeline to ensure your production systems are at the latest and greatest while keeping development environments separate you won’t run untested code.
SCADA
Modern SCADA systems like Ignition and our publicly available Ignition Exchange resource Metatools make version control a breeze. Legacy SCADA systems usually have their files stored in a non-human readable format that is also difficult to manage in version control systems.
Ignition exposes the various files in a text-readable format that you can use to track changes in a version control system. You can check code into the version control system while it is being worked on so everyone can see the latest and greatest copy.
You can still manage legacy SCADA systems in version control systems, however you won’t have the granularity provided by the change logs.
PLCs
Most commonly used PLC software is backed up to a computer’s filesystem. This works in practice, as you can easily manage version numbers with descriptive filenames.
Where this approach breaks down is when someone makes changes to the system, and those changes aren’t also pulled into the computer’s filesystem. This usually happens during the startup of a new system, deployment of a new project, or even during routine maintenance activities.
This isn’t necessarily a fault of the PLC programmers. It is very easy to get any copy of the PLC program, or even start with no copy at all and upload the latest version of the code from the PLC itself. All it takes for this backup approach to stop working is for someone to grab an older version of the program, or make changes to the latest version and not save them back to the filesystem.
To get around this hurdle, many PLC programmers are using tools like AssetCentre from Rockwell and VersionDog. These tools let you check in the latest version of the code to their version control system. To make changes, you will need to check it out of the system, make your changes, then check it back in. Once you’ve made the changes, you can track change history across all of the versions, and the tools will also send out alerts when changes are made to the code. If something should be changing you’ll know about it, and more importantly you’ll know if changes happen without your knowledge. Take a deeper dive into PLC backup strategies and build a solid foundation with our post about PLC code backups and version control.
Databases
From a backup perspective, databases have two modes of operation. First, the database schema which contains the database configurations, table structures, queries, stored procedures, etc. And second, the data contained in the database.
Taking backups of the database schema is built into most of the configuration tools you might use to manage the database itself, regardless of what system you are using. This is a relatively simple process and you can store the backup in version control.
On the other hand, data can be much more complicated to backup. Often, there will be large amounts (gigabytes or terabytes) of data. Moving this much data around can put a strain on your network and will require a lot of additional storage space. In many cases, companies will use a High Availability architecture so the database itself will manage backing up the data to other database instances, while keeping copies of it on hand if you need to restore data from any given point in time. This can also reduce risks from a cybersecurity perspective because you will always have a copy of the data available if you need to restore it.
Typically, databases are controlled by the IT department, so a lot of the work managing database backups and restoration will be their responsibility.
Business System Integrations
It’s also very important to keep backups of everything outside the scope of OT systems that still need to integrate with other tools to generate production schedules, quality control test information, shipping/receiving records, and inventory management. While these business systems are traditionally managed by an IT department, tools like Ignition allow you to build many of the integrations into Ignition directly, and they will be handled by your SCADA version control.
Continuous Integration and Continuous Deployment
The ability to automatically push tested and validated code from the version control system to production is another great benefit of Version Control systems. There are a number of tools available to enable this feature, including pushing changes to Ignition servers, and a lot of this work will typically be managed by IT.
Backups
Taking regular backups of all of your relevant technology systems is extremely important. In the event you get hit with a ransomware attack you want to be able to get back up and running as quickly as possible, and also not lose weeks or month of development time because you didn’t have a regular backup scheduled.
Ignition also has built in configuration options for automating backups of your various gateways, we highly recommend enabling this if you don’t have it already, even if you aren’t using version control.
Please keep in mind it is not enough to only take backups. You also need to periodically review the backups and restore them in a development environment to ensure they are working. The worst thing to experience finding corrupted backups when you need to restore a system, and the last working version is over a year old!
Version Controlled Systems
If you are using version control systems for SCADA, PLCs, databases and any other tools you basically get backups for free from your version control system.
Computers, Hardware, and Software
While this post is mostly focused on the various technologies and tools that run operations, it is vitally important to keep backups of your computers, the software you use to run everything, and all of your relevant license information.
You don’t necessarily need to run full backups of every computer you have, however you do need to document all of the network and security configurations, user and network policies, licensing applied to each machine, and which software including versions are installed.
You will also want to back up any network hardware configurations you have, and anything else you would need if you had to get the system up and running from zero.
User Accounts and Roles
In addition to all of the hardware and software you have set up, you will want to make backups of all of your user accounts and roles. This will help you get back up and running in the case of a catastrophic event.
We recommend using tools like 1Password or LastPass to help manage user passwords.
System Configurations
Finally you need to document all of your various system configurations for SCADA, PLCs, databases, business systems, software, licensing, and anything else you might need to get up and running again.
Disaster Recovery
Disaster recovery plans and documentation are a critical tool for any facility. Once your backup and version control strategies are in place, the most important piece of the puzzle is creating a plan for how to restore everything to a good, working state when necessary.
For any projects we work on at Corso Systems, we typically create a disaster recovery plan as part of the initial system setup. We take note of everything we do to get a system up and running from the initial computer and account setup, software installation and configuration, network setup, along with how to deploy everything from backups and/or version control systems.
Disaster recovery can be a daunting task if you haven’t done it before, especially if you have many systems that need disaster recovery plans. To help you manage backups and disaster recovery scenarios we offer our Corso Care packages to take that burden off your shoulders.
You will need to document all of the equipment, computers, and devices you have, each step for setting everything up, WHERE your backups live so you know how to find them. It’s also important to document who is responsible for setting things up, and the order of operations to take to get everything back up and running.
Disaster Recovery planning could be its own series of posts, if you need help to make sense of everything you have please contact us with the details of your project. We’ve completed a lot of the hard work already, and can help you get your plans off the ground.
Wrapping Up
Even if you refuse to implement any level of cybersecurity risk management, everything in this post is still absolutely crucial for any manufacturing company or critical infrastructure organization to note and implement. We’ve seen systems go completely offline due to lightning strikes, major power issues, and even from construction crews digging in the wrong spot and taking out network and electrical infrastructure!
Getting back up and running in these scenarios (or in a worse scenario: a ransomware attack), having backups and a disaster recovery plan will help you manage the stress and chaos involved in starting everything back up.
The worst case scenarios we have seen are when companies have been hit with ransomware and do not have a current backups of many their systems. It can take weeks or months to get everything back online instead of just hours or days if they had had a proper plan in place.
Please take this post to heart and make sure you have everything you need in case disaster strikes, wherever it may come from.
Need to Implement Version Control, Backups, or Create a Disaster Recovery plan?
Corso Systems can help!
Get started now by scheduling an intro call with Cody Johnson in sales, or contact us with your project details and challenges.
