Digital Transformation: Data Migration Process and Tools
Any company considering a Digital Transformation must utilize their data in the best ways possible. For many companies, this specifically points to to their Process Historian. Is your Process Historian costly and/or poorly supported by the vendor? Do you wish that you could use cutting edge data analysis tools instead? Are you locked in by multiple years of data across many thousands of tags? Good news: you don't need to be locked down any longer!
Data Migration: A Better Way
Let's be honest, no one makes it easy to bring all of your old data into a new Historian. This is a multi-pronged problem. Most companies focus on building better technology for themselves—they don't have time to go deep enough into another system to understand how to export everything. Plus, they don't want to disparage, or even mention other companies in their marketing. Conversions are a one-time task and may not be critical for getting the sale in the first place. In the worst cases, the old Process Historian simply goes into hibernation and is retrieved when someone needs old data.
At Corso Systems, our job is to integrate systems. We make software that doesn't or shouldn't talk to other software spill all of its secrets. We reduce the number of tools our customers need for running their facilities. At Corso Systems, we "Say Yes and Figure It Out". And, we think that converting a Process Historian from one platform to another is a fun problem to solve. We've been solving this exact problem for companies for years. But now, we have a better, more automated data migration process and tools. Check out our post: Ignition Explained - Tag Historian to learn more.
Why Automate Your Conversion?
Reason 1: Manual Migrations Take Forever
The most painful way to migrate data is by hand. Sorting through existing tags, then creating them in the new historian—then manually exporting and importing a bunch of files. This was the process on the first Wonderware to OSI PI Conversion we ever did. (Read the case study) Thankfully this wasn’t too terrible since the project had less than 5,000 tags with just two years of data—and we used the best tools available at the time. Today, we need better solutions for systems with hundreds of thousands tags, and over a decade of data. Time for a better solution!
Manual Migrations Are Expensive
Consider the previous example with 5,000 tags and two years of data. Let's stack the deck and assume everything will work perfectly—of it doesn’t in the real world. Assume that miraculously all the data looks good on first view—there are no strange strings from ten years ago and when an event occurred when the data went to 0 and/or the quality went to bad. Then, assume you have a world class Process Historian expert working on your solution… which you probably don't. Now, let’s use the extremely fast time of five minutes per tag to migrate it by hand. Then we'll also assume that the full two years of data is also good and we don't have to worry about it. So, even in that (unrealistic) best case scenario, 5,000 tags x 5 minutes per tag still equals 25,000 minutes or 416.67 hours or about 10.5 person weeks of work. This amount of time also doesn’t include breaks, software issues, and normal issues that are not just straight work hours. At $200/hour, this comes to $83,334 or about $16.67/tag. Ok, that seems a little expensive, but does it scale?
10,000 tags
50,000 Minutes
833.33 Hours
$166,666 Cost
20,000 tags
100,000 Minutes
1,666.67 Hours
$333,332 Cost
50,000 tags
250,000 Minutes
~2 Person YEARS
$833,334 Cost
100,000 tags
500,000 Minutes
~4.2 Person YEARS
$1,666,668 Cost
Now, consider the cost with non-world-class experts. What if you’re having some people work on the issue internally at your facility. They may need twenty minutes or more per tag! And while you’re probably not paying them $200/hr, for just 10,000 tags at twenty minutes each that’s almost an entire person-year worth of work—without including breaks, other maintenance, or any other issues that they also need to handle!
Want to talk about opportunity cost?
Data Validation is Crucial
Have we convinced you that manually moving large amounts of data doesn’t scale? Good. Now, let's talk about an important feature of our data migration tool: Data Validation.
Our Process Historian Data Migration tool does the initial data imports to validate communications, data integrity, and the overall import process. Once the initial import is validated, you can generate the scripts for an automated import of the bulk of the data.
The first step is to set up communications with both Process Historians. Once established, navigate to the tag selection screen to pick the tags you want to import. Then select the period of time to import, the target historian, and then click "Import". This will grab all of the data, generate the tags in the new Historian, import everything, and generate a list of any import errors or anomalies. The anomalies be quarantined and re-imported once the issues have been resolved. We also have a page to query each historian and validate that the data matches up, highlighting anything out of the ordinary.
Step By Step: The Data Import Process
A typical import process…
Use our Process Data Import tool to generate all of the tags in the new Historian and begin collecting data. Typically, we do this either at the start of the workday, or the start of a particular hour so there’s a defined point in time for when to start backfilling data.
Import a single tag's data for a small period of time (example: one hour), to validate that everything is functioning.
Expand the list to a larger group of tags for the same period to validate any network or performance impacts with a larger dataset.
Expand the process to the entire tag list to validate the import for all tags.
Try a larger time period for all of the tags (example: twelve hours), to validate performance.
Import a day's worth of data.
Finally, import a week's worth of data.
Move to automated import mode once you have validated that everything is working properly.
Data Migration: Day by Day
Our Process Historian Data Migration tool breaks up the imports into daily chunks. This reduces the amount of reporting you need to handle if there are any issues. It’s also simpler to go back and re-import anything as needed, since the overall import datasets are reduced. This happens on the backend, whether you are doing an automated import or a manual import. This approach also helps to increase performance for the overall system.
Depending on the Process Historian capabilities, we also allow you to configure which datasets the imported data will go into. This makes it easy to test by creating datasets strictly for testing which you can delete with no impact to the production data. It also makes it easy to sync the testing datasets with the production data once everything has been validated. This gives you the best performance possible.
The next step is to validate that the new Process Historian data is integrated into your existing analysis and reporting tools. This is where you will truly realize the value of an integrated system with imported data. Generally, this process involves pulling data from both systems for spot checks of reports, trends, etc. and verify that everything matches.
Why did we build this migration tool?
Importing data by hand is extremely time consuming—and time is money. Even with the most detail oriented people on the case, data migration can be very tedious and error prone. Trying to get each tag import process into the single digit minutes can still mean staring down multiple person-years of labor required to complete the project. Our tool strips away a lot of this risk. We automate the validation process so you can focus on fixing the small number of anomalies instead of having to manually validate gargantuan datasets.
Our tool also makes it easy to help our customers get to the bottom of why they want to change their Process Historian in the first place. If the data isn't locked into a single platform, then choosing a new Process Historian becomes about finding the best tool for the job. This includes taking into consideration support, training, and any other potential tradeoffs you will get with a software package.
Keep in mind that we build the tools we need, and make them better as we use them. Our common projects are converting between Wonderware, OSI PI, FactoryTalk Historian (PI), Ignition, and Canary Labs Process Historians. If you don't see your Historian listed, let us know. If you have questions, please reach out so we can make your life easier. Check out our Canary Series, where we talk about Acquiring Assets and Transitioning Tags.
Can YOU Use This Migration Tool?
The short answer is yes. If you're going through a migration process, Corso Systems can use this tool to migrate your data for you. At the moment, the cost will be dependent on what Historians you're using and migrating to, what your data looks like, and the level of manipulation needed. If you're another service provider looking to use the tool as a service or externally, we'le also consider working with you. We realize that there are a lot of facilities considering migrating to new process historians.
We also realize that there are many really good companies out there who provide top notch services and we’d consider letting you lease this tool. Contact us for specific information and pricing and we can walk through it! Want to know more about expanding your Process Historian and look at Manufacturing Execution Systems? Check out our MES Guide!