Digital Transformation Data Migration Process and Tools
Any company looking to go through a Digital Transformation process needs to find ways to utilize their data in the best ways possible. For many companies this is specifically points to to their Process Historian.
Is your Process Historian costly? Poorly supported by the vendor? Wish that you could be using cutting edge data analysis tools?Are you locked in with multiple years of data across however many thousands of tags? You don’t have to be locked down any longer.
A Better Way
Let’s be honest, no one makes it easy to bring all of your old data into your new Historian. This is a multi-pronged problem. Most companies focus on building better technology for themselves. They don’t have the time to go deep enough into another system to understand how to export everything. They don’t want to disparage, or even mention other companies in their marketing. Conversions are a one time thing and may not be critical to get the sale in the first place. Worst case the old Historian simply goes into hibernation and you dig it out when you need old data.
Corso’s job is to integrate systems. We make software that doesn’t or shouldn’t talk to other software spill all of its secrets. We reduce the number of tools our customers need to run their facilities. At Corso Systems, we “Say Yes and Figure It Out”. Converting a Process Historian from one platform to another is a fun problem for us to solve. We’ve been solving this problem one at a time for years. Now we’ve come put with a better, more automated data migration process and tools.
Check out Ignition Explained Tag Historian.
Why Automate Your Conversion?
Manual Takes Forever
The most painful way to migrate data is by hand.
Sort through the existing tags, create them in the new one, export a bunch of files and import them. This was the process on the first Wonderware to OSI PI Conversion we ever did. (Read the case study)
This wasn’t the most terrible being less than 5,000 tags with 2 years of data. We used the best tools available at the time. Today we need a better solution for hundreds of thousands tags, and over a decade of data. Time for a better solution.
Manual Is Expensive
Using the previous example above of 5,000 tags and two years of data.
Let’s stack the deck and assume that everything is going to go perfectly (in the real world, it doesn’t.)
Miraculously all the data looks good on first view. There are not strange strings that we were using 10 years ago and when an event occurred, the data went to 0 and/or the quality went to bad. Say that you have a world class Process Historian expert working on your solution (which you probably don’t.) We’ll use the extremely fast time of 5 minutes per tag to be done by hand. Then we’re going to also assume that the two years of data is also good and we don’t have to worry about this.
5,000 tags x 5 minutes a tag = 25,000 minutes or 416.67 hours or about 10.5 person weeks of work. This is not including anything like breaks, software issues, normal things that are not just straight work hours.
At $200/hr, this is $83,334 or about $16.67/tag
Ok, that seems a little expensive, but does it scale?
- 50,000 Minutes
- 833.33 Hours
- $166,666 Cost
- 100,000 Minutes
- 1,666.67 Hours
- $333,332 Cost
- 250,000 Minutes
- ~2 Person YEARS
- $833,334 Cost
- 500,000 Minutes
- ~4.2 Person YEARS
- $1,666,668 Cost
Now consider the cost when we’re looking at non-world class experts. Say some people internally at your facility. We could easily be in the 20 plus minute range per tag. You’re (probably) not paying these folks $200/hr but for just 10,000 tags at 20 minutes a tag, we’re almost an entire person year. This is not including breaks, other maintenance, no other issues that they are working.
Want to talk about opportunity cost?
Validation is Crucial
Convinced you enough that manually moving large amounts of data is not scalable? Good then let’s talk about an important feature that our data migration tool: Data Validation.
This Process Historian Data Migration tool does the initial data imports to validate communications, data integrity, and the overall import process. Once the initial import is validated, you can generate the scripts for an automated import of the bulk of the data.
The first step is to set up communications with both Historians. Once established you go the tag selection screen, pick the tags you want to import, the period of time to import, the historian you want the data to go into, and click “Import”. This will grab all of the data, generate the tags in the new Historian, import everything, and generate a list of any import errors or anomalies encountered. These will all be quarantined and can be re-imported once the issues have been resolved.
We also have a page to query each historian and validate the data matches up, highlighting anything out of the ordinary.
Step By Step
A typical import process looks like this:
- Use our tool to generate all of the tags in the new Historian and begin collecting data. Typically we do this either at the start of the work day, or the start of a particular hour so we have a defined point in time for when to start backfilling data.
- Import a single tag’s data for a small period of time, say 1 hour to validate everything is functioning.
- Expand the list to a larger group of tags for the same period to validate any network impacts, or performance impacts with a larger dataset.
- Expand to the entire tag list to validate the import for all tags.
- Move to a larger time period for all of the tags, say 12 hours to validate performance.
- Import a day’s worth of data.
- Finally import a week’s worth of data.
- Move to automated import mode once you have validated things are working properly.
Day by Day
Our Process Historian Data Migration tool breaks imports up into daily chunks reducing the amount of reporting you need to handle if there are any issues. It also makes it simple to go back and re-import anything as needed by reducing the size of the overall import datasets. This happens on the backend, no matter if you are doing an automated import or a manual import.
This also helps increase performance for the overall system.
Depending on the Historian capabilities, we also allow you to configure which datasets the imported data will go into. This makes it easy to do testing by creating datasets strictly for testing you can delete with no impact to the production data. It also makes it easy to then sync the testing datasets with the production data once everything has been validated. This gives you the best performance possible.
The next step, is to validate the new Historian data is integrated into your existing analysis and reporting tools. This is where you will truly realize the value of an integrated system, with imported data. Generally this process involves pulling data from both systems to do spot checks of reports, trends, etc. and verify everything matches.
Why did we build this tool?
Importing data by hand is extremely time consuming. Time is money. It can be very tedious and error prone, even with the most detail oriented people on the case. Try to get each tag import process into the single digit minutes. You can still quickly be staring down multiple person-years of labor required. Our tool strips a lot of this risk away. We automate the validation process so you can focus on fixing the usually small number of anomalies instead of having to manually validate gargantuan datasets.
It also makes it easy to help our customers get to the bottom of why they want to change the Historian in the first place. If the data isn’t locked into a single platform, then it becomes about having the best tool for the job. This includes support, training, and any other potential tradeoffs you will get with any software package.
Keep in mind, we build the tools we need, and make them better as we use them.
Our common projects are converting between Wonderware, OSI PI, FactoryTalk Historian (PI), Ignition, and Canary Labs Process Historians. If you don’t see your Historian let us know. If you have questions please reach out and we can make your life easier.
Check out our #FullCanary Series, where we talk about Acquiring Assets and Transitioning Tags.
Can YOU use this tool?
The short answer is yes.
If you’re going through a migration process, Corso can use this tool and migrate your data for you. At the moment, the cost will be dependent on what Historians you’re using and migrating to, what your data looks like and the level of manipulation needed.
If you’re another service provider looking to use the tool as a service or externally, we’re also looking to look at working with you. We realize that there is a lot of facilties that are looking at transiting historians. We also realize that there are many really good companies out there that provide top notch services and we’re looking to let you lease this tool.
Looking for specific information or pricing? Drop Dave a line at (773) 241.0004 or Dave@CorsoSystems.com and we can walk through it!
Want to know more about expanding your Process Historian and look at Manufacturing Execution Systems? Check out THE Ultimate MES Guide!