Even today, organizations still rely on the slow, painful, error-prone manual process of working with data. This can involve scanning various documents, OCR with the mandatory deciphering of bad handwriting, adding needed metadata, properly naming documents to fit the organization standards, saving digital files in the right format, moving documents to other physical and digital location, etc. Just reading the description of this process is enough to give you a headache.
Is There Light at the end of the ECM Migration tunnel?
Now think about the scary process of moving all that data into an ECM system. Enterprise Content Management systems are massive systems built to answer most of the modern company’s data needs. But, taking all those scanned files, all those disjointed excel spreadsheets, tons and tons of data stored in different databases is scary, to say the least.
Companies can spend thousands of work hours in processing all their data and still not be out of the woods when it comes to eliminating human error. There are far too many moving parts, far too many variables, and every step of the way is yet another chance of forgetting something, overlooking a small detail, and end up with a failed migration into an ECM.
The old saying goes applies. If I could get a Nickel any time the phone rang with a stuttered voice on the other end of the line saying “Um, guys, umm, we have a problem. The Payroll database, we kind of dropped the ball and, um, we can’t really make sense of what we have. It’s just a series of numbers, then a series of names, then a series of commas, colons, semicolons. Please help!”
Our data migration teams have gone through all kinds of data migration near-death scenarios and we’ve come out more focused, more successful, with a streamlined and even automated process of data migration. This goes both for data migration into an ECM, and data migration from one ECM into another.
When doing Automated ECM migration the key step is to come up with a repeatable and scalable process of taking all kinds of data on the Input of the process and send out structured, properly formatted data on the Output of this process. This usually results in a very simplified data intake because we’ve removed all or most of the manual steps. When a migration is properly architected, the flow should look something like this:
- All of the organizations’ paper documents are scanned and OCRed in batches, and very little human intervention is required.
- Document metadata is formatted according to the target ECM system needs.
- The digital files are automatically identified, categorized, located, and named according to the organization’s rules.
- The files are automatically stored in workflows.
- New files are noted in the master directory.
Automating the whole process of ECM migration means that operations that previously have taken weeks to complete, especially on large groups of documents, now take just a couple of hours.
Why Automate ECM Data Migration?
Most of the times, people jump at the pitch of automating anything. But companies are a bit more conservative to hop onto new glittering trends when it comes to their data. And this is actually a good thing. The Shiny Metal Object Syndrome caused some pretty big headaches for pretty big organizations.
Sometimes, however, there is a lot to be gained from automating an ECM migration, especially when this is done by a seasoned team of experts. Here are just a few benefits to keep in mind:
- Save time: Migrating your content manually is a time-consuming and complex process where you can spend maybe months of constant work. Automating the whole process can perform the migration in just a few days, or even hours (depending on the amount of data).
- Avoid data loss and duplication: Instead of missing some data or copying it twice, you can do things automatically. Translating your organization’s’ databases directly to the new database structure is a much easier process. With an automated approach, you can avoid data loss and duplication and simplify the whole process.
- Reduce human involvement: The good thing about automating the content migration is that you can start when you want and whenever you want. Automating your content migration will reduce the delays and errors connected with manual work.
- Skip reinventing the wheel: Whatever your case may be, chances are that somebody somewhere was in a very similar situation. So there is probably somebody out there that has the experience, and more importantly, the tools to do an ECM migration properly, the first time around. Tools like Caliente have done automated ECM migration far too many times for you to avoid the fact of its existence.
Automating the process of migration is completely different from the manual process. It is very easy to trace back steps and find where data originated and this makes fault-fixing much, much easier than any manual process. When you factor in ECM data migration tools like Caliente, the whole Automation idea takes a much more approachable, down-to-earth look where you get to see benefits much faster. Not to mention that the period from starting the ECM migration to the day your organization starts using the ECM will be much shorter.
Data migration is an important part of every intranet redevelopment. It is a challenging process and it can become extremely tedious if a lot of content awaits migrating into an ECM. Most of the times, this loosely organized content is the stumbling block for many organizations to even consider ECM Migration.
The good news is that many other organizations have already migrated onto an ECM, or migrated from an old ECM to a newer, stronger, better-supported ECM. The processes and technology needed to make data migration a manageable risk are already out there. There are teams who have migrated millions of documents and terabytes of data. Chances are, regardless of the specificity of your organization, someone out there has a process and a tool to make the data migration fully automated.
One such tool is Caliente. It’s a data migration solution specifically designed to handle large-scale data migration to and from various enterprise content management platforms and electronic formats including, but not limited to, Documentum, Alfresco, and Microsoft SharePoint. An automated data migration tool like Caliente will help you simplify the whole process of migration by providing a library of templates and tasks that can be easily performed with one click.
Our data migration experts have moved enormous amounts of disjointed documents into an integrated ECM. If you have any concerns about your specific situation, or you have some more general questions, feel free to comment below and we’ll get back to you with some answers, insights or advice.
Many organizations get cold feet when facing the daunting process of migrating to a new ECM. Migrating content that is unattributed, unorganized, and unclassified can fill up the ECM with huge quantities of unwanted data, or it may entirely derail the project.
Today’s import tools are tailored for user-driven import tasks or one-time migration efforts. Both solutions are working for specific scenarios in which the functionality is meeting the requirements, but for some import projects, they are not providing the full solution.
A more flexible method is to set up a protocol and integration/import framework that gives a simple and familiar development process for multi-system integration. That import/integration framework should have a high-level upgrade tolerance so that changes in the target or the source can take place independently, so long as content requirements do not change.
Armedia’s Caliente is such an import framework that is providing a flexible environment for the unattended and development batch processing of metadata and content to be loaded into the content repository.
Caliente is a content migration tool specifically designed for the large-scale content migration to and from different ECM platforms and storage solutions. To name just a few: Alfresco, Microsoft SharePoint, and Documentum.
Caliente Case Study
Caliente was used to migrate millions of structured and unstructured pieces of content into a new ECM Documentum based Case Management system. The system was developed to support a large US Government Enforcement and Intelligence Agency to manage investigative cases electronically and help the cause of fighting terrorism.
The vision of this organization was to replace the existing mainframe structured and paper-based data case system with a modern system that supports electronic workflow, records, comprehensive search, evidence tracking, and among other features rich reporting. The ultimate goal for the new system was to support their current functional priorities, including the most important priority – preventing terrorist attacks.
The Agency’s new content repository had many differently structured objects than the mainframe application. This created some challenges as mapping structured data from the old system to content object in the new one was a non-trivial task.
Another challenge occurred in the performance since there were millions of existing legacy record and the new Case Management system can only be shut down for limited periods during weekends.
After recognizing that their current solution did not integrate with Documentum, the System Integrator turned to Armedia to support the migration with the use of Armedia’s Caliente. The specific task was to migrate structured and unstructured content from an IBM mainframe-based file system to the Documentum platform.
The approach was taken to upload legacy metadata to a normalized Oracle database tables and move associated files from the mainframe to temporary file storage. Caliente was used to load the content files from the temporary file system storage, and the Oracle tables, into the Documentum repository.
Caliente loads data in a pipeline process so a single query from the source database can create several Documentum objects. From the Oracle table, a row of records can be loaded into several Documentum object in a single pass with one SQL query. Caliente here helped to efficiently map the simpler model to the richer, new model.
To reduce the time of migrating millions of legacy content objects, multiple instances of Caliente were run at the same time.
Caliente has successfully migrated the administrative case files from the mainframe to the Documentum repository. Caliente succeeded to create 8 million new Documentum objects in less than 4 days. These 8 million objects covered around 40 custom Documentum object types, including a mix of non-sysobject and sysobject object types.
Integration of multiple content sources with content management repositories is becoming a growing business issue, and it cannot be efficiently addressed by collecting one-off app development projects or depending on user-driven migration tools. Configurable and repeatable middleware that simplifies the content exchange between systems will decrease the total cost of ownership. It will also protect organizations from expensive remediation when the target system or the source has to be replaced or upgraded.
The documents are essential to any organization and during the migration you need to make sure all documents are migrated to the new system without any negative impact on the integrity of the documents or your business.
Armedia’s Caliente offers a low-cost option that adjusts the supportability of packaged integration with the flexibility of custom development. Because of its open architecture and flexibility, Caliente can support the most common import scenarios facing organizations today.
Content migration is a complex process for every organization. It is an enterprise-wide data management process that is tedious and demands huge amounts of resources. Content migration is a challenging process since it can become extremely difficult if you have a lot of content to migrate. Therefore, many organizations are approaching a content migration process from the perspective of performing the bare minimum for meeting their objectives. That’s why organizations are focusing on what tool to provide for the content migration.
There are two methods for managing your organization’s content migration – automated or manual. So, you can either form a team to generate and migrate all the content manually, or you can make a use of numerous automated tools and make the whole migration process easier.
Manual Content Migration
Manual content migration is not always the harder method. It depends on the environments between which the data is transferred and the volume of data that needs to be migrated.
Now, let’s suppose that your organization has a data storage with 200 categories and 2000 products with additional descriptions and attributes. You have chosen an ECM platform with better features than your current one and now, all you have to do is transfer all data including hundreds of categories and thousands of products. How long will it take to manually transfer the database? The answer is – weeks, maybe months.
Manual content migration usually adds to the complexity of the whole process because of the following issues:
1. Slows down the whole process
Increasing the number of people that will do the job by hand only goes so far to resist this. You will either outsource the tedious work, or you will tie up your internal resources to do that by hand.
During a content migration process, content model changes are extremely common. Using the manual method for your content migration only means redoing the already migrated content.
3. Double entry of the content
A manual content migration can last weeks or even months. During that period all changes of the already migrated content must be logged and copied into two systems. The alternative is freezing the long content. This is not an issue with the automated content migration method.
4. Lack of improvements
Manual content migration is not promoting any intelligent improvements which are possible using the automated content migration method such as metadata enrichment, syntax checking, and data conversions.
The manual method of content migration should be considered only if heavy data editing is required or when the content is divided across various locations. This includes auditing your data to create data inventory and then evaluating the quality of each piece of content based upon your previously determined strategy.
Automated Content Migration
The era of automation increases the popularity of services and technologies which aim to make every organization’s burden easier and routine tasks optimized. It is a key part of eCommerce, and it is available for the task of automated content migration.
The manually repeating methods for copying over the content of the same nature now can be easily automated. If your organization is one that has a huge volume of content of the same nature, then a huge part of the migration can be automated. Using the automated method for your organization’s content migration will bring you a bunch of benefits:
1. Time saving
As we mentioned, migrating your content using the manual method is more complex and time-consuming process. You may spend months to complete the whole process if everything goes according to your plan. You can avoid this difficulty with the help of automated content tools like Caliente. The content migration will be automatically performed in just a few hours, and you will save much of your valuable time.
2. Service Availability
The good thing about automated content migration is that the whole process is performed with only minimal human involvement. You can start the migration anytime you want and from everywhere. This method excludes the errors and delays connected with human factors such as working hours, employee’s vacation, etc.
3. Excludes data duplication and loss
Using the manual method, you can miss or copy twice some data, and the more data you have, the bigger is the risk of mistaken data transfer. On the other hand, automated migration connects directly to your organization’s database and copies required content to the new database. Using automated content migration tool like Caliente will help you exclude the data duplication or loss and it will simplify the whole migration process.
4. No coding skills required
By migrating your content automatically, you don’t have to have special skills and deal with any programming tasks. The automated content migration tool will lead you through the migration steps, giving you hints what information you need to provide and things to do next. These tools are very helpful and are minimizing any risk of mistakes.
Using automated content migration tools will help you ease the burden and save time when you have a high volume of content to migrate. Choosing the manual method will only save you money, but the money you will spend on automated content migration tool don’t worth the efforts and time your organization will spend. On the other hand, choosing the automated method will make the whole migration process accurate, effortless and risk-free.
Migrating your content manually will contribute to natural and inevitable mistakes. Automating your migration will accelerate transactions, locate documents quickly and minimize human errors. Automated tools can help your organization save thousands of hours and also ensure a sustainable process for maintaining content.
Automated content migration tools like Caliente will help you automate the whole process of migration by giving you a library of tasks and templates that can be performed with one click. With Caliente, you can import content from external feed and process unlimited content files daily. Caliente is customized migration tool designed for large content migrations. It can migrate from and to various electronic formats, including but not limited to, ECM Documentum, ODBC compliant databases, eRoom, Alfresco, SharePoint and hot folders.
An automated content migration tool is a must have especially if you are an organization with a high volume of data. It will help you simplify the whole migration process and will save you a lot of efforts and time.
The automated content migration is our choice. What is yours? Let us know below.
As you know from a previous post, it bothers me that out-of-the-box, D2 does NOTHING. In an effort to help validate D2 v4.5 installations and provide a test environment for my DQL Editor widget, I developed a simple “starter” application that I named “D-Top”.
D-Top provides simple, unrestricted navigation of a Docbase, shows default
dm_sysobject properties, versions, audit trails, renditions, object locations, and workflow tasks. It provides search capabilities, virtual document support, and bookmarks for checked out objects and subscribed objects. You can create and import documents, as well as check them in and out. Nothing has been “customized” for specific users, roles, object types, ACLs, or lifecycles.
I had several goals in mind while developing this application:
- First, to provide a way for developers and users to quickly validate D2 installations. If this application works and allows you to perform basic repository functions, you can conclude that the installation is correct and is ready for you to develop your own applications.
- To provide a “starter application” (more generic than the HR Sample configuration provided with D2 v4.2) upon which to build other applications.
- Lastly, so that this application now does something useful out-of-the-box (more or less).
View showing relationships, history, and an attribute dump
If you are interested in the “D-Top” D2 starter project, you can download it here. Once downloaded, import it into D2-Config like you would import any other such configuration. See the EMC Documentum D2 v4.5 Administration Guide for instructions regarding importing such configurations.
If you are interested in what else Armedia can do with this application, leave us a comment.
This software is provided “as-is,” without any express or implied warranty. In no event shall Armedia be held liable for any damages arising from its use.
My article for the EMC Proven Professional’s 2015 Knowledge Sharing Contest, A Language Translation Service for Documentum, has been published! You can read the article for yourself on the ECN in the Proven Professionals’ area (if you have access), or on my personal blog. I am interested in any feedback you have on the article and the integration.
Back in February, I posted a video demonstrating how the translation service worked with Webtop, and in June I posted a short video interview I gave at EMC World 2015 discussing the article.
If this integration could benefit your company, department, or organization, leave us a comment, we’d be happy to discuss it with you.