The Great Migration: The Do’s and Don’ts of Large Data Migrations 

Every financial institution (FI) understands data’s transformative powers. While FIs have more data than ever to analyze, storing and accessing it quickly for insights is challenging. A lack of data completeness plagues FIs and prevents them from justifying every action with an audit trail at a moment’s notice without errors. 

These shortcomings have led to firms realizing the need to migrate their data to new archives. However, once firms begin exploring a migration, a whole host of issues crop up, leading to a patchwork of multiple archives and solutions. 

What are these challenges and how can your firm navigate them successfully? We explored these questions in a recent webinar moderated by Alex de Lucena, director of product strategy at Shield. The panel featured Simon Altit, director – EMEA at Insentra, Tom Rimmer, regional sales manager at Shield, and Anthony Diana, partner at Reed Smith LLP. 

The fear of migration 

At first glance, moving your data makes intuitive sense, much like moving to a new house. The old house is harder to maintain, is a bit rough around the edges, and isn’t suitable for your current needs. Recognizing the benefits of moving to a new house, or new technology, is easy. 

However, the benefits are the only simple facts you’ll see.  is not only complex, it rests on a bevy of fears from conservative executives and uncovers a host of new issues and headaches. 

Rimmer lists a few of these concerns. “What’s the risk of data leakage?” he says. “How do firms protect it? How do they do that safely? How can they make sure it [the data] will still deliver the outcomes they want?” 

The concerns don’t stop there. Firms must consider regulatory risk and ensure data in their new archive is just as discoverable as in their legacy one. Rimmer also points out that identifying the right stakeholders to guide and sign off on the migration is a challenge, given the wide scope of data migration. 

Diana says that legal and compliance departments often don’t understand the complexities of a migration, posing further challenges. “They don’t know enough to say ‘what are the risks that I have to start mitigating?’ because people don’t explain it to them,” he says. 

Altit points out that costs also breed uncertainty. “A big driver around these programs is the TCO of getting out of their existing platform and into the new one,” he says. De Lucena extends the point by saying hostage costs (the cost of migrating data from a legacy vendor to a new one) also cloud the TCO picture. 

“You’re trying to sell something internally—an opportunity—knowing that the cost is going up,” he says. “And where’s the benefit you’re selling? It’s just [another] fee. That can really make people hesitate.”  

This situation breeds uncertainty that makes data migration a tough sell. 

These challenges and unanswered questions breed a fear of migration making the status quo a seemingly safer place. This begs the question: Is it worth it?  

Painting a picture of the firms migrating their data 

De Lucena identifies 3 categories of firms looking to migrate their data: 

1. The desperate 
2. The brave
3. Everyone else

The desperate 

The desperate firms are driven to migrate due to shortcomings in their legacy technology stack. Much of these shortcomings boil down to legacy tech falling short of handling new kinds of data feeds. “A lot of the legacy platforms don’t store much more than mail and in occasional scenarios, a little bit of instant message,” Altit explains. “With the amount of regulatory and compliance requirements, they just can’t consolidate all that information in a single place, which is easy to access, easy to search, etc.” 

Aging infrastructure is another key driver for these firms, irrespective of whether that infrastructure is on-prem or in the cloud. “We’re seeing some of those legacy cloud vendors as well that pose issues,” Rimmer says. “Firms are caught by these vendors due to hostage costs in there like cost to export, etc.” 

De Lucena adds that legacy systems struggling to maintain data integrity is another driver.  

“When it was all email coming in, firms could say 99.99% is here and come up with something to automatically replay it,” he says. “But now you’re getting a Slack feed from a third-party. And they don’t have the tooling.” 

Diana says data deletion within legacy systems—or the lack of it—is creating needless risk. “I tell my clients, all you’re sitting on is risk,” he says. “Whether it’s a litigation risk or a risk for privacy, it’s all risk. You’re paying for storage of data that is just risk.” 

These drivers scare some companies into initiating data migration projects. But what about the other end of the spectrum? There are companies realizing the benefits of new technology and migrating their data proactively. 

The brave 

De Lucena categorizes these companies as “brave” organizations, saying they recognize the benefits of the cloud and digital transformation. Altit says these moves are as much about delivering optimal services for these companies and not just about a move to the cloud. 

“It is also about a service strategy for a lot of these businesses,” he explains. “They want to get rid of a lot of the ongoing administrative overhead and the investment that’s required to maintain a lot of these [legacy] platforms.” 

Everyone else 

Then there’s every other firm that recognizes the need to migrate its data but doesn’t. Diana says a firm’s risk perception around old data leads to a lack of action. “It is often a lot easier to say, “we’re going to go to a new archive”,” he says. “The harder sell is, “What do we do with the old stuff?” Some people have had data there sitting there for twenty years. They haven’t been deleting it.” 

“From both an IT perspective and legal perspective there’s a lot of fear associated with it,” he continues. “A lot of people say, well, I’ll let it die in the vine.” 

But this is a flawed way of thinking. “Firms think they will apply some type of retention legal holds and delete it,” Diana says. “Now my perspective is it’s not as passive an exercise as they would think. You still have to analyze what’s there.” 

As a result, data lives on, creating workarounds that complicate the picture. “What you’re doing is kicking it down to somebody else,” Diana says. 

So, what should firms do to avoid falling into this trap? 

Developing a migration plan and avoiding pitfalls 

Rimmer says aligning stakeholders and having conversations about alignment and work processes are critical. Altit points to an example of how aligning stakeholders early in the process saved time in a migration. 

“We were doing work for a global investment bank and one of the decisions that was collectively made early on was to bring both their outside counsel and their auditors in right from the get go,” he says. “We had mapped out how we were going to approach it to meet what were quite challenging demands in terms of information that both the council and the auditors wanted and the way they wanted to present it better.” 

Altit reckons this initial work cut an additional year from the project. Rimmer adds that examining vendor contracts for handcuffs is important during this phase. 

“One of the areas that we put into our contracts is outlining that your data is your data,” he says. “And I think that’s really important for people to look at those contracts as they go into the migration with the new vendor because sometimes that’s not the case.” 

Diana says setting expectations across the organization is critical. 

“Ultimately, someone has to sign off and say, “we’re decommissioning this legacy archive” that could be billions of messages,” he says. “Someone has to put their signature, and that is always one of the hardest things to do. You have to prepare them.” 

Diana also adds that setting expectations around error rates at this stage is important since legal and compliance expect 100% accuracy. “Have a solutions design document upfront that lists expected errors,” he says. “They just have to accept the risk that some of this data is going to be corrupted.” 

Altit adds that explaining the causes of data corruption is critical. “We’re now at a point where these platforms have existed for 20, if not 30 years,” he says. “The platform the customer is in today is not the first one they were in. And so there is a raft of additional issues that come with data that’s already been migrated once and potentially converted and all of the other components that come with that.” 

Understanding this upfront helps firms present information to migration service providers and consultants, resulting in mitigation plans that enable error acceptance and sign-offs. 

Altit adds that verifying platform use cases before migrating platforms is critical. “A big part of the transition is making sure the data is easy to consume similarly,” he says. “The way the use cases used to work in legacy platforms don’t align with the way the use cases work today.” 

Educating users about these changes and making sure they’re comfortable with them is important. Once migrated, reconciling data boils down to 3 steps, according to Altit: 

1. Getting an inventory of all data in the legacy platform. 
2. Verifying whether data in the new platform aligns with data the migration tool discovers. 
3. Verifying whether data reported by migration tooling arrives in the target platform. 

This kicks off the testing and validation phase. “I can’t emphasize enough how important that is because [you can’t] make it [the migration] go faster by moving the data and then dealing with it when it’s in the new archive,” Diana says. 

Instead, he says, embrace the time it takes to test and validate. “Moving the data is the easy part. It’s the testing, the validation, understanding what it is, dealing with the anomalies, that takes a long time.” 

Removing fear from migrations 

Data migrations might pose a seemingly insurmountable challenge at first, but careful planning can help companies execute one successfully. Ultimately, modern technology offers companies several benefits. But realizing those benefits needs some preparation and careful execution. 

With a project that can bring so many benefits across the organization, it’s no longer a question of if you’ll migrate but when. Above all the panelists agree on 1 thing: Communication and planning are key. Before starting a new migration project, consult the experts and align your organization to realize a successful conclusion.   

• Watch The Webinar