Data Integration in the Cloud – or Not

Data integration is critical for companies that need to combine systems and databases after a merger or acquisition, or need to link legacy systems to new business applications. This need is vast throughout commercial and public industries, and the task can be daunting, especially for CIOs and IT managers of small- to medium-sized businesses, whose IT staffs and budgets may be constrained. Add to that the rise of cloud-based data integration services and the decision becomes even more confusing:

“Should we implement data integration on-premise, or choose a cloud-based Software as a Service (SaaS) solution provider?”

There are clear advantages to either approach, but for many SMBs, the hybrid approach — where some data integration processes take place in one or multiple clouds and others remain behind the firewall — is the most practical solution.

Reality Check

It’s not always feasible for companies to move all their data integration functions to the cloud. Instead, some find it more practical to “dip their toes” into the cloud for a host of reasons. The trick is figuring out which toes to dip in — or, more explicitly, which business processes to move. You want to be confident that your data integration processes will work as well in the cloud as they do on-premise. No one makes that decision without some trepidation. Choosing a cloud services provider that specializes in data integration and can meet your technical and regulatory requirements will help ensure a successful transition.

You can test the waters by moving one or more of your data integration projects to the cloud while leaving the rest on-premise. Establishing new integration processes in the cloud or moving customer-facing B2B processes to the cloud are much easier to do than moving the more complex back-office integration processes and will also provide a healthy ROI much more quickly.

The key to deciding what projects to move hinges on a slew of technical concerns, including the cost of data translation, connectivity, performance, scalability and security, all of which must be fully vetted with cloud services providers.

Data Translation Woes

Moving data around, whether it’s on-premise or in the cloud, is technically easy. Data translation is the tricky part and, therefore, can be expensive. Many companies scrap their plans to move data integration processes to the cloud simply because of the cost of data translation. If the cloud provider doesn’t use the same tools as you use, it could mean throwing away your investment and paying your provider to reproduce data translation events either as part of a professional services engagement or as part of your subscription cost. There are two ways to overcome this barrier:

1. Adopt the hybrid approach, whereby existing data translation stays on-premise and all of your new or modified processes get implemented in the cloud; or
2. Find a provider that can do data translation at a lower cost. Fortunately, the data integration industry is evolving, and some data integration cloud service providers are already using a new generation of intelligent mapping tools that significantly speed up the task of moving on-premise data translation processes into the cloud, which correspondingly lowers the cost.

If you’re at the juncture of trying to decide whether to license a new tool for on-premise data integration or outsource to the cloud, you can hedge your bet by identifying vendors that do both. These companies license their data integration tools for on-premise use and use those same tools to handle data integration in the cloud as part of their service offering. This future-proof strategy gives you the flexibility to implement a more modern data integration solution on-premise now while removing the technical and cost barriers of moving to the cloud in the future.

Connectivity Concerns

One of the biggest worries people have about cloud computing is losing their connection. Despite the recent, multi-day global loss of service by BlackBerry users, loss of connectivity within the cloud is actually very rare.

A cloud provider is judged on availability. For that reason, it builds its IT infrastructure for high availability and disaster recovery to ensure that service won’t be interrupted even in the event of a natural disaster. Companies are often surprised to see that their own history of downtime is worse than that of cloud providers.

Performance Planning

The only real issue between on-premise and cloud computing performance that might be a concern to some organizations is latency. Cloud computing can never achieve the same low latency as tightly coupled servers on the same LAN segment, but the difference will be negligible so long as the bandwidth is adequate.

If you anticipate the potential for minor latency differences being an issue — and for most companies it’s not — then make sure this is addressed when your data integration solution is being re-architected for the cloud.

System Scalability

Besides being able to add hardware resources easily to handle expanding data volumes — a hallmark benefit of cloud computing — the ability to scale bandwidth is also an important consideration when moving business processes to the cloud. With internal IT infrastructures, systems are typically housed in close proximity to one another on a LAN segment, which can be optimized to improve performance. The downside of this optimization is that it creates bottlenecks that eventually inhibit growth, simply because the systems are so tightly coupled that they become rigid.

Decoupling some of these tightly integrated systems so that some of the data integration processes are performed in the cloud while others remain on-premise forces you to re-architect or at least re-think your systems, resulting in a much more flexible architecture that can scale more easily. In a nutshell, the cloud model forces a good technical design that allows companies to avoid organically creating the performance “fixes” of the past that limit internal scalability.

Security Sensitivities

Data security is another concern for SMBs that are considering moving data integration processes to the cloud. It’s important to find out what provisions the cloud provider has for security, from protecting sensitive data at rest and destroying it at the end of its lifecycle to how it handles access controls. This becomes especially important for companies that must comply with industry data security standards or privacy laws.

For example, if you’re planning to store or process credit card numbers in the cloud, make sure the cloud service provider is PCI DSS compliant or can provide you with the audit information you will need for your auditors. If your organization collects data globally, it is also going to be important to ask where the cloud provider will keep the data to ensure compliance with data protection laws within those jurisdictions.

Expert Help

While there are a lot of moving parts associated with transitioning your data integration processes to the cloud, the weight of having to figure out how to assemble those parts to make them hum doesn’t have to rest solely with you. If you choose the right cloud services provider, it will have both the data integration solutions and the expertise to work through the technical issues to help you come up with the best solution. Because they’ve lived through numerous conversions with a multitude of different organizations and have already solved a plethora of process and technical complexities, experienced data integration cloud service providers are in the unique position of understanding the nuances of moving on-premise legacy systems to the cloud.

In short, cloud providers add expertise to your organization that can be difficult and expensive to add otherwise.

Practicality Wins Out

What it really comes down to is gaining an understanding of the technical issues of data integration in the cloud, weighing your options and deciding what to put in the cloud and what to keep on-premise. It doesn’t have to be all or nothing.

In fact, most SMBs will find that it makes perfect sense to migrate data integration processes to the cloud slowly and incrementally. This approach will allow your organization to become familiar with the cloud, gain trust in it and start seeing the ROI more quickly than if you had moved everything at once.

The hybrid approach will likely be around in most organizations for some time, and that’s OK. Fast-forward to the not-so-distant future and we’re likely to see the true rise of the Intercloud, where an organization’s computing and data integration processes are managed by multiple cloud services providers, and where data flows securely between clouds, applications and users without a hitch.