Data Migrations to Cloud as a Service (DMaaS)
An innovative solution to execute the data migrations from Onpremise to Cloud more easily than ever before, efficiently and cost effectively. Data migrations will not be painful anymore using the solution described here.
Abstract
For IT teams, tasked with App Modernization and Cloud Migration strategy, Data migrations from legacy onpremise environments (& systems) to Cloud will be the crucial aspect of the transformation and a dominant deciding factor of the successful cloud migration strategy. Data migrations to cloud, typically involve Database Migrations in most of the cases in many enterprise IT teams and a successful migration can have an enormous payoff but the consequences of a failed execution could be dire as well.
Typically, App Modernization will involve breaking down the monolith into microservices or modular, leveraging the distributed data architectures, simplified data management and evolution, simplified system integrations, leveraging the modern COTS solutions or services for standard business processes, etc. App Modernization specifically in Telco enterprises using Cloud migration strategy will involve Hybrid Cloud management with core objectives such as Agile delivery, rapid innovation, security & regulatory compliance and seamless integration with Onpremise systems, COTS applications/services and 3rd party systems/services. DB migrations from legacy Onpremise systems to Cloud will be the majority of the effort, time and cost involved in such App Modernizations. There’s no one-size-fits-all approach to database migration, but there are some key factors and unique solutions anyone attempting DB migrations should consider for a relatively painless experience. This whitepaper describes a solution that will enable Enterprise IT teams to execute the DB migrations to Cloud cost effectively and efficiently with an assured DevEx and OpEx.
Problem Statement
Although there are data migration services and tools available from respective hyperscalers, the database migrations in most of the cases will not be a straightforward journey using those out of the box tools from cloud provider. IT teams in many enterprises have implemented data management and associated applications using traditional RDBMS databases in a manner that is highly bespoke and unique to the organization. That makes the database migrations to cloud also to be bespoke and requires high customization that works best for the given It team or application or data management aspect.
As observed with many database implementations across the IT teams, the database migrations to cloud becomes complex, painful, risky and bespoke due to the factors such as
- The nature of usecases & scenarios for the applications
- Organic evolution of data management patterns using traditional RDBMS systems
- Complex data/entity models and respective data access mechanisms using legacy architecture principles and application design patterns
- Legacy Data management and associated processes that are influenced by specific database technology & capabilities
- Use of specific database native capabilities
- One size fits all implementations using same RDBMS system for everything
- Same database adoption for varied data access patterns
- Hard coupling of applications with database using direct “local access” to database along with network and infrastructure affinity
Database migrations to cloud are challenging and cumbersome in many cases. Enterprise IT teams will have the following requirements to execute the DB migrations to Cloud and drive cloud strategy successfully.
- Cost effective DB migrations
- Efficient DB Migrations (easily recoverable, easy roll back, easy replications, data consistencies, etc)
- Dynamic Data Modelling and DB design for target state — simplified, automated, extendible, quicker, customizable
- Repeatable & Reusable Exercise — templates, jump start kits, MDK
- Seamless DB Operations for migration (Day-0 & Day-1) — migration monitoring, DI/DQ checks, Agile Migration Process, resiliency setup, high availability, Data backups/snapshots, data replications, audit reports, KPIs, SLAs, etc
- Homogeneous and Heterogenous DBs used by various Apps
- Agile Cloud Migrations for flexibility & speed
Solution
In order to make database migrations to cloud less painful, less cumbersome and more efficient, a new solution has been described in this whitepaper which is not available in the industry yet. Enterprise IT teams can use this solution in many formats as suitable as it satisfies almost all the requirements that many IT teams will have in general for database migrations to cloud.
The framework described here is called as Database Migrations to Cloud as a Service (DMaaS) is unique and innovative which comprises of the following components to make it an efficient solution for making database migrations to Cloud painless and less cumbersome and yet cost effective.
- Data Integration Platform (DIP)
- Migration Amplifying Kit (MAK)
- Migration Assurance
- Migration Pipeline
The following diagrams shows the key components of the framework for database migrations to Cloud for different scenarios.
Data Integration Platform (DIP)
This is the heart of the solution which will enable various data management aspects required for database migrations from onpremise to Cloud. Largely, this will consists of the following key capabilities.
- Data Profiling, Mapping & Analysis
- Data Model Transformations from source to target (one type of DB to another, one-to-many distributed DBs/Data stores, etc)
- Schema Conversion & Migration
- Data Reconciliations
- Extract, Transform, Load and Distribute (ETLD)
- Data Annotations
- Metadata management of Source & Target database(s)
- Data Catalogues
DIP is the core engine of the migrations in this framework which can be extended for any type of data migrations but not for RDBMS database alone. Ab Initio is a good example to use as DIP in this framework.
DIP in this framework should not be assumed as a traditional ETL tool or to integrate Data Lakes or Data Warehouses. DIP in this framework is an integration platform for the data flows from one system to another. DIP in this framework is not proposed as any intermittent storage layer rather it acts as a Data Bridge for Migrations from Source to Target.
Migration Amplifying Kit (MAK)
This is the automation of the actual migration solution which will be used to migrate any database to Cloud leveraging DIP and respective Cloud DMS. This automation enables the IT teams to templatize and repeat the database migrations so that teams can perform the database migrations more easily and efficiently. The MAK consists of the following capabilities to drive the required automation.
- Play Books for Migration Guidance & Recommendations
- Reusable Templates for migrations
- Run Books for migration pipeline (sequencing, feature flags, etc)
- Abstraction of DB Operations (Source & Target) for Migration with the integration of respective Cloud DMS
- Data Backups & Snapshots
- Data Replications & Synchronizations
- Data Security & Protection (Encryptions/Decryptions, Masking, etc) - Migration Assurance
- Migration Visualization
MAK and DIP are complimentary to each other and mutually inclusive in this framework. MAK will enable the automation for the pipeline, process, workflow, expert intervention as required for the data migrations.
MAK proposed in this framework can be customized as suitable for the given capabilities and process workflows as suitable for the given IT team and data migration usecase.
Migration Assurance
Cloud migration is complex, involving people, processes, and technology where database migrations will add additional risk and challenges in execution for the IT teams. But this solution makes it easier than ever, with automation to execute the database migrations to Cloud efficiently. This solution will provide the framework to measure and provide assurance for the IT team using the following constructs for executing the database migrations with high assurance.
- Migration Operations
- Notifications & Alerts
- Reports
- Dashboards
- Traceability
- Error Handling & Recovery
- Security Posture and Compliance - Migration Test
- Acceptance & Regression Test
- DI & DQ Validations & Reports
- DB Posture for Pre & Post Migration - Migration KPIs
- Zero Down Time
- Zero Data Loss
- Resiliency (Rollback, Failover & Recovery)
- Data Integrity
Assurance for the migration will be provided by leveraging the MAK and integration with other monitoring tools that are in use by the IT teams in Onpremise environment and as well as the Observation tools/capabilities provided in the target cloud environment. MAK in this framework can be integrated with ITSM and as well as DevOps tools as necessary to provide the required level of automation for monitoring and assurance of the migration solution.
Migration Governance
As the framework is quite open and allows teams to evolve it rapidly and extend it for wider number of usecases, it is necessary to have high level governance so that reusability, efficiency and effectivness can be ensured while the teams autonomously using it.
At high level the governance will include the following key aspects which can be managed by a central team to make it efficient and enable the wider adoption across the organization.
- Migration SLAs
- Cost per migration
- Burnt Down Effort
- Time to Market - Migration Maintenance
- Migration catalogues & patterns
- Pipeline repos
- Upgrades
- Evolution
Migration Pipeline
Leveraging DIP and MAK, this solution will automate the database migration as a continuous and repeatable pipeline for consistency, reliability and execute the database migrations in an Agile manner. IT teams can execute the database migrations more quicker and easier using the pipeline automation with the following capabilities.
- Migration Pipeline leveraging respective DIP & MAK
- Migration Feature Flags
- Migration Assurance
- Application Trials and Rollout
A Sample Migration pipeline can be implemented as follows
Usecases
The solution described here can be used for any type of Database migrations to Cloud. Typically, database migrations will fall under the following categories depending on the usecase, architecture strategy and path for application modernization or migration to Cloud.
- Onpremise Database Rehost (lift’n’shift) to Cloud
- Re-platform or Refactor of Onpremise Database and migrate to Cloud [same type of DB (eg. RDBMS) but different DB at source and target]
- Re-architecture/transformation of Onpremise Database for the Application Modernization in Cloud (RDBMS to NoSQL/DocumentDB, Single RDBMS to distributed set of different DBs, etc)
IT teams across the organization will have different applications per domain yet they will be using same traditional RDBMS systems leveraging the enterprise license model. But the migration of databases to Cloud will largely vary from one application to other. This solution enables the application team to execute the database migrations more predictably, reliably and efficiently.
Conclusion
Depending on the App Modernization and Cloud strategy, IT teams could migrate into a new database platform on cloud, maximizing the benefits of a new target database and cloud infrastructure. Before migration, it is important to perform analysis regarding the various aspects such as data replications, downtime and service disruption, data backups, operations, etc as suitable to the target application architecture and the respective DB solution considered in the target cloud environment. Although all major cloud providers have respective tools and services for DB migrations, it becomes overwhelming for the IT teams to learn and execute different ways for different clouds. It requires customisation and bespoke implementations for the DB migrations that will be unique to the application, organization practices and processes and business needs.
Within an Enterprise, most of the times, the nature of DB migrations would be similar for a given source & target DB solution and the DB operations will be similar along with sharing of enterprise licenses. So, considering the risk and effort involved in the DB migrations to Cloud it is absolute necessary and essential to have a bespoke, flexible and automated migration solution that can be used for various App teams within the Enterprise.
The solution and framework described here is flexible and can be extended and upgraded to manage any type of Data migrations from onpremise to cloud or from one cloud to another or from one platform to another or from one data source to another or from one technology to another beyond databases migrations.
This solution will enable the enterprise IT teams to execute DB migrations to Cloud efficiently and cost effectively. The solution described here can be productized within the Enterprise Cloud CoE/Migration Hub so that application teams across the LoBs can consume it as an Insource Service and hence make DB migrations less painful and also assured enough aligning to the cloud migration strategy.
