Data Migrations with Movebot
Learn how to run a data migration with Movebot at a high level.
Introduction
Movebot was built for data migrations and transfers. Our product has been specifically built to provide fast, simple, and cost-effective migrations.
Performing a data migration in Movebot is a relatively simple process that can be set up in a few minutes. This guide will give you an overview of how a migration is modeled and executed in Movebot with an explanation of some key concepts to help you navigate the project.
Key Concepts in Movebot
When configuring a migration in Movebot, there are a couple of key concepts that we have adopted to help make the process simple. It's useful to quickly skim through them to understand how we approach migrations.
#1 Copy vs Migrate
Migrations in Movebot are not really migrations at all.
Movebot perhaps should have been called CopyBot as that's essentially what it does 99% of the time. Moving data infers that the content is no longer available at the source. Movebot doesn't move the data out of the source as in most cases this is not ideal and can be downright dangerous.
For this reason, Movebot copies the data and does not delete it or remove permissions during the migration process. For us, that is the job of the team running the migration, not the machines.
#2 The Project
A project is a collection of transfer jobs and settings.
Projects in Movebot are a collection of transfers that relate to the same source and destination. When configuring a migration, regardless of its size and scope, you must create a project. Projects use the same configuration settings and connect to the same source and destination systems.
In Movebot, you can create as many projects as you like and storage connections can be shared between them. For example, you can create a Google Workspace connection for a Google Workspace to SharePoint migration, then later create a project for Dropbox to Google Workspace and use that already-configured Google Workspace connection.
Projects include:
A source and destination connection configuration
Permission migration configuration
Project-wide content inclusion and exclusion options
Project-wide sanitization options
Platform-specific options
Ignored files
#3 Transfers and Jobs
Transfers and jobs are children of projects. They can be thought of as an individual job or folder mapping between the source and destination locations.
Transfer Mappings are analogous to Jobs, Transfers, Migrations.
They are executed in relative isolation to other transfers
They are composed of a single source folder to a destination folder
All types are mapped with Transfer Mappings. Users, Team Drives, Servers, Network Drives, External Servers etc.
Movebot simplifies migrations by treating everything as folders. Transfer mappings are effectively folder mappings, where folders can be users, mailboxes, network drives, subfolders, or virtually anything the connected storage system supports.
Launching migrations is done by starting transfers.
Transfers can be launched and executed more than once. Each time a transfer is executed, Movebot executes a scan and builds a picture of the migration.
#4 Files and Folders
When executing transfers, transfer mappings include child Files and Folders.
During the scan stage of a transfer, the scan will build out a complete picture of child folders and files under the transfer, including all targeted files and folders under this transfer.
#5 The Scan Stage
The first thing Movebot does when executing anything is run a scan.
Scanning is a fundamental step in the process and a tool in itself. The scan stage maps out and identifies files that are going to be moved, and how those files can and should be transferred to the destination.
The scan identifies and resolves a lot of issues before the transfer process begins and generates a snapshot of what we see in the source and how it will appear in the destination.
The scan functionality in Movebot is used internally as part of the migration, but it is also exposed as the Discovery Scan function on a project level. A discovery scan is purely informational and provides a high-level overview that can be used to scope out and plan your migration.
#6 The Transfer Stage
After confirming the scan results, you can initiate the transfer. Movebot handles all of the infrastructure management and scaling for you and automatically optimizes for best results.
While the transfer is ongoing you have access to the Performance Advisor. This gives you real-time stats about the transfer that is currently in progress.
Once the transfer is complete, Movebot will generate a post-transfer report so you can review your migration results.
#7 The Delta Migration
Deltas update changed and added files
The delta functionality in Movebot is an overlooked gem. Most migration tools require a hard cutover, but with Movebot, the cutover process can be managed by the admin in a way that best suits them.
In Movebot, delta migrations iteratively migrate changes that have occurred since the first transfer using the modification time and some other smarts by looking at a scan of both the source and the destination folders.
During a delta migration, Movebot will only transfer and update files that have either been updated or newly created in the source.
Deltas can be run as many times as needed, including months after the project was initially set up.
Quick Start Guide
Transferring or migrating data with Movebot generally comes down to the following process.
Pro Tip: Run a small migration/transfer first. Don't jump in the deep end too quickly
We recommend that you run a small test migration before proceeding with a large number of transfers. This will allow you to familiarise yourself with Movebot and gain confidence in configuration.
Step 1: Creation of a project
Projects in Movebot are a logical unit of separation. Projects have a source and destination connection, configuration settings, and child transfers.
To create a project, visit Movebot and Click Create Project. During the configuration stage, you will need to select or create a source and destination connection. You can find detailed information on setting up various connections under
Step 2: Project Discovery Scan
With any large migration, we recommend executing a project scan before adding folders. This step is completely optional, but very useful as it provides a high-level overview of the source and highlights potential issues with moving the data to the destination, before you get started.
The project scan is also a useful tool for adding folder mappings. On each folder or user, it provides:
A hierarchical view of files and folders
Totals and summary data
File and folder sizes
Conflicts and issues
Overview of mapping status
Sharing and permission information
Step 3: Mapping/Add Transfers
Before any data will be transferred, you will need to configure Transfer Mappings. Transfer mappings tell Movebot to move data from one location to another. They form the base unit of operations in Movebot and live under a project.
Creating transfer mappings is done from the project dashboard and can be done by selecting the source and destination folders individually or by uploading a CSV file of the mappings.
Step 4: Launch Transfers
Once transfers are created, they can be launched. Launching transfers stages them for execution.
Step 5: Investigate Failures
Movebot attempts to remedy all failures automatically, but there are often edge cases that we can't accommodate. Failures are shown under a job and can occur for various reasons.
It's important that you investigate Critical and High Severity errors. Ignoring these will most likely result in data loss if they are not fixed.
Errors caused by intermittent issues will most likely be resolved in the next stage.
Step 6: Deltas and Cutover
The final and often most frequent stage in a migration with Movebot is running Deltas. With a full migration/transfer like what you launched in Step 4, Movebot will move all data from the source to the destination. With a Delta, we will only move files that have changed or are new in the source and not yet moved to the destination.
If files have changed or failed during the transfer in Step 4, they will be picked up with a delta.
Conclusion
There are a lot of nuances with Migrations that cannot be covered in a short document. If you are migrating from one of the major platforms, we recommend reading the Overview and Connections which will give you a good grounding.
Getting Help
Our support rocks, really it does, and it's free. Reach out to the team with any questions and we are always more than happy to lend a hand.
Last updated