How Our Partnership Migrated 3TB of Data in 6 Weeks
With a great team and the right tools, migrating to the cloud can be quick and easy. Our SharePoint migration team was presented with a challenging cloud project that consisted of splitting a SharePoint Online tenant, managing multiple site owner communication channels, and locking down the source sites after migration.
There were nearly 3,000 sites making up almost 3TB of data that would be migrated on a very aggressive timeline. Migration channels were to include pre and post migration emails as well as a banner on each site home page. We were also required to prevent changes on the source sites during and after each migration in order to avoid incremental migrations.
Migrating to the cloud presents a few challenges that include on-premises features not migrating to SharePoint Online, large lists not completely or correctly migrating and speeds. Since the data was already in the cloud, we were not concerned with features not migrating but we were presented with a few very large lists exceeding 100,000 items that would make the migration even more challenging.
With the aggressive timeline and nearly 3TB of data, our biggest concern was migration speed. There are speed limitations while migrating to the cloud on a single physical machine and even with one of the best tools available, Metalogix Content Matrix, we would likely only average between 200 and 500MB/hour. This limitation was a huge risk that could prevent us from not finishing on time.
Our solution to avoid this risk was Azure. We used 12 Azure VMs that included a domain controller, a SQL Server and 10 VMs with Metalogix Content Matrix all located in the same region as our target tenant. The SQL Server was used to store all the log files created by the 10 migration VMs allowing us to quickly gather statistical analysis and in turn improve our migration speeds.
- 250 MB/Hour/migration VM
- 20 hours/day
- 6 days/week
- 10 weeks
- 10 migration VMs
Figure 1: Azure Configuration
In addition to selection of technology, there was also a month of planning. Our planning included finding sites that contained large lists and/or managed metadata and determining the size of each site. Out-of-box, SharePoint Online includes a report that provides the site sizes but doesn’t give much visibility into large lists or managed metadata without a lot of manual searching. To discover sites with large lists and managed metadata, our team built a custom app that quickly searched through each site and produced a report with that information.
The initial planning was used in order to build out a day by day schedule. It was important that each site in the inventory be scheduled for a specific day in order to provide proper communication with the customer. Each site owner was notified one week prior to migration and the day after migration was complete via email.
Our team finished in 6 weeks which was 3 weeks ahead of schedule. This was largely due to the effort of the team to insure each of the migration VMs were running 24/7. Also, the statistics gathered from the log databases were critical to our success. It gave us the insight we needed to make appropriate changes to our daily and weekly schedule.
Figure 2: Statistics pulled from SQL log databases for 43 days of migrating with Metalogix Content Matrix.
Migrating to SharePoint Online can be challenging and timely, but with good planning and excellent resources can be done quickly and with minimal issue.