Scalable Cross-Cloud Data Migration to Amazon S3 with Distributed rclone

Migrating petabytes of data across different cloud providers to Amazon S3 presents significant operational challenges for organizations. Traditional data transfer methods often struggle at this scale, leading to difficulties in tracking copied data, managing failures, and requiring constant manual intervention. To address these complexities, AWS has developed and outlined a distributed migration architecture that leverages several AWS services to create a scalable, resilient, and automated pipeline. The proposed architecture is built on three distinct layers: Discovery, Queueing, and Execution. The Discovery layer, powered by Amazon ECS Fargate, is responsible for automatically identifying and enumerating objects from the source cloud provider. These objects are then batched into self-contained messages and pushed to an Amazon Simple Queue Service (SQS) queue, forming the Queueing layer. This approach ensures fault isolation and enables built-in retry logic, significantly reducing the need for manual intervention during transient failures. The Execution layer consists of a fleet of Amazon EC2 instances running rclone workers. These workers scale up and down dynamically based on the SQS queue depth, managed by an EC2 Auto Scaling group. This elastic scaling allows for aggregate throughput well beyond what a single machine could achieve. The rclone workers process the batches from the SQS queue, securely retrieving credentials from AWS Secrets Manager, and then transfer the data to Amazon S3. This distributed architecture is designed to be provider-agnostic, supporting migrations from S3-compatible storage providers such as IBM Cloud Object Storage, Google Cloud Storage, and Azure Blob Storage. The solution also integrates with Amazon CloudWatch for comprehensive observability, providing logs for troubleshooting and metrics for performance monitoring throughout the migration process. A single AWS CloudFormation template can deploy the entire infrastructure, including pre-configured IAM roles, simplifying deployment. In testing, this architecture successfully migrated a 2.7 PB dataset in approximately two weeks with a compute cost of around $2,000, demonstrating its efficiency and cost-effectiveness for large-scale cross-cloud data migrations.