Migrating data from Alibaba OSS to AWS S3

Cloud technology

It can offer a lot of benefits, such as cost savings, mobility, reliability, high-speed computation, etc to businesses irrespective of its size. And it’s no wonder why almost every business is drawn towards cloud technology. In the past few years, I’ve helped many companies from around the world embrace cloud technology. Working in cloud services is fun. But do you what is more fun?

Switching the cloud service provider!

These days, it is a common practice for businesses to migrate from one cloud service provider to another. Services provided, the performance of the cloud infrastructure, the cost of using the cloud infrastructure, geographical presence, etc are the main factors that influence a business owner to switch the cloud service provider. In this article, I’ll show you how to migrate data from Alibaba’s OSS (Alibaba Cloud Object Storage Service) to AWS’s S3 (Amazon Simple Storage Service)

A couple of years ago, I was working for MAP CLUB -the biggest lifestyle coalition loyalty program in South East Asia. MAP CLUB used Alibaba’s OSS as its cloud infrastructure. But, due to some reasons, they wanted to switch to the AWS cloud. And the herculean task I had on my plate is to migrate the data from OSS to S3. S3 lets you store all your files in ‘buckets’. This means that you can categorize your media according to project names, which makes access much easier. Doesn’t seem like a bad idea, right?

There are two ways through which I could achieve this migration

1. Download all the files from OSS to an independent database and then upload them again to S3
2. Find a tool that would allow me to transfer everything from OSS to S3 directly

The first option is not at all feasible. It’d take a lot of time and more than that, who is patient enough to do such things? People rarely go past the third page of Google results. Well, I went past the tenth page and I still couldn’t find a data transfer tool that I could use. I realized that building my data transfer tool would be a lot easier than waiting for a solution to show up.

So, that’s it. I built the package and transferred all the files and folders from OSS to S3 in no time. Since there is no tool available for this data migration, I think it’s best to opensource it. So that you can use it if you want to. The best part is that I built the tool such that it can be used on any platform, be it Linux, Windows, or even Mac.

To set up and run this tool, you can follow these steps:

Clone or download the project from the Github repository
Set up the S3 and OSS configuration in the config.toml in the project root folder

# This is a TOML document.
[oss]
endpoint = "oss-ap-xxxx-5.aliyuncs.com"
accesskeyid = "XxXXXXXxXXXxxxXx"
secretaccesskey = "XxXNxXxNxXXxxXxxxxxxxxXXXLXxxX"
bucket = "bucket-name"

[s3]
region = "ap-xxxx-1"
accesskeyid = "XXXNXXXNXXNNXXN"
secretaccesskey = "XNXNxxNXnXnnXNXNXnxnnnxXnXnX"
bucket = "bucket-name"
token=""

macOS/Linux:

In the case of macOS/Linux, open the terminal and go to the project directory. Inside the project directory, there will be an executable file osstos3. Execute osstos3 after configuring the config.toml. If there is an error in the configuration it will be printed in the terminal.

$ cd project-directory
$ ./osstos3

Windows:

In the case of windows, open cmd and go to the project directory. Inside the project directory, there will be an executable file osstos3.exe.Execute osstos3.exe after configuring the config.toml. If there is an error in the configuration it will be printed in the terminal.

cd C:\download-path\project-directory
start osstos3.exe

If you have already installed GO in your system, you can also run by specifying the go run command. All you have to do is go to the project directory and follow the below commands. Then run the command to execute the primary function in the osstos3.go file.

$ cd project-directory
$ go run osstos3.go

Configuring Go and Taking Build for Different Operating System

The GO executable can be created not only for Mac, Linux, and Windows but also for other platforms like android, Darwin, FreeBSD, and many more. You can visit the official GO documentation link by clicking here and setting up your system. The documentation by the digital ocean has also mentioned easy methods for building and installing Go programs for different platforms.