It can offer a lot of benefits, such as cost savings, mobility, reliability, high-speed computation, etc to businesses irrespective of its size. And its no wonder why almost every business is drawn towards cloud technology. In the past few years, I’ve helped a lot of companies from around the world to embrace cloud technology. Working in cloud services is fun. But do you what is more fun?
These days, it is a common practice for businesses to migrate from one cloud service provider to another. Services provided, the performance of the cloud infrastructure, the cost of using the cloud infrastructure, geographical presence, etc are the main factors that influence a business owner to switch the cloud service provider. In this article, I’ll show you how to migrate data from Alibaba’s OSS (Alibaba Cloud Object Storage Service) to AWS’s S3 (Amazon Simple Storage Service)
A couple of years ago, I was working for MAPCLUB -the biggest lifestyle coalition loyalty program in South East Asia. MAPCLUB used Alibaba’s OSS as its cloud infrastructure. But, due to some reasons, they wanted to switch to the AWS cloud. And the herculean task I had on my plate is to migrate the data from OSS to S3. S3 lets you store all your files in ‘buckets’. This means that you can categorize your media according to project names, which makes access much easier. Doesn’t seem like a bad idea, right?
There are two ways through which I could achieve this migration
1. Download all the files from OSS to an independent database and then upload them again to S3
2. Find a tool that would allow me to directly transfer everything from OSS to S3
The first option is not at all feasible. It’d take a lot of time and more than that, who is patient enough to do such things? People rarely go past the third page of Google results. Well, I went past the tenth page and I still couldn’t find a data transfer tool that I could use. I realized that building my own data transfer tool would be a lot easier than waiting for a solution to show up.
So, that’s it. I built the package and transferred all the files and folders from OSS to S3 in no time. Since there is no tool available for this data migration, I think its best to opensource it. So that you can use it if you want to. The best part is that I built the tool such that it can be used on any platform, be it Linux, Windows or even Mac.
To set up and run this tool, you can follow these steps:
Clone or download the project from the Github repository
Set up the S3 and OSS configuration in the config.toml in the project root folder
# This is a TOML document. [oss] endpoint = "oss-ap-xxxx-5.aliyuncs.com" accesskeyid = "XxXXXXXxXXXxxxXx" secretaccesskey = "XxXNxXxNxXXxxXxxxxxxxxXXXLXxxX" bucket = "bucket-name" [s3] region = "ap-xxxx-1" accesskeyid = "XXXNXXXNXXNNXXN" secretaccesskey = "XNXNxxNXnXnnXNXNXnxnnnxXnXnX" bucket = "bucket-name" token=""
In the case of macOS/Linux, open the terminal and go to the project-directory. Inside the project directory, there will be an executable file osstos3. Execute osstos3 after configuring the config.toml. If there is an error in the configuration it will be printed in the terminal.
$ cd project-directory $ ./osstos3
In the case of windows, open cmd and go to the project-directory. Inside the project directory, there will be an executable file osstos3.exe.Execute osstos3.exe after configuring the config.toml. If there is an error in the configuration it will be printed in the terminal.
cd C:\download-path\project-directory start osstos3.exe
If you have already installed GO in your system, you can also run by specifying the go run command. All you have to do is go to the project directory and follow the below commands. Then run the command to execute the main function in the osstos3.go file.
$ cd project-directory $ go run osstos3.go
The GO executable can be created not only for Mac, Linux, and Windows but also for other platforms like android, darwin, FreeBSD and many more. You can visit the official GO documentation link by clicking here and set up for your own system. The documentation by the digital ocean has also mentioned easy methods for building and installing Go programs for different platforms.