Migrate to SAP Commerce Cloud: Migrate Media with AzCopy
12 min read
AzCopy is a command-line utility that can be used to copy blobs or files to
or from an Azure Blob Storage account. This article describes the migration of SAP Commerce media files to
SAP Commerce Cloud Media Storage. Before reading this article it is recommended that you review the
introduction in Migrating to SAP Commerce
Cloud: Data and Media Migration.
Table of Contents
During a media migration your media are copied from the existing on-premise source to the target instance of SAP Commerce Cloud. The migration occurs in a like-for-like environment to ensure data segregation and security, that is, for example, Development to Development, Staging to Staging, and Production to Production.
The following diagram
outlines how you can go about migrating your media. More detailed explanations for each of the phases are
covered in individual sections below.
This phase should define which environments will be involved in the media migration: the typical setup involves three environments (Development, Stage, and Production), however, every customer is different and you may find you need to migrate fewer or additional environments. For example, some development environments might be re-initialized when deployed. Therefore, their data and media could be excluded from a particular migration project. The actual migration of media should be performed environment-by-environment, tweaking and improving along the way.
Using a Self-Service VPN Tunnel
AzCopy allows transfer over HTTPS however it is strongly recommended to establish connectivity over VPN for added security during the transfer so no unauthorized person can hook, record, or steal proprietary data before, during, or after the migration. This is a self service process available directly in the SAP Commerce Cloud Portal and described in the documentation.
A Shared Access Signature (SAS) Token is a single-use identity token URI that grants restricted access rights to Azure Storage resources. A client who is in possession of the SAS token can make a request against Azure Storage with just the SAS URI, and the information contained in the SAS token is used to authorize the request. In order to be able to copy the media files to the targeted SAP Commerce Cloud media blob storage, a temporary SAS Token will need to be created for each environment via a Support Ticket. One Ticket will be needed per environment. SAS Tokens are set to expire after 2 weeks. When the SAS Token expires a new request will be needed. Please allow up to 1 week for ticket to be executed. The following example shows how that request would be created. Make sure that the environment name is mentioned, because multiple environments of the same type can exist.
Please note that ALL the information in the ticket between square brackets are mandatory. The IP to be allowlisted is particularly important as it will be the only IP which is allowed communication to the media blob store in SAP Commerce Cloud.
Media Migration to Commerce Cloud with AZCopy - SAS Tokens Commerce Subscription ID: [link to your Cloud Portal subscription] Storage Account Name: [Storage Account Name] Cloud Environment: [Commerce Cloud Environment Name] Purpose: On Prem to Commerce Cloud Media Migration Incoming IP Address: [100.100.100.101] Please create a SAS Token for the cloud environment mentioned above and send them. Thank you.
Ideally, the you would want to run AzCopy on a system that has direct access to the source media files, such as an application server with a mounted NFS folder containing the sys_master folder of the SAP Commerce media. The following script shows how you can run AzCopy:
# Log in to the server as a regular user # Download the tar.gz file to the home folder: $ wget https://aka.ms/downloadazcopy-v10-linux # Untar the archived files: $ tar -xvf downloadazcopy-v10-linux # Go into to the untar'ed folder to locate AzCopy executable: $ cd azcopy_linux_amd64_10.0.4/ # Verify that the program can run: $ ./azcopy
Once the allowlisting is complete and you're in possession of the needed SAS tokens, a browser can be used to test access using a URL similar to the example below. The Storage Account Name is set in the media.globalSettings.cloudAzureBlobStorageStrategy.connection property that can be read in the hybris Admin Console (hAC) of one of your environment endpoints.
https://[storage account name].blob.core.windows.net/[container name]?[SAS Token]&comp=list&restype=container
Once the AzCopy installation is complete and library is accessible, test connectivity by listing the contents of a container from the SAP Commerce Cloud's media storage.
#list destination to see if connected and accessible ./azcopy list "https://[storage account name].blob.core.windows.net/[container name]?[SAS Token]"
For media storage SAP Commerce provides different strategies which define how and where medias are stored - see Media Storage Overview for more details. Usually customers are using localFileMediaStorageStrategy which stores media on the file system. For SAP Commerce Cloud the cloudAzureBlobStorageStrategy is used, which stores media in Azure Blob Storage and is using containers to structure the data.
When a build is deployed to an SAP Commerce Cloud environment with the initialization option, by default, SAP Commerce will create the following containers for the typically used media types.
|Media Type||Source Location||
For each Media type, an AzCopy command is needed to copy the data to the correct container. In the following script example, it will copy the catalogsync from the source and fully copy that folder and all its content into the target container. With this block of code, scripting the migration to handle all media types should be straight forward.
#catalogsync example ./azcopy copy "../sys_master/catalogsync" "https://[storage account name].blob.core.windows.net/sys-master-catalogsync/catalogsync?[SAS Token]" --recursive=true --log-level ERROR
Some customers might have other media types created from customizations utilizing the Commerce Media Storage Strategy. In general the mapping should be done by analyzing the table "mediafolders"; the "p_qualifier" column is the media type and the "p_path" is the folder name in the old system. Now the mapping should map from .../sys-master/<path> to sys-master-<qualifier>/<path>.
|Media Type||Source Location||
The strategy to migrate those custom media types is the same, with the additional operation to create the target container that will ultimately get the migrated files.
#customMediaType example ./azcopy make "https://[storage account name].blob.core.windows.net/sys-master-customMediaType?[SAS Token] ./azcopy copy "../sys_master/customMediaType" "https://[storage account name].blob.core.windows.net/sys-master-customMediaType/customMediaType?[SAS Token]" --recursive=true --log-level ERROR
The mapping gets a little bit more complicated if subfolders are used. Here is an example: Let's assume a customer has two brands ("fishy" and "tonic") and stores the images for the different brands in different folders. To make the example more interesting the qualifier for the media have the old-brand names in it ("salt" and "pepper"). The mediafolders tables has these entries:
With the localFileMediaStorageStrategy the different images are stored in
In SAP Commerce Cloud these have to go into the new blob containers
The mapping should look like this
It is not possible by just looking at the folder structure and folder names in the local filesystem to determine the correct container names and folders in Blob Storage. The correct mapping can only be done by also looking at the mediafolders table.
Mapping for other Media Storage Strategies
The mapping that was described so far is only for localFileMediaStorageStrategy, if another Media Storage Strategy is used in the source system the mapping might be different.
Here is for example the simple mapping for the s3MediaStorageStrategy for Amazon S3 storage:
Folder in AWS S3
Container and Folder in Blob Store
AzCopy creates a lot of log files, which is a nice feature because it allows it to continue working where it left off if the process gets interrupted. In case of very large folders (e.g. the images folder) it might be a problem, because the disk that stores the logs would run out of space. A mitigation could be to perform the copy process not for the complete images folder with one call to AzCopy, but split the task in several chunks. After each successful chunk the logs can be deleted with
./azcopy jobs clean
As was explained earlier the container name in Azure Blob Store is derived from the "p_qualifier" column in the table "mediafolders". This can cause some problems, because there are naming restrictions for container names in Azure (see https://docs.microsoft.com/en-us/rest/api/storageservices/Naming-and-Referencing-Containers--Blobs--and-Metadata). This means that you may have to adjust the container name that can be used and also the content in the "mediafolders" table.
AzCopy can also support delta migration using the sync option to facilitate pre-cutover and cutover activities reducing the migration time further. This approach is meant to be an extension to the current process, which will give customers a different option to use in case a large amount of media is to be migrated.
You should now have an understanding of the fundamental steps of
migrating media into your SAP Commerce Cloud in the Public Cloud environment(s). For more information
regarding migration please see the rest of the articles in the Migrate to SAP Commerce