Importing Offers from Amazon S3 into SAP Marketing Cloud
Price promotions are a major category of sales promotions where companies reduce the selling price of a product or service to entice customers to buy. Companies often lower prices of their products and services in order to attract more buyers which means that price promotion is a major part of sales promotion. For SAP Marketing Cloud, offers are usually created by external systems and subsequently imported into SAP Marketing Cloud. In our case, we will take a look at Price Promotion which is one type of offer.
Offers are kept in SAP Marketing Cloud and contain information such as:
- Basic information on the offer
- Time validity and status of the offers
- A list of locations where the offer is valid
- A list of products and product categories
- Contacts to whom the offers are sent
- Offer content
In this article, we will focus on the scenario on how to import offers created by external system to SAP Marketing Cloud. As there are many customers that use Amazon Simple Storage Service (Amazon S3) to store and retrieve data, we will store the offers in an Amazon Web Services(AWS) S3 bucket. The process of reading offers, converting and mapping into an appropriate structure will be done in SAP Cloud Platform Integration (CPI). Since the offers don't include all the necessary data, we will have to make one external call to enrich the offers during the processing in SAP CPI before importing the offers into SAP Marketing Cloud.
We will show you how to use XSLT mapping to generate OData request to fetch additional information, how such data can be temporarily stored in memory using HashMap, and how to split large volumes of data into individual offers to be imported into SAP Cloud Marketing. When the import of the offers is completed, we will delete the file from S3 bucket to avoid repeated processing. Furthermore, we would like to show you that some approaches may significantly lower performance especially when processing large volume of data. Therefore, it’s important to conduct various test scenarios in order to find out how the system performs when processing large data.
Table of Contents
To access AWS S3, we will use a Authenticating Request, where the header value also includes a signature. The detailed description of all the steps on how to calculate a signature is provided at: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html#example-signature-calculations.
We will not deal with the steps on how to create an S3 bucket, how to create a test user, and how to grant a user a bucket permission right here. This is out-of-scope for this article.
The prerequisites for reading a file from an S3 bucket are:
- An existing AWS S3 bucket
- Existing credentials composed of an AWS Access Key and an AWS Secret Key
In SAP CPI, we will create an iFlow that reads a file from an S3 bucket and converts it into the appropriate format. Then, we will use parallel multicast processing in order to let one branch make an external call and temporarily store data for further processing. The next step will be to enrich offers with data acquired by the external call. This article is not intended to provide a ready to run integration scenario. For use in real-life scenarios, it is necessary to consider how to proceed in case of an error message. It is not the purpose of this article to give a complete solution, but rather to provide guidelines on the best practices on how to process large files provided by external systems.
For better clarity, our use case is displayed in the diagram below. The whole process is described in three steps:
Step 1: Integration with an AWS S3 bucket to read the offers from the file
Step 2: Data enrichment, here we will:
- Use a sequential multicast in iFlow to make an HTTP call
- Demonstrate how to use XSLT to generate an HTTP request
- Show how to temporarily store data in HashMap and enrich a message
Step 3: Importing the offers into SAP Marketing Cloud with extra focus on:
- How to use an XML parser to modify an XML message
- How to get rid of namespaces in an XML message
Step 1: Integration with AWS S3
While reading files from an S3 bucket, we will use the REST API. Our request will be authenticated by AWS which requires valid credentials. We will make a REST API call directly from our Groovy code based on the information in the link: https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html
We need to create a signature using the credentials and include the signature in our REST API request. Suppose we have a file (containing the offers) saved in an S3 bucket, and we already have the credentials composed of an Access Key and a Secret Key to access the S3 bucket.
Below is a code extract necessary to make an HTTP request to AWS S3. Most of the code can be found here: https://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-java
Once we are done with the groovy script in our iFlow, we add a Request-Reply element and choose an HTTP adapter which enables us to send an HTTP request to AWS S3. In this article, we are not going to deal with Externalization to avoid hard-coded values in integration flows.
The “Address” is the URL of AWS S3 that is to be called from the Request-Reply element.
Enter the Endpoint of the iFlow into the Address field, and in the Authorization Tab, enter the login credentials of the SAP CPI tenant. Once we complete the first part of the scenario, we can save and deploy our iFlow, and then we can test it using Postman.
The response will show the content of the file (offers) from an AWS S3 bucket. In our example, we can see that the offer consists of basic information such as Offer ID, Offer Name and Offer Description, Offer Validity From - To, and also sub-entities like Marketing Location and Products.
Now, let’s take a look at the second part of our scenario which describes the process of conversion, enrichment, and finalization of offers intended for Marketing. As we already mentioned at the beginning, the offers we receive from the AWS S3 bucket are incomplete and therefore we need to add other necessary entities and attributes.
Before that, we should take a look at OData Service Metadata for offers. The following URL allows us to get the metadata file for the offers API service:
The import of offer data is always started through the Import Headers entity and, in order to provide bulk processing, a deep insert on the offer entity. The offer OData resource represents an imported offer and provides basic offer header attributes that can be imported. Resource Path:
The offers data structure pictured below will be enhanced with an “OfferContent” sub-entity containing attributes like content source URL and content target URL.
We will assume that the subject of the offer are products that we have already saved in SAP Marketing Cloud under a unique product ID. Those products already include the image URLs and the target product image URLs. Our next objective will be to carry out an HTTP request during the offers processing in SAP CPI in order to obtain the missing product URLs.
Step 2: Data Enrichment - Offers
One of the approaches how to enrich data with missing information, is to make an individual request for each offer. This would mean that if there were, for example, 100 000 offers, the same number of requests to SAP Marketing Cloud would also have to be sent and it would take extremely long time to process the data. Therefore, we will take a different approach.:
- First, we will search for all unique product IDs in the file from AWS
- Second, then we will generate one OData batch request so that we can receive all product image URLs from SAP Marketing Cloud, and temporarily save them in the memory
- Next, we will match these image URLs with the product IDs mentioned in the first step
The picture below (a part of the iFlow process) displays the offer enrichment process:
After receiving the offers from the S3 bucket and removing unnecessary namespaces, we will use the sequential multicast pattern which sends the same message into two branches in a specified order. This pattern is important because the first branch won’t be executed until the second branch has been completed.
- In Branch 1, we will use an XSLT mapping to remove whitespaces and newlines from the message at the tag level.
- In Branch 2, we will use a predefined XSLT mapping to create a custom batch request for a OData service to fetch/obtain the product image URLs.
What the “XSLT mapping 1” looks like is shown below:
As we already mentioned, the input into the XSLT processing is the message (the offers), and the output will be the batch request that contains a list of GET operations, as shown below:
To send the batch request we created in the step above to SAP Marketing Cloud Result, we will use the Request-Reply pattern. We don’t have to go into much detail on OData usage, as there are a lot of articles on this topic.
The service we are going to use is:
Once the request has been sent to SAP Marketing Cloud, the response including URLs we get, will look as follows:
In the image above, you can see an example of one product containing the necessary <d:ProductImageURL> and Web site <d:WebsiteURL> URLs received from SAP Marketing Cloud. In the next step, we will use Groovy scripts to read this response and create a HashMap to temporarily save the products together with URLs.
For our use case, we will assume that all products with image URLs are stored in SAP Marketing Cloud. In real-life cases, when the product’s URL we are querying doesn’t exist, we will receive a 404 response. Such situation should be dealt with.
The following code snippet from the Groovy script above shows a way of storing multiple values for the same hash key. In our example, we store values such as ProductOrigin, WebsiteURL, ProductImageURL for each “Product ID”.
As we used the Multicast step to process the message in two branches, we will bring the branches together using the Join and Gather elements as shown in the picture below.
The next step will be adding the missing links to images to the products. This step is done using the Groovy script “Modify XML”. First, we will read the message body using XmlSluper, then we will filter out all products from the offers, fetch the URLs from the HashMap and add them to the products.
The „Modify XML“ is shown below :
If we invoked the iFlow at this point in time, we could see the following payload enriched with the OfferContent element.
In the following step, we will use the splitter which enables a single message to be split into multiple partial messages that can be processed individually. The splitter is often used in cases when the external system sends large messages containing hundreds of thousands or millions of records and there is an assumption that the same amount of records will be sent to the receiver system. In such cases, a situation may arise when the receiver system will not be able to process such large message and then performance problems or memory consumption issues may occur. One of the solutions is to use the splitter to create individual messages to be processed.
In the next part of the processing, we will focus on the mapping functionality. We will work on the XML transformation, where we have to provide the XML schema of the source and the target message by uploading respective XSD files.
However, we have to make one additional step in the data processing by adding two attributes related to the communication channels to each offer. In our case, the communication channels will be 'EMAIL' and ‘ONLINE_SHOP‘.
The “Groovy Script 1” that performs these steps, is shown below:
In the snippet below, there is a part of the payload showing one particular offer and two communication media, EMAIL and ONLINE_SHOP, created using the groovy script above.
At this moment, we have all the data ready and we can move on to the next step, which is mapping. But before that, we will make one additional step where we remove the namespace prefix from source payload. For this, we will use the XSLT element.
Step 3: Mapping and Importing Offers
The last step to import offers into SAP Marketing Cloud, is the mapping which transfers data into the required format. Here, we must provide the XML schema of the source and the target message by uploading respective XSD (for a source message) and EDMX (for a target message) files to SAP CPI. There are free online tools which will take an XML instance document and output a corresponding XSD schema.
Once we upload the XSD file of the source message to our iFlow, we will generate an EDMX file which stores the schema of the entities encapsulated in the OData service, including their fields and relationships. In our case, the CUAN_OFFER_IMPORT_SRV OData service will be used.
To invoke the OData service, we will implement a Request-Reply pattern scenario.
To invoke an external OData source, we need to configure several parameters. In order to get the EDMX file, we will use the Query Editor to create the correct the OData service endpoint when connecting to the service provider. In other words, the Query Editor is used to model the access to the OData source
After clicking on the Step 2 button, the Query Editor connects to the service and retrieves its metadata information. From the Fields list, we will select the required fields by checking their respective checkboxes. This information is again retrieved from the service’s metadata information.
After finishing this step, you’ve completed the configuration of the OData Adapter. After that, the EDMX file is automatically created and added to the Resources tab in our iFlow. This file will be used for the mapping step in our iFlow. The source message for the mapping step is the structure defined in the automatically generated XSD file, and the target message is described in the generated EDMX file.
Once the configuration of the mapping activity is completed, you can now save, deploy, and run your new integration flow. What the imported offer looks like is shown in the picture below:
In this article, we specified how to use AWS S3 to read and/or delete an object stored in an S3 bucket. We described one of the methods on how to get additional data by an external call during the processing. We showed you how to make one batch call to fetch all required data instead of making thousands of individual calls. We also demonstrated how to use the XSLT mapping capability to generate OData request.