CX Works

A single portal for curated, field-tested and SAP-verified expertise for your SAP C/4HANA suite. Whether it's a new implementation, adding new features, or getting additional value from an existing deployment, get it here, at CX Works.

Implementing Your Data Retention Policy for SAP Marketing Cloud

Implementing Your Data Retention Policy for SAP Marketing Cloud

You might have heard statements such as "data is the new oil" multiple times during presentations or conferences. While this article is not about the validity of statements like these, it is about the effect of data being continuously collected. One of the noticeable effects is that the amount of data grows substantially over time, sometimes exponentially.

This article will elaborate why this can become an issue and how you can address it with a data retention policy. We will look into the definition of retention policies and how implement those with SAP Marketing Cloud. 

Table of Contents


Motivation

Image One: Reasons for Data Deletion and Archiving


Most of the reasons for storing data for marketing are obvious. High-quality data is essential for all marketing processes, from personalized campaigns to product recommendations and campaign success analytics. In addition, there are legal regulations like the Sarbanes-Oxley (SOX) Act for U.S. public companies, which defines how long each type of business data must be kept.

Why should you care about data deletion and archiving? There are four main reasons to think about this topic: Compliance, Performance, User Experience, and Cost.


Compliance is based on legally enforced regulations and corporate standards and policies. Legal requirements and various data protection and privacy regulations define the boundary conditions. Regulations can require you to only keep data, if consent is given or when it's necessary to keep it. Corporate standards comply with legal regulations and add an additional layer, for example, applying the European-Union General Data Protection Regulation (GDPR) not only in the EU but also in other countries. Here are a few exemplary regulations that are important to many companies: 

  • General Data Protection Regulation (GDPR) - European Union
  • Privacy Act - Australia
  • Personal Information Protection Act (PIPA) - Japan

The performance of the system benefits when a data retention and deletion policy is implemented. Accessing large tables slows down the performance and database tables will be partitioned when the number of records grows over 2 billion (2^31). The performance might decrease with an increasing number of partitions. Overall, a well defined data retention policy helps to keep the system in a good shape with a managed data throughput.


Huge amounts of data can effect the user experience and become a beast for end-users. If you strive for a high degree of easy self-consumption of your campaign automation solution, you should limit the amount and complexity of data to what is essential for the marketing business. Less is more.


The software license of SAP Marketing Cloud regulates the overall number of contacts in the system. The license cost is based on this. If you have reached the number of contacts defined in your license, you must either contact SAP to increase the licensed contacts or delete outdated and unnecessary contacts. 


Disclaimer: Please be advised that nothing in this article constitutes legal advice. We strongly recommend that you involve your legal deputies to check and approve the concepts you want to implement.


Data Deletion and Data Archiving

When talking about data retention, there are two fundamentally different concepts: data deletion and data archiving. Both are typically part of the definition of Data Lifecycle Management. 

Data deletion is known as the technical or physical destruction of data irretrievable by ordinary commercially available means. This means that data is no longer accessible and available for further processes. 

Data archiving means that data which is no longer actively used is moved to a separate type of storage for long-term retention. A future usage of archived data is generally possible, for example, for data recovery. 


SAP Marketing Cloud supports data deletion as described in this article. Data archiving would need to be implemented on a per project basis as of the most recent update of this article (SAP Marketing Cloud 1905). However, in most cases there are no legal reasons to retain marketing data. In most cases, data should be deleted for the reasons described above.


Defining Your Data Retention Policy

Image Two: Drivers for Your Data Retention Policy


Before defining your data retention policy, you should understand what the main drivers for your data retention policy should be. In general, you want to declutter your system of marketing data that you are obligated to delete or that no longer serves any useful purpose. While there is no silver-bullet approach across all markets and industries, there are some typical considerations for building up your data retention policy. 

The main drivers for the data retention policy should be compliance, costuser experience and performance. The first step is to check compliance with the legal & data protection and privacy regulations to identify data that you are obligated to delete or that you must keep. Another aspect is cost through license compliance, as described in the previous chapter. User experience is mainly based on business value and should be qualified through business user interviews and quantified  through structured data analysis in the system. Performance implications (based on high data volume) should be quantified through the specific applications in SAP Marketing Cloud. 

As-Is Analysis

You should start with an as-is analysis in your system. The leading questions are: What data is currently persistent in the system and what are the measures to be taken to be compliant, increase business value and performance? 

The most important objects which should be part of every data retention policy for SAP Marketing Cloud are interactions, interaction contacts, and target groups. These objects typically lead to a high data volume in the system. Based on your system and your scenarios, there are likely more objects to consider. However, since the approach is very similar, we will focus on these three objects. The following section should help you to analyze the data volume of these objects. 

Interaction contacts can best be checked through the Browse Contact Data app. In addition, the segmentation might also help. The typical criteria are the validation status, the age of the contact's record, or the number of interactions of the interaction contact. Following the drivers from above, check for compliance, cost (license compliance), user experience (for example, data rarely used across the segmentation models) and performance (for example, mass anonymous contacts). 

Interactions can best be checked using the Browse Interaction Data app. Using this app, you can analyze interactions, identifying which ones should be deleted due to compliance, bad user experience due to low business value, or for performance reasons.

Target groups are another important object due to the persistence of the target group members for static and dynamic target groups. The number of target group member records can grow quickly, especially if target groups are built through the target group set operations (for example, intersect dynamic target group A with dynamic target group B). You can best check target groups through the corresponding target groups app, which gives you the status, created/changed timestamps, and the number of members. For target groups, compliance and cost is typically not relevant, however, user experience and especially performance should be analyzed. 

One example of an as-is analysis result is shown below. In this example, we have the typical inbound and outbound interactions and both anonymous and known contacts in the system. The four drivers and the related analysis is shown in the following table:


Data Object

Total Number of Records

Compliance

Cost

Selection Criteria

User Experience (Based on Business Value)

Performance Impact

Interaction Contact


4,000,000

Not applicable

< 5,000,000 due to license compliance

Anonymous contacts

Very low

  • not used in any campaign

High

  • Anonymous:
    • 500,000 (~1/8 of all contacts), of that 400,000 records older than 3 months
  • Not anonymous:
    • 3,500,000 (~7/8 of all contacts)

Contacts with no interaction in last 12 months

Low

  • used in ~5 campaigns

High

  • No interaction in last 12 months:
    • 1,000,000 (~1/4 of all contacts)
  • At least 1 interaction in the last 12 months:
    • 3,000,000 (~3/4 of all contacts)

Interaction

12,000,000

Not applicable

Not applicable

Click through, or e-mail open, or e-mail outbound, or website visit

Medium

  • mainly used for contact analysis and segmentation

Very high

  • 8,000,000 (~2/3 of all interactions), of that 5,000,000 records older than 6 months
  • 4,000,000 (~1/3 of all interactions) interactions of other types
Target Group 200 Not applicable Not applicable Completed target groups

Low

  • not used in any active campaigns

Very high

  • 120 completed target groups with 3,000,000 target group members
  • 80 non-completed target groups

...


Defining Your Data Retention Policy

The next step towards data retention should be to define your data retention policy based on the as-is analysis. 

This table shows an exemplary data retention policy for the objects based on the previous example. Please note that the retention period is best set to (N*3)+1 month to allow for a better data analysis, for example, 13 months for a year-to-year analysis. In this example, anonymous contacts will be deleted due to low business value and license compliance for overall number of contacts. In addition, the retention period is set to 4 months since the reporting on contacts is based on the last 3 months and most anonymous contacts are older than 3 months as well. 


Object Selection Criteria Retention Period
Interaction Contact Anonymous contacts

4 months

Contacts with no interaction in last 12 months

13 months

Interaction

Click through, or e-mail open, or e-mail outbound, or website visit 7 months
Sales order 25 months
Other interaction types 13 months
Target Group

All target groups which are not assigned to running campaigns

7 months



To define your data retention policy, it is important to consider all objects and their different types stored in the system. For each of these, a clear retention period should be defined. It is crucial that you define your data retention policy driven by compliance, cost, user experience, and performance. Again, we strongly recommend that you involve your legal deputies to check and approve the concepts you want to implement.

Implementing Your Data Retention Policy

Options for Deleting Objects

Now that you have defined your data retention policy, it's important to understand how you can implement it. In general, there are two options to delete data: Deletion through marketing application jobs and deletion through the respective applications. 

Marketing Application Jobs

With the app Application Jobs, you can schedule and monitor application-related jobs. It's possible to either execute application jobs directly (ad hoc) or schedule them in the future periodically. The jobs can reduce your workload by running in the background specific repetitive tasks for you. 

Depending on the application job, there are different selection criteria and parameters. Several marketing application jobs are available for data deletion. The list of all marketing application jobs is available under Marketing Application Jobs. 

Most application jobs for data deletion follow a two-step approach.


Step 1: Flag Object(s) for Deletion

With these application jobs, you can specify objects that are no longer needed (based on selection criteria and parameters). The objects are flagged for deletion and are not longer visible for the business user and can't be used in marketing processes.

Depending on the object type (for example, interaction or contact), there are different selection parameters (for example, the interaction type for interactions or validation status for contacts). One important parameter for contact deletion using the "Flag Contacts for Deletion Based on Target Group" application job, is the live target group. Using the live target group, you can use segmentation to flexibly define the selection criteria for contacts which are derived on the fly. 

In addition, the object-independent parameters are the "retention period" and the "test/productive mode". The retention period is used to select objects older than the specified number of days. By using the test mode, you can get the number of objects that match the selection criteria to validate it first. You can then execute the job again in productive mode. One example is the "Flag Interactions for Deletion" application job.

One example for an application job for flagging contacts for deletion based on target groups is shown below. The example is based on the data retention policy above - deleting contacts with no interactions in the last 12 months after 390 days/13 months. Please note, that the selection of contacts has been done using a live target group through the segmentation.


Image Three: Flag Object(s) for Deletion - Example


When objects are flagged for deletion using the deletion jobs, they no longer display on the frontend anymore. We strongly recommend that you always run the jobs in test mode first to validate the number of objects to be deleted.


Step 2: Delete Flagged Object(s)

With these application jobs you can delete objects which have been flagged for deletion. The data is physically removed from the database. 

The exact same object-specific selection criteria and parameters ("retention period" and "test/productive mode") are available as the corresponding application job for flagging. An example is the "Delete Flagged Interaction" application job. 

You can find an example of a marketing application job for deletion below. This example is also based on the previous exemplary data retention policy. Previously flagged interactions (click through, e-mail opens and website visits) are deleted after 210 days/7 months. 


Image Four: Delete Flagged Object(s) - Example


The following video shows you how you can use the respective application jobs: 

Video 1: Data Lifecycle Management for SAP Marketing Cloud


In addition, there is a small number of marketing application jobs which you can use to delete objects without flagging them as a first step. These jobs do not work in production systems. One example is the "Delete Interactions" application job.


We recommend that you always use the two-step deletion jobs instead of the one-step deletion jobs as a precaution - even in quality systems.

If you have follow-on processes for which deletion information could be relevant, we strongly recommend that you schedule the dates of the deletion jobs far enough apart from the flag for deletion jobs. For example, do not schedule them to run on the same day. In this way, other processes that run between the two dates, can react to the fact that objects have been flagged for deletion.

One example could be a running integration using the marketing contact API (API_MKT_CONTACT_SRV) which returns the "IsMarkedForDeletion" flag. In this example, an integrated system could also trigger a deletion of data related to the flagged contact.


Applications

For an ad hoc deletion (of a typically limited number) of objects you can also leverage the in-app deletion mechanism. This ad hoc deletion is typically not part of the automated data retention process. 

In-app deletion is available for a number of business objects, one example is the segmentation models app: 


Image Five: Deletion in "Segmentation Models" Application


General Considerations

There are a couple of general considerations and recommended practices around data retention and data deletion.


It is crucial that you double check, if deletion may affect one of your processes. Typical processes could be based on campaigns, segmentation models, reporting views, scores, recommendation models but also custom apps or processes in systems which rely on integrated data.


SAP Marketing Cloud prevents you from deleting dependent data which is still used in the application (for example, deleting target groups which are still used in active campaigns). However, custom extensions and dependencies from custom business objects are not included in this dependency check for usage. Therefore, you should also double-check that you are deleting the right data and in the right sequence. 


Confirm that you delete data in the right sequence, for example, contacts after the related interactions have been deleted or target groups after related campaigns are stopped. SAP Marketing Cloud checks for known dependencies, but you should double check especially data in custom business objects.


One example could be a custom app which uses interaction data through the interaction API. SAP Marketing Cloud will not prevent you from deleting interactions which are only consumed by the custom app and not by any standard processes. 


Conclusion

Now, you know why you should care about data retention and how you can define and technically implement your data retention policy for SAP Marketing Cloud. This article also contained multiple recommended practices related to data retention and data deletion. 

If you're interested in learning more about approaches and strategies to define and implement your data retention policy, we are happy to help you with our Technical Design Guidance service.