Webhook Do's and Don'ts - Building Scalable Webhook Integrations
SAP Customer Data Cloud's Extensibility suite allows you to quickly and flexibly customize your authentication and consent flows to suit your business needs. Webhooks is one of the most powerful extensibility tools in the suite, offering a way to extend CIAM flows by sending a notification to your server when a specific event has occurred. With Webhooks, you can implement your own code, and set it to trigger when the relevant hook is fired. To assist in navigating through some of the complexities of a Webhooks architecture, this article aims to share best practices from our SAP CDC Experts.
Table of Contents
Endpoint timeouts during high traffic events
One of the most important things to consider before you proceed with setting up Webhooks within SAP Customer Data Cloud is your capacity to both provision an endpoint and produce a Message Queue application that can listen for notifications on that endpoint.
A typical pitfall of Webhooks implementations is when the notification endpoint attempts to process each of the events, and only once it has finished, returns a 2XX success code. The HTTP POST notification request may include up to 100 multiple events in a single notification, and waiting too long to respond may cause the notification to be resent unnecessarily.
SAP Customer Data Cloud will wait 10 seconds before considering the notification unsuccessful. Next, the notification will be resend at increasing intervals up to one hour. It is here where the implementation pitfall becomes critical, as the endpoint is unable to receive, process, and respond such a high volume of events (live events + retries). This common scalability problem is illustrated in the diagram below.
Figure 1. Common pitfalls of implementing Webhooks
Events Queue DB
To avoid the issues described earlier, the recommended approach is that once a notification is received, the endpoint returns a 2XX Success immediately, before continuing to process it. A Message Queue system should be the application used to log and then asynchronously process the events. Further downstream functionality can then be executed; this is illustrated in the diagram below.
Figure 2. Webhooks Recommended Architecture
- Once an event is triggered in CDC, the webhook will trigger the configured endpoint.
- The endpoint will receive the payload and will validate the request:
- by retrieving the JWT sign from the x-gigya-sig-jwt header field and verifying via JWT validation process (Webhooks 2.0)
- by running a check of the X-Gigya-HMAC-SHA1 header field to verify the authenticity to the request.
- The endpoint acts accordingly to the validation result:
- If it's a valid request, the endpoint will respond with 200 status to notify CDC that the request was correctly received.
- If it's not a valid request, the endpoint will respond with a different status (500, 400, etc..).
- The Endpoint will then enqueue the request in the message queue.
- Once a request is available for processing, the Message Queue serve the event to the logic layer.
- The logic layer can then process the payload (according to the client requirements), potentially making additional calls to CDC to retrieve additional data, and propagating changes to downstream systems.
- Once the processing is complete, the event is removed from the Message Queue.
SAP Customer Data Cloud will often send the same event, which can be identified by the id field.
Therefore it's good practice to filter duplicate request by the id field.
Verify the authenticity of the request
Security is of paramount important for SAP Customer Data Cloud. To ensure that a given notification originated from SAP Customer Data Cloud and that none of its data has been tampered with in transit, notifications are always sent to the endpoint with a hash of the payload/body. It is up to the endpoint to ensure the authenticity of the notification and to process the received payload without compromising security.
There are different versions of Webhooks. Newer versions of Webhooks contain more information in the payload and notification can be validated with a JWT.
To verify the hash of Version 1.0:
- Create an HMAC-SHA1 hash of the JSON payload using a User Secret as the key.
- Compare your hash to the one received in the X-Gigya-HMAC-SHA1 header. If the hashes match, the notification is considered valid.
For further information on signing notifications using a Secret Key, please see the accounts.webhooks.set documentation.
To verify the hash of Version 2.0:
In Webhooks 2.0 the notification will be signed with a JWT and it can be verified with the accounts.getJWTPublicKey endpoint without having to exchange secrets with the Webhook recipient.
- Retrieving the JWT from the HTTP POST x-gigya-sig-jwt header
- Extract the hash from the subject claim ("sub") of the JWT. This is the hash of the event body
- Retrieve the public key by calling the accounts.getJWTPublicKey endpoint
- Verify the hash by applying SHA256 on the received Webhook notification body (using the retrieved public key) and comparing it to JWT subject
For further information on JWT verification process, please see Validate A JWT from SAP Customer Data Cloud.
Webhook notifications are sent in near real-time, in an asynchronous way. SAP Customer Data Cloud will commit the database operation and send a response to the client application without waiting for the Webhook endpoint to finalise processing.
Another typical pitfall when implementing Webhooks for SAP Customer Data Cloud is chaining operations that could cause a race condition (software flaw).
A typical example of a race condition may occur when a Webhook needs to add an external ID to the account upon creation, and the client application encounters an account without an external ID. The account may, for a brief period, be in an unwanted state before settling back to the designed state. In this case, the system expected the Webhook to complete the update operation in the backend before the user was logged in for the first time.
It is crucial to recognize and eliminate race conditions in a Webhook integration before they cause problems. Some examples where race conditions can occur include:
- A Webhook updating an account in SAP Customer Data Cloud: Client applications must be designed to tolerate such glitches, as the account might hold null, outdated, or unexpected values for a short period of time, while the asynchronous update operation is completed in the backend.
- A Webhook searching an account by UID in SAP Customer Data Cloud: Consider, for example, a Webhook that search in the Audit Log for the operation that triggered the event. This search might return inconsistent results, as short delays are possible between the writing of data and its availability in queries like audit.search, ds.search, or accounts.search.
When possible, retrieve account information within a Webhook using the accounts.getAccountInfo endpoint.
Choosing the right integration tool for your needs
To help support ever-changing business objectives, SAP Customer Data Cloud gives you a wide variety of customization solutions to achieve your desired outcomes and limit the cost and time to market of initial deployments and future changes. Although Webhooks provide a way of extending and customizing authentication, registration and consent flows, there are certain scenarios where its application is not ideal.
Implementing Webhooks "the right way" requires a fair amount of elements:
- Setting up a server-side endpoint to receive the notifications
- Implement a Message Queue system to enqueue the events
- Perform an additional call to SAP Customer Data Cloud to retrieve account's information, as some events will only include a UID in the event data, but no profile/data fields
Due to its asynchronous characteristic, Webhooks are ideal for server-to-server integration in high traffic solutions. However, it might not be the best tool for other use-cases like validations or synchronous data enrichment. For instance, a Webhook cannot stop a user from creating an account, as the operation has already been committed.
Based on the above, we recommend evaluating all the different extensibility tools available in SAP Customer Data Cloud and its capabilities, in order to choose the right technology for your integration requirements.
|Server / Client||Asynchronous, server-to-server||
|What are the triggers?||After a specific event occurs||Within the SAP Customer Data Cloud flow||Client-side screen-set triggers|
|Event type||Gigya webhooks||Extension endpoints||Screen-set events|
|Useful for||Downstream flows and application||Real-time decision making||Real-time response to user’s input|
This article has presented expert guidance for implementing Webhooks with SAP Customer Data Cloud, taking in consideration best practices from our professional services teams and previous implementations. We've given you a detailed description of the common pitfalls during implementation, such as scalability issues due to lack of a Message Queue system. We have covered some important Webhooks security considerations as well as a capabilities comparison against some other extensibility tools.
To access all our community or out of the box product documentation, please check out our List of Online Resources.