Overview

Once you've completed the SAP Commerce Platform upgrade, which includes porting your configuration, custom code, and data, test your solution thoroughly before it's deployed into production. This article covers all the considerations you should take into account during this phase of your upgrade project.

Table of Contents

Test Considerations

The goal of this phase is to give you full confidence in all aspects of your solution, before it goes into production. The code should be deployed into your production-like environment (sometimes called pre-prod or UAT) and should be accessible to the business users that will be using the system in production. Although the focus is usually on the regression and acceptance testing, there are other tests that should be run because changes to the SAP Commerce Platform codebase or migrated customizations could have altered the way your solution currently runs in production. Without thorough and realistic testing, it is hard to understand if there are any negative impacts and how large they might be.

We recommend you run all these tests because it reduces the risk of some big unknown issue occurring. However, we understand that it might not be possible because of time and budget. Your project plan should account for as many tests as possible, but worst-case scenario you could run the set of tests you would run for a normal deployment. If a test fails and a new release candidate is created, make sure you run through all of the tests again. This applies for every test you run. As shown in the diagram below, deployment and regression testings should be done in sequence. Whereas, acceptance, security, and performance testing can be done in series after regression testing is completed.

Test Data

As mentioned above, you want to run your tests using the most realistic data and test scenarios possible. Although you most likely aren't able to use real production data (due to PCI compliance), you should try to emulate your test environment with similar data. This means, you should try to replicate the number of orders/carts/promotions/customers you have in production, with the focus on data that has caused issues in the past. This will help you in identify issues that would come up in production.

Another critical mistake that is often made is the realistic data is not able to load recurrently. If you can't start with a similar test data set for each type of the tests, it will make validating each test much harder. One potential option is to create one or more IMPEX files that can be toggled on and run as part of an initialization/system update. This can be incorporated into your build/deployment process using the Ant targets.

Deployment Testing

There is nothing worse than having the code signed off after a successful round of testing and then something in the build fails when it's put into production. Testing the application is critical, but it's also important to test the entire deployment process from beginning to end before you run the tests below. This make sure:

• Your deployment scripts work. The person/team that will be running the deployment in production should be running the full deployment in your lower environment before final testing. This ensures they are comfortable with all steps, regardless of whether they are manual or automated, and  it gives them the opportunity to ask questions if needed.
• Any issues found during testing that are traced back to issues in the deployment (in other words, not code related) can provide the deployment team with the opportunity to fix things before production. 
• You have a baseline timing for how long it will take in production. This can help with scheduling calls and organizing key members of the team.

You should also test any rollback scenarios to ensure they work. If you don't plan on having a rollback, this gives the team a chance to see potential issues that could occur in production and work together to determine how to solve them.

Be aware that the typical upgrade related deployments are rather different from typical development related deployments. There are many additional steps, like orphaned types cleaning, executing manual SQL scripts, importing impex scripts, or executing scripts in SAP Commerce Administration Console (Scripting Console). That is why testing upgrade deployments is important.

Regression Testing

This testing should be completed throughout the lifecycle of the project, but it is essential to run the whole suite of regression tests against the potential candidate for production. Regression tests should be automated as much as possible because they can be long, tedious, and need to be run against each build. This will allow you to cover more of your solution in a consistent and timely manner. If there aren't automated tests, then someone should be executing every unit and integration tests that are coded, as well as run through standard test scenarios for your site (e.g., login, browse, add to cart, checkout with credit card). Once the regression is completed and successful, you can move on to Acceptance testing.

Acceptance Testing

If you're completing a like for like upgrade you should be using the same Acceptance criteria and Acceptance testing you used for your original deployment. This is because none of the acceptance criteria should have changed. If you're completing an extended upgrade make sure you added new test cases and their relevant acceptance criteria, for every new feature you include. This whole set of test cases should be run by the business users who will be using the system. The acceptance criteria should be met before the business QA team signs off on the release candidate that will go to production.

Security Testing

Security issues could be introduced through an unknown bug in the target version of the platform or it could be created by some updates to customizations in your solution that are required to make your solution work on the target version. Your target version should be more secure, due to updates to 3rd party libraries and bug fixes, but without running through security testing, you might be vulnerable when your upgraded code makes it into production. Running through your full security suite of tests on the release candidate should be a critical step before signing off on the deployment to production.

Performance Testing

Without running through your set of performance tests, you won't know whether any of the known metrics you have for your source version will apply to the target version. Running your performance tests on your release candidate is essential to ensuring your site will perform the same, if not better when it reaches production. In the unlikely scenario the performance does degrade, at least the business will be aware of it and can choose to continue (knowing it is a potential risk). Because they know the risk, they can start to put a plan in place to mitigate it (if it become an issue in production). However, if the performance results are significantly below the baseline of your source version, you should re-evaluate the timeline of going to production and focus on fixing the critical performance issues.

Conclusion

This article introduced you the importance of testing your new SAP Commerce version before deploying it on production. This step will help you avoid issue on production and maintain your acceptance criteria. Once you've completed all the activities you should move on to Deploy an Upgrade of SAP Commerce.