Starting in October 2023, the SAP SuccessFactors Future of Work Research Lab scientists did something few had attempted at the time: we measured the business impact of AI. We partnered with six customers and surveyed their employees before and after they used SAP SuccessFactors AI use cases in their daily work. We embarked on this research program because AI’s potential was receiving so much hype, but there was little data on what AI was actually delivering to end-users.

This blog covers 1) highlights from the program 2) lessons learned and 3) calls to action for those looking to do this work. We hope to empower others to apply a scientific lens to questions around AI’s ROI and in doing so, keep the human implications of AI transformation at the forefront.

1) Key research findings from the program

Joule saves serious time: Joule was our most-studied use case. We saw impressive results in terms of efficiency gains with employees completing self-service tasks 86% faster. Employees reported feeling optimistic, excited, and satisfied after using Joule.  The top benefit for Joule was helping make boring and repetitive tasks easier. We also found an interesting ripple effect of using Joule on impressions of one’s employer: 57% of employees reported that using Joule made them feel their company wants to make their work life easier.

AI-assisted goals are still motivating:

The SAP SuccessFactors Performance and Goals, development goal creation and the SAP SuccessFactors Performance and Goals, performance goal creation use cases were of particular interest to our research team due to our expertise in organizational psychology, which has a storied history of goal-setting research. We wondered whether having AI assistance in goal setting would undermine identification with the goal and motivation to work towards it. Interestingly, we found that goals written with AI were just as useful, valuable, and motivating as manually written ones and enjoyment of the goal setting process was higher with AI assistance (an average score of 3.88/5 enjoyment with AI assistance compared to 2.50/5 without AI assistance). However, AI-assisted goals did score lower on alignment to company objectives, indicating that capturing full organizational context is an area for further improvement.

2) Advice for those looking to do this research

Looking back at these research partnerships, some key lessons learned come to mind:

Start with your winning use cases and broaden as needed. Targeting use cases that employees are excited about and eager to use will improve your chances of getting a meaningful sample size. This approach can also build momentum for use case adoption.

Time your survey launches thoughtfully. Triple check that everything is working smoothing with both the AI use cases in question and with your tech stack more broadly. Even subtle glitches in user experience will skew your results.

Consider the feasibility of your design. We ran two surveys per customer, one before the AI use cases were live and one after. This created added complexity, including precise pre- and post-survey launch timing, finite data collection windows, and decreased participant motivation to complete the second survey. Weigh your options when you design your surveys – it would be better to run one successful post-launch survey than several that have low uptake. You can also consider other designs like diary studies or AB testing. If you are intending to pursue more in-depth, qualitative data collection doing so with a pilot group might be best to ensure participant commitment.

3) Call to action

Our team’s experience doing this research firsthand left us with renewed resolve about the importance of this work. We urge internal and external practitioners to take up the mantle of doing this research. Otherwise, you run the risk of relying on assumptions, unable to track what impact AI tools are actually having the workforce.

A final, critical call to action for those intending to do this work is the importance of bringing nuance to these conversations and designs. In times of great uncertainty and change, stakeholders may look for a simple narrative, a key statistic, or a cut and dry ROI. However, behavioral science research is rarely that straightforward – human interactions with AI are multifaceted, different employee segments may react differently, and these dynamics may rapidly change over time. For example, an AI use case could both dramatically save time but also undermine employee job satisfaction. When considering what you measure in AI business impact research, advocate for measuring more than just efficiency – constructs like enjoyment, usefulness, and emotions are a good place to start. What you measure is a reflection of what your company values and signals those values directly to the employees who are responding to your surveys.

For more insights from our research study, please check out our PowerPoint deck and infographic.