Developing the Business Case for AI solution
AI solutions have a huge potential for businesses to leverage data and gain a competitive advantage. However, one common inhibiting factor for business investment is the uncertainty about the benefits case for AI. This article provides practical guidance on how to structure the benefits measurement for AI use cases and translate technical metrics to financial KPIs.
Introduction
The most common request that I have received from business teams when implementing innovative technology solutions is the benefits case i.e. top-line growth or bottom-line cost reduction?
Historically, technology solutions like SCM, CRM, MES, or even ERP have always had measurable functional KPIs. These are easy to define and measure; for example
CRM / Sales KPIs — conversion rates, sales cycle length, customer lifetime value, customer acquisition cost etc
Supply chain KPIs — Ontime in full, Order fulfilment lead time, forecasting accuracy, inventory days etc
Operations KPIs — Cost of production, throughput, overall equipment efficiency, inventory buffer, quality loss, etc
Finance — Cost of transaction processing, analysis and control (risk, variance, contribution…), AP/AR days, Days to close etc
Many times, benefit projection is done at the beginning of the project. Few revisit the actual number after a long 1 to 3 year implementation period.
As my old school consultant used to say — “you need to adopt best practice and be in the upper quartile for industry benchmark KPIs”.
On the contrary, AI solutions present an immense opportunity to leverage a vast amount of data, apply machine learning, deep learning technologies to gain a competitive advantage in a short period of time. However, the complexity of AI solutions and their use for decision making, present difficulty in translating technical KPIs like model accuracy into financial KPIs.
Generally, the benefit levers applied for AI solutions are improved productivity of people, efficiency in process execution, and reduced costs.
In order to develop the benefits case, we need to understand the intent of AI solutions. Based on my industry experience, I will discuss the top 3 intents for AI solutions, and present an approach to measure the linkage, relationship between the AI technical KPI to the financial KPIs.
Intent 1 — Analyse a vast amount of data for business insights and decision making
Typically use cases in this area include market intelligence solutions, knowledge advisors, and more. These use cases leverage technologies in the area of Big Data, Automation, Search, and Reporting.
Technical KPIs for this purpose include
Volume, Variety, Velocity and Veracity of data
Data processing time (Time needed for data to be available for Query, Collection, Analysis, Reporting)
Accuracy of data (consistency, completeness, or accuracy)
Dashboard/report responsiveness (time to load, drill down, change or update)
Search / ranking metrics in terms of Mean reciprocal rank, Precision at k, Normalized Discounted Cumulative Gain
Since the use case enables business users in decision making, quantifying the benefits on productivity metrics like reduction in hours of work is not enough to provide justification for investment. It is important to broaden the business case to the eventual benefits that the organization will derive from the insights.
Case Study: AI for market insights in Trading
In this AI solution, the commodity movement unstructured data was captured from external sources. This data helped the analysts and traders to take decisions on what to trade, with whom to trade, and how to price the trade.
Initially, the team measured solution benefits for increase in productivity of analysts i.e. reduction in hours of work per FTE. Although this KPI is important, it does not justify the million dollar investment. As the team started to build the business case further, they expanded to the eventual benefit from the system. The team quantified the benfits in terms of delta increase in revenue from baseline based on the AI recommendation for more profitable trades. This helped to justify the benefits of the solution which was manyfold of the investment required.
It was encouraging for the team to have an honest business owner who felt better enabled by the market insights delivered by the AI solution and supported the final benefit case.
Another, powerful way to measure AI solution benefits and to show value to the business is Retrospect, i.e. compare the AI recommendation to the human decision and calculate the % improvement if the AI decision was executed. It is important in this case to document the baseline properly and the formulas for calculating benefits over baseline.
Sometimes it may be beneficial to let the business use the AI solution for two or three months and then ask them the question — “If the AI solution was taken away, how much would you pay to bring it back?”
Intent 2 — Predict the future
Typically, use cases in this area include demand forecasting, promotion effectiveness, yield prediction, predictive maintenance, and more. These use cases leverage technologies in the area of advanced analytics and machine learning.
Technical KPIs for this purpose include
Regression metrics (Mean square error, Mean absolute error and more)
Classification Metrics (confusion matrix, F1-score, Area under the curve, Receiver operating characteristic and more)
Statistical Metrics (Correlation, R square)
In order for the technical KPIs to be useful to business, they need to be translated and linked to business KPIs. Some examples of business KPIs are as follows
Increase in sales (Right Product, price, place, and promotion)
Better resource allocation (inventory, budgets and more)
Reduction in maintenance cost, reduction in downtime
Identifying the right problem to solve using an AI solution is the most difficult task. Today, businesses are working with many “Commercially off the shelf applications” (COTS) and have developed business rules with decades of experience. There is an ecosystem of partners in terms of marketing agencies, trading partners, OEMs, insurance agencies, and other outsourcing vendors. Getting benefit from an AI solution requires re-designing the processes across the existing applications, ecosystem partners, and people. In the case study below, we see why understanding the cost drivers and their relative contribution is important before embarking on an AI solution.
Case Study: Predictive maintenance for Haul Trucks in Mining
In this AI solution, the objective was to build a predictive maintenance model for the Haul trucks using the data captured in from onboard vehicle logs. The solution relied on the Remaining useful life (RUL / Cox regression) technique and provided a health score for the trucks to prioritize maintenance. The benefit was based on reduction of downtime and maintenance costs.
However, as the business started using the solution, predicting health score was seen as less of a priority, since the fuel consumption increased as the truck extended time for operations. Hence the team pivoted to energy consumption and identified the predictors for fuel burn rate in terms of road conditions, speed, elevation gain, weather, idle time and more.
Eventually, the AI solution saved thousands of dollars in running costs per truck per year, however success of the solution was the result of applying AI for the right problem where even few percentage point reduction in fuel burn saved significant dollars.
Lastly, sometimes due to compliance, regulatory requirements a business case may not be required, like in Safety use cases, however, it is always beneficial for the project sponsor to define KPIs for measuring success.
Intent 3 — Enrich customer experiences
Typical use cases in this area include Chatbots, recommendation engines, search tools, visual analytics, and more. These use cases leverage technologies in the area of Natural language processing, Deep learning, and Computer vision.
Technical KPIs for this purpose include
Natural language process metrics (F1 score, Perplexity, Word error rate, Bilingual Evaluation Understudy Score)
Deep Learning/computer vision metrics (defined by a loss function, area of the union, Inception score, and more)
Often business KPIs in this area are based on cost reduction i.e productivity improvements. However when these solutions go-live, they are always questioned on their ability to improve the consumer experience.
Case Study: Marketing pre-sales chatbot
In this AI solution, the objective was to build a marketing pre-sales chatbot on the website that helped potential customers to understand product features and recommend them the most suited product for their needs. The idea was to provide a personalized experience similar to a sales advisor in-store.
Initially, the team built a huge repository of product data, intents and ingested it into the chatbot to cover all possible questions. Then they tested the chatbot with the view to see what can I ask to make it go wrong.
Changing their mindset, the team looked as to how they could provide the most relevant information to the customer and redirect unanswered queries to other communication channels. Eventually the team stabilized the chatbot with a limited set of data and improved overall customer experience. Some of the KPIs that the team developed to measure customer experience were:
*Total no. of unique users, Active users, Engaged users, New vs Returning Users
*Average session time, #?? of interactions per User, No. of users who leave a message
*Response time, No. of clicks to links
*Content — Most popular intent or Which intent has the most exits
*Confusion Rate = Number of times the chatbot had to fall back / Total Messages Sent
*Bot satisfaction rate
These types of AI-based chatbots have had their highs and lows. The ability to have a chatbot that can answer as effectively as an experienced customer service agent has sometimes not been met. Successful implementation have three things in common — definitive functional scope, exhaustive training, and the right set of KPIs.
One of the tools, that I have seen teams use effectively is the Opportunity Canvas. This tool is used to synthesize the value proposition, document benefits, and investment metrics.
Conclusion
To conclude, defining the business case for AI solutions is important in terms of business metrics of top-line growth or bottom-line cost reduction.
Investing in multiple AI pilots in the hope to see which one will provide maximum ROI may not be the best strategy for the organization.
My recommendation for successful development and execution of AI solutions with a solid business case is:
1. Leadership support for developing a long term AI roadmap and strategy
2. Ability to redefine and re-engineer existing processes to enable AI solutions to scale
3. Committed investment to develop and continually enhance AI solutions with the view that each iteration of the solution provides incremental value
As Clayton Christensen rightly said ‘AI solutions are a form of “disruptive innovation” and we should first invest in the right AI strategy and then develop roadmap for the AI solutions’.
Wish you all the best for your AI solutions journey
