Machine learning: A new tool for better forecasting

Demand forecasting is difficult, and most demand forecasting conducted today produces disappointing results and significant forecast errors. It cannot easily identify trends in the demand data, and its limited ability to understand the underlying causes of demand variability makes that variability seem worse than it would if demand drivers were clearly understood. And because it is manually intensive, it suffers from persistent bias and poor planner productivity.

"Supply Chain Shaman" Lora Cecere puts it bluntly. In her excellent book, Bricks Matter, she writes, "Within an organization, the words 'Demand Planning' stir emotions. Usually, it is not a mild reaction. Instead, it's a series of emotions defined by wild extremes including anger, despair, disillusionment, or hopelessness." She goes on to say that planning teams are dismayed by demand planning's challenges, and further claims that leaders are not optimistic about making improvements to planning processes and technologies.

What makes forecasting demand so challenging? Rather than appearing as a logical series of numbers, in today's business environment demand more often seems like a pattern of partially constrained chaos. Demand is increasingly influenced by multiple internal and external factors that drive it up and down in ways that can't be understood by simply looking at a historical time-series of aggregated demand buckets. Instead, demand should be viewed as being driven by a complex series of indicators that can be nearly impossible to manage with traditional forecasting algorithms.

However, a new technology called machine learning can help companies address demand-forecasting challenges by reliably modeling the numerous causes of demand variation. Machine learning is a computer-based discipline in which algorithms can actually "learn" from the data. Rather than following only explicitly programmed instructions, these algorithms use data to build and constantly refine a model to make predictions. I'll explain in more detail later, but first I'd like to describe several business scenarios where companies have employed machine learning in their demand forecasting. See if any of these scenarios suggest familiar attributes in your own business.

Lots of promotions. Every year, the Italian dairy producer Granarolo S.p.A. runs thousands of consumer promotions, creating forecasting scenarios for 34,000 unique stock-keeping unit (SKU) promotions. And it gets worse: Demand spikes can amount to an extraordinary 30 times baseline sales. (For more about these challenges, see the Granarolo sidebar.)

This is a common predicament. Expenses for advertising and promotions can add up to more than 20 percent of sales for many consumer products companies. Yet according to Michael Kantor, founder and chief executive officer of the Promotion Optimization Institute, only about 1 in 50 brands is able to forecast demand uplift reliably enough to guarantee consumer product availability and to evaluate the economic returns on those promotions. Without improved technology, few companies can forecast effectively in such a promotion-heavy environment. (For an example, see the sidebar about Groupe Danone.)

Lots of new products. The United Kingdom-based electronics distributor Electrocomponents plc is a top-ranked global distributor with 500,000-plus in-stock items. The company introduces 5,000 new products every month and fulfills more than 44,000 same-day orders every day from its operations in 32 countries. A few new products a month is one thing, but predicting demand for such a vast array of new products is more than a demand planner can reasonably be expected to handle. Plus, new products, by definition, are difficult to forecast. Nevertheless, planners can tap into external data to help them predict initial demand and thus decide how much marketing budget to invest in launching a new product.

Lots of "long-tail" demand. Companies whose e-commerce business is growing find themselves having to forecast demand for more slow-moving, "long-tail" items that customers order infrequently and in small quantities. Outliers are naturally hard to predict, making inventory planning notoriously difficult. Even if you can predict the average demand for certain products, you probably can't predict the demand spikes. This makes it nearly impossible to maintain a balance—having enough on hand to satisfy sudden spikes without adding unnecessary inventory and eventually holding "dead stock."

Growing complexity. Planning wasn't so complicated when Granarolo started out in the 1960s as a local collective of milk producers, but gradually complexity intensified as the company grew into a multinational concern comprising eight brands and hundreds of different dairy products, and utilizing various delivery modes. Its basic software was never designed to handle this kind of growth, and what resulted was progressively inaccurate forecasting that needed time-consuming manual activity to fine-tune. Granarolo's situation is typical of modern supply chains, which continue to increase in complexity.

Extreme seasonality. The United States-based heating, ventilation, and air conditioning (HVAC) manufacturer Lennox International Inc.'s forecasting was complicated because of its high number of SKUs (each of which had its own unique demand pattern) and a significant stock of slow-moving parts, and because it is an extremely seasonal business. Further complicating matters was the company's plans to greatly expand its distribution network, as detailed in the Lennox sidebar. There was no way the manufacturer could manage this level of complexity and variability without adopting a highly automated demand planning system.

Just too much data. In all of these companies we find a pattern that is common to most of today's businesses: a proliferation of new data. I'm referring here primarily to market and logistical data that can help companies better predict demand. Having to manage huge volumes of diverse and ever-growing data streams is more than most planners (and planning systems) can handle. Trying to incorporate them into a forecast using spreadsheets or traditional planning tools is frustrating, often futile, and can be extremely costly.

The companies in the scenarios above share an intrinsic level of complexity and scale that makes it almost impossible for planners to generate reliable forecasts. They are no longer simple and predictable businesses, able to forecast based on historic sales volumes—if they ever were! Their planners were overwhelmed.

In many cases we see, people don't start contributing to forecasts until the very end of the process. So, rather than providing input to help generate an accurate forecast in the first place, they're collaborating to adjust the forecast "output." This approach is inefficient. While some late-stage "crowd wisdom" can be useful, it can also introduce bias. A typical example is when a sales organization artificially adjusts a forecast to match revenue targets.

What else do these companies have in common? They all turned to machine learning in order to increase forecast reliability. This decision dramatically slashed inventory costs and at the same time provided better, more efficient service to customers. It also meant that planners no longer had to waste time manually overriding or adjusting forecasts.

Let's examine how machine learning enabled these improvements.

What machine learning is and does
Machine learning systems were designed to handle forecasting models that can incorporate many kinds of data. Rather than following traditional programmed instructions, machine learning systems reduce demand variability by capturing and modeling all the relevant attributes that shape demand while filtering out the "noise," or random and unpredictable demand fluctuations.

As a result, they learn from the data that they process and modify their operations accordingly. For example, a machine learning system that uses Web data to quickly detect successful new products will find and learn which demand indicators—such as Web page hits, specification downloads, and time on site—are most reliable, and then will update its model over time as consumer behavior changes.

Machine learning can interpret the effect of stimuli (such as trade promotions and advertising) and demand indicators (such as social media activity) originating from each distribution channel. As information proliferates, the data concerning these causes and demand indicators become both more accessible and more manageable over time. Machine learning systems therefore can integrate and usefully model these important new data sources, including detailed market data, machine telemetry, and social media feeds, in ways that are simply not possible with legacy planning systems.

What does this mean in practical terms? For one thing, it means companies can take advantage of valuable data signals that are generated closer to the consumer, including data from points of sale and social media channels. This enables companies to understand the impact of demand drivers such as media, promotions, and new product introductions, and to then use that knowledge to significantly improve forecast quality and detail.

Could you benefit from machine learning?
Would machine learning technology be beneficial for your supply chain? One way to know is by finding out whether your old planning system may be causing escalating costs. Here are three potential signs of this problem, and how machine learning can help to address them:

Inflated safety-stock levels. You can't trust your safety-stock levels to deliver the required service levels, so you keep them artificially high. By taking more demand variables into account, machine learning can help companies with a diverse range of SKU profiles, including long-tail items, to set optimal, lower levels they can trust.

Planning team "burnout." Your team is spending too much time manually adjusting and evaluating forecasts, and often is still not able to deliver them accurately enough or on time. This leads to poor productivity and morale. Machine learning takes more demand variables into account and weights each according to its significance, resulting in much more accurate forecasts. This helps planners succeed in their roles and frees up time for them to refine forecasts using their personal insights and business knowledge.

An inefficient sales and operations planning (S&OP) process. Your consensus forecast from the S&OP is unreliable, or the collaboration process behind it is too slow to adapt to the dynamic nature of the market and SKU behavior. Machine learning's high level of automation can improve the quality of the short- and mid-term forecast by picking up key trends from transactional and promotional data and providing actionable insights about those trends, thereby making the S&OP process more efficient and effective in achieving your business objectives.

If any of these situations resonate, it's likely time to take a closer look at machine learning technology. This doesn't have to mean "ripping and replacing" your existing software. Granarolo, for example, implemented machine learning technology alongside its existing systems to boost performance. Companies that implement machine learning often find that it is easy to use, and that its ability to learn from existing data means that it takes relatively less time to implement, deliver benefits, and pay for itself.

In the not-too-distant future, most supply chains will rely on software that uses machine learning technology to analyze much larger, more diverse data sets. For companies that are serious about tackling today's complex forecasting problems, this new technology will prove an invaluable tool.