When Good Data Lies: The Hidden Risks of Data-Driven Pricing

When Good Data Leads to Bad Decisions

What if your data is accurate, your models are statistically sound, and your pricing recommendations are still wrong?

For years, the progress of data-driven pricing has followed a seemingly obvious principle: collect more transactions, add more explanatory variables, improve predictive accuracy, and optimize more efficiently. Retailers now combine scanner data, loyalty transactions, digital journeys, competitor prices, promotion histories, inventory records, and external signals. Machine learning can recover complex relationships from this material; optimization can turn estimated demand into a seemingly exact price. All of this represents genuine progress.

Yet the sophistication of the calculation makes a particular mistake easier to commit: confusing precision with validity. The output may be expressed to the nearest cent, accompanied by an elasticity coefficient, a forecast accuracy score, and a confidence interval. None of those features, on its own, proves that the recommendation would improve commercial outcomes if implemented.

There are three distinct questions. What did customers do? This is a descriptive problem. What are they likely to do under the conditions represented in our data? This is a predictive problem. What would they do if we deliberately changed our price or commercial policy? This is a causal and decision problem. The third cannot always be answered by improving the first two. The difference is recognized in causal econometrics, statistical learning, and research on pricing with demand uncertainty; it is not a new concern introduced by generative AI (Villas-Boas and Winer, 1999; den Boer, 2015).

Three examples illustrate where confidence goes wrong. An e-commerce model may describe a market regime that no longer exists. A loyalty-program segmentation may rediscover purchasing behavior created by the retailer’s own discounts. A demand model may predict very accurately given actual weather, while the future weather required to use that model remains unknown. The same underlying limitation runs through all three: historical data is evidence about the process that generated it, not automatic proof about every decision we might make next.

1. When Historical Data Describes a World That No Longer Exists

The e-commerce market before, during, and after COVID-19

Consider online retail before, during, and after the pandemic. Before 2020, retailers had years of observations from which to estimate seasonality, channel preferences, promotional effects, and correlations between prices and demand. Then lockdowns, changes in shopping access, supply disruptions, and shifts in household priorities altered the environment abruptly. Many consumers bought online more often; some discovered new brands, channels, and fulfillment arrangements. Later, as restrictions were lifted, behavior changed again. Some habits persisted and others did not. Research on pandemic-era consumption documents the disruption and possible persistence of new shopping habits (Sheth, 2020).

At each stage, firms collected more data. But accumulating observations was not necessarily the same as learning more about the market in which the next price decision would be made. A pre-pandemic relationship might fail during lockdowns; a model retrained on lockdown demand might then perform poorly after reopening. The historic observations were not fabricated. They described genuine transactions under genuinely different conditions.

The fundamental issue: distribution shift and structural breaks

A forecasting model trained in one environment is often used as if it described another. Formally, the conditional relationship between purchases and their drivers may change over time:

PrtQ∣P,X  Prt+1Q∣P,X.

Here Q represents demand, P the price, and X other observed drivers. This is an illustrative representation of a changing conditional distribution, not a claim that every pandemic shock necessarily changes the structural price effect. Sometimes only the distribution of customers or product availability changes; sometimes the conditional demand relationship changes as well. The diagnostic task is to distinguish these possibilities, rather than treat all changes as one generic failure of machine learning.

Research on concept drift studies learning under changing relationships (Gama et al., 2014). Econometric methods for structural-break detection address changes in model parameters and the timing of regime shifts (Bai and Perron, 2003). The idea of invariance across environments provides a further theoretical connection: relationships that are genuinely causal may be more stable under certain interventions than associations based on changing background conditions, provided the assumptions of the causal model hold (Peters, Bühlmann, and Meinshausen, 2016). Invariance is a testable research direction, not a guarantee that a historical elasticity will transfer unchanged to every new market.

Why more observations may not solve the problem

Suppose a retailer estimates low price sensitivity from transactions recorded during a period of exceptional demand and limited availability. Consumers may have faced few substitutes, urgent needs, and shipping constraints. An elasticity estimate from that period may be internally consistent and still be a poor guide to a later, more competitive market. Adding a large amount of similar pandemic-period data may make the historical coefficient appear more stable without making it more transportable to the new conditions.

There is another complication. Observed sales are not always unconstrained demand. In a simple inventory-limited setting, sales are approximately the smaller of customer demand and available inventory. If both a high-price and a low-price scenario exhaust the same limited stock, observed sales can look similar even though underlying demand differs. This is a conceptual illustration; actual retail data also require attention to replenishment timing, partial availability, substitutions, and the fact that inventories can themselves respond to price choices.

The solution is not to discard old data automatically. Historical observations across several environments can be extremely valuable for understanding how behavior varies with context. Instead, pricing teams should test performance by regime, store format, product group, and time; compare rolling-origin forecasts rather than one random holdout; inspect changes in price support; and record whether the candidate price is being evaluated in a market similar to the one that generated the training data.

Read the full research article — register for free access