Saturnalia Price Variation Predictive Model Accuracy: Bordeaux 2016 after 4 years

You can download this analysis as a PDF file by filling the form at the end of the page.

Our Predictive Model is based on the assumption that price variation from release price (RP) of Bordeaux top wines is influenced, among other things, by the relative quality of recent vintages and the relationship between RP and quality, as described by critics’ scores.

Whilst these aren’t the only variables involved – think about external market forces, such as the current Covid pandemic –, our algorithms can explain a significant amount of the variance.

We set our prediction time in the medium term: 3 years after release. This means that the first set of data available to test our model accuracy is the 2016 vintage, which was released in 2017 and assessed in 2020, at the end of the En Primeur campaign (which this year took place about 3 months later than usual due to the Coronavirus pandemic).

The question we try to answer here is obvious: does the model work or not?

The question is very straightforward, but the answer is not so simple.

The first thing that comes to mind when trying to assess whether a prediction model is correct or not is to draw a comparison between predicted and observed values and, successively, calculate the difference between these two measures.

However, when a predictive model is built, a common problem arises: you need a base to which you can compare the results. To eliminate this issue and to evaluate the accuracy of a predictive model, a solution is to use a less complex, or “naïve model”, that works as a sort of guideline.

We took the same approach to verify our accuracy. Our “naïve”, or basic model of prediction is the average variation from Release Price (RP) after 3 years from release, for the last 11 vintages since 2005. That’s to say, future prices are predicted on the basis of past performances.

The difference between predicted prices of the “naïve model” and the observed prices, and the difference between Saturnalia model predicted prices and the observed prices are called MAE (Mean Absolute Error).

You would expect a more sophisticated model to have more accuracy (lower MEA) than a more “naïve” one, which is exactly what we have found for the 2016 Bordeaux prices.

The relationship between the two MAEs is useful to understand how more accurate the better model is, and it’s called MASE (Mean Absolute Scaled Error)

In essence, Saturnalia model was 49% more accurate than the simpler model based on Chateaux past performances.

65% of the sample (91 Chateaux) had a prediction error (difference between predicted and observed) between 0-12%, and 90% of the sample between 0-24%.

When we look at the type of variation, we observe that Saturnalia Prediction Model tends to take a more “conservative” approach, meaning that the error is skewed towards lower values, when compared to the rather higher observed gains.

Observing the difference in the direction of price variation, i.e. when Saturnalia model predicted a gain or a loss in price of a wine whilst the actual values showed the opposite sign, only 15% cases were different in sign, whilst 85% had the same direction of change.

In conclusion, whilst our predictions can only explain the amount of variation in price that is due from internal factors (wine quality, release price, etc.), and it doesn’t account for external factors such as the general state of the economy, Brexit, Covid, tariffs, etc, we believe that we have been able to show that it can be a useful tool to inform purchase decisions on top Bordeaux wines that have been released in the market in the last 3 or 4 years.

If you have any questions or if you want to get in touch with us to learn how to use our data to inform your trading and investing decisions, please subscribe to our newsletter or send us an email by filling the form down below