On the Accuracy of the Predictive Model for Saturnalia early Score: Bordeaux 2018 and 2019

You can download this analysis as a PDF file by filling the form at the end of the page.

Since Saturnalia inception, we have been asked many questions about our predictive models, and particularly our satellite-based prediction model of wine scores, which come out soon after harvest, and before any critic has the chance to actually taste the wines.

So far, we have been focusing our attention on Bordeaux. This is due to several factors: a rather large number of individual Châteaux consistently producing high quality output; terroir-driven wines; wines that are highly relevant for the market and that are regularly tasted by a large number of wine critics.

We could spend some time explaining the rationale behind our proprietary method (and we have actually done it in many ways, for example click here to read the demonstration page of Saturnalia on the European Space Agency site, or click here to see how our demos for growers and companies actually work), but in the end everyone wonders: does it work or not?

Simple question, which requires a less simple answer.

First of all, how do you measure if a prediction model is working or not? Usually, you compare the predicted and observed values, of course. And that is where we face the first hurdle: wine scores are not an exact science, and there can be a wide variability of scores for the same wine of the same vintage from different wine critics. And this variability can exist even for the same critic when he or she tastes the wines in subsequent tasting sessions.

This is particularly true for En Primeur wines, which are tasted from barrels, after a few months from the harvest. That is why many critics give ratings for these wines in a bracket of 1 or 2 points, to be revised when the wine is put into bottles (Check our comparative analysis to gather more details about score variability for Bordeaux 2019).

This variabilty is a known fact by the trade, and that is why most professionals rely on benchmark critics. For Bordeaux wines Liv-ex, for example, uses the scores of Robert Parker until 2012 and Neal Martin after that. For this analysis, we have done the same.

Secondly, it is interesting to calculate the difference between observed and predicted scores. This calculation gives you a value, but how useful is it if you can’t compare it with something else?

Again, this is a known problem for all predictive models; for example, this happens also for sales or stock prices or currency values forecast. An efficient method to draw conclusions on the accuracy of a more sophisticated prediction model is to use a less sophisticated, or “naïve” model, as a baseline comparison.

We took this approach to verify our accuracy. Our “naïve”, or basic model of prediction, is the average of our benchmark critic’s score for the last 15 vintages for each one of the Châteaux analysed. That is to say, future scores are predicted on the basis of past performances.

The difference between this “naïve” average and the observed score, and the difference between Saturnalia predicted score and the observed score are called MAE (Mean Absolute Error).

You would expect a more sophisticated model to have more accuracy (lower MAE) than a more “naïve” one, and this is exactly what we have found in both the vintages (2018 and 2019) for which we made our predictions available before the publication of the critics' scores.

The relationship between the two MAEs is useful to understand how more accurate is the better model, and it’s called MASE (Mean Absolute Scaled Error)

This is what we found (same wines analysed for both vintages):

In essence, whilst we observe a wider error variation in 2019 than in 2018 (which may also tells us something about the consistency of tasting notes and quality variation within the region for this vintage), in both cases Saturnalia model was between 39 and 79% more accurate than the simpler model based on Châteaux past performances.

In conclusion, while our predictions do not intend to replace scores from wine critics (we do also love a sip!), we can now offer a proved and tested method to provide valuable estimates of harvest value, down to Châteaux level, up to 6 months before the rest of the industry.

If you have any questions or if you want to get in touch with us to learn how to use our data to inform your trading and investing decisions, please subscribe to our newsletter or send us an email by filling the form down below.