On the Accuracy of the Predictive Model for Saturnalia early Score: Bordeaux 2018 and 2019

June 29, 2020

Find out more about Saturnalia predictive model and discover how it works, as it manages to compute the scores for the future vintage immediately after the harvest and before critics' tastings.

You can download this analysis as a PDF file by filling the form at the end of the page.

Since Saturnalia inception, we have been asked many questions about our predictive models, and particularly our satellite-based prediction model of wine scores, which come out soon after harvest, and before any critic has the chance to actually taste the wines.

So far, we have been focusing our attention on Bordeaux. This is due to several factors: a rather large number of individual Châteaux consistently producing high quality output; terroir-driven wines; wines that are highly relevant for the market and that are regularly tasted by a large number of wine critics.

We could spend some time explaining the rationale behind our proprietary method (and we have actually done it in many ways, for example click here to read the demonstration page of Saturnalia on the European Space Agency site, or click here to see how our demos for growers and companies actually work), but in the end everyone wonders: does it work or not?

Simple question, which requires a less simple answer.

First of all, how do you measure if a prediction model is working or not? Usually, you compare the predicted and observed values, of course. And that is where we face the first hurdle: wine scores are not an exact science, and there can be a wide variability of scores for the same wine of the same vintage from different wine critics. And this variability can exist even for the same critic when he or she tastes the wines in subsequent tasting sessions.

This is particularly true for En Primeur wines, which are tasted from barrels, after a few months from the harvest. That is why many critics give ratings for these wines in a bracket of 1 or 2 points, to be revised when the wine is put into bottles (Check our comparative analysis to gather more details about score variability for Bordeaux 2019).

This variabilty is a known fact by the trade, and that is why most professionals rely on benchmark critics. For Bordeaux wines Liv-ex, for example, uses the scores of Robert Parker until 2012 and Neal Martin after that. For this analysis, we have done the same.

Secondly, it is interesting to calculate the difference between observed and predicted scores. This calculation gives you a value, but how useful is it if you can’t compare it with something else?

Again, this is a known problem for all predictive models; for example, this happens also for sales or stock prices or currency values forecast. An efficient method to draw conclusions on the accuracy of a more sophisticated prediction model is to use a less sophisticated, or “naïve” model, as a baseline comparison.

We took this approach to verify our accuracy. Our “naïve”, or basic model of prediction, is the average of our benchmark critic’s score for the last 15 vintages for each one of the Châteaux analysed. That is to say, future scores are predicted on the basis of past performances.

The difference between this “naïve” average and the observed score, and the difference between Saturnalia predicted score and the observed score are called MAE (Mean Absolute Error).

You would expect a more sophisticated model to have more accuracy (lower MAE) than a more “naïve” one, and this is exactly what we have found in both the vintages (2018 and 2019) for which we made our predictions available before the publication of the critics' scores.

The relationship between the two MAEs is useful to understand how more accurate is the better model, and it’s called MASE (Mean Absolute Scaled Error)

This is what we found (same wines analysed for both vintages):


In essence, whilst we observe a wider error variation in 2019 than in 2018 (which may also tells us something about the consistency of tasting notes and quality variation within the region for this vintage), in both cases Saturnalia model was between 39 and 79% more accurate than the simpler model based on Châteaux past performances.



In conclusion, while our predictions do not intend to replace scores from wine critics (we do also love a sip!), we can now offer a proved and tested method to provide valuable estimates of harvest value, down to Châteaux level, up to 6 months before the rest of the industry.

If you have any questions or if you want to get in touch with us to learn how to use our data to inform your trading and investing decisions, please subscribe to our newsletter or send us an email by filling the form down below.

Share by Email
Jancis Robinson and Jane Anson’s Bordeaux 2020 assessments compared with Saturnalia predictions of November Following our last article on Bordeaux 2020, we now discuss what Jancis Robinson and Jane Anson stated by comparing our satellite-derived predictions published back in Nov 2020. July 15, 2021
Bordeaux 2020: satellite-derived Saturnalia predictions of November compared with critics’ assessments In November 2020 we derived our quality predictions for Bordeaux 2020. Now that the En Primeur disclosed its wines, we wanted to see if our predictions were aligned with the critics' statements. June 23, 2021
Saturnalia announces a new partnership with Vindome Saturnalia teams up with Vindome, the first mobile dedicated to fine wine trading, letting users secure more transparent and informed deals. April 26, 2021
Saturnalia and JF Tobias team up to unveil the En Primeur 2020 JF Tobias has included Saturnalia unique, satellite-derived insights in its latest report on the upcoming 2020 vintage of Primeurs. April 23, 2021
FREE Barolo 2016 excerpt Get a preview of our Barolo report and download our excerpt for free! March 10, 2021
Saturnalia report for Barolo 2016 is now available Find out all about this magnificent vintage: 2D and 3D vineyards models, quality analyses and comparison across vintages for 38 Barolo MGA. All in one report. Feb. 17, 2021
Cookies disclaimer
I agree

Our site saves small pieces of text information (cookies) on your device in order to deliver better content and for statistical purposes. You can disable the usage of cookies by changing the settings of your browser. By browsing our website without changing the browser settings you grant us permission to store that information on your device.