Friday, 3 July 2020

HadEX3 released

Yesterday (July 2nd 2020) the paper describing HadEX3 appeared as an Accepted Article in JGR-A.  We have also made the underlying gridded dataset and plots available on the hadobs website.  In due course the data files will also be available at, once some final reformatting has been completed, and also in the CEDA Archive.

Friday, 1 May 2020

Almost there

I've been very quiet with the updates since last summer.  Partially this is just forgetting to update the blog, but we have been making progress on the HadEX3 dataset.  

During the latter part of 2019, we played around with options for the gridded dataset.  I fine-tuned the station selection and some of the quality control scripts to ensure we have a good balance between the widest spatial coverage and the best quality of stations.  As a result, for the final grids, stations which end before 2009 are not included.  A number of sources end before then, and their absence from the network in the final decade causes apparent inhomogeneities in some regions.  The stations will still be made available where possible.

The results of these are that the released version of HadEX3 will be at a finer resolution than HadEX2, using 1.875x1.25 degrees (so x4).  The "global" trends for the land surface don't change much at all, but as our station network is denser in many regions, we can present greater spatial detail.

One thing that we have been able to do is a more detailed comparison between the trends and spatial patterns of the indices which use a reference period (e.g. TX90p, CSDI, R95p) as input data for these cannot be combined if they use different periods.  This limitation does affect the spatial coverage at the moment, but we are asking contributors to calculate indices with the other reference period where possible to make this comparison as complete and robust as we can.  This will allow any future collections of these indices to understand how different reference periods affect the absolute values and hence also long term trends.

The final bits of data came in during November in time for us to submit the paper before the end of 2019 to JGR-A. The reviews which came back earlier in 2020 were positive, but it has taken a little longer to deal with them with the changes that everyone has undergone as a result of COVID-19.  However, I'm pleased to say that we have submitted the revised manuscript back this week.  Once the peer review process has completed, I'll make the dataset publicly available.

Wednesday, 31 July 2019

Quick update

Just a very quick update on this.  We have now received all the data we were expecting to receive - and so can start adjusting some of the processing criteria. 

However, to give an indication of the number of stations and rough coverage for trend analysis, I thought I'd put the latest plots here.  Once the processing criteria are settled on my intention is to put a draft document together.
Fig 1 - Stations in the Rx1day grids
Fig 2 - Stations in the TXx grids
As you can see, we have stations now covering the majority of the globe.  However, we want to ensure that the hard work from colleagues who have submitted data is recognised, and include as many stations as makes sense.  Currently, we're requiring that stations report relatively continuously during the period when they do - so can only have a maximum of 2 missing years, and I think that this is reducing the final numbers from over Brazil, India and elsewhere. 
Fig 3 - Rx1day trends (only from grid boxes with 66% completeness)
Fig 4 -TXx trends (only from grid boxes with 66% completeness)
And from these trend plots, the coverage is much reduced.  This arises from knock on effects, fewer stations being currently selected because of the completeness criterion.  Those that do may result in a too sparse network to ensure grids are calculated (which is more likely for indices which have a short spatial correlation).  And when they are, they may not be for sufficient years to calculate the trends. 

Fig 5 - Timeseries of Rx1day, compared to GHCNDEX and HadEX2

Fig 6 - Timeseries of TXx, compared to GHCNDEX and HadEX2
What is interesting is that, despite the limitations of the spatial coverage, the long term trends in quasi-global averages match very well with the other ETCCDI datasets available (GHCNDEX and HadEX2).

We're seeing what can be done to improve the spatial coverage without impacting the integrity of the dataset, so watch this space.

Wednesday, 29 May 2019

Filling in the Globe

Just a quick update this time.  Work has gently been going on behind the scenes to include more data and continue to fill in the globe.  Thank you to the many folks who have spent time to process the data they have kindly sent over for inclusion in this project.  We now have data from China, India, Myanmar, Brunei and Malaysia, with more on the way from some other countries.  For some indices this has really reduced the regions for which we do not have coverage, though of course this depends on the interpolation our gridding routine can perform.

To give an idea of where we are at, below are the same plots from the end of last month, but with the current coverage. These are all for the 2.5 x 3.75 degree gridding options using 1961-90 as the reference period (where appropriate). We have also made trial versions using smaller grid boxes (1.25 x 1.875 deg), a simpler (non-interpolating) gridding method, and also using 1981-2010 as a reference period, which allows the inclusion of some pre-calculated indices from some of our contributors.

Fig 1 - stations for CDD annual
Fig 2 - stations for CDD monthly
Fig 3 - stations for TNn annual
Fig 4 - stations for TNn monthly

And of course, the trends are here (and yes, we've not yet looked at the colour maps for monthly CDD!):
Fig 5 - linear trends for CDD, Annual
Fig 6 - linear trends for CDD, January
Figure 7 - linear trends for CDD, July
Figure 8 - linear trends for TNn, Annual
Figure 9 - linear trends for TNn, January
Figure 10 - linear trends for TNn, July

As you can see, in some cases we have more complete spatial coverage for the annual indices, rather than in individual months, which tends to arise when we have not been given the monthly index values.

We're still working on including more data from new sources and also check that the station selection criteria are sensible - to see if we can improve the underlying station coverage where possible.

Tuesday, 23 April 2019

Latest data update

In the last few months we've been gently gathering more data, and adapting the routines used to handle it to improve the coverage that we have.  

GHCND - this was using just the HCN over the US, but we've now expanded this to include the GSN and so have world wide coverage.  There is a risk that by using these data we are including stations which have significant inhomogeneties in them.  However, in regions with other data sources, the effect should be minimised as a result of the other stations present.  And in regions with no other data, having some data is better than none!

ACRE - we have adapted how this dataset is processed by Climpact2.  As few stations will have overlaps with the reference period we are using (1961-90 by default) the quality control removes these stations, and so no indices are calculated - even those where the reference period isn't used.  We now use a temporary reference period for each station and remove the indices which rely on it at a later stage.

Arabian, West African & South African data - these have been calculated using a more recent reference period (1981-2010).  Again, we've adapted how these are handled to make sure as many stations are retained and those requiring the base period are removed.

What our intention is that we will release a version for a 1981-2010 reference period - where this situation is reversed. Of course, for raw data where the indices are calculated using Climpact, this isn't a problem. And for ECAD, LACAD and SACAD we can revert to the underlying daily data from EOBS, "LAOBS", and "SAOBS" and recalculate the indices.

We are continuing to source data - over Brazil, China, India, Iran, Japan, and Mexico - and also hope to obtain more data over south-east Asia in due course.  We have set a deadline of June 2019 for data submissions to allow the finalisation of the paper.

The latest station distribution plots are below.  This time we're showing the annual coverage and also for one of the months - as these tend to have more stations pulled through.  The reason for this is that there are data presence requirements on a monthly and annual basis - and it is more likely to have insufficient data to calculate an annual quantity, than to calculate a monthly quantity.
Fig 1 - Consecutive Dry Days - Annual
Fig 2 - Consecutive Dry Days - Monthly
Fig 3 - Minimum Tmin - Annual
Fig 4 - Minimum Tmin - Monthly

In some cases the data that have been kindly provided only contain the annual indices, and so the coverage is different between annual and monthly maps shown below.

And here are the trend plots (simple linear values) for the same indices, also monthly (January and July) and annual.  We note that the colour scales are the same for the annual and monthly maps - which we will reassess and adjust in future runs.
Fig 5 - trends for CDD, annually, in January and in July
Fig 6 - trends for TNn, annually, in January and in July
The next step is to run some different versions of this dataset, using the alternative reference period (1981-1990) and also using smaller grid boxes, to more closely match the resolution of the current generation of GCMs.

Thursday, 7 March 2019

Improved spatial and temporal coverage

Since the last post, more data have arrived and been included in the latest run of HadEX3.  These include temperature and rainfall over Canada and Chile, and rainfall over Australia.

With the Angular Distance Weighting gridding, these additions are starting to fill in the globe for some indices - those which have long correlation length scales.  However, our quest for further data is not over, and we are indebted to our colleagues around the world for their efforts in providing data and calculating the indices.

Station Distribution for TX90p (annual)
Station distribution for R10mm (annual)
The location of all the stations which currently contribute to HadEX3 for two example indices are shown above.  As is clear, we still have large gaps, in Africa and Asia, as well as parts of South America, but we are working on filling these in the next few months.  However, as always, if you are able to send daily Tx, Tn and Precipitation data, or know of collections we could use, please get in touch!
Trend in TX90p (annual)
Trend in R10mm (annual)
Using the preliminary data the above two panels show the linear trends in these two example indices.  These plots are automatically produced, so the colour-scales are not necessarily the best. What is also clear is that some stations in central-north Africa seem to have some unexpected values for TX90p which are influencing the region surrounding them.  These may be true measurements, but are in stark contrast to neighbouring regions.  Similarly over south-eastern Asia there are strong trends not occurring in neighbouring grid-boxes.  As these are preliminary plots, we are not commenting on the correctness, just that we should take a look and check.
Timeseries of TX90p, comparing HadEX3 with earlier products (and matching to HadEX2 coverage).

Timeseries of R10mm, comparing HadEX3 with earlier products (and matching to HadEX2 coverage) 
What we can also do at this stage is compare the "global" average (land surface, where there are data) to previous products in the "Climdex Family".  Even at this point, with relatively large gaps in the spatial coverage, there is some agreement with HadEX2 (and more so when HadEX3 is coverage matched). For TX90p, as this is an index using a reference period, then GHCNDEX also matches relatively well.  However, for R10mm, although the short-timescale variation is reasonable, there is an offset to GHCNDEX, most likely due to different regions having data. This is also a likely cause for the "edge-effects" in the most recent years, as there are no stations in Asia and few in Africa post-2014 (see below).
HadEX3 stations for R10mm in 2005
HadEX3 stations for R10mm in 2015

Wednesday, 21 November 2018

Including more data

Since the test run of the HadEX3 codes in July (see last post) we've had submissions of more data from colleagues around the world (Thank you folks!).  Some of these have come in the form of the raw daily Tmax, Tmin and precipitation accumulations, whereas others have been the ETCCDI indices.

We're starting to fill in the land surface of the globe.  The GHCND (HCN only), ECAD, LACAD and SACAD still form a strong backbone in some parts of the world, but these have been supplemented by data from New Zealand, South America, Honduras, Russia, Western Africa (rainfall only), Australia (temperature only so far), North Africa and the Middle East and Pacific Island states.  The latest run of the code has included most of these new sources.  Thank you to all who have contributed data so far, your efforts are very much appreciated.

For the early period, we also have received data from the ACRE consortium to add in.  Naturally, many of these rescued data do not cover the reference period used for the indices based on percentile thresholds (1961-90), but for other indices these provide a valuable increase in the data coverage for the early part of the record.  To tempt you, here are some plots of the current station distribution for a couple of indices.

Fig 1. Distribution of stations used for TXx (annual)
Fig 2. Distribution of stations used for R10mm (annual)

We know of data over Canada (T), Southern Africa, Australia (P) which we should get in the next couple of months.  If you are able to provide data over China, India, Eastern Africa then the would be wonderful, but, as always, for any data that could contribute, we'd love to hear from you!

Fig 3. Percentage of land grid boxes with non-missing data at each latitude over time for (top) TXx, and (bottom) R10mm (annual values).