Extension to include ET-SCI indices

When it was developed, HadEX3 contained the usual set of ETCCDI (WMO Expert Team for Climate Change Detection and Indices) indices that were also part of HadEX2.  The intention with the creating HadEX3 was to improve the spatial coverage and also extend the temporal coverage to more recent times (from ending in 2010 for HadEX2 to 2018 in HadEX3).  We also moved to a new reference period, and found that there were non-linearities in how some indices behaved as a result.

However, the ETCCDI indices were developed from a climatological perspective, as the plan was to be able to both monitor changes in extremes and also perform detection and attribution studies.  Over recent years, assessing the impact of our changing climate on sectors relevant to human society and infrastructure have become more important. After all, we know the climate is clearly warming, and have a much better understanding of how the tails of the temperature distribution are changing.

The approach used by the ETCCDI of developing indices, running workshops and providing standardized software was adopted by the WMO Expert Team on Climate Risk and Sector-specific Climate Indices (ET-CRSCI) and subsequently the Expert Team for Sector-specific Climate Indices (ET-SCI) to develop indices to monitor changes in climate conditions relevant to a number of societal sectors.  These “ET-SCI indices” extend the ETCCDI indices and enable those working in sectors such as agriculture, health and water resources to more directly determine how relevant climate extremes are affecting them. The combined set of indices developed by both the ETCCDI and ET-SCI are termed the “Climpact indices,” and for a full list see https://www.climdex.org/learn/indices/. Indices defined by both Expert Teams are applicable across different sectors, and currently fall under the WMO Expert Team on Climate Information for Decision-Making (ET-CID).

We have recently released an extension to HadEX3 which incorporates the additional indices defined by the ET-SCI.  We have where possible used the same input data and approach as for the original HadEX3 (for the ETCCDI indices), and so the additional indices have been released with the version number of 3.0.4 to tie into the current set.

Again, versions are available relative to 1961-90 and 1981-2010 for those indices where a reference period is used in their construction.  For full details of the extension, see Dunn et al, (2024), and the data are available at https://www.metoffice.gov.uk/hadobs/hadex3. 

Please do get in touch if you find this dataset useful, have comments or suggestions, or have found issues with any of the data files.

 

Dunn, R. J. H., Herold, N., Alexander, L. V., Donat, M. G., Allan, R., Bador, M., et al. (2024). Observed global changes in sector-relevant climate extremes indices—An extension to HadEX3. Earth and Space Science, 11, e2023EA003279. https://doi.org/10.1029/2023EA003279

 

HadEX 3.0.4

We have updated HadEX to version 3.0.4.

We were alerted to an error in the calculation of the GSL index in the Climpact2 code for stations in the Southern Hemisphere.  This has been corrected as of their version 1.3.0.  We have re-run the HadEX3 code for this index, and uploaded the new data files.  Although we have updated the metadata (version number) in all files, the data changes are for GSL only.

The updated calculation only affects the Southern Hemisphere stations.  A number of our data sources for this hemisphere were of daily data and so we were able to recalculate these as part of the HadEX3 build.  For other data sources which were precalculated indices, most did not use Climpact2 to calculate GSL.  The only ones which did were stations in Indonesia calculated using Climpact2 during a workshop in Singapore held in 2019.  Only a handful of stations could contribute to the gridded data over this region, so we have just withheld these few stations when createing the HadEX3 grids for this update.

Figure 1 - trends in GSL for HadEXv3.0.4

References

Meteorological Service Singapore (2019). Second workshop on ASEAN Regional Climate Data, Analysis and Projections (ARCDAP-2): Workshop report. Retrieved from https://public.wmo.int/en/events/workshops/second-workshop-asean-regional-climate-data-analysis-and-projections-arcdap%E2%80%932

HadEX 3.0.3

Time for another minor version update.  We have addressed two issues in this latest release.

1) There was an error in how the Rx1day and Rx5day data were being handled for one of the West African data sources (January was being used in place of the Annual values).  We have fixed this, which also ensures that the monthly values for these indices are also carried through to the final grids.  You can see the difference in the resulting trends for this region in the two maps in Figure 1a & 1b, though there are only very small changes in the global annual trends.

 

(a) HadEX 3.0.2 Rx1day trends

(b) HadEX 3.0.3 Rx1day trends

 

2) In this latest update we have included information from some of the flags set in GHCND when converting these daily observations in preparation for calculating the indices.  We cannot remember whether the previous functionality was intentional or if the flags had been ignored inadvertently.  We now set observations which have failed the bounds check ("X"), the streak/frequent value check ("K") and the duplicate check ("D") as missing.  For some indices the HadEX QC would have picked up e.g. the bounds check, but not for all.  Keeping real extremes is clearly important in this case.  Therefore, we have decided to use only these tests as we wanted to remove only those observations which are clearly erroneous and would have an impact on the indices. 

For the gridded fields, this change makes very little difference, as the blending between stations performed by the Angular Distance Weighting algorithm smooths out the effect of erroneous values (as can be seen [or not!] in Figure 1).  However around an extra 20 stations are now included in the gridded fields as these no longer fail our own QC. For users of the station data, these changes will be apparent.

HadEX 3.0.2

We have just released an update to HadEX3 (version 3.0.2).  It was brought to our attention that the land-sea mask used had an offset against the actual land sea boundary, something that was not spotted on the global-scale maps but was clear on regional cut-outs.

The net effect of this error was that grid boxes on the south/west were more likely to be categorised as sea rather than land, with the opposite for those on northern/eastern coasts.  No grid box values or locations away from the coasts have changed.  As an example, we show the trend maps for version 3.0.1 and 3.0.2 below (the Pacific coast of South America shows the change clearly).

HadEX 3.0.2 (TNn trend, 1950-2018)

HadEX 3.0.1 (TNn trend, 1950-2018)

 

This has been corrected for all indices and reference periods and as of 5th November, the new netCDF and image files are available online at Hadobs and Climdex.  No other changes were made to the input data, and so there are only very minor changes to grid boxes on coastlines.

HadEX 3.0.1

We have released an updated version of the HadEX3 dataset, version 3.0.1. The files on HadOBS, Climdex and CEDA have been updated.  We outline the reasons and changes over 3.0.0 below.

A user contacted us regarding some curious data values they found in Southeast Asia in two indices, DTR and TN90p (61-90).  Although we have updated the metadata (version number) in all files, the data changes are for these two indices only.

For DTR, we identified a handful of stations where the values were sufficiently spurious (values of many tens of degrees).  

For TN90p, theoretically, the climatologically expected values are 10% (one in ten days are over the 90th percentile for the minimum temperatures).  There are a number of reasons (see below) why this is not going to be exactly true in the final gridded dataset, but values should still be within a few percent.  The maps of the climatology values were not close to 10% over parts of Southeast Asia.  We found a single station where the values were not reasonable.

In both cases, we have manually removed these stations from the selection procedure and re-run the build of the dataset (re-calculating the DLS, and recreating the grids etc).  As there has been a change to the underlying data we have updated the version number.  Our reason for incrementing the "z" in the x.y.z format is that only a few stations have been withheld, a small geographic area was affected for only two indices and the bad values were only in a few years of the input data.

Why are the climatological values for the percentile indices not identically equal to 10%?

Firstly, there are a number of methods available to calculate the percentile values for these quantities, interms of how the discrete distribution is handled to extract the threshold value, and so there will be some variation from that alone.  The handling of any missing daily data during this reference period can be quite varied betweeen routines. We have tried to mitigate this to use a single routine where we have done the conversion from daily data to ETCCDI indices.  But there are a number of codes available, and we do not know which was used when indices have been submitted for use in HadEX3. Also, specifically for the ETCCDI indices, our calculations include work o specifically avoid inhomogeneities for these indices which mean the calculations are more complex than the simplest version of extracting the percentiles (Zhang et al, 2005).

The percentile thresholds determined for doing the counts above/below are determined from the daily data.  However, the monthly and annual index values used in the dataset (on a station basis) are only available if this underlying data are sufficiently complete.  If too many days within a specific calendar month or year are missing, then the index value cannot be accurately determined, even if the threshold value (10th/90th percentile) has been calculated. Therefore when calculating the climatologies from the gridded fields, these missing months and years will affect the average value, reducing the likelihood of it equalling 10% exactly.  

There could also be the impact of quality control algorithms.  For the HadEX3 scripts, the QC processes flag entire stations, rather than individual years or months, so this should not have an effect, but any upstream QC performed on the indices may have flagged individual years or months and so further affect the completeness, and hence the climatological values.

Finally, these indices are measured in percent.  However, for regions of the world which have comparatively low annual temperature variability, this makes the threshold exceedances very sensitive to any impact from the issues outlined above.  And of course, the gridding process blends together stations, and so if e.g. missing years in one station affect the back-calculated climatology value, then this will be drawn through into each grid box this station contributes to.

References:

Zhang, X., G. Hegerl, F. W. Zwiers, and J. Kenyon, 2005: Avoiding Inhomogeneity in Percentile-Based Indices of Temperature Extremes. J. Climate, 18, 1641–1651, https://doi.org/10.1175/JCLI3366.1.

HadEX3 on CEDA and Climdex

The gridded HadEX3 data have been available on the homepage since publication.  However, we also have made the data available on the Centre for Environmental Data Analysis (CEDA - formerly the BADC archive) and also on the www.climdex.org website.

Also, the subset of the underlying station data that we are allowed to make available are now on www.climdex.org.

HadEX3 released

Yesterday (July 2nd 2020) the paper describing HadEX3 appeared as an Accepted Article in JGR-A.  We have also made the underlying gridded dataset and plots available on the hadobs website.  In due course the data files will also be available at www.climdex.org, once some final reformatting has been completed, and also in the CEDA Archive.

Almost there

I've been very quiet with the updates since last summer.  Partially this is just forgetting to update the blog, but we have been making progress on the HadEX3 dataset.  

During the latter part of 2019, we played around with options for the gridded dataset.  I fine-tuned the station selection and some of the quality control scripts to ensure we have a good balance between the widest spatial coverage and the best quality of stations.  As a result, for the final grids, stations which end before 2009 are not included.  A number of sources end before then, and their absence from the network in the final decade causes apparent inhomogeneities in some regions.  The stations will still be made available where possible.

The results of these are that the released version of HadEX3 will be at a finer resolution than HadEX2, using 1.875x1.25 degrees (so x4).  The "global" trends for the land surface don't change much at all, but as our station network is denser in many regions, we can present greater spatial detail.

One thing that we have been able to do is a more detailed comparison between the trends and spatial patterns of the indices which use a reference period (e.g. TX90p, CSDI, R95p) as input data for these cannot be combined if they use different periods.  This limitation does affect the spatial coverage at the moment, but we are asking contributors to calculate indices with the other reference period where possible to make this comparison as complete and robust as we can.  This will allow any future collections of these indices to understand how different reference periods affect the absolute values and hence also long term trends.

The final bits of data came in during November in time for us to submit the paper before the end of 2019 to JGR-A. The reviews which came back earlier in 2020 were positive, but it has taken a little longer to deal with them with the changes that everyone has undergone as a result of COVID-19.  However, I'm pleased to say that we have submitted the revised manuscript back this week.  Once the peer review process has completed, I'll make the dataset publicly available.

Quick update

Just a very quick update on this.  We have now received all the data we were expecting to receive - and so can start adjusting some of the processing criteria. 

However, to give an indication of the number of stations and rough coverage for trend analysis, I thought I'd put the latest plots here.  Once the processing criteria are settled on my intention is to put a draft document together.

Fig 1 - Stations in the Rx1day grids

Fig 2 - Stations in the TXx grids

As you can see, we have stations now covering the majority of the globe.  However, we want to ensure that the hard work from colleagues who have submitted data is recognised, and include as many stations as makes sense.  Currently, we're requiring that stations report relatively continuously during the period when they do - so can only have a maximum of 2 missing years, and I think that this is reducing the final numbers from over Brazil, India and elsewhere. 

Fig 3 - Rx1day trends (only from grid boxes with 66% completeness)

Fig 4 -TXx trends (only from grid boxes with 66% completeness)

And from these trend plots, the coverage is much reduced.  This arises from knock on effects, fewer stations being currently selected because of the completeness criterion.  Those that do may result in a too sparse network to ensure grids are calculated (which is more likely for indices which have a short spatial correlation).  And when they are, they may not be for sufficient years to calculate the trends. 

Fig 5 - Timeseries of Rx1day, compared to GHCNDEX and HadEX2

Fig 6 - Timeseries of TXx, compared to GHCNDEX and HadEX2

What is interesting is that, despite the limitations of the spatial coverage, the long term trends in quasi-global averages match very well with the other ETCCDI datasets available (GHCNDEX and HadEX2).

We're seeing what can be done to improve the spatial coverage without impacting the integrity of the dataset, so watch this space.

Filling in the Globe

Just a quick update this time.  Work has gently been going on behind the scenes to include more data and continue to fill in the globe.  Thank you to the many folks who have spent time to process the data they have kindly sent over for inclusion in this project.  We now have data from China, India, Myanmar, Brunei and Malaysia, with more on the way from some other countries.  For some indices this has really reduced the regions for which we do not have coverage, though of course this depends on the interpolation our gridding routine can perform.

To give an idea of where we are at, below are the same plots from the end of last month, but with the current coverage. These are all for the 2.5 x 3.75 degree gridding options using 1961-90 as the reference period (where appropriate). We have also made trial versions using smaller grid boxes (1.25 x 1.875 deg), a simpler (non-interpolating) gridding method, and also using 1981-2010 as a reference period, which allows the inclusion of some pre-calculated indices from some of our contributors.

Fig 1 - stations for CDD annual

Fig 2 - stations for CDD monthly

Fig 3 - stations for TNn annual

Fig 4 - stations for TNn monthly

And of course, the trends are here (and yes, we've not yet looked at the colour maps for monthly CDD!):

Fig 5 - linear trends for CDD, Annual

Fig 6 - linear trends for CDD, January

Figure 7 - linear trends for CDD, July

Figure 8 - linear trends for TNn, Annual

Figure 9 - linear trends for TNn, January

Figure 10 - linear trends for TNn, July

As you can see, in some cases we have more complete spatial coverage for the annual indices, rather than in individual months, which tends to arise when we have not been given the monthly index values.

We're still working on including more data from new sources and also check that the station selection criteria are sensible - to see if we can improve the underlying station coverage where possible.

Latest data update

In the last few months we've been gently gathering more data, and adapting the routines used to handle it to improve the coverage that we have.  

GHCND - this was using just the HCN over the US, but we've now expanded this to include the GSN and so have world wide coverage.  There is a risk that by using these data we are including stations which have significant inhomogeneties in them.  However, in regions with other data sources, the effect should be minimised as a result of the other stations present.  And in regions with no other data, having some data is better than none!

ACRE - we have adapted how this dataset is processed by Climpact2.  As few stations will have overlaps with the reference period we are using (1961-90 by default) the quality control removes these stations, and so no indices are calculated - even those where the reference period isn't used.  We now use a temporary reference period for each station and remove the indices which rely on it at a later stage.

Arabian, West African & South African data - these have been calculated using a more recent reference period (1981-2010).  Again, we've adapted how these are handled to make sure as many stations are retained and those requiring the base period are removed.

What our intention is that we will release a version for a 1981-2010 reference period - where this situation is reversed. Of course, for raw data where the indices are calculated using Climpact, this isn't a problem. And for ECAD, LACAD and SACAD we can revert to the underlying daily data from EOBS, "LAOBS", and "SAOBS" and recalculate the indices.

We are continuing to source data - over Brazil, China, India, Iran, Japan, and Mexico - and also hope to obtain more data over south-east Asia in due course.  We have set a deadline of June 2019 for data submissions to allow the finalisation of the paper.

The latest station distribution plots are below.  This time we're showing the annual coverage and also for one of the months - as these tend to have more stations pulled through.  The reason for this is that there are data presence requirements on a monthly and annual basis - and it is more likely to have insufficient data to calculate an annual quantity, than to calculate a monthly quantity.

Fig 1 - Consecutive Dry Days - Annual

Fig 2 - Consecutive Dry Days - Monthly

Fig 3 - Minimum Tmin - Annual

Fig 4 - Minimum Tmin - Monthly

In some cases the data that have been kindly provided only contain the annual indices, and so the coverage is different between annual and monthly maps shown below.

And here are the trend plots (simple linear values) for the same indices, also monthly (January and July) and annual.  We note that the colour scales are the same for the annual and monthly maps - which we will reassess and adjust in future runs.

Fig 5 - trends for CDD, annually, in January and in July

Fig 6 - trends for TNn, annually, in January and in July

The next step is to run some different versions of this dataset, using the alternative reference period (1981-1990) and also using smaller grid boxes, to more closely match the resolution of the current generation of GCMs.