Extension to include ET-SCI indices

When it was developed, HadEX3 contained the usual set of ETCCDI (WMO Expert Team for Climate Change Detection and Indices) indices that were also part of HadEX2.  The intention with the creating HadEX3 was to improve the spatial coverage and also extend the temporal coverage to more recent times (from ending in 2010 for HadEX2 to 2018 in HadEX3).  We also moved to a new reference period, and found that there were non-linearities in how some indices behaved as a result.

However, the ETCCDI indices were developed from a climatological perspective, as the plan was to be able to both monitor changes in extremes and also perform detection and attribution studies.  Over recent years, assessing the impact of our changing climate on sectors relevant to human society and infrastructure have become more important. After all, we know the climate is clearly warming, and have a much better understanding of how the tails of the temperature distribution are changing.

The approach used by the ETCCDI of developing indices, running workshops and providing standardized software was adopted by the WMO Expert Team on Climate Risk and Sector-specific Climate Indices (ET-CRSCI) and subsequently the Expert Team for Sector-specific Climate Indices (ET-SCI) to develop indices to monitor changes in climate conditions relevant to a number of societal sectors.  These “ET-SCI indices” extend the ETCCDI indices and enable those working in sectors such as agriculture, health and water resources to more directly determine how relevant climate extremes are affecting them. The combined set of indices developed by both the ETCCDI and ET-SCI are termed the “Climpact indices,” and for a full list see https://www.climdex.org/learn/indices/. Indices defined by both Expert Teams are applicable across different sectors, and currently fall under the WMO Expert Team on Climate Information for Decision-Making (ET-CID).

We have recently released an extension to HadEX3 which incorporates the additional indices defined by the ET-SCI.  We have where possible used the same input data and approach as for the original HadEX3 (for the ETCCDI indices), and so the additional indices have been released with the version number of 3.0.4 to tie into the current set.

Again, versions are available relative to 1961-90 and 1981-2010 for those indices where a reference period is used in their construction.  For full details of the extension, see Dunn et al, (2024), and the data are available at https://www.metoffice.gov.uk/hadobs/hadex3. 

Please do get in touch if you find this dataset useful, have comments or suggestions, or have found issues with any of the data files.

 

Dunn, R. J. H., Herold, N., Alexander, L. V., Donat, M. G., Allan, R., Bador, M., et al. (2024). Observed global changes in sector-relevant climate extremes indices—An extension to HadEX3. Earth and Space Science, 11, e2023EA003279. https://doi.org/10.1029/2023EA003279

 

HadEX 3.0.4

We have updated HadEX to version 3.0.4.

We were alerted to an error in the calculation of the GSL index in the Climpact2 code for stations in the Southern Hemisphere.  This has been corrected as of their version 1.3.0.  We have re-run the HadEX3 code for this index, and uploaded the new data files.  Although we have updated the metadata (version number) in all files, the data changes are for GSL only.

The updated calculation only affects the Southern Hemisphere stations.  A number of our data sources for this hemisphere were of daily data and so we were able to recalculate these as part of the HadEX3 build.  For other data sources which were precalculated indices, most did not use Climpact2 to calculate GSL.  The only ones which did were stations in Indonesia calculated using Climpact2 during a workshop in Singapore held in 2019.  Only a handful of stations could contribute to the gridded data over this region, so we have just withheld these few stations when createing the HadEX3 grids for this update.

Figure 1 - trends in GSL for HadEXv3.0.4

References

Meteorological Service Singapore (2019). Second workshop on ASEAN Regional Climate Data, Analysis and Projections (ARCDAP-2): Workshop report. Retrieved from https://public.wmo.int/en/events/workshops/second-workshop-asean-regional-climate-data-analysis-and-projections-arcdap%E2%80%932

HadEX 3.0.3

Time for another minor version update.  We have addressed two issues in this latest release.

1) There was an error in how the Rx1day and Rx5day data were being handled for one of the West African data sources (January was being used in place of the Annual values).  We have fixed this, which also ensures that the monthly values for these indices are also carried through to the final grids.  You can see the difference in the resulting trends for this region in the two maps in Figure 1a & 1b, though there are only very small changes in the global annual trends.

 

(a) HadEX 3.0.2 Rx1day trends

(b) HadEX 3.0.3 Rx1day trends

 

2) In this latest update we have included information from some of the flags set in GHCND when converting these daily observations in preparation for calculating the indices.  We cannot remember whether the previous functionality was intentional or if the flags had been ignored inadvertently.  We now set observations which have failed the bounds check ("X"), the streak/frequent value check ("K") and the duplicate check ("D") as missing.  For some indices the HadEX QC would have picked up e.g. the bounds check, but not for all.  Keeping real extremes is clearly important in this case.  Therefore, we have decided to use only these tests as we wanted to remove only those observations which are clearly erroneous and would have an impact on the indices. 

For the gridded fields, this change makes very little difference, as the blending between stations performed by the Angular Distance Weighting algorithm smooths out the effect of erroneous values (as can be seen [or not!] in Figure 1).  However around an extra 20 stations are now included in the gridded fields as these no longer fail our own QC. For users of the station data, these changes will be apparent.

HadEX 3.0.2

We have just released an update to HadEX3 (version 3.0.2).  It was brought to our attention that the land-sea mask used had an offset against the actual land sea boundary, something that was not spotted on the global-scale maps but was clear on regional cut-outs.

The net effect of this error was that grid boxes on the south/west were more likely to be categorised as sea rather than land, with the opposite for those on northern/eastern coasts.  No grid box values or locations away from the coasts have changed.  As an example, we show the trend maps for version 3.0.1 and 3.0.2 below (the Pacific coast of South America shows the change clearly).

HadEX 3.0.2 (TNn trend, 1950-2018)

HadEX 3.0.1 (TNn trend, 1950-2018)

 

This has been corrected for all indices and reference periods and as of 5th November, the new netCDF and image files are available online at Hadobs and Climdex.  No other changes were made to the input data, and so there are only very minor changes to grid boxes on coastlines.

HadEX 3.0.1

We have released an updated version of the HadEX3 dataset, version 3.0.1. The files on HadOBS, Climdex and CEDA have been updated.  We outline the reasons and changes over 3.0.0 below.

A user contacted us regarding some curious data values they found in Southeast Asia in two indices, DTR and TN90p (61-90).  Although we have updated the metadata (version number) in all files, the data changes are for these two indices only.

For DTR, we identified a handful of stations where the values were sufficiently spurious (values of many tens of degrees).  

For TN90p, theoretically, the climatologically expected values are 10% (one in ten days are over the 90th percentile for the minimum temperatures).  There are a number of reasons (see below) why this is not going to be exactly true in the final gridded dataset, but values should still be within a few percent.  The maps of the climatology values were not close to 10% over parts of Southeast Asia.  We found a single station where the values were not reasonable.

In both cases, we have manually removed these stations from the selection procedure and re-run the build of the dataset (re-calculating the DLS, and recreating the grids etc).  As there has been a change to the underlying data we have updated the version number.  Our reason for incrementing the "z" in the x.y.z format is that only a few stations have been withheld, a small geographic area was affected for only two indices and the bad values were only in a few years of the input data.

Why are the climatological values for the percentile indices not identically equal to 10%?

Firstly, there are a number of methods available to calculate the percentile values for these quantities, interms of how the discrete distribution is handled to extract the threshold value, and so there will be some variation from that alone.  The handling of any missing daily data during this reference period can be quite varied betweeen routines. We have tried to mitigate this to use a single routine where we have done the conversion from daily data to ETCCDI indices.  But there are a number of codes available, and we do not know which was used when indices have been submitted for use in HadEX3. Also, specifically for the ETCCDI indices, our calculations include work o specifically avoid inhomogeneities for these indices which mean the calculations are more complex than the simplest version of extracting the percentiles (Zhang et al, 2005).

The percentile thresholds determined for doing the counts above/below are determined from the daily data.  However, the monthly and annual index values used in the dataset (on a station basis) are only available if this underlying data are sufficiently complete.  If too many days within a specific calendar month or year are missing, then the index value cannot be accurately determined, even if the threshold value (10th/90th percentile) has been calculated. Therefore when calculating the climatologies from the gridded fields, these missing months and years will affect the average value, reducing the likelihood of it equalling 10% exactly.  

There could also be the impact of quality control algorithms.  For the HadEX3 scripts, the QC processes flag entire stations, rather than individual years or months, so this should not have an effect, but any upstream QC performed on the indices may have flagged individual years or months and so further affect the completeness, and hence the climatological values.

Finally, these indices are measured in percent.  However, for regions of the world which have comparatively low annual temperature variability, this makes the threshold exceedances very sensitive to any impact from the issues outlined above.  And of course, the gridding process blends together stations, and so if e.g. missing years in one station affect the back-calculated climatology value, then this will be drawn through into each grid box this station contributes to.

References:

Zhang, X., G. Hegerl, F. W. Zwiers, and J. Kenyon, 2005: Avoiding Inhomogeneity in Percentile-Based Indices of Temperature Extremes. J. Climate, 18, 1641–1651, https://doi.org/10.1175/JCLI3366.1.

HadEX3 on CEDA and Climdex

The gridded HadEX3 data have been available on the homepage since publication.  However, we also have made the data available on the Centre for Environmental Data Analysis (CEDA - formerly the BADC archive) and also on the www.climdex.org website.

Also, the subset of the underlying station data that we are allowed to make available are now on www.climdex.org.

HadEX3 released

Yesterday (July 2nd 2020) the paper describing HadEX3 appeared as an Accepted Article in JGR-A.  We have also made the underlying gridded dataset and plots available on the hadobs website.  In due course the data files will also be available at www.climdex.org, once some final reformatting has been completed, and also in the CEDA Archive.