Precipitation in HadISD

We have included precipitation accumulations in HadISD since its launch (netCDF field “precip1_depth”), as this information is used as part of the quality control suite to check for high humidity periods in the dewpoint depression check.  We wanted to make the HadISD fully traceable, so that users could check our quality control decisions for themselves, should they wish to.  These precipitation accumulations are not quality controlled and so we have urged users to take care when using these data in their analyses.  

In the ISD data format there are four possible entries for the precipitation data.  These are indicated by character code “AA1” to “AA4”.  Each of these has period, depth, condition and quality entries.  To assist in the quality control of the dewpoint temperature fields, we extracted the first of these four precipitation fields.  The netCDF names we assigned these variables are “precip1_depth” and “precip1_period”, as these were from the first ISD precipitation field.   

Recently, Kimberly Channell of the Great Lakes Integrated Sciences and Assessments at the University of Michigan highlighted a confusion with the description of the precip1_depth field in the netCDF files.  The metadata for versions up to and including v2.0.2.2017p states “Depth of Precipitation Reported over time period”. This, combined with the hourly time stamps, could easily result in an assumption that the precip1_depth field only contains hourly accumulations.  Furthermore, our naming of the netCDF variable inadvertently supports this interpretation, that the “1” in the “precip1_depth” suggests hourly accumulation values.  Unfortunately, neither of these are the case.  

The accumulation period for the precip1_depth is given by the precip1_period.  Even if there are timestamps every hour, the accumulation period may be a mix of time periods (from hourly to daily). We now appreciate that the metadata for these two variables could have been clearer, and that our chosen naming could be confusing, especially without knowledge of the ISD naming conventions.  We apologise to users if these issues have caused problems with their analyses.  To properly use the precipitation information, the depth information should be combined with the period.

Within one of the ISD precipitation fields, it is possible to have a number of accumulation periods, rather than just a single one across the entire length of the station record.  The ISD is itself made up of a number of underlying databases, drawing observations from across a variety of observation networks (e.g. SYNOP, METAR, GTS).  Each of these may have a different accumulation period, and also conventions as to the time of observation of e.g. 24 hour accumulations (and to which day these are assigned).  These have been combined together to form threaded records for single station locations where possible during both the ISD and HadISD development.

It may be that a station report type (e.g. GTS) was the primary source in the early period (e.g. filling AA1), but that for a later period, a different source with a different standard accumulation period has a higher priority in a merging process, and so supersedes this.  Therefore, at observation times where both sources have data, this could move the hourly accumulations down into the later precipitation fields (AA2-4), resulting in the interleaving of the different accumulation periods in the first entry.  The example station in Figure 1 exhibits behaviour consistent with this.  It is possible that further hourly accumulation values are present in AA2-4 of the ISD file for this station, but as we have not extracted those, they are not available to users of the HadISD at this time.
 

Figure 1 (top) precip1_depth, and (bottom) precip1_period for 724380-93819 (Indianapolis Airport).  This station has been merged from two ISD stations (99999-93819 and 724380-93819).  As the precipitation information is not quality controlled, likely erroneous observations like the ~160mm in the late 1970s are still present in the data files (using HadISD v2.0.2.2017p).

Therefore, for any given station in the HadISD, it is very likely that the period over which the precipitation depth has been accumulated is not constant over the entire record.  However, there still may be valid precip1_depth measurements present at each hourly timestamp, but these may be a combination of hourly1, but also 3, 6, 12, and 24 hourly measurements.  We advise users wishing to take advantage of the precipitation information to make plots like those in Figure 1 to check for themselves what data have been included.

In light of this possible confusion from our netCDF variable names, we have/will take a number of actions:

1)    Added notes to the HadISD webpages to clarify our naming scheme and inform users about the need to use both the “precip1_depth” and “precip1_period” fields.  We’ve also improved the metadata for these two variables on the webpages too.
2)    Improve the metadata of the “precip1_depth” and “precip1_period” fields in the netCDF files in the next update (v2.0.2.2017f).
3)    In the longer term, extract all four ISD precipitation fields where available, and attempt to disaggregate into 1, 3 , 6, 12, and 24 hourly accumulation fields within the netCDF files.  However, it is unlikely that we will be doing any quality control on these data and so we will still advise caution when using these. 

We note again that the precipitation information in HadISD is not quality controlled at the moment. 

Please do get in touch if you would like more information.

HadISD v2.0.2.2017p

We have just released version 2.0.2.2017p of HadISD on the Hadobs website.  The data now cover 1931/1/1 to 2017/12/31.

Downloading the data from the ISD finished on Monday 15th January and the quality control and other processes ran over the following days.

There are 8103 stations in this version of HadISD, a full 2000 more than in HadISD version 1.0.x.  However, there have been no changes to the quality control tests over v2.01.2016f.

As always, if you notice anything untoward in the dataset please do get in touch.  We intend to run a final version in a few months time if there have been changes to the ISD data in 2017 or earlier years in the intervening time.

We hope to move to monthly updates during 2018, which entail some minor changes to the QC code, but which should not impact the annual update methods.  We will post on here in due course when this project is nearing completion.