Link: https://mesonet.agron.iastate.edu/request/download.phtml
One of the unique datasets that the IEM curates is an archive of airport weather station data in METAR format. This ancient format is still being used today to disseminate much of the world's airport weather station data. The IEM attempts to process a real-time feed of this information from NOAA and among many other things, makes a simple download interface available for anybody to use.
So while the download interface is simple, the actual usage of the data can sometimes be anything but due to a quirk with how METAR data reports precipitation data. The reporting of precipitation data within METAR accumulates precipitation until an hourly reset time is reached, at which the accumulation is set back to zero. The implications of this are significant:
- You can not simply total up the reported precipitation within the download dataset from the IEM due to some hours having multiple reports and thus double counting of precipitation.
- The hourly precipitation reset time varies by station and generally matches the routine hourly reporting time for the station.
- The hourly reset time has changed over the years for some stations.
- Most of the state-owned AWOS sites do not have a concept of "special" METAR reports and just report 2-3 or more times per hour.
- During active weather, special reports may be generated at a time slightly before the routine reporting time and if no further precipitation is accumulated, the routine reporting time is skipped (the horror).
Oh boy, we are getting way down into the weeds here. Let us consider a practical example of actual reports from Ames from 4 July 2022:
KAMW 041138Z AUTO 28010G18KT 1SM +TSRA BR FEW007 BKN020 OVC038 22/22 A3001 RMK AO2 LTG DSNT ALQDS RAB05 TSB10 P0056 T02220217 KAMW 041153Z AUTO 26004KT 2SM TSRA BR FEW007 SCT020 OVC043 22/21 A2998 RMK AO2 VIS 1 1/2V3 LTG DSNT ALQDS RAB05 TSB10 SLP143 P0069 60069 70069 T02220211 10233 20217 53011
So at 11:38z, the precipitation total is 0.56 inches and then at the reset time of 11:53z, the total is 0.69 inches. The total precipitation for the "hour" is 0.69 inches and not the summation of the two reports. The 11:53z report covers the 10:53 to 11:53 period.
The long standing problem here for the IEM database is that all the reports made during the hour were lumped into one classification without any delineation offered on the download page. Rewording, you could not download just the hourly reports at the reset time and not also include specials.
Another implication is that for some applications like wind roses, one may only want the routine report included within a climatology calculation so not to bias any values to active weather situations that coincide with more special reports being generated. The IEM website tools had lots of boilerplate and complexity attempting to work around that very issue.
Why hasn't the IEM fixed this issue up until now? The curt answer is that if it was easy, it would have been done already. Additionally, a dedicated interface existed whereby the IEM attempted to compute hourly precipitation totals and shield end-users from this horror.
There are processes currently running over the billion plus observations within the IEM database attempting to delineate out the routine vs specials reports. Once this completes, the simple download interface will be updated to support this delineation.
And finally, a note for downstream API / library users that presently send report_type=2 to the IEM server to fetch data. This will continue to be supported and return the same combined routine and special reports, so hopefully there is no breakage from this change.
As always, please contact us with any questions/concerns you have!