Link: https://mesonet.agron.iastate.edu/request/download.phtml
One of the unique datasets that the IEM curates is an archive of airport weather station data in METAR format. This ancient format is still being used today to disseminate much of the world's airport weather station data. The IEM attempts to process a real-time feed of this information from NOAA and among many other things, makes a simple download interface available for anybody to use.
So while the download interface is simple, the actual usage of the data can sometimes be anything but due to a quirk with how METAR data reports precipitation data. The reporting of precipitation data within METAR accumulates precipitation until an hourly reset time is reached, at which the accumulation is set back to zero. The implications of this are significant:
- You can not simply total up the reported precipitation within the download dataset from the IEM due to some hours having multiple reports and thus double counting of precipitation.
- The hourly precipitation reset time varies by station and generally matches the routine hourly reporting time for the station.
- The hourly reset time has changed over the years for some stations.
- Most of the state-owned AWOS sites do not have a concept of "special" METAR reports and just report 2-3 or more times per hour.
- During active weather, special reports may be generated at a time slightly before the routine reporting time and if no further precipitation is accumulated, the routine reporting time is skipped (the horror).
Oh boy, we are getting way down into the weeds here. Let us consider a practical example of actual reports from Ames from 4 July 2022:
KAMW 041138Z AUTO 28010G18KT 1SM +TSRA BR FEW007 BKN020 OVC038 22/22 A3001 RMK AO2 LTG DSNT ALQDS RAB05 TSB10 P0056 T02220217 KAMW 041153Z AUTO 26004KT 2SM TSRA BR FEW007 SCT020 OVC043 22/21 A2998 RMK AO2 VIS 1 1/2V3 LTG DSNT ALQDS RAB05 TSB10 SLP143 P0069 60069 70069 T02220211 10233 20217 53011
So at 11:38z, the precipitation total is 0.56 inches and then at the reset time of 11:53z, the total is 0.69 inches. The total precipitation for the "hour" is 0.69 inches and not the summation of the two reports. The 11:53z report covers the 10:53 to 11:53 period.
The long standing problem here for the IEM database is that all the reports made during the hour were lumped into one classification without any delineation offered on the download page. Rewording, you could not download just the hourly reports at the reset time and not also include specials.
Another implication is that for some applications like wind roses, one may only want the routine report included within a climatology calculation so not to bias any values to active weather situations that coincide with more special reports being generated. The IEM website tools had lots of boilerplate and complexity attempting to work around that very issue.
Why hasn't the IEM fixed this issue up until now? The curt answer is that if it was easy, it would have been done already. Additionally, a dedicated interface existed whereby the IEM attempted to compute hourly precipitation totals and shield end-users from this horror.
There are processes currently running over the billion plus observations within the IEM database attempting to delineate out the routine vs specials reports. Once this completes, the simple download interface will be updated to support this delineation.
And finally, a note for downstream API / library users that presently send report_type=2 to the IEM server to fetch data. This will continue to be supported and return the same combined routine and special reports, so hopefully there is no breakage from this change.
As always, please contact us with any questions/concerns you have!
Link: https://mesonet.agron.iastate.edu/request/asos/1min.phtml
For many moons, the IEM has been processing an archive of one minute interval data from Iowa ASOS (primary airports) sites provided by the National Center for Environmental Information (NCEI, formally NCDC). This news item details an expansion of data processed by the IEM to provide data from all available sites provided by NCEI.
Twice a day, NCEI connects to many of the ASOS weather stations in this country to download its one minute interval data tables. NCEI then collects up this information and provides it in monthly files on their website here. For the lay person and even me, the weather programming nerd, the dataset provided by NCEI is difficult to work with and doesn't really have documentation on what exactly the data format is.
The IEM has made a best attempt effort to ingest this information and provide a simple download of the data found in these files. For some stations, data dates back to the year 2000.
While NCEI updates their available files twice per day, the IEM is just syncing them once every morning (around 8 AM). There is some delay with the most recent observation available for a given site, but generally a days worth of one minute data arrives within the next two days. Rewording, if some big event happens at your local ASOS site and you want to see the one minute data, this service won't have it for a day or two in the future.
Link: https://mesonet.agron.iastate.edu/request/download.phtml
I was poking around the Internet on a lazy Saturday morning and noticed that the NOAA MADIS project was now producing High Frequency METAR product. This is a cryptic term to describe observations coming from the primary weather sensors at most airports in the country. We only routinely get hourly data from these sensors, but 5 minute and even 1 minute data is collected but not widely made available. Note: on a once per month basis, the IEM processes a 1 minute interval feed of this information provided by the National Center for Environmental Information (NCEI).
The MADIS project appears to have a connection with the FAA data collection system so as to get a feed of this information! I coded up an ingest of this dataset and it is now augmenting what the IEM provides for ASOS/METAR data.
So this means you will start to see additional observations in the site history and METAR download.
I am also generating "fake" METARs for this data feed to include within the archive. I'd be interested to hear of troubles folks have as they see an increased volume of reports and these auxillary obs. These generated metars contain the string "MADISHF" in the remarks section so to hint where it came from.
Update 9 Jun 2016: So an unfortunate quirk has been found with this dataset and I recently confirmed the quirk with the MADIS Project. The FAA transfers the temperature and dew point data to the NWS in units of whole degree Celsius instead of whole degree Fahrenheit that the ASOS sites actually record the data in. This means that the regular METAR feed, which uses a T-group notation to remove ambiguity when the conversion to Fahrenheit takes place, does not always align well with the 5 minute MADIS data. It also means that sometimes the computed daily extremums can be one degree warmer or cooler than what was actually observed. Oh the fun!
Update 8 Jul 2016: This issue is causing too much confusion on the website. So I have stopped the ingest of the temperature and dewpoint data from this feed. It will simply appear as null values within the various apps. I continue to persue this matter with the FAA and NWS, so hopefully it can be fixed someday.
Update 24 Jul 2017: The issue of temperature precision is known to be a limitation of a MS-DOS based program that runs at the ASOS sites. The ASOS program managers are in the process of updating the software and hope to fix this issue in the process of software upgrades. The completion date is 2023.
Update 19 May 2023: Well, the year is 2023 now and one may wonder the status of this being fixed. To my knowledge, it is unclear what is happening and getting a NWS manager to acknowledge this issue is very difficult. There is also a massive ASOS modernization effort ongoing over the coming years, so hopefully this gets magically fixed during that process.
Link: http://mesonet.agron.iastate.edu/request/download.phtml
The ASOS download interface has been improved to support requesting multiple stations (a long requested feature) and improvements have been made to how time zones are handled. Please let me know of any issues you encounter downloading data or suggestions for improvements you may have.