Climate to the Rescue! Rebuilding the Lost HRRR Weather Model Archive

Shortly after the start of the year, University of Utah Professor John Horel, and PhD student Brian Blaylock, discovered that the Center for High Performance Computing (CHPC) object-storage system, known as Pando, had experienced a system failure. Pando was a relatively new, perhaps even experimental, data archive solution at the University of Utah, so hiccups were bound to happen. It had been heralded as an affordable, easy-to-use, and reliable long-term archive solution that made it very attractive to academic researchers who are dealing with increasing requirements by funding agencies to maintain archives of the data they use for research grants.

This hiccup turned into a catastrophe when it was announced that Pando was beyond repair and all data stored on it was lost forever (Fig. 1). This loss included 50 TB of one of the two known long-term archives of the National Oceanic and Atmospheric Administration’s (NOAA’s) High Resolution Rapid Refresh (HRRR) weather model. Climate LLC holds the only other known available archive.

The HRRR weather model is widely used by atmospheric scientists and is considered the best weather model for predicting short term (out to 18 hours) weather conditions over the United States. It assimilates more observations and data sources into its initial conditions than other weather models. It also has a superior high resolution 3-km spatial grid and is one of the only weather models updated every hour of the day. Climate has relied heavily on its output as a data layer for our Climate FieldView™ digital agriculture platform as well as various research programs. Our Weather Science team also used it to support the inversion risk and 3-foot wind forecasts in Monsanto’s RRXtend Spray app. We need an historical archive of past HRRR forecasts to train and validate these models to ensure robust and accurate performance. While NOAA only provides two days of HRRR forecasts, in 2016 we began saving each forecast to Amazon Web Services (AWS) cloud storage, and the University of Utah saved forecasts to Pando before it failed.

The University of Utah gave other researchers access to their HRRR archive in March 2017 and reached more than 175 registered users across many universities, agencies, and companies before the archive was lost in January 2018 (Fig. 2). Users included National Weather Service, NASA, and United States Air Force scientists as well as employees at Lockheed Martin, ACME AtronOmatic, and Vaisala. Some of the more common uses of HRRR forecasts for these users included downstream model development and verification (like at Climate LLC), HRRR forecast performance metrics, and retrospective case study analysis.

When our Weather Science team learned of the Pando failure, we acted quickly to backfill the University of Utah’s HRRR archive so the atmospheric science community could continue advancing our understanding of this great weather model. We are excited to report that the Pando system has been rebuilt with several changes and new failsafes, and HRRR forecasts are being backed up to a separate data archive system. Archived HRRR forecasts from Climate LLC are now available to download to registered users from this University of Utah download page.