A couple of people have been curious about the value of this Old Weather project. It often feels like we're working in the dark and, although Kevin occasionally posts news about the project, I asked him for an update.
This is what he wrote:
There are a number of posts that are worth reading, or re-reading, in the forum topic Old Weather news and results and related science newsYou got it -- for those working with the off-line material like Stuart we can go forward with southern hemisphere ships that we have imaged, or customized for others that may have a particular interest. I am sorry to be so slow to blog, but I do have a number of things not quite ready to post there, including the first look at some WWII output. For the overall picture, the following is part of some text I just wrote to include our work in the coming UN Decade of Ocean Science. I was going to develop it into a blog post, but I think it might be useful to share now w/ old hands on the forum if you'd like, though it isn't packaged for public distribution yet.
+++
The bottom line issue is that our fundamental marine-met climate data set - the International Comprehensive Ocean Atmosphere Data Set (ICOADS) - is heavily burdened by legacy issues. The impasse that has long been recognized is the substantial loss of metadata (especially US origin) that occurred when the primary marine-met data was first punched on IBM cards in the early 1950s, and then amplified through sequential migration to an optical analog format, to magnetic tape, and so on to today's digital form. The other issue is that only a tiny fraction (~3%) of the available U.S. data were punched. There is a discussion in AR5 Section 2 (See Box 2.1 and Section 2.4.2); likely unchanged in AR6.
This issue affects the shape of the global and hemispheric temperature curves in nearly every ICOADS-derived product, from the widely used NOAA Global, NASA GISTEMP, and Hadley Centre gridded products to NOAA ERSSTv5, and especially in the mid-20th century period. It also likely affects most extended reanalyses, including the state-of-the-art NOAA-CIRES-DOE 20CRv3, due to dependence on SST and Night Marine Air Temp. (NMAT) as a boundary condition or assimilated variable.
What has changed in just the last 2 months is we have been able to recover much of the lost metadata, including ship/platform IDs, instrument details and methods, for the majority of the U.S. assets between ~1920 and 1970. Millions of hourly weather records. We can now trace an observation back to the person who made it. This granular forensic analysis is made possible by the massive amount of primary source material now available online, and by the contributions of thousands of citizen-scientists (e.g. https://www.zooniverse.org/projects/krw ... eather-ww2).
It is now highly likely that the presumed data artifacts in the mid-century data can be diagnosed on a physical basis and possibly corrected. The current thought, once this is done, is that the appearance of mid-century climate variability will diminish and the global temperature trend lines will steepen (e.g. Chan and Huybers 2021 https://doi.org/10.1175/JCLI-D-20-0907.1). If I were to speculate even further, the accelerated regional rates of change we are seeing today may well be better explained.
+++
And on another front, I just received an enquiry from National Geographic Education asking to use our polar data as a demonstration set in a new GIS tool they are building for school kids. If this will go forward, you and others who've contributed so much to that data will be (with your concurrence) named as the authors.
I'll list some interesting facts from some of the articles, so you won't have to read them all the way through. There are many other articles that I haven't included, but which are also interesting and informative.
Old Weather in the News
- Old Weather Time Machine opens a treasure trove for reaearchers
- The weather map at the start of the article shows the Galveston hurricane of 1915 using the ICOADS data. Farther down the articles is an image of a hand drawn weather map from that time.
- Using NOAA’s Global Forecast System, researchers reconstructed the global atmosphere from surface pressure readings, sea temperature and sea ice observations from archival records, some transcribed by citizen volunteers. From this data, the model estimates temperature, pressure, winds, moisture, solar radiation and clouds.
- “This tool lets us quantitatively compare today’s storms, floods, blizzards, heat waves and droughts to those of the past and figure out whether or not climate change is having an effect,” Compo said. “This should be useful for climate attribution research.”
Enriching our understanding of long-ago events
Scientists have also used the previous versions of the “old weather” data to discover unknown hurricanes, study the climate impact of old volcanic eruptions, investigate the timing of bird migrations, and even explore the economic impact of diseases spread by the tsetse fly in sub-Saharan Africa. - 20CRv3 uses millions more observations than previous versions of the reanalysis, especially for earlier periods. The new reanalysis includes up to 25 percent more available observations for years prior to 1930. Running the model and crunching all this data required astronomical computing resources. To accomplish this third upgrade, the Department of Energy donated 600 million cpu hours to crunch 21 million gigabytes of data at the National Energy Research Scientific Computing Center.
- Learning from the past to understand the future: historical records of change in the ocean
- Figure 2 shows ships' positions in the ICOADS database. Old Weather data is shown in yellow.
- NOAA and the University of Washington (Cooperative Institute for Climate, Ocean & Ecosystem Studies) have been collaborating with the U.S. National Archives since 2011. During this period, the project has produced high-resolution digital images of 4 618 volumes of federal ship logbooks, dating between 1844 and 1955. These are all publicly available worldwide on the National Archives Catalog. These assets have so far produced about 1.5 million new-to-science hourly weather records via the Old Weather citizen-science project. As shown in figure 2a, more than 600 000 weather and sea-ice observations pertaining to the Arctic have been further enhanced by painstaking reconstruction of ship tracks to hourly resolution using the ‘dead reckoning’ and pilot information contained in the logbooks (i.e. data on the ship’s course and distance run, bearings and ranges from known landmarks).
- The enormous potential for data rescue is illustrated by the size of the U.S. collection that remains largely unutilized. Beginning in 1847, the logbooks of the U.S. Navy, Coast Guard/Revenue Cutter Service and Coast Survey contain 24-hourly weather records per day, and include 7–10 variables per hour, although not all variables were uniformly acquired in fact until after the U.S. Civil War (1861–1865). There are roughly 22 700 logbooks in the National Archives that date between 1801 and 1941. Until 1915, most volumes contain about one year’s worth of observations, and then from 1915–1941 logbooks were generally bound in monthly volumes. Conservatively estimating that only half of these logbooks contain all 24-hourly observations that would amount to 75 500 000 weather records to be recovered. There are undoubtedly tens of millions more unrecovered weather records from the World War II era and after.
- The National Archives of Denmark contain huge collections of logbooks. Starting as early as the mid-seventeenth century, it has been possible to identify more than 7 000 archive boxes, filling more than 700 metres of shelves with logbooks and other maritime data. Only a very small part of this data has been digitized to date.
- In recent years, NIWA, (The National Institute of Water and Atmospheric Research in New Zealand), has driven the Southern Weather Discovery (SWD) citizen science platform hosted on Zooniverse (southernweatherdiscovery.org), recovering ~250 000 Southern Hemisphere marine weather observations, promoting meteorological data rescue and completing experimentation on replicated data keying (Fig. 2b). They are also actively collaborating with Microsoft on an Artificial Intelligence (AI) for Earth project that is comparing manually transcribed observations and those completed by automatic means.
- The principal challenges faced by the marine data rescue community fall largely into two categories: access to historical records and conversion from manuscript to digital format.
...
In the second category, conversion to an actionable digital format is also a significant bottleneck. At present, this step relies on manual transcription, either via double-blind keying or citizen-science (crowdsourcing). These approaches are quite worthwhile if targeting particular regions or time periods with sparse data, such as the Arctic or Southern Ocean, or a discrete research question. However, a large-scale conversion of the vast quantities of unused marine weather data that are known to exist will require an efficient AI/ machine-learning solution.
- The Old Weather Blog has a couple of interesting articles and links to articles that were written in Phases I-III. OW Whaling (OWW) would be Phase IV and Old Weather Arctic (OWA) would be Phase V.
- The Education and Outreach topic has a link to the Reuters article about OW.
- New Discovery - storm quakes. USS Jamestown reported them in October 1880.
- Initial results with the Applied PIOMAS sea ice model The conclusions are that our ice data have confirmed that the PIOMAS model gives good results. (Basically, it uses weather data to predict sea ice. People could have said the ice edge forecasts for the late 1800s and early 1900s were theoretical because there were no ice reports to confirm it. Our OWA ice observations proved that, when the model uses historic weather data, the results are valid.).
- These are rather good results and a very big thank you to everyone who is working so hard on this.
...
The observations you all have recovered nails this down - it's easy enough to argue that a sea-ice model is getting wrong when there are no obs to validate, but not now.
Implications? The heat budget of the Pacific inflow into the Arctic would likely have been very different, and this would have long term impacts on the multiyear ice cover in Arctic basin consistent with current obs. Our understanding of the state of the Bering Sea ecosystem since the 1970s may be seen in a new context...especially for ice-obligate marine mammals. There will be more.
- Sharing our results
- We need not only to make our results available to everyone, but also to make sure they stay available for the next hundred years. So we are sending our new records to the International Comprehensive Ocean-Atmosphere Dataset which, as it says on the tin, is the international repository for our sort of records. ICOADS is freely available to any user, and actively managed to make sure nothing gets lost over the years.
- We've produced an impressive batch of results since we started the project, and I'm sometimes asked what, exactly is being done with them all. Some of the results, like those I've posted on the blog, and the ship histories on navalhistory.net, you already know about, but we are starting to have a much wider impact.
Weather & Climate:
There are at least 5 major climate reconstructions currently in progress using our new weather observations:
20CR Version 3 (NOAA/University of Colorado)
Era-Clim (ECMWF)
SODA (Texas A&M)
HURDAT (NOAA)
HadISST2 (Met Office)
Ship histories:
New museum exhibits inspired by or involving our work are in planning or preparation at:
The National Maritime Museum
The National Museum of the Royal Navy
The British Library
- Old Space Weather: sightings of aurorae and sunspots
- Old Weather volunteers have already found at least one example where sunspots were reported in a ship's log (presumably observed through fog or high cloud so the sailor on watch didn't severely damage his eyes!). Naked eye observations of sunspots are very rare and such observations are very important in putting other historical observations into context (there are many reports of sunspots in ancient Chinese and Korean records for example).
