Archive by Author | Philip

Free at last

We highlight improvements to the International Comprehensive Ocean-Atmosphere Data Set (ICOADS) in the latest Release 3.0 (R3.0; covering 1662–2014). ICOADS is the most widely used freely available collection of surface marine observations, providing data for the construction of gridded analyses of sea surface temperature, estimates of air–sea interaction and other meteorological variables. ICOADS observations are assimilated into all major atmospheric, oceanic and coupled reanalyses, further widening its impact. R3.0 therefore includes changes designed to enable effective exchange of information describing data quality between ICOADS, reanalysis centres, data set developers, scientists and the public. These user-driven innovations include the assignment of a unique identifier (UID) to each marine report – to enable tracing of observations, linking with reports and improved data sharing. Other revisions and extensions of the ICOADS’ International Maritime Meteorological Archive common data format incorporate new near-surface oceanographic data elements and cloud parameters. Many new input data sources have been assembled, and updates and improvements to existing data sources, or removal of erroneous data, made. Coupled with enhanced ‘preliminary’ monthly data and product extensions past 2014, R3.0 provides improved support of climate assessment and monitoring, reanalyses and near-real-time applications.

Sounds exciting, doesn’t it? Well, it’s even more exciting than it sounds, because that’s the abstract of Freeman, E., Woodruff, S. D., Worley, S. J., Lubker, S. J., Kent, E. C., Angel, W. E., Berry, D. I., Brohan, P., Eastman, R., Gates, L., Gloeden, W., Ji, Z., Lawrimore, J., Rayner, N. A., Rosenhagen, G. and Smith, S. R. (2016), ICOADS Release 3.0: a major update to the historical marine climate record. Int. J. Climatol.. doi:10.1002/joc.4775 which has just hit the scientific literature, and which describes the latest version of the International Comprehensive Ocean-Atmosphere DataSet, the collection of marine weather data.

Everything about oldWeather has been free from the start: our ambition has always been to make the information in our logbooks openly available for anyone to use, and our results already have seen use in datasets, reanalyses, historical and personal projects, … But whenever anyone asks me “What are you doing with the results of this project?”, I’ve always answered “We’re going to put the new observations in ICOADS – to make them available for all future uses, in climate and other fields”. With ICOADS R3.0, we have finally achieved this: ICOADS is internally arranged in ‘decks’ (a reminder that data collection is older than the digital computer) – it now includes decks 249 “World War I (WW1) UK Royal Navy Logbooks” and 710 “US Arctic Logbooks” – clearly illustrated in figure 1.

I’ve been working as a scientist for a while now, but the publication of a new paper is still something of an event. Scientific papers come in many forms: some describe wild new ideas, brave experiments, or dramatic breakthroughs – this one is nothing like that; it just reports the work of many people, over several years, scavenging observations from wherever we can find them, systematising them, quality controlling them, analysing them, and now releasing them. It makes up for its lack of drama by being useful – the surface marine record is one of the most widely used datasets in all of climate: our observations are used directly in the monitoring datasets that measure climate change, they are assimilated in all reanalyses, they provide boundary conditions and validation for the models we use for predicting future change, and they provide calibration to palaeoclimate reconstructions of the deep past.

So from now on, if you’ve contributed to oldWeather, keep an eye out for any new climate results. Whether it’s a new global temperature record, a prediction of climate for the next generation, a study of changes in flood or drought risk, a government report on climate impacts and adaptation, … anything really. When you see it, square your shoulders and stand a little taller – that result owes something to you.

Better bad weather with oldWeather

A spaghetti-contour plot of a storm in 1916 as reconstructed by the 20th Century reanalysis.  On the left, without oldWeather observations, on the right with the oldWeather ships added.

A spaghetti-contour plot of a storm in 1916 as reconstructed by the 20th Century reanalysis. On the left, without oldWeather observations, on the right with the oldWeather ships added. The yellow dots mark observations used in the reconstruction – the additional yellow dots in the right image are from Royal Navy ship logs transcribed by oldWeather.

Next week sees an important event in the calendar of the observational climatology community: the ninth annual meeting of the Atmospheric Circulation Reconstructions over the Earth (ACRE) project, in Maynooth, Ireland. I’ll be there to talk about the new knowledge we are generating with oldWeather, and I thought I’d share a sneak preview here.

The picture above is a pair of contour plots: on a map contour lines mark places with the same height and they are used to show the shape and size of hills. Here the lines mark places with the same atmospheric pressure, and they show the size and shape of a valley in the atmosphere – an atmospheric depression – a storm. The picture is messy because this is also a spaghetti plot – I have 56 different maps of the same storm (the individual ensemble members of the 20th century reanalysis) and I’ve drawn all 56 in the same image.

In an ideal world we’d know exactly the size and shape of this storm, so all 56 maps would be exactly the same and the plot would show pin-sharp simple contours. We’re not there yet, but adding the oldWeather observations to the reconstruction has made things a lot better – our map of this storm is much more precise than it was before.

Too windy for Zeppelins

100 years ago today – 31st May 1916, saw the start of a major fleet action between the British and German navies: The battle of Jutland.

This is right in the middle of the period covered by the original oldWeather project, so you’d think we had all the logbooks and observations, at least from the British half of the battle, but alas, it’s not so. The Grand Fleet sounds impressive, and with as many as 40 major warships surely was impressive, but it didn’t travel much: The doctrine of ‘Fleet in being’ means that all those battleships stayed in port as a threatening influence rather than travelling to distant locations, and that puts them right at the bottom of our priority list for transcription, and we’ve never looked at them.

So we don’t have the Grand Fleet, but we can still reconstruct the weather of the battle, and our observations still make a major contribution to the reconstruction:

Weather (contours are pressure, streamlines show wind and temperature), as reconstructed by 20CR version 2c, for the battle of Jutland. Dots mark observations used to make the reanalysis - red dots are observations from oldWeather, yellow dots are observations from other sources

Plenty of our ships are contributing to that weather reconstruction – in port on the West coast, on patrol or convoy duty in the North Atlantic, and they are doing a good job describing the dominant weather feature – the low pressure moving into the Norwegian Sea. But the North Sea around Jutland is pretty bare of observations – two major fleets, and we have almost nothing from either of them. The reason we don’t have them is that we don’t need them – the European weather stations give us a pretty good picture of the weather anyway, and the ships were only out of port for two days, so they wouldn’t be a huge asset to science; but it’s still a pity from the historical perspective, we’ll keep an eye out for a future opportunity, both in the UK and in Germany.

The weather did, apparently, play a part in the battle – strong winds grounded the Zeppelin fleet that would otherwise have been scouting and bombing on 31st May – and it was one of those Zeppelins that engaged (on June 1st) the only member of our fleet to have participated in the action at all: HMS Fearless was not big enough to mix it with the battleships and battlecruisers; but she was there, and you can read the story we rescued from her logs at naval-history.net (and see the observations she made in the video above).

The battle had no winners – 9,823 men died, and 25 ships were sunk, including one (HMS Invincible) who’s story helped inspire the start of oldWeather.

neōn katalogos

It’s not all about the shiny and the new – we should appreciate, also, the virtues of the classics: In particular classic oldWeather, our original and ongoing project to rescue data from the US Government Arctic logbooks, which has now transcribed more than three million (3,000,000) weather observations.

All the contributors I could not tell nor name, nay, not though ten tongues were mine and ten mouths and a voice unwearying, but now I will tell the leaders of the ships and the ships in their order:”

  • Of the Albatross (1884); leelhat and Hanibal94 were captains, with steeleye and jd570b and Zovacor, with 569 more. They brought 150,734 weather observations, rich in pressures, temperatures, and wind directions.
  • Of the Albatross (1890); hurlock and Ravendrop were captains, with p3nguin53 and listritz and 1049 more. They brought 62,931 weather observations.
  • Of the Albatross (1900); Danny252, hurlock and pommystuart were captains, with HHTime, JanetET-S and wendolk with 482 more. They brought 57,991 weather observations.
  • Of the Bear, veteran of many campaigns; lollia paolina, gastcra and Hanibal94 were captains, with DennisO, jil and pommystuart, with 410 more. They brought 349,015 weather observations
  • Of the Concord; pommystuart and gastcra were captains, with Hanibal94 and MAPurves, and 1207 more. They brought 380,191 weather observations.
  • Of the Corwin; gastcra, pommystuart and lollia paolina were captains, with but 24 more. They brought 9,588 weather observations.
  • Of the Jamestown (1844); kimma001 was captain, with gastcra and Zovacor and 92 more. They brought 83,533 weather observations.
  • Of the Jamestown (1866); leelhat, Hanibal94 and kimma001 were captains, with 445 more. They brought 128,922 weather observations.
  • Of the Jamestown (1879); lollia paolina was captain, with gastcra, LouisaEvers, smith7748 and 475 more. They brought 93,696 weather observations
  • Of the Jamestown (1886); leelhat was captain, with lollia paolina with 385 more. They brought 82,624 weather observations.
  • Of the Jeannette; gastcra, Clewi and jil were captains, with with 67 more. They brought 42,982 weather observations and much knowledge of the ice.
  • Of the Patterson; Hanibal94, gastcra and asterix135 were captains, with helenj, avastmh and 101 more. They brought 334,146 weather observations.
  • Of the Perry; leelhat and Hanibal94 were captains, with exim_202, elizabeth_s, and rbertin1068, with 427 more. They brought 7,352 weather observations.
  • Of the Pioneer; Hanibal94 was captain, with gastcra and helenj and 86 more. They sought out 182,586 weather observations.
  • Of the Rodgers; leelhat was captain, with Hanibal94, avastmh and 50 more. They saved 19,718 weather observations from the fire.
  • Of the Rush; lollia paolina was captain, with leelhat and researchib with 368 more. They carried 25,174 weather observations.
  • Of the Thetis; lollia paolina was captain, with jil, leelhat, KookyBird and 716 more. They brought 220,493 weather observations.
  • Of the first Unalga; Hanibal94 and propriome were captains, with gastcra and Caro, with 92 more. They brought 136,001 weather observations
  • Of the Second Unalga; Hanibal94 was captain, with gastcra, Caro, and 36 more. They brought 10,395 weather observations
  • Of the Vicksburg; leelhat and lollia paolina were captains, with 393 more. They brought 357,525 weather observations
  • Of the Yorktown; Lekiam and lollia paolina were captains, with gastcra with 737 more. They brought 279,546 weather observations
  • Of the Yukon; gastcra and Hanibal94 were captains, with 80 more. They brought 31,111 weather observations

Collectively awesome

We launched the new www.oldWeather.org a month ago, which means that the volunteers using the site have provided quite a bit of new data, and we can start to analyse it. This is one of my favourite moments in any project – first blood, when we get the initial sense of what we’ve got, how it’s going to work, what we can learn from it.

One of the golden rules of statistical analysis is “first plot the data” – always start by making a simple visualisation, so you can be sure you understand what you’ve got, and you’re not missing anything obvious. But the oldWeather data is not easy to plot: the database contains records from hundreds of people making thousands of annotations on dozens of different logbook pages; what, exactly, should we look at?

So I’ve taken inspiration from Listen to Wikipedia, and asked ‘what would it look like if we could see (and hear) the data as it came in – in (accelerated) real time?’ The video below shows every contribution to www.oldWeather.org over a three hour period on December 3rd 2015. The number of pages shown is the number of volunteers contributing at each point in time. Each box drawn, and sound played, is one annotation, a contribution to the project. Blue boxes contain weather data, yellow boxes ship positions, orange boxes dates, and red boxes other events; pages that have moved on to the transcription phase have grey boxes.

Listen to oldWeather: December 3rd 2015.

December 3rd was when we launched the new site, so we can see a large change in the number of people participating as they learn about the launch. It’s instantly clear that it’s working – we are collecting annotations and transcriptions in quantity, as we hoped. There is much to be learned from careful examination of visualisations like this, but mostly I think it shows the power of the project – the awesome capability of collective public science.


Note that this shows only data from the new Panoptes version of oldweather. It does not include data from the whaling site – I’m still working on that.

A double century of centennial records

Australian cricketer Herbie Collins

Very few achieve 200

What was life really like in the Royal Navy 100 years ago? Where did the ships go? How did the crews spend their days? What were the noteworthy, and the routine, events in their lives?

The Royal Navy logbooks we worked through in the original version of oldWeather provide a uniquely powerful insight into these questions – they are primary records of exactly what happened. But they are not easy to use – hard to read, not indexed, or searchable, and often full of obscure technical language.

When we transcribed the weather in the logs we caught many of the historical events as well, and we were able to make a formatted history file for each ship – linking each logbook page image to transcribed events and information from that day, and we assembled those ship histories on our partner website naval-history.net.

Those history files made from the raw transcriptions are a good start, but they are far from perfect: Some events we caught cleanly, some only half-stopped, and our decisions on what to leave were usually good, but not always. So our team of volunteer editors have been working through the raw files editing and improving them: reviewing the decisions made in the heat of transcription, correcting mistakes, merging multiple versions, adding missing events, incorporating pithy commentary and expert summaries of key points, and adding maps of the ship journeys.

As so often with oldWeather, this has been a lot of work – a major task tackled with care and patience by an increasingly-expert team of volunteers. Their achievement is clear to see, comparing the edited histories (in bold on this page) with the raw versions shows a huge improvement in clarity, accuracy, completeness, and value. And the score of the editing team has mounted steadily – they have just released their 200th edited ship history.

To get to 200 ships edited is an awesome achievement, but of course we still have power to add: HMS Cricket is done, as are Cardiff, India, New Zealand and Sydney; but Dunedin, Durban, Perth, Delhi and Capetown are yet to be conquered. Are you available for selection?

There she blows!

on_lookout
We’ve spotted the first signs of new ships on oldWeather. We are partnering with the New Bedford Whaling Museum to read and transcribe their collection of whaling ship logs. Whalers are not just historically important and interesting, their hunting grounds were well into the Arctic, so we’re hoping to get some particularly good sea-ice records from these logs.

These new logs look quite different to the government ship logs we are used to – they don’t have a uniform page structure with tables of data – so we are adding them in their own separate section, and the way to mark the page and enter data is a bit different. Please have a look at the new logs, and experiment with the new interface – it will take a while to get used to, but help and advice is available, as always, through the forum.

We are also adding one other new thing: a way to add our own value to individual log images. We’ve always concentrated on transcribing the contents of the logs, but up to now we have not had a good way to add our own comments to them: or mark interesting pages, or flag mistakes made by the log-keeper, or tag pages to make groups. We are adding the ‘Talk’ system (as featured on other Zooniverse projects) to oldWeather to let us do this, and I’m really looking forward to what we can do with it.

If you like the old oldWeather, don’t worry, it’s still there as before, but now we have the added opportunity of oldWeather-whaling.

Floating Forests

Rockfish around kelp Monterey Bay Aquarium
Kelp is, perhaps, more important than you might guess: Not only does it thicken your toothpaste, it supports whole marine ecosystems where it grows. It is important enough to have a Zooniverse project devoted to watching it grow, through Landsat images.

Satellite imagery is a great way to monitor the world – providing frequent, comprehensive pictures of the whole planet. But in-situ observations also have their place: people on the ground, interacting directly with the system being monitored, can often provide a detail and precision that the satellite records lack.

One of the unexpected joys of oldWeather is that it provides in-situ observations of a vast range of different things. Most often kelp is mentioned in the logs simply as a highlight of a day at sea:

Fine weather. Light breeze from South. At 2.30 took in and furled the sails. Passed a piece of kelp. [Yorktown, May 1892.]

4 to 8 a.m. Overcast but pleasant. Airs from NE. Passed some kelp. [Same, a day later.]

Sometimes we see the kelp interacting with the environment:

Saw a large patch of kelp with a dozen seals hauled out on it. [Rush, June 1891]
Sighted a whale and a bunch of kelp. [Yorktown, May 1892]
At daylight passed much drift kelp, to one large batch a boulder about 3 ft in diameter was attached [Patterson February 1885]

Occasionally they do seem to be actively surveying it:

Steaming along at various speeds, locating outer limit of kelp beds off La Jolla, fog gradually increasing, log hauled in. [Pioneer, spring 1923].
Continued sounding passing inside of Aleks Rock. No signs of kelp were seen. [Patterson].

But the most interesting mentions feature it as a hazard to navigation. I suspect most of our log-keepers would see definite benefits in any decline of kelp:

Slowed down a few minutes on account of kelp. [Concord, August 1901]
4:15 Kelp ahead, full speed astern … Ran about 1/2 mile SWxW and ran into kelp again. Wreck bore E 1/2 N. Stopped and backed away from it [Patterson].
found four masted schooner “Watson A. West” in the kelp on the outer edge of the shoal, broadside to the beach, close in and in dangerous position [Unalga, October 1916].
Between six and seven o’clock, patent log registered only 3.9 knots: hauled in rotator and found it fouled with kelp; cleared it, and allowed 2.6 knots for the discrepancy. [Commodore Perry, July 1896].
Found spar buoy #16, two hundred yards NE of true position and in kelp. [Commodore Perry, February 1903].
At 10.36 sighted what appeared to be a pinnacle rock. Stopped ship lowered boat and after inspection the object proved to be a much worn spar, heel up, with kelp attached. [Yorktown, June 1894].

We don’t have that many observations of kelp – we probably won’t be much help to the Floating Forests team mapping the distribution, but we do have our own viewpoints to add – aspects that the satellites will never see:

Thanks to all those who’ve transcribed such records from the logs.

Mentioned in despatches

We're in there: Figure 9 from Cram et al. 2015.

We’re in there: Figure 9 from Cram et al. 2015.

oldWeather does not produce that many research papers: The process of science is changing fast at the moment – moving away from individual researchers and small groups working independently, and towards much larger consortia working together with big datatsets on major problems – fewer small papers, more big science. As a major data provider, closely linked into the international climate research community, we fit nicely into this new model.

But the academic papers industry is still out there, and now and again one appears that involves us directly. One of the major datasets we contribute to is the International Surface Pressure Databank, and Tom Cram and colleagues have a new paper about the data (including our contributions) and how it’s assembled and distributed through the excellent Research Data Archive at NCAR. It’s open access – free for all to read (here’s the link) – why not check out the oldWeather references and see exactly how our results are being used by researchers.

International Archives Day

Today, 9th June, is International Archives Day, where archives around the world unite to celebrate their remarkable resources by sharing an iconic image from their collection. oldWeather is no archive, but we certainly do appreciate them, and we aim to add value to their collections through our transcription work.

So here’s an image in which we saw some special value – this is from the collection of the UK National Archives: The logbook of HMS Tarantula, from August 1920. Thanks to generations of archivists for preserving these records for use today; without their skill and dedication this record, and all the others from which we have learned so much, would have been lost forever.

Logbook of HMS Tarantula, August 1920

On Sunday 29th Aug. aneroid was compared with Standard Mercurial in HMS Carlisle and found to be reading 0.26 ins. too low. An examination of the aneroid showed it to contain an ants’ nest and be otherwise defective. This aneroid is being surveyed and until a new one is received the Captain’s private Baragraph will be used. This barograph was corrected on 29th August.