Archive by Author | Philip

In search of lost weather

Imagine you have a free hour, one wet weekend, so you settle down to a little light reading:

That’s 8,000 pages. If you read them all again the following weekend (to catch the subtleties that escaped you the first time), and then again, and again, and again, and again, and again; you’d still be 6,000 pages short of matching the work we’ve done reading the logbooks of USS Bear.

So congratulations to lollia paolina, gastcra, Hanibal94, DennisO, jil, pommystuart, LarryW, smith7748, tastiger, and every one of the 402 other crew members – on an achievement of epic proportions: From the 20,930 pages of the Bear’s logs (each read 3 times, remember), we’ve recorded 349,015 weather observations, each with several components (wind speed, barometer, air temperature, etc.) making more than 2.9 million data points.

And of course it’s not just weather, those logs also provided 22,957 dates, 6427 longitudes, 2947 people, 189 animals, 19,489 places, and more … 8,872,438 characters in all.

As before, I’ve used the transcriptions to make a movie version. But the sheer size of the achievement causes problems even here: I thought that a maximum movie length of 10,000 seconds was way more than enough, but not for the Bear. So while I sort that out, here’s just the first installment: 1884-1890.

A daughter:

If you look at you might get a feeling of deja-vu – a sense that you’ve seen something similar before.

oldWeather is not yet four, a bit young to be having children. But that’s four internet years, so maybe it’s time: We’ve contributed DNA to plenty of other projects, but Weather Detective is our first direct descendant.

As with all children it’s a separate person – with its own science team, volunteer community, logbooks, and interface. They are friends, but they will be doing things differently from us (and we’re not too old to learn from their approach).

Congratulations to Christa Pudmenzky, USQ, ABC, and all involved. We wish you fair winds and a following sea.

Arts and Music

Vice-Admiral robert Fitzroy

Robert Fitzroy

Charles Darwin

Charles Darwin

Noted naval exploring captain; surveyor and hydrographer; Vice-Admiral; pioneering weather forecaster, founder of the Met Office (first Meteorological Statist to the Board of Trade); governor of New Zealand. Robert Fitzroy was a man of parts, who made a great impact on the world.

But he sits in the shadow of Charles Darwin, who accompanied him on his 1831-6 circumnavigation in HMS Beagle. This September (8-13) the Progress Theatre in Reading, jointly with WAM, the Festival of Weather, Arts and Music, is staging Juliet Aykroyd’s play ‘Darwin and Fitzroy’. Each day, the performance is supported by a side event celebrating Fitzroy’s life and influence, and on Thursday September 11th that side event is me, talking about oldWeather. (We have not put the Beagle on oldWeather (yet) but I’ve got her weather observations from the 1831-6 circumnavigation).

Why not come along? Tickets are £10 (£8 concessions) – it’s an amateur performance, that’s just to cover the theatre costs. There is a different side-event on each day, but if you choose to come on Thursday, I’ll be happy to see you.

New crew members at has been a core partner since the very start of oldWeather: That’s where we publish all the historical events we find in the logbooks – these are being carefully edited into ship histories (UK and US) by a team of volunteers. is also run by a group of dedicated (and expert) amateurs, and adding the vast quantity of new material we are transcribing is a lot of work. Some of our volunteer editors have taken on the additional responsibility for this, and Gordon (who runs has added them to his crew:

  • Navigator Maikel prepares and publishes completed RN and US ship histories. In addition, he has published a large backlog of RN ship histories that were waiting for the addition of corrected navigation data and Journey Plotter maps (Maikel built the Journey Plotter program in his spare time), and published new US logs ready for editing. Not only that; he has created individual pages for the ships in the overhauled US index, and made many behind the scenes improvements, such as redirection pages and improvements to the RN index page.
  • Paymaster Janet sends out transcribed logs for editing and receives the completed edits in return. She checks them for style and obvious errors before making them available for Maikel to publish on the Naval-History website. She also keeps track of the status of all available logs – waiting to be edited, being edited, complete and published, or awaiting publication or updates; she also keeps record of which logs have been reserved by which editors. Janet also has checked the new US ship files for errors and supplied missing service histories.
  • Leading Stoker Howard does a lot of background work. He converts oldWeather’s plain text transcription files into the formatted Word files that Janet sends to the editors. He also creates many Journey Plotter maps for those editors who request his help. He corrects or adds latitudes/longitudes as necessary that allow Journey Plotter to produce maps – both of the ship’s full voyage and of more detailed sub-sections. Leading Stoker is his choice of rank; he keeps the fires burning.
  • Leading Telegraphist/Writer Caro advertises, using Naval-History’s accounts, new and updated ship histories on the social media, getting the word out on Facebook, Twitter and Google+. She also rewrites important sections of text for the Naval-History site and supplies artwork on occasion. She created the new logo for that is starting to appear on the site and in social media.

The atmospheric conditions have been very unfavourable lately

It rained, and it rained, and it rained. Never in all the UK records, and they go back goodness knows how long, 103 years, or even 247 years? – never had we seen so much rain. Days and days and days.

The UK is too small to have its own weather, we participate in the weather of the North Atlantic region, or indeed the whole world. So to understand why the last UK winter was record-breakingly wet, we need to look at atmospheric behaviour on a large scale. I’ve turned to MERRA – an hour-by-hour reconstruction of the whole atmosphere – and made the video of northern hemisphere weather above. (There’s a lot going on, I recommend watching it in full-screen, press the ‘X’ on the control bar).

The key feature is the sequence of storms that spin off North America, and then head out into the North Atlantic in the Prevailing Westerly Circulation (anti-clockwise in the projection of the video). In November these storms mostly follow the standard path northwest to Greenland or Iceland and Scandinavia, but in December the weather changes: North America becomes much colder and the path of the storms moves south, driving the bad weather straight at the UK. This persistent pattern of a cold North America and southerly Atlantic Storm Track is the outstanding feature of the winter, and it shows up even more clearly a bit higher in the atmosphere – the Upper-Level Winds have a simpler structure, as they are not complicated by contact with the Earth’s surface.

Wind and temperature at 300hPa (30,000 feet or 9km altitude) of winter 2013/4 centred on the North Pole. Data are from MERRA.

The temperature difference between cold polar air and warmer southerly air stirs up an overturning circulation, and the rotation of the Earth turns this into a strong anti-clockwise (westerly) rotating wind – the Polar Vortex. As early as 1939, Carl-Gustaf Rossby realised that this circulation would not be smooth and stable, and the characteristic undulations (Rossby waves) have a major impact on our weather. It’s a series of these waves that push cold polar air much further south than usual over eastern North America, producing a Very Cold Winter in those parts, shifting the storm tracks south and causing the wet, stormy weather in the UK.

But of course I’m not really interested in modern weather – that’s too easy, with ample satellite observations and tremendous tools like MERRA to show us what’s going on. The challenge is in providing the long-term context needed to understand these modern events – is there a consistent pattern, if not, what’s changed. And it just happens that a previous Markedly Wet UK Winter occurred 99 years earlier, in 1914/5, and we’ve been rescuing logbook observations for that time so we can use them to make improved studies of that winter.

Surface weather of winter 1914/5 centred on the North Pole: Sea-ice, (white shading), pressure (contours), wind (vectors), temperature (colours) and rain/snow (black shading). Data are from the 20th Century reanalysis (scout run): yellow dots mark available surface pressure observations, and fog masks regions where the analysis is very uncertain (because there are too few nearby observations).

This time we use the Twentieth Century Reanalysis (more precisely a test version of 20CR updated to benefit from oldWeather-rescued observations). In some areas (most obviously the high Arctic) there are no observations so the analysis is too uncertain to be useful, but over the US, UK, and Atlantic storm-track region we can reconstruct the weather of that year.

Again, the picture is clearer if we look at the upper-level circulation:

Wind and temperature at 300hPa (30,000 feet or 9km altitude) of winter 1913/4 centred on the North Pole. Data are from the 20th Century reanalysis (scout run): yellow dots mark available surface pressure observations, and fog masks regions where the analysis is very uncertain (because there are too few nearby observations).

Do we see the same picture in 1914/5 as in 2013/4? Reality tends to be somewhat messier than the simple explanations that scientists treasure – but I think we do see the same pattern: a persistent tendency for cold, polar air to extend south over North America, and a North Atlantic storm track shifted to the south.

We can say quite precisely what happened last winter, and (thanks, in part, to oldWeather) how last winter compared to previous Exceptional Winters. However the obvious follow-on question is ‘Why did the polar vortex behave like that, and can we predict when it’s going to do it again? We’re still working on that one.

A talk by Kevin in Washington


At the stroke of noon next Wednesday (May 14th), Kevin is giving a talk on our work at the Archives in Washington DC. All welcome, but note the details on the flyer.

A number considerably above the average

To achieve a million once may be regarded as good fortune; to do so twice looks like carefulness, skill, enthusiasm, and dedication.

We’ve been working on the US Arctic logbooks since July 2012, and in that time we’ve rescued one million weather observations (actually 1,081,641 as of March 11th). That’s an awesome achievement – even more awesome than it seems at first, as a single weather observation is a complex object containing a lot of information. Our 1,081,641 observations comprise:

  • 960,221 air pressures,
  • 2,782,755 temperatures (950,351 dry-bulb, 765,270 wet-bulb, 597,556 barometer and 469,578 sea),
  • 1,020,223 present weather codes,
  • 1,049,692 wind directions,
  • 1,019,268 wind forces,
  • 864,586 cloud fractions, and
  • 846,859 cloud types.

To record all that, we’ve typed 21,511,795 characters – about 20 times the length of a typical novel.

However you look at it, that’s a tremendous achievement.

Code-breaking and oldWeather

Marcus Cocceius Nerva Caesar Augustus. 13th Caesar.

Marcus Cocceius Nerva Augustus. 13th Caesar.

Jr nyjnlf unir na rlr gb gur shgher, urer ng byqJrngure, naq jr’ir abgvprq bhe sevraqf ng gur nepuvirf cubgbtencuvat fbzr arjre ybtf – H.F. fuvcf sebz gur zvq-20gu praghel – nf jryy nf gur byqre barf jr’er hfrq gb. Gur nepuvirf ner abg cubgbtencuvat gurz sbe hf – gurl ner cneg bs nabgure cebwrpg – ohg gurl ner trarebhf crbcyr, naq gurl jvyy tvir hf n pbcl bs gur ybt obbx vzntrf vs jr jnag gurz. Qb jr jnag gurz?

Jryy bs pbhefr jr qb – bhe ybir sbe ybtobbxf arire snvyf, naq jr ner qrgrezvarq gung gur xabjyrqtr va gurz jvyy abg cnff njnl. Ohg jr pna’g qb rirelguvat evtug abj. Fb jr qb, hasbeghangryl, unir gb or frafvoyr, naq frg cevbevgvrf, naq lnqn lnqn lnqn.

Gur zbfg vzcbegnag guvat gb xabj nobhg nal cbffvoyr arj ybtf, vf jurgure nalbar unf ybbxrq ng gurz orsber: unir gurve jrngure bofreingvbaf nyernql orra genafpevorq? Gb svaq bhg, jr arrq gb ybbx va VPBNQF (gur vagreangvbany qngnonfr jurer jr fgber bhe jrngure erpbeqf) naq frr jung gurer vf sebz nal fuvc jr ner vagrerfgrq va – ohg gurer’f n pngpu.

Zbfg bs hf ersre gb n fuvc ol vgf anzr; bapr gur bjare unf qrpvqrq gb anzr n fuvc UZF Jbatnaryyn gura gur erfg bs hf fubhyq whfg sbyybj nybat. Bhe cerqrprffbef, ubjrire, jub qvtvgvfrq fbzr ybtobbx jrngure erpbeqf qrpnqrf ntb, qvq abg nterr. Creuncf gur anzrf jrer gbb ybat gb svg bagb gur Ubyyrevgu pneqf gurl hfrq sbe qngn fgbentr, ohg jungrire gurve ernfba gurl hfrq pbqr ahzoref nf fuvc vqragvsvref. Fb vs jr jnag gb svaq gur erpbeqf bs gur HFF Nexnafnf va VPBNQF, jr arrq gb xabj gung ure pbqr ahzore vf 01033 – naq, va trareny, jr qba’g.

Ohg gur byqJrngure grnz ner n pncnoyr ohapu, naq Xriva naq Znex ner sngubzvat guvf zlfgrel. Jvgu uryc sebz VPBNQF rkcregf ng APQP naq ABPF gurl ner qrpbqvat gur VPBNQF vqragvsvref fb jr pna yvax gur byq bofreingvbaf jvgu gur Nepuvir’f ybtobbxf naq cvpx bhg gubfr fgvyy haernq.

Certificated (2)


Thanks again to the sponsors, organisers, and judges, of the 2013 International Data Rescue Award in the Geosciences, for adding to our project laurels.

Old New Zealand, HMS New Zealand, & new New Zealand

Atmospheric pressure along the route sailed by HMS New Zealand in 1919. The blue band shows the range of our estimates before oldWeather, the black points the new observations we provided, and the red band the revised analysis range incorporating our observations.

Atmospheric pressure along the route sailed by HMS New Zealand in 1919. The blue band shows the range of our estimates before oldWeather, the black points the new observations we provided, and the red band the revised analysis range incorporating our observations.

This week, atmospheric scientists are gathering in Queenstown, New Zealand, for the fifth general assembly of the SPARC program (Stratosphere-troposphere Processes And their Role in Climate). We’ve mentioned New Zealand before: both as a country who’s isolation means that its historical weather is poorly documented, and as a Battlecruiser in the original oldWeather fleet. In September 1919 the two met: the battlecruiser visited the country, giving us an opportunity to make a major improvement in reconstructing the climate of the region.

As we showed back in October, we’re now re-doing our analysis of global weather, so we can see exactly how much the observations we’ve recovered from HMS New Zealand have improved our knowledge of the climate of New Zealand (the country). The figure above (made for the SPARC meeting) shows our estimates of the weather in each region visited by HMS New Zealand during her circumnavigation in 1919: blue for before oldWeather, and red a new revision using our observations. The width of the band indicates uncertainty – narrower is better – and the improvement we’ve made is very large.


Get every new post delivered to your Inbox.

Join 1,389 other followers