Archive | Data RSS for this section

Crossovers (2)

oldWeather has its own venue for high-quality science gossip, but in the late nineteenth century there were no internet forums; instead they had a printed magazine: Hardwicke’s Science Gossip: A Monthly Medium of Interchange and Gossip for Students and Lovers of Nature.

Today we’d call those ‘Students and Lovers of Nature’ scientists, so our friends at the Zooniverse have set up another transcription project, to read Science Gossip and related magazines, and find out what the citizen scientists of 100 years ago were interested in.

There is one thing in particular that we are also interested in – it’s a safe bet that the pages of those magazines contained information on the weather of the time; comments and perhaps original observations. At oldWeather we specialise in ship’s logbooks, but we are not fussy, and we value weather observations from any source. So we’ve added another tag to this new project for marking pages: #oldweather for any weather observations that turn up.

Thanks to @VanHyningV, @SiobhanLeachman, our own Jil, and the whole Science Gossip project community, for this Zooniverse/documentary records/historic weather crossover.

Crossovers (1)

Florence Augusta Merriam Bailey was an eminent ornithologist round about a century ago. The Smithsonian have her diary from 1887, and are running a volunteer project to transcribe it. Despite the strong hints given in the printed page headers of her diary, Florence rarely included weather observations, but there is at least one:

Minus 26 degrees (presumably F), and down to minus 30 in the valley. That’s pretty cold even for New York state (where Florence was at the time). Why so cold? Now, that’s something we can help with, with our rich collection of historical observations and renalysis. I don’t usually do forecasts (there’s another part of the Met Office for that), but as this one’s for 128 years ago maybe we can stretch a point:

Forecast for Monday, January 3rd, 1887: The high pressure currently over the central United States will strengthen and move eastward, bringing clear skies, northerly winds, and very cold weather.

A forecast for Florence Bailey in 1887: Uses data from the Twentieth Century Reanalysis ( - version 2c) and shows the places where we have observations (yellow dots) sea-level pressure anomalies (contours: solid=low pressure and dashed=high), and temperature anomalies (red=unusually hot, blue=unusually cold).

Thanks to @SiobhanLeachman, @MeghaninMotion, @TranscribeSI, and Florence Bailey herself; for this citizen science/documentary records/historical weather crossover study.

A history of the world in 1,399,120,833 observations

Locations of the weather (surface pressure) observations being used to reconstruct global weather and climate: Observations coverage (1851-2008) in version 3.2.9 of the International Surface Pressure Databank (that used in the Twentieth Century Reanalysis version 2c).

There’s a lot of history hiding in even purely scientific datasets. This movie shows just the locations of the 1.4 billion observations in the International Surface Pressure Databank (1851-2008), and in it I think I can see:

  1. The constraints on sailing-ship trade routes imposed by the global wind fields.
  2. The transition from sail to steam in shipping (late nineteenth century).
  3. The opening of the Suez canal in 1869 (01:30).
  4. The Famous Arctic voyage of Nansen’s Fram (03:20).
  5. The heroic age of Antarctic exploration (starting at about 04:00).
  6. The opening of the Panama canal in 1914 (05:10).
  7. The first world war (05:10).
  8. The second world war (07:00).
  9. Major administrative changes in India (08:00).
  10. The introduction of drifting buoys (1978: 10:20)
  11. And, sadly, a reduction in observations coverage in the last couple of decades as participation in the Voluntary Observing Fleet declines.

Of course these observations are not all that were made. Many more historical observations exist (on paper, or in restricted access collections), but these are the ones that are currently available to science. The process of rescuing the observations has also left its mark on the coverage – including right at the beginning of the video, where the coverage of ship observations reduces sharply in 1863 – the end of Matthew Fontaine Maury‘s pioneering data collection work. Various subsequent rises and falls in coverage result from the work of many other scientists and teams; including, of course, a large group of Royal Navy ship observations in the period around the First World War (starting about 05:00) clearly distinguishable just from their locations, as Naval ships move in a quite different pattern from commercial shipping. (Our US Arctic ships are not in this database yet – they will be in the next version).

Seas of red


Into the Top500

This visualisation, comparing two reconstructions of the weather of 1918 (each using the oldWeather observations), took four supercomputers to make: The blue contours are from the ERA20C reanalysis, run on a pair of IBM Power775s at ECMWF; the red contours are from 20CRv2C, run on Hopper - NERSC's Cray XE6; and the post-processing and rendering was done on Carver, NERSC's iDataPlex.

The Met Office, where I work, has just finalised an agreement to buy a new supercomputer. This isn’t that rare an event – you can’t do serious weather forecasting without a supercomputer and, just like everyday computers, they need replacing every few years as their technology advances. But this one’s a big-un, and the news reminded me of the importance of high-performance computing, even to observational projects like oldWeather.

To stand tall and proud in the world of supercomputing, you need an entry in the Top500: This is a list, in rank order, of the biggest and fastest computers in the world. These machines are frighteningly powerful and expensive, and a few of them have turned part of their power to using the oldWeather observations:

Two other machines have not used our observations yet (except for occasional tests), but are gearing up to do so in the near future:

My personal favourite, though, is none of these: Carver is not one of the really big boys. An IBM iDataPlex with only 9,984 processor cores, it ranked at 322 in the list when it was new, in 2010, and has since fallen off the Top500 altogether; overtaken by newer and bigger machines. It still has the processing power of something like 5000 modern PCs though, and shares in NERSC’s excellent technical infrastructure and expert staff. I use Carver to analyse the millions of weather observations and terabytes of weather reconstructions we are generating – almost all of the videos that regularly appear here were created on it.

The collective power of these systems is awe-inspiring. One of the most exciting aspects of working on weather and climate is that we can work (through collaborators) right at the forefront of technical and scientific capability.

But although we need these leading-edge systems to reconstruct past weather, they are helpless without the observations we provide. All these computers together could not read a single logbook page, let alone interpret the contents; the singularity is not that close; we’re still, fundamentally, a people project.

43 years in the North

The voyages of the Bear, Corwin, Jeannette, Manning, Rush, Rodgers, Unalga II, and Yukon

Today is the fourth birthday of oldWeather, and it’s almost two years since we started work on the Arctic voyages. So it’s a good time to illustrate some more of what we’ve achieved:

I’m looking at the moment at the Arctic ships we’ve finished: Bear, Corwin, Jeannette, Manning, Rush, Rodgers, Unalga II, and Yukon have each had all of their logbook pages read by three people; so it’s time to add their records to the global climate databases and start using them in weather reconstructions. From them we have recovered 43 ship-years of hourly observations – more than 125,000 observations concentrating on the marginal sea-ice zones in Baffin Bay and the Bering Strait – an enormous addition to our observational records.

The video above shows the movements of this fleet (compressed into a single year). They may occasionally choose to winter in San Pedro or Honolulu, but every summer they are back up against the ice – making observations exactly where we want them most.

So in our last two years of work, we’ve completed the recovery of 43-ship years of logbooks, and actually we’ve done much more than that: The eight completed ships shown here make up only about 25% of the 1.5 million transcriptions we’ve done so far. So this group is only a taster – there’s three times as much more material already in the pipeline.


Sometimes there is just no word powerful enough to describe the achievements of oldWeather.

Back in March we reached a million, and since then we’ve powered on from that milestone, now having added an additional five hundred thousand observations to our tally. That’s two new observations every minute, night and day, 7 days a week: Come rain or shine; snow or sleet; ice, fire, or fog.

The atmospheric conditions have been very unfavourable lately

It rained, and it rained, and it rained. Never in all the UK records, and they go back goodness knows how long, 103 years, or even 247 years? – never had we seen so much rain. Days and days and days.

The UK is too small to have its own weather, we participate in the weather of the North Atlantic region, or indeed the whole world. So to understand why the last UK winter was record-breakingly wet, we need to look at atmospheric behaviour on a large scale. I’ve turned to MERRA – an hour-by-hour reconstruction of the whole atmosphere – and made the video of northern hemisphere weather above. (There’s a lot going on, I recommend watching it in full-screen, press the ‘X’ on the control bar).

The key feature is the sequence of storms that spin off North America, and then head out into the North Atlantic in the Prevailing Westerly Circulation (anti-clockwise in the projection of the video). In November these storms mostly follow the standard path northwest to Greenland or Iceland and Scandinavia, but in December the weather changes: North America becomes much colder and the path of the storms moves south, driving the bad weather straight at the UK. This persistent pattern of a cold North America and southerly Atlantic Storm Track is the outstanding feature of the winter, and it shows up even more clearly a bit higher in the atmosphere – the Upper-Level Winds have a simpler structure, as they are not complicated by contact with the Earth’s surface.

Wind and temperature at 300hPa (30,000 feet or 9km altitude) of winter 2013/4 centred on the North Pole. Data are from MERRA.

The temperature difference between cold polar air and warmer southerly air stirs up an overturning circulation, and the rotation of the Earth turns this into a strong anti-clockwise (westerly) rotating wind – the Polar Vortex. As early as 1939, Carl-Gustaf Rossby realised that this circulation would not be smooth and stable, and the characteristic undulations (Rossby waves) have a major impact on our weather. It’s a series of these waves that push cold polar air much further south than usual over eastern North America, producing a Very Cold Winter in those parts, shifting the storm tracks south and causing the wet, stormy weather in the UK.

But of course I’m not really interested in modern weather – that’s too easy, with ample satellite observations and tremendous tools like MERRA to show us what’s going on. The challenge is in providing the long-term context needed to understand these modern events – is there a consistent pattern, if not, what’s changed. And it just happens that a previous Markedly Wet UK Winter occurred 99 years earlier, in 1914/5, and we’ve been rescuing logbook observations for that time so we can use them to make improved studies of that winter.

Surface weather of winter 1914/5 centred on the North Pole: Sea-ice, (white shading), pressure (contours), wind (vectors), temperature (colours) and rain/snow (black shading). Data are from the 20th Century reanalysis (scout run): yellow dots mark available surface pressure observations, and fog masks regions where the analysis is very uncertain (because there are too few nearby observations).

This time we use the Twentieth Century Reanalysis (more precisely a test version of 20CR updated to benefit from oldWeather-rescued observations). In some areas (most obviously the high Arctic) there are no observations so the analysis is too uncertain to be useful, but over the US, UK, and Atlantic storm-track region we can reconstruct the weather of that year.

Again, the picture is clearer if we look at the upper-level circulation:

Wind and temperature at 300hPa (30,000 feet or 9km altitude) of winter 1913/4 centred on the North Pole. Data are from the 20th Century reanalysis (scout run): yellow dots mark available surface pressure observations, and fog masks regions where the analysis is very uncertain (because there are too few nearby observations).

Do we see the same picture in 1914/5 as in 2013/4? Reality tends to be somewhat messier than the simple explanations that scientists treasure – but I think we do see the same pattern: a persistent tendency for cold, polar air to extend south over North America, and a North Atlantic storm track shifted to the south.

We can say quite precisely what happened last winter, and (thanks, in part, to oldWeather) how last winter compared to previous Exceptional Winters. However the obvious follow-on question is ‘Why did the polar vortex behave like that, and can we predict when it’s going to do it again? We’re still working on that one.

A number considerably above the average

To achieve a million once may be regarded as good fortune; to do so twice looks like carefulness, skill, enthusiasm, and dedication.

We’ve been working on the US Arctic logbooks since July 2012, and in that time we’ve rescued one million weather observations (actually 1,081,641 as of March 11th). That’s an awesome achievement – even more awesome than it seems at first, as a single weather observation is a complex object containing a lot of information. Our 1,081,641 observations comprise:

  • 960,221 air pressures,
  • 2,782,755 temperatures (950,351 dry-bulb, 765,270 wet-bulb, 597,556 barometer and 469,578 sea),
  • 1,020,223 present weather codes,
  • 1,049,692 wind directions,
  • 1,019,268 wind forces,
  • 864,586 cloud fractions, and
  • 846,859 cloud types.

To record all that, we’ve typed 21,511,795 characters – about 20 times the length of a typical novel.

However you look at it, that’s a tremendous achievement.

Code-breaking and oldWeather

Marcus Cocceius Nerva Caesar Augustus. 13th Caesar.

Marcus Cocceius Nerva Augustus. 13th Caesar.

Jr nyjnlf unir na rlr gb gur shgher, urer ng byqJrngure, naq jr’ir abgvprq bhe sevraqf ng gur nepuvirf cubgbtencuvat fbzr arjre ybtf – H.F. fuvcf sebz gur zvq-20gu praghel – nf jryy nf gur byqre barf jr’er hfrq gb. Gur nepuvirf ner abg cubgbtencuvat gurz sbe hf – gurl ner cneg bs nabgure cebwrpg – ohg gurl ner trarebhf crbcyr, naq gurl jvyy tvir hf n pbcl bs gur ybt obbx vzntrf vs jr jnag gurz. Qb jr jnag gurz?

Jryy bs pbhefr jr qb – bhe ybir sbe ybtobbxf arire snvyf, naq jr ner qrgrezvarq gung gur xabjyrqtr va gurz jvyy abg cnff njnl. Ohg jr pna’g qb rirelguvat evtug abj. Fb jr qb, hasbeghangryl, unir gb or frafvoyr, naq frg cevbevgvrf, naq lnqn lnqn lnqn.

Gur zbfg vzcbegnag guvat gb xabj nobhg nal cbffvoyr arj ybtf, vf jurgure nalbar unf ybbxrq ng gurz orsber: unir gurve jrngure bofreingvbaf nyernql orra genafpevorq? Gb svaq bhg, jr arrq gb ybbx va VPBNQF (gur vagreangvbany qngnonfr jurer jr fgber bhe jrngure erpbeqf) naq frr jung gurer vf sebz nal fuvc jr ner vagrerfgrq va – ohg gurer’f n pngpu.

Zbfg bs hf ersre gb n fuvc ol vgf anzr; bapr gur bjare unf qrpvqrq gb anzr n fuvc UZF Jbatnaryyn gura gur erfg bs hf fubhyq whfg sbyybj nybat. Bhe cerqrprffbef, ubjrire, jub qvtvgvfrq fbzr ybtobbx jrngure erpbeqf qrpnqrf ntb, qvq abg nterr. Creuncf gur anzrf jrer gbb ybat gb svg bagb gur Ubyyrevgu pneqf gurl hfrq sbe qngn fgbentr, ohg jungrire gurve ernfba gurl hfrq pbqr ahzoref nf fuvc vqragvsvref. Fb vs jr jnag gb svaq gur erpbeqf bs gur HFF Nexnafnf va VPBNQF, jr arrq gb xabj gung ure pbqr ahzore vf 01033 – naq, va trareny, jr qba’g.

Ohg gur byqJrngure grnz ner n pncnoyr ohapu, naq Xriva naq Znex ner sngubzvat guvf zlfgrel. Jvgu uryc sebz VPBNQF rkcregf ng APQP naq ABPF gurl ner qrpbqvat gur VPBNQF vqragvsvref fb jr pna yvax gur byq bofreingvbaf jvgu gur Nepuvir’f ybtobbxf naq cvpx bhg gubfr fgvyy haernq.