Archive by Author | Philip

A number considerably above the average

To achieve a million once may be regarded as good fortune; to do so twice looks like carefulness, skill, enthusiasm, and dedication.

We’ve been working on the US Arctic logbooks since July 2012, and in that time we’ve rescued one million weather observations (actually 1,081,641 as of March 11th). That’s an awesome achievement – even more awesome than it seems at first, as a single weather observation is a complex object containing a lot of information. Our 1,081,641 observations comprise:

  • 960,221 air pressures,
  • 2,782,755 temperatures (950,351 dry-bulb, 765,270 wet-bulb, 597,556 barometer and 469,578 sea),
  • 1,020,223 present weather codes,
  • 1,049,692 wind directions,
  • 1,019,268 wind forces,
  • 864,586 cloud fractions, and
  • 846,859 cloud types.

To record all that, we’ve typed 21,511,795 characters – about 20 times the length of a typical novel.

However you look at it, that’s a tremendous achievement.

Code-breaking and oldWeather

Marcus Cocceius Nerva Caesar Augustus. 13th Caesar.

Marcus Cocceius Nerva Augustus. 13th Caesar.

Jr nyjnlf unir na rlr gb gur shgher, urer ng byqJrngure, naq jr’ir abgvprq bhe sevraqf ng gur nepuvirf cubgbtencuvat fbzr arjre ybtf – H.F. fuvcf sebz gur zvq-20gu praghel – nf jryy nf gur byqre barf jr’er hfrq gb. Gur nepuvirf ner abg cubgbtencuvat gurz sbe hf – gurl ner cneg bs nabgure cebwrpg – ohg gurl ner trarebhf crbcyr, naq gurl jvyy tvir hf n pbcl bs gur ybt obbx vzntrf vs jr jnag gurz. Qb jr jnag gurz?

Jryy bs pbhefr jr qb – bhe ybir sbe ybtobbxf arire snvyf, naq jr ner qrgrezvarq gung gur xabjyrqtr va gurz jvyy abg cnff njnl. Ohg jr pna’g qb rirelguvat evtug abj. Fb jr qb, hasbeghangryl, unir gb or frafvoyr, naq frg cevbevgvrf, naq lnqn lnqn lnqn.

Gur zbfg vzcbegnag guvat gb xabj nobhg nal cbffvoyr arj ybtf, vf jurgure nalbar unf ybbxrq ng gurz orsber: unir gurve jrngure bofreingvbaf nyernql orra genafpevorq? Gb svaq bhg, jr arrq gb ybbx va VPBNQF (gur vagreangvbany qngnonfr jurer jr fgber bhe jrngure erpbeqf) naq frr jung gurer vf sebz nal fuvc jr ner vagrerfgrq va – ohg gurer’f n pngpu.

Zbfg bs hf ersre gb n fuvc ol vgf anzr; bapr gur bjare unf qrpvqrq gb anzr n fuvc UZF Jbatnaryyn gura gur erfg bs hf fubhyq whfg sbyybj nybat. Bhe cerqrprffbef, ubjrire, jub qvtvgvfrq fbzr ybtobbx jrngure erpbeqf qrpnqrf ntb, qvq abg nterr. Creuncf gur anzrf jrer gbb ybat gb svg bagb gur Ubyyrevgu pneqf gurl hfrq sbe qngn fgbentr, ohg jungrire gurve ernfba gurl hfrq pbqr ahzoref nf fuvc vqragvsvref. Fb vs jr jnag gb svaq gur erpbeqf bs gur HFF Nexnafnf va VPBNQF, jr arrq gb xabj gung ure pbqr ahzore vf 01033 – naq, va trareny, jr qba’g.

Ohg gur byqJrngure grnz ner n pncnoyr ohapu, naq Xriva naq Znex ner sngubzvat guvf zlfgrel. Jvgu uryc sebz VPBNQF rkcregf ng APQP naq ABPF gurl ner qrpbqvat gur VPBNQF vqragvsvref fb jr pna yvax gur byq bofreingvbaf jvgu gur Nepuvir’f ybtobbxf naq cvpx bhg gubfr fgvyy haernq.

Certificated (2)


Thanks again to the sponsors, organisers, and judges, of the 2013 International Data Rescue Award in the Geosciences, for adding to our project laurels.

Old New Zealand, HMS New Zealand, & new New Zealand

Atmospheric pressure along the route sailed by HMS New Zealand in 1919. The blue band shows the range of our estimates before oldWeather, the black points the new observations we provided, and the red band the revised analysis range incorporating our observations.

Atmospheric pressure along the route sailed by HMS New Zealand in 1919. The blue band shows the range of our estimates before oldWeather, the black points the new observations we provided, and the red band the revised analysis range incorporating our observations.

This week, atmospheric scientists are gathering in Queenstown, New Zealand, for the fifth general assembly of the SPARC program (Stratosphere-troposphere Processes And their Role in Climate). We’ve mentioned New Zealand before: both as a country who’s isolation means that its historical weather is poorly documented, and as a Battlecruiser in the original oldWeather fleet. In September 1919 the two met: the battlecruiser visited the country, giving us an opportunity to make a major improvement in reconstructing the climate of the region.

As we showed back in October, we’re now re-doing our analysis of global weather, so we can see exactly how much the observations we’ve recovered from HMS New Zealand have improved our knowledge of the climate of New Zealand (the country). The figure above (made for the SPARC meeting) shows our estimates of the weather in each region visited by HMS New Zealand during her circumnavigation in 1919: blue for before oldWeather, and red a new revision using our observations. The width of the band indicates uncertainty – narrower is better – and the improvement we’ve made is very large.

Documenting World War 1 at Sea

We chose our first batch of logbooks to cover the period of the First World War, as our climate records from this period were particularly poor. This gave us not only invaluable new climate information, but also a new look at a key historical period which is about to reach its centenary.

To use our historical results, we teamed up with Gordon and In Gordon’s words: ‘Our present world has been shaped by World War 1 – as much a maritime war as World War 2. Not just the Battle of Jutland or the Allies near-defeat by the U-boats, but Mediterranean, Belgian coast, South West & South Africa, East Africa, Persian Gulf, German raiders, Atlantic convoys, North Russia.’ We need to present our logbook records so they can contribute to public and scholarly understanding of the period.

The transcribed and edited logbook records are now a major component of, where they are described as:

British warship log books of the World War 1 era, totalling some 300,000 pages. The logs of over 300 ships have been transcribed, and most are online. They include coverage of Battle of the Falklands, Northern Patrol, Dardanelles, East Africa, trans-Atlantic convoys, Indian Ocean, China Station, amounting to some 60-70 percent of all major warship movements 1914-18, outside of British home waters.

But they are not enough on their own, we should combine them with other sources of information. Naval-History.Net has prepared for the centenary for some years, using contemporary sources where possible and more recent research where available. Current projects include:

  1. Chronology providing the political and military background to the war at sea.
  2. Naval Operations by Corbett and Newbolt – many of the excellent plans are online including all the Battle of Jutland – and the three volume Merchant Navy histories by Hurd.
  3. Navy despatches and relevant Army despatches from the London Gazette. Also Royal Navy honours and gallantry awards by award and by Gazette date. Includes Medal index/database by name.
  4. Royal Navy and Royal Marine casualties (researched by Don Kindell working with the Naval Historical Branch (MOD)), as well as those of the Dominion Navies and U.S. Navy & Marine Corps.
  5. Royal Navy warships and auxiliaries from the invaluable “Ships of the Royal Navy 1914-1919″ by Dittmar and Colledge. Although still in progress, all warships and many of the auxiliaries are listed by name and by type/class.

The honourable work of data rescue

Gil Compo (centre) accepts the honourable mention oldWeather was awarded in the The 2013 International Data Rescue Award in the Geosciences, from organisers Kerstin Lehnert (IEDA, left) and Bethan Keall (Elsevier, right).

Gil Compo (centre) accepts the honourable mention for oldWeather from the 2013 International Data Rescue Award in the Geosciences, with organisers Kerstin Lehnert (IEDA, left) and Bethan Keall (Elsevier, right).

One of the fun parts of working as a scientist is going to conferences, and in the geosciences, conferences don’t come much bigger than AGU. The American Geophysical Union’s 46th annual Fall Meeting ran last week in San Francisco, and it brought together more than 22,000 scientists for a week of presentations, discussions, celebrations, and beer.

Our man at AGU this year was Gil Compo, and he represented oldWeather at an important side event: The prize ceremony for the 2013 International Data Rescue Award in the Geosciences. We didn’t quite win this prize (the winner was the excellent Nimbus Data Rescue Project), but the judges liked us a lot, and we were awarded an honourable mention. So well done to all the oldWeather participants on a further well-deserved honour, and thanks to the award sponsors and organisers.

Every scientist’s must-have accessory, at any large conference or meeting, is a poster: This is a large sheet of paper (typically A0, or about 4′ by 3′) covered with artistically arranged images and results from your project, which you attach to a wall or display board, and use as a visual aid. Kevin made an excellent poster for us, combining images from all aspects of the project. You can see it on display in the background of the photo above, and if you’d like your own copy, it’s on our resources page.

Scientific progress goes …

The oldWeather distributed supercomputer

Nobody succeeds alone, and that’s doubly true of oldWeather: not only are we legion in ourselves – a community of thousands working on logbook weather, but even as a project we are embedded in a community – we have friends and relations.

Our close relations, of course, are the other Zooniverse projects: That’s a diverse family – from the paterfamilias to the newest member, united by shared principles and the talents of the core team. But we also have more distant relatives. oldWeather is neither the first, nor the biggest, climate and weather citizen science project. (CPDN) turned ten this year, and they have a very different way of doing science.

Many of the experiments climate scientists would like to do are impossible in practice: What would happen to the weather, for example, if we were to induce artificial volcanoes as a way to cool the planet? To investigate these questions, we do simulations – we build computer models of the climate system and do the experiment in the model. We have learned an enormous amount by doing this, but it does take a lot of computer time. CPDN asks volunteers to let their desktop computers contribute to this work – most of the time we use only a small fraction of the power of our computers, so this work can be done entirely in your computer’s spare time – it does not interfere with your normal use.

CPDN is also part of a family: There are lots of volunteer computing projects sharing the infrastructure provided by the Berkeley Open Infrastructure for Network Computing (BOINC) and you can contribute to any you choose.

Several of the oldWeather community have doubled their efficiency by doing citizen science and volunteer computing simultaneously: while the people are reading logbooks, their computers are simulating the climate, or Neutron stars, or malaria, or the Milky Way, or … I’d like to congratulate the oldWeather BOINC group on their tremendous contribution both to oldWeather and to volunteer computing.

Too low, terrain!

Cockpit of a NOAA P3.

Cockpit of a NOAA P3.

oldWeather is telling us a great deal about how the present climate is different from that of 100 years ago, but to make maximum use of that information, we also want to know exactly how the present climate is behaving. This will help us link our observed changes in surface weather to the basic physics of the ocean and atmosphere. To learn about the present climate we collect a rich and detailed set of observations from research ships, aircraft, and satellites.

Last year, Kevin was out making such measurements from a ship, on a research cruise in the Bering Strait. This field season he’s back out there, but he’s gone up in the world. For some purposes ground level is too low, and satellites are too high, and to fill this gap NOAA have two research aircraft (affectionally known as ‘Kermit’ and ‘Miss Piggy’). Kevin’s group have got some time on one of them, they are trying to “quantify the air-ice-sea interactions and lower atmospheric structure in the marginal ice zone, with the ultimate goal of being able to infer how recent reductions in sea ice extent in autumn will impact the atmosphere“.

The research aircraft is complex and well-equipped: According to Kevin “The NOAA WP-3 is instrumented like ten satellites. So we are able to collect a vast array of data from deep oceanography with AXCTD and AXBT expendables, SST and surface microwave emission (wind/waves/ice), upward/downward radiation, up to 22 thousand feet where we deploy dropsondes from above the clouds to characterize the structure of the atmosphere. On a survey we collect flight level data continuously while deploying AX instruments about every six minutes.

To do all that effectively requires close cooperation between the crew of the aircraft and the scientists – that’s Kevin’s job. He’s sent back this video to give us a taste of what it’s like. It looks exciting – they spend a lot of time travelling at 200 knots, only 200 feet off the ground, much to the distress of the auto-pilot – but it’s hard work: one flight means 8-10 hours flight time + 2 hours for briefings before and after.

See more about this mission on the NOAA website.

Frost in the South

We’ve looked at the world from the top; this is the view from beneath: Antarctica in the centre, South America at top, South Africa right, Australia and New Zealand bottom left. Streamlines show near-surface wind, colours indicate temperature, dots mark rain and snow. All data are from the Met Office global analysis.

One reason why weather forecasting and climate research are hard is that the atmosphere is complicated: There’s a lot going on – all sorts of different motions and changes occurring simultaneously all over the world. So while it’s often useful to use simplified views – perhaps to look only at mean-sea-level pressure, for example – it’s also good sometimes to embrace the complexity, and remind ourselves why we need a supercomputer to keep track of it all.

So this time I’ve put as much as possible in the video: sea-ice, wind speed and direction, temperature and even rainfall. It’s still only a tiny fraction of the full three dimensional atmospheric state that our forecast models have to simulate, but there’s plenty to look at: We can see not only the small-scale complexity of the winds, but also some larger-scale patterns: the strong clockwise circulation around Antarctica formed by the southern hemisphere westerlies, the cyclones forming in that strong flow, and atmospheric waves folding outwards.

This isn’t really old weather, it’s almost new – from only last month. But I used this example because it illustrates that the weather is not only complicated and interesting, it also matters. If you set the video to September 16th you’ll see a low pressure (clockwise circulation) off Marie Byrd land, linking with a high pressure (anti-clockwise circulation) in the south-east Pacific. These combined to channel cold Antarctic air up toward central Chile, which contributed to a late frost which cost their fruit industry an estimated $1 billion. Expect to pay extra for peaches, cherries, and even Cabernet Sauvignon, as a result.

Brightening the world

We launched oldWeather three years ago today (October 12th, 2010). It was an exciting but scary moment – would she float? We’d done everything we could, but you’re never quite sure until the splash has settled.

One thing we did know at launch was where we were going: The map of past climate variability and change contains some very large blank areas – great expanses of space and time where we knew almost nothing of what the weather had done. Ours was a voyage of exploration: We would sail, via the archives, into these regions and rescue their weather observations, adding systematically and permanently to the scientific records on which our understanding of the climate is based.

And it’s worked very well. As with any research project we’ve encountered plenty of surprises along the way, but they’ve been good surprises – we knew about the weather in the logs, but we didn’t realise just how much else was in there. So we’ve added detailed ship histories, maps, geographical databases, illustrations of the course of WW1, tales of life on board, …

But our primary aim is still the weather, and we’ve recovered an enormous account of historical weather information, more than 1.6 million new observations from our original set of Royal Navy logs alone. These new basic observations are a permanent foundation on which scientists all over the world can build new reconstructions and products, and today we can see such a building appear.

Gil Compo and colleagues, from NOAA/CIRES/University of Colorado, are using our new observations in an atmospheric reanalysis (20CR). Essentially they combine surface weather observations (such as ours) with information on sea temperature and sea-ice, and a physical model of the atmosphere, to make a detailed and comprehensive picture of the global weather. It takes some of the world’s largest supercomputers to do this analysis: 20CR was produced at the US National Energy Research Scientific Computing Center and the US Oak Ridge Leadership Computing Facility. But it’s worth the effort – not only do they make a global weather reconstruction, but they also calculate the accuracy of their reconstruction, and we can compare their new reconstruction with one they made earlier, to see how much difference our observations have made.

So the video above has four components:

  1. The weather. The reanalysis calculates everything about the weather: winds, temperatures, clouds, rainfall, the jet stream, … but I can’t show all that in one video so we’re only seeing mean-sea-level-pressure. The solid black contours show where this is low (bad weather), and the dashed contours where it is high (good weather).
  2. The observations. Grey dots mark observations we’ve had since before oldWeather started. Yellow dots mark new observations. Most (but not quite all) new observations are from oldWeather. (We are only part of a wider recovery program).
  3. The fog of ignorance. Grey fog marks the areas where we still don’t have enough observations to say exactly what the weather was doing.
  4. The glow of discovery. Yellow highlighting marks the areas where the reconstruction is much better than it was before (mostly because of our new observations).

That’s a lot to get in one image, but it’s the yellow that matters. Our work has cleared the fog, and illuminated the weather, over a huge area of land and ocean. The improvement stretches over about 20% of the Earth’s surface – more than 100 million square kilometres – and is there for every hour of the 9+ years covered by the Royal Navy logs we read.

That’s not a bad return for our three years hard work.


Get every new post delivered to your Inbox.

Join 1,305 other followers