A history of the world in 1,399,120,833 observations

Locations of the weather (surface pressure) observations being used to reconstruct global weather and climate: Observations coverage (1851-2008) in version 3.2.9 of the International Surface Pressure Databank (that used in the Twentieth Century Reanalysis version 2c).

There’s a lot of history hiding in even purely scientific datasets. This movie shows just the locations of the 1.4 billion observations in the International Surface Pressure Databank (1851-2008), and in it I think I can see:

  1. The constraints on sailing-ship trade routes imposed by the global wind fields.
  2. The transition from sail to steam in shipping (late nineteenth century).
  3. The opening of the Suez canal in 1869 (01:30).
  4. The Famous Arctic voyage of Nansen’s Fram (03:20).
  5. The heroic age of Antarctic exploration (starting at about 04:00).
  6. The opening of the Panama canal in 1914 (05:10).
  7. The first world war (05:10).
  8. The second world war (07:00).
  9. Major administrative changes in India (08:00).
  10. The introduction of drifting buoys (1978: 10:20)
  11. And, sadly, a reduction in observations coverage in the last couple of decades as participation in the Voluntary Observing Fleet declines.

Of course these observations are not all that were made. Many more historical observations exist (on paper, or in restricted access collections), but these are the ones that are currently available to science. The process of rescuing the observations has also left its mark on the coverage – including right at the beginning of the video, where the coverage of ship observations reduces sharply in 1863 – the end of Matthew Fontaine Maury‘s pioneering data collection work. Various subsequent rises and falls in coverage result from the work of many other scientists and teams; including, of course, a large group of Royal Navy ship observations in the period around the First World War (starting about 05:00) clearly distinguishable just from their locations, as Naval ships move in a quite different pattern from commercial shipping. (Our US Arctic ships are not in this database yet – they will be in the next version).

Pioneer! O Pioneer!

Markup of the transcribed log for the year 1923.

We start 2015 with another big achievement – we’ve completed all 14 years of records (1922-1935) from the Pioneer. That’s more than 60,000 new weather observations from nearly 11,000 logbook pages. Congratulations to the lightning-fingered captain Hanibal, lieutenants gastcra, helenj, jill, pommystuart, and the 84 other crew, on another tremendous piece of work.

As a Coast and Geodetic Survey ship, Pioneer behaves quite differently from our earlier vessels – rarely venturing out into the open ocean and often staying in one place for extended periods of time. Possibly as a result of this, her logs rarely include latitude and longitude positions, preferring instead the current port or land location. This does make tracking her location more difficult, but with our excellent database of port and place locations and a little effort we can estimate a good location for almost every day.

The locations of the Coast and Geodetic Survey Steamer Pioneer, over 1922 to 1935. Records from all 14 years are combined into a single calendar year for this visualisation.

A centennial: The Battle of the Falkland Islands

oldWeather forum moderator Caro has been showcasing the history in our logs, by tweeting, every day, excerpts from the logs of exactly 100-years ago (follow along here). The terse style of the logs is a good match for Twitter, but on some days so much happened that we’d like to go into more detail. December 8th, 1914 was such a day, so Caro has written this post:

It’s been said before: oldWeather is not just about the weather. We transcribe history too and few of the historical narratives to emerge from our WWI ships’ logs can compare to the events that took place on this day, 8 December, 100 years ago: the Battle of the Falkland Islands. The logs of all nine Royal Navy ships involved ― Bristol, Canopus, Carnarvon, Cornwall, Glasgow, Inflexible, Invincible, Kent, and Macedonia ― have given our transcribers and editors first-hand accounts of one of the most important sea battles of WWI.

Back on November 1, Admiral von Spee and his German cruisers had defeated a Royal Navy squadron near Coronel, Chile. British losses were heavy; the ships Good Hope and Monmouth were lost and with them the lives of about 1600 men. Glasgow and Otranto escaped. The British Admiralty, realising the danger of the German ships escaping into the South Atlantic and disrupting the Allies’ operations along the African coast; or sailing around the Horn to attack the now almost defenceless British base in the Falkland Islands, sent a squadron to the South Atlantic to track down von Spee’s cruisers. Eight Royal Navy warships assembled at Port Stanley in the Falkland Islands on December 7. The old battleship Canopus had been set in place as guardship for Port Stanley, resting on the mud, since mid-November.

On 8 December, the German cruisers Scharnhorst, Gneisenau, Nürnberg, Dresden and Leipzig, together with three auxiliary vessels, gathered to attack the Falklands and raid the British facilities there. Gneisenau and Nürnberg detached from the rest of the German squadron and moved to attack the wireless station and port facilities of Port Stanley. The two raiders were seen by a hilltop spotter who reported their approach to Canopus, waiting out of sight behind the hills.

The logs continue the story:

  • 9.19am Canopus: Opened fire fore & aft 12” turrets on Gneisenau & Nürnberg
  • 930am Canopus: Ceased fire. Enemy retreated
  • 9.30am Glasgow: Weighed and proceeded
  • 9.50am Kent: Proceeded to follow enemy. 3 more German cruisers reported in sight, Scharnhorst, Leipzig, and Dresden
  • 10.15am Glasgow: As requisite keeping touch with enemy; squadron weighing and proceeding from Port William
  • 11.43am Carnarvon: Bristol ordered to take Macedonia & destroy transports
  • 12.57pm Inflexible: Opened fire at extreme range on Leipzig, firing 12 rounds of 12 inch, apparently making no hits
  • 12.57pm Invincible: Invincible opened fire
  • 1.25pm Invincible: Enemy’s light cruisers observed to spread to starboard
  • 1.33pm Invincible: Scharnhorst & Gneisenau opened fire
  • 1.35pm Invincible: Cornwall, Kent & Glasgow ordered to chase enemy light cruisers
  • 2.51pm Inflexible: Opened fire on Gneisenau, 15,200 yards, Invincible engaging Scharnhorst, the leading ship in line ahead
  • 3.00pm Glasgow: Opened fire & engaged Leipzig with 6″ gun
  • 3.30pm Bristol: Fired 2 rounds fore 6″ and ordered Santa Isabel and Baden, German colliers, to stop; crews ordered to abandon ships. German crews transferred to Macedonia
  • 4.01pm Inflexible: Scharnhorst listing heavily to starboard, two funnels gone, and ship on fire. Ceased firing on her
  • 4.15pm Carnarvon: Opened fire [on Scharnhorst]
  • 4.17pm Carnarvon: Scharnhorst turned over & sank bow first; cease fire
  • 5.00pm Kent: Kent proceeded in chase of Nürnberg
  • 5.30pm Cornwall: Opened fire [on Leipzig] with 6″ guns & continued action with all guns
  • 5.40pm Macedonia: Opened fire on Baden
  • 5.48pm Inflexible: Finally ceased firing [on Gneisenau]. Signalled to Carnarvon, “I think enemy have hauled down their colours”
  • 6.02pm Invincible: Gneisenau sinks. Invincible, Inflexible and Carnarvon proceeded at full speed to pick up survivors
  • 6.45pm Kent: Opened fire and finally ceased fire at 6.57pm; Nürnberg sank at 7.25 pm
  • 6.50pm Cornwall: Enemy [Leipzig] on fire fore and aft
  • 7.00pm Bristol: Macedonia ordered to remain till colliers sunk and proceed to Port Stanley with crews
  • 7.23pm Kent: Stopped and endeavoured to pick up [Nürnberg] survivors
  • 7.53pm Macedonia: Baden sank
  • 8.15pm Macedonia: Opened fire on Santa Isabel
  • 9.00pm Cornwall: Stopped; lowered port boats to pick up [Leipzig] survivors
  • 9.23pm Cornwall: Leipzig foundered
  • 9.30pm Macedonia: Santa Isabel sank

The German auxiliary Seydlitz and light cruiser Dresden escaped. Almost 1900 German seamen lost their lives; 10 British were killed.

One hundred years on, we remember all those who died at Coronel and the Falklands and in the battles to come.

Credits reel II: This time it’s colourful.

Today is the last Thursday in November, and our friends in the U.S.A. are celebrating Thanksgiving. This festival has not caught on here in the UK, so I’m spared the turkey, and the pumpkin pie.

But I do know about being thankful, and today I’m particularly thankful for the 19,683 people who have transcribed at least one logbook page for oldWeather. Every one has made a contribution, from those who visited only once, to those who have done thousands of pages, and help guide and drive the project and its community. I’m proud to count them all as co-investigators.

So it’s an appropriate day to release a revised version of the project credits reel:

This project has nineteen thousand, six hundred and eighty three contributors.

Seas of red

         
           
         
   
         
         
         
       
       
 
       
         
       
           
           
         
       
           
             
       
     
           
         
     
     
       
     
       
         
           
             
       
         
         
     
             
           
           
           
           
       
       
       
         
       
     
         
             
       
             
           
     
           
         
     
         
     
           
       
 
         
         
     
       
       
         
           
       
       
         
           
           
         
             
         
           
     
           
     
           
         
           
     
     
             
         
           
       
         
       
         
       
           
     
         
       
       
   
         
       
     
             
       
         
         
       
   
         
         
           
             
     
       
         
       
   

Into the Top500

This visualisation, comparing two reconstructions of the weather of 1918 (each using the oldWeather observations), took four supercomputers to make: The blue contours are from the ERA20C reanalysis, run on a pair of IBM Power775s at ECMWF; the red contours are from 20CRv2C, run on Hopper - NERSC's Cray XE6; and the post-processing and rendering was done on Carver, NERSC's iDataPlex.


The Met Office, where I work, has just finalised an agreement to buy a new supercomputer. This isn’t that rare an event – you can’t do serious weather forecasting without a supercomputer and, just like everyday computers, they need replacing every few years as their technology advances. But this one’s a big-un, and the news reminded me of the importance of high-performance computing, even to observational projects like oldWeather.

To stand tall and proud in the world of supercomputing, you need an entry in the Top500: This is a list, in rank order, of the biggest and fastest computers in the world. These machines are frighteningly powerful and expensive, and a few of them have turned part of their power to using the oldWeather observations:

Two other machines have not used our observations yet (except for occasional tests), but are gearing up to do so in the near future:

My personal favourite, though, is none of these: Carver is not one of the really big boys. An IBM iDataPlex with only 9,984 processor cores, it ranked at 322 in the list when it was new, in 2010, and has since fallen off the Top500 altogether; overtaken by newer and bigger machines. It still has the processing power of something like 5000 modern PCs though, and shares in NERSC’s excellent technical infrastructure and expert staff. I use Carver to analyse the millions of weather observations and terabytes of weather reconstructions we are generating – almost all of the videos that regularly appear here were created on it.

The collective power of these systems is awe-inspiring. One of the most exciting aspects of working on weather and climate is that we can work (through collaborators) right at the forefront of technical and scientific capability.

But although we need these leading-edge systems to reconstruct past weather, they are helpless without the observations we provide. All these computers together could not read a single logbook page, let alone interpret the contents; the singularity is not that close; we’re still, fundamentally, a people project.

43 years in the North

The voyages of the Bear, Corwin, Jeannette, Manning, Rush, Rodgers, Unalga II, and Yukon


Today is the fourth birthday of oldWeather, and it’s almost two years since we started work on the Arctic voyages. So it’s a good time to illustrate some more of what we’ve achieved:

I’m looking at the moment at the Arctic ships we’ve finished: Bear, Corwin, Jeannette, Manning, Rush, Rodgers, Unalga II, and Yukon have each had all of their logbook pages read by three people; so it’s time to add their records to the global climate databases and start using them in weather reconstructions. From them we have recovered 43 ship-years of hourly observations – more than 125,000 observations concentrating on the marginal sea-ice zones in Baffin Bay and the Bering Strait – an enormous addition to our observational records.

The video above shows the movements of this fleet (compressed into a single year). They may occasionally choose to winter in San Pedro or Honolulu, but every summer they are back up against the ice – making observations exactly where we want them most.

So in our last two years of work, we’ve completed the recovery of 43-ship years of logbooks, and actually we’ve done much more than that: The eight completed ships shown here make up only about 25% of the 1.5 million transcriptions we’ve done so far. So this group is only a taster – there’s three times as much more material already in the pipeline.

Sesquimillional

Sometimes there is just no word powerful enough to describe the achievements of oldWeather.

Back in March we reached a million, and since then we’ve powered on from that milestone, now having added an additional five hundred thousand observations to our tally. That’s two new observations every minute, night and day, 7 days a week: Come rain or shine; snow or sleet; ice, fire, or fog.

The Weather of HMS Beagle

As I’ve mentioned previously, last Thursday I was warm up man for Charles Darwin and Robert Fitzroy (finally, a job truly worthy of oldWeather) – I was giving a talk about the project at the Progress Theatre in Reading.

HMS Beagle isn’t (yet) one of our ships, the observations from her 1831-6 circumnavigation had been rescued before oldWeather started; but I could use what I’ve learned from analysing the oldWeather observations to show the route of the ship, the weather they experienced, and the effect of their observations on our reanalyses for the period.

The route of HMS Beagle, and the value of her weather observations. The inset graph show the Beagle's pressure observations (as black dots), and the analysed mean-sea-level-pressure (from scout run 3.3.8 of the Twentieth Century Reanalysis): The pale blue band gives the range of first-guess pressure estimates (at the location of the ship), and the dark blue band the analysis rage. The dark band is consistently narrower than the light one, showing the more precise estimates of the weather obtained by assimilating the observations.

Follow

Get every new post delivered to your Inbox.

Join 1,545 other followers