Old Zooniverse hands know all about serendipity – Hanny’s Voorwerp was an unplanned and unexpected discovery from Galaxy Zoo, and has generated a great deal of new science. It could be that involving lots of people in the process of science (the citizen science approach) is a particularly good way to generate surprising and unexpected discoveries.
When OldWeather started, the science team were clear about what we wanted – weather observations please, and lots of them – but we were also aware that there was a lot of other material in the logbooks, and that we needed to enable those reading the logs to record whatever they thought was important. I only really do weather, but I’m not immune from the general fascination that the logs exert over those of us who use them, so I’m very curious about what will emerge from our collective studies – what will the citizen scientists decide is worth recording?
I haven’t yet seen anything as dramatic as a Voorwerp, but quite a few people are getting interested in the sickness records in the logs, and their relationship to the well-known ‘spanish flu’ outbreak in 1918. Many of the logs contain a record of the ‘Number on Sick List’; we didn’t ask for this number to be recorded, but some people have decided to record it anyway, and so far the database has accumulated values from almost 10,000 log pages from 126 different ships. I don’t have any professional expertise in medicine or epidemiology, but I couldn’t resist the temptation to have a peek at the data, and see what had been found.
The figure above shows all 10,000 values, arranged by date. We need to be careful when looking at the data, because I haven’t been able to quality control the sickness counts the way we do with the weather observations, so certainly a few of these points will be errors; but we can see a basic pattern. There are generally only a few people off sick, but occasionally there is a short-lived spike in the number – sometimes to hundreds on the list at once.
A closer look shows that the spikes are infectious outbreaks on a single ship. The biggest spike is seen in the logs of HMS Africa
On September 2nd 1918 the Spanish flu caught up with HMS Africa, and the cases mounted fast – reaching a peak with 476 people ill on the 9th. The Africa was a big ship, a King Edward VII-class battleship, but even so 476 people is nearly 2/3 of the crew; and it must have been a major challenge keeping her operational. The logbook, however, gives few hints of the struggle; and certainly they managed to continue their weather observations through the outbreak – the Navy’s dedication to duty is justly famous. Though virulent, the outbreak was brief; By September 21st the Africa had overcome the virus, but the victory was not without cost – log pages over the infection period regularly record deaths from influenza.
So congratulations to all those who’ve been recording sick-list counts – there’s definitely some interesting material there, and it’s great to see the collective intelligence of the project generating new research directions. I can’t say whether the sick-list counts will lead to new published science in the end – I’m only a climatologist – but I look forward to finding out.
To use the weather records in the OldWeather logbooks, we need to know not only what the observed temperature and pressure were and the date and time of the observation, but also the position of the ship at the time the observation was made (its latitude and longitude).
We are collecting quite a bit of position information in the logs: if the ship is in port, we get the port name; if at sea, the latitude and longitude at noon (and sometimes at 8am and 8pm as well). From this information it’s relatively straightforward to estimate the ship position at any time of the day (we just draw a line between the noon positions and use positions along this line). But, as with all the data we collect, this doesn’t give us exact positions – any of various problems might cause the positions to be inaccurate, and in using the observations we have to make allowances for these inaccuracies.
What might go wrong:
- The port name is not always enough to uniquely identify the ship’s location. There is a Devonport in Plymouth in the UK – but there is another one in Auckland, New Zealand. HMS New Zealand was in Devonport on valentine’s day 1919, but which?
- The latitude and longitude might be wrong, either because of transcription error, or because of an error in the log. A common such error is confusion between east and west longitudes (or between north and south latitude).
- The noon observations are not perfect: they rely on the accuracy of the chronometers and observations used to calculate them. Dead-reckoning and observed noon positions are commonly a few minutes (maybe 10 miles) apart.
- When in port, we only have the name of the port, not the precise position of the ship in the associated harbour or anchorage. On December 16th 1919 the New Zealand’s position was given as ‘in Panama Canal’: The Panama Canal is 48 miles long , where were they exactly?
- Estimating a position at midnight (say) by assuming it’s half way between the preceding and following noon positions assumes the ship is sailing all the time on the same course at constant speed. This is rarely true, so interpolating positions at times of observation from noon positions will introduce an error. In theory, this could produce a big error – if a ship travelled at full speed (say 30 knots) in a straight line between noon and midnight, and then turned around and returned to its starting point for noon the next day, our estimate of the position at midnight would be out by about 400 miles; but ships almost never actually behave like this: as we’ve seen in the routes of the New Zealand, and the Goliath, Gloucester, and Glowworm, ships generally either hang around in one place, or move fairly directly from one point to another. This estimation does introduce an error into positions, but it’s usually modest, rarely more than 30 or 50 miles.
The first two of these will produce big errors in the positions (chosing the wrong Devonport is close to being the biggest error possible), so they are serious, but also easy to spot. These are a nuisance (because we have to correct them), but hardly ever a problem. The last three points are hard to correct, and do introduce modest errors in the observations positions.
The figure shows the noon positions of HMS Pegasus on her Voyage from Plymouth to Arkhangelsk and back to Dundee in 1919 – ably digitised by captain Uldis Ohaks, lieutenants Manock and elizabeth, and their crew.
One position is clearly in error – the visit to Baffin Bay (marked in red) is impossible. Checking the log page for that day shows that, most unusually, both the latitude and longitude of this point are wrong. The longitude has been incorrectly captured by our system as 59 degrees west, while it is actually 59 minutes (i.e. 1 degree) west. We have correctly captured the latitude given in the log (68 degrees north) but this must be an error in the log, because that page also refers to sighting landmarks in Orkney and Caithness, so the ship must be around 10 degrees south of this. We have to guess the actual position, and, in this case it seems likely that the ship was actually at 58 degrees north, and the entry in the log is a typo.
The other positions clearly don’t contain big errors, but it’s clear that the ship didn’t always travel in a straight line between her noon positions – the lines linking them on the map often cross land, in England, northern Norway, and the Kola Penninsula. In these cases, the actual midnight position of the ship is probably about 100 miles from the straight-line-guess.
It’s tempting to fix these problems: we could improve our ship position estimates noticeably by using more sophisticated methods of tracking them. For example, the logs often contain hourly course and speed information, so it would be possible to digitise this and make hourly position estimates by dead reckoning; when within sight of land the logs often contain bearings to points on shore, which could also, in principle, be used to derive a more precise route for the ship. So we could certainly do better (at the cost of a great deal of work) but we’ll never have perfect information on the ship positions, so it’s worth asking first, how precisely we need to know them.
The weather shows itself both in very local effects (showers, contrails, frost hollows) and in very big effects (such as the drought, and now flooding, in Australia). A good illustration of this is the excellent Oldweather authors poster – if you wanted to paint this image you’d need a large brush for North Africa and the tropical oceans, and a very small brush to capture the fine detail. We can capture this detail with modern satellites, but to do the same from ship observations we’d need millions of ships, densely packed over all the oceans – an obvious impossibility. So when doing historical weather reconstructions from ship data we have to forget about the small-scale effects, but it’s still really important to get the large-scale effects correct. Throw away the small brush, and concentrate on using the big one to best effect.
It turns out that the combination of the number of weather observations we can collect, the accuracy of their pressure and temperature measurement, and the power of the computers available to us, mean that we can’t reconstruct weather effects at smaller scales than about 200 miles. (For the Oldweather period – we can do better in the present day). And that’s with the new observations we’re producing; without them in many areas we can’t reconstruct the weather at all.
This means that, in practice, a ship position error of a few 10s of miles is not a big problem, and doing all the work necessary to get more precise positions would hardly help us at all. It’s much more useful to spend our effort on entering more data, and that’s what we’re doing.
So if your ship seems to have taken the M1 (road) out of London instead of the more conventional choice of using the Thames and North Sea, don’t worry – the observations are still useful. Unauthorised excursions to Greenland, Baffin Bay, and other far-flung locations are unacceptable, however, and will have to be corrected.
For Day 18 of the Zooniverse Advent Calendar, we’re following-on from Day 12 and I have created this image of a Royal Navy ship built up of the words from the HMS Goliath logs – captained by Zooniverse user Roar. There are some great words hidden in here – it makes a compelling read.
Quite a few people have asked why we don’t have to input the time of each weather observation. It’s a sensible question, and we do need the observation times, particularly for tracking fast-changing weather events like moving fronts. But one of the clever features of the the oldweather website is that we don’t have to enter the times – they are automatically collected through the process of entering the weather data.
To do this we take advantage of a symmetry between space and time in the logs (scientists love symmetries). The top of each log page corresponds to the beginning of the day, and the bottom of the page corresponds to the end of the day. So the further down the page an entry is, the later in the day it was taken. We record the position of the push-pin for each entry digitised from the page and from that push-pin position, we can find the time associated with that entry. The image below shows the positions of all the weather observations entered from the logs of HMS Bacchante (thanks captain richbr15, lieutenants dazedandconfused and davemcg, and all the crew).
As with most books, there are two sorts of pages: left-hand (red dots) and right-hand (blue dots). They have different margins in our images, so they don’t line up precisely horizontally, but their vertical position is the same, and that’s what gives us the time.
The Bacchante recorded the weather at the end of each watch: so at 4, 8 and 12 a.m., and the same times in the afternoon. They also recorded it at the end of the first dog watch (6 p.m.) – so we should expect to see three equally-spaced groups of points in the top half of the figure (the morning), and four, more irregularly spaced, groups in the bottom half (the afternoon). This is exactly what we see, and it’s clear that for the vast majority of the observations, we can easily say which watch they are associated with, and so when they were taken.
There are a few observations that are not quite so easy – we can see some smaller clusters of observations above and to the left of the main clusters; but again, it’s easy to see which watches there observations correspond to. There are also a few observations in irregular positions – lost in time and space – but these are only a tiny fraction of the total.
So it’s going to be easy to find the time of observation in the usual case where the observations are 2 or 4 hours apart. For the diligent few log-keepers who recorded observations every hour or even more frequently, we will have to be a bit cleverer; and use the differences between the observed weather values, as well as the position on the page, to group the observations into clusters and assign them to times.
All this, of course, relies on having accurate positions on the page for each observation, which means lining up the entry box with the observation text carefully each time when entering the data. So far, we’ve done well at this (as the figure shows); I’ve come to expect no less from oldweather, but it’s still a pleasure to see.
For Day 12 of the Zooniverse Advent Calendar I have created this image of a Royal Navy ship built up of the words from the HMS Invincible logs – captained by Zooniverse user clibby34. This is something we’ve been playing around – the idea of using the text from logs in interesting ways – with for Old Weather and we’d love your feedback on it. In the meantime you have a great image to play about with.
As part of the Zooniverse Advent Calendar we have been creating some author posters, as a thank you to our users. This poster shows the average global temperature over the Earth in April 2003 made up of the nearly 8,000 names of people who have taken part so far and agreed to have their name published.
You can download the larger version of the poster (7000 pixels, 19MB) or the smaller size (3500 pixels, 4.5MB). Do what you like with this post – print it, share it. Help spread the word about the wonderful Old Weather project! Good luck finding your name…
Oldweather is now really hitting its stride, with a stream of ships reaching completion and so becoming available for analysis. One of those that has completed recently is the battlecruiser HMS New Zealand, which was named for the country that funded her construction, and carried Admiral of the Fleet John Jellicoe on a tour to India, Australia, New Zealand and Canada in 1919. Her logs from this circumnavigation are those we’re looking at in Oldweather.
Long voyages like this are particularly desirable as a source of weather observations, because the same ship, crew and weather instruments experience and record a wide range of different weather conditions; from the hot calms of the doldrums, through the steady trade-wind regions, to the stormy conditions of the roaring forties. On her circumnavigation New Zealand sees all of these conditions, and the change in weather conditions shows up clearly in her barometer records – Captain toucans, Lieutenants keybasher, jdulak, Cyzaki, and all her crew have provided a good picture of the weather in 1919, over a wide section of the world.
So the records digitised from the New Zealand, like those from the other ships completed so far, have been entered with accuracy and skill; but there is always one component of the data that presents us with more trouble than the others, and for the Oldweather ships, it’s the ship positions. Many of you will have noticed that the maps showing the ship positions as they are being entered sometimes show unlikely or impossible positions – HMS New Zealand’s positions are not unusually problematic, but, because the ship travelled so widely, they make a good example to illustrate the issue. The figure below shows the positions digitised for the New Zealand.
It’s clear where the ship was going, and it’s also clear that sometimes we’re getting the positions wrong. In this case, all the wrong positions have the same cause – we know the longitude, but not whether it’s East or West of Greenwich.
This image shows a typical at-sea position entry – we can see that the latitude is 2 degrees 17 minutes, and the longitude is 88 degrees 5 minutes, but the letters showing the position as south of the equator (`S’) and east of Greenwich (`E’) are a bit detached from the position entries. The position of these letters seems to vary from log to log – sometimes they are put immediately after the position values, sometimes merely nearby, as here. But either way, please always enter them along with the positions, so this example should be entered as latitude “2 17 S” and longitude “88 5 E”.
Even if we always enter the direction letters, we won’t get rid of all position errors of course – sometimes the letters are missing in the log, sometimes the log-keeper makes a mistake entering the position. It’s important to faithfully reflect the logs, so we shouldn’t try to fix such problems. But please do enter them however they appear on the page.
Those of you who return to the main site today will notice that the counter has jumped ahead. Having reviewed the data that we’ve received for the first few ships to complete their voyage, we’ve decided that we only need three separate transcriptions for each ship, rather than five. This should speed up the process considerably, and make it easier to get things done. Full speed ahead!
P.S. You can read more about this on the forum
I wanted to say something about the results from our fifth finished ship HMS Inflexible, so I’ve proccessed all the measurements and made a full set of plots showing trends in the weather over the voyage. The results are excellent, but sometimes even I get a bit tired of barometric pressures and wet-bulb temperatures, so instead of posting them here I’ve sneaked over to the other side of the project to have a look at the history data – the log entries that we’ve selected for inclusion.
The sort of history we’re doing in OldWeather: collecting and synthesising large quantities of diverse material, is an example of a new research trend sometimes called ‘digital humanities’. Because it’s new, the methods for making use of the events we’ve been collected are not yet well established (and anyway I’m no historian) so I’m not sure what the best way to use the events we’ve recorded is. But it’s always a good start to plot the data and see what you’ve got, so I’ve fed all the events recorded from the Inflexible into Wordle (www.wordle.net) and made this:
It’s a bit like the blob plots I’ve posted before – everything is in there (I’ve stripped out a few really common words like ‘the’ and ‘ship’), but more popular entries are bigger. Inflexible is most famous for being in the battle of the Falkland Islands (like her sister ship – HMS Invincible), but I think this image gives a more balanced indication of the real day-to-day story of the ship: Routine activities and references to other ships loom large, and only right down in the small print are the signs of drama: ‘Firing’, ‘Killed’, ‘Died’, and ‘Sunk’. I was baffed at first by the frequent reference to ‘Sabretache’ – a sort of cavalryman’s handbag, but it turns out that it is the name of another ship – a French destroyer.
So thanks and well done to captain clibby34, lieutenants sarek and instigator008, and all the crew. And just in case you’re interested in the wind direction, barometer height etc., I’ve put that on line here.
Stuart and I have been performing some routine maintenance of the Old Weather engines here at Zooniverse HQ (swabbing the decks etc). As we were messing about with the database rigging, we realised that we could quickly create a map of the current positions of the entire Old Weather fleet. So here it is: a snapshot of Old Weather right now.
You can see that almost all of the ships are to be found in water (phew!) and those that aren’t are often river boats. We cannot explain the boat in the middle of Greenland. We’d like to provide this kind of snapshot – a sort of Old Weather Live – as we currently do for Moon Zoo. Keep a lookout for updates along these lines.