One of the main uses of the weather observations that we are collecting is in new reanalyses – reconstructions of weather and climate over the last few decades or centuries. This week, dozens of scientists working on weather and climate reconstruction are meeting for a workshop on reanalyses and historical weather observations, hosted by the Royal Netherlands Meteorological Institute at De Bildt.
This is an opportunity to tell everybody working in the field just how much we’ve achieved with oldWeather over the last 11 months, so I’m giving a presentation highlighting our results. As you’ll have seen from earlier blog entries, there’s plenty to present – so my biggest challenge is in working out how to give credit to all the project participants: It’s a firm rule in science that you should credit all your collaborators in any project, but there are 9566 people who’ve made a significant contribution to oldWeather (at the last count). So to list them all I’ve borrowed a technique from the movies, and made a credits video – this video is being premiered at the meeting (as part of my talk).
Of course it’s not enough just to have lots of people involved, we’ve also got to generate lots of new scientific results. So I’ll also be showing another video – less detailed, but faster and much more colourful – showing the 841848 new weather observations that we’ve generated.
Edmond Halley is best known for his comet, but he was one of the great polymaths – as well as making astronomical discoveries he was also a notable meteorologist: he did important early work understanding the trade winds and monsoons. It’s less well known that that he was also a Naval Officer: in 1699 he was granted a commission as captain in the Royal Navy, and he commanded HMS Paramour (a pink) on an expedition into the South Atlantic to investigate the variation of the compass.
His main concern was with magnetism, but as a man of wide interests, Halley took with him examples of those two exciting modern scientific instruments: the thermometer and the barometer. I can’t find the logbook of the voyage, but Halley’s notes have survived: they were published by Alexander Dalyrmple, in 1775, as part of “A collection of voyages chiefly in the Southern Atlantick Ocean“. They date from 220 years before the logbooks we’re used to in OldWeather, but to anyone who’s looked at our logbooks they are oddly familiar: records of latitude, longitude, wind force and direction and, in the left-hand margin, thermometer and barometer readings.
In 1699 the barometer had been around for more than 50 years, and the barometer records in Halley’s account are clearly in the familiar inches of mercury. But the thermometer did not become a reliable, precision instrument until about 1725, when Fahrenheit invented the mercury thermometer with a standardized, calibrated scale. So when Halley says the temperature is ’33’ it’s not immediately obvious how this should be interpreted. Careful scholarship has established, however, that Halley was using a thermometer designed by Robert Hooke, and lavishly described in his book Micrographia:
The Stems I use for them are very thick, straight, and even Pipes of Glass […] above four feet long […] [filled] with the best rectified Spirit of Wine highly tinged with the lovely colour of Cocheneel, which I deepen the more by pouring some drops of common Spirit of Urine, which must not be too well rectified, […]
From Hooke’s description we can convert Halley’s reported units into modern equivalents at least approximately – Halley’s ’33’ was about 8°C.
The diary entries are mostly routine accounts of the movements of the ship, but occasionally he puts in longer and more interesting reports: here’s an example from Thursday 1st February 1700, when they were close to South Georgia, in the cold waters of the Southern Ocean:
[…] between 4 and 5 we were fair by three Islands as they then appeared; being all flat on the top, and covered with Snow milk white, with perpendicular Cliffs all round them […] The great height of them made us conclude them land, but there was no-appearance of any tree or green thing on them, but the Cliffs as well as the tops were very white, our people called A by the name of Beachy-Head, which it resembled in form and colour. And the Island B in all respects was very like the land of the North-foreland in Kent, and was at least as high and not less than 5 miles in front, […]
The following day they were disconcerted to discover that these ‘islands’ had moved, and fled north to warmer waters. This is the first recorded sighting of a tabular iceberg.
Halley’s observations are probably not of great value to climate scientists: his instruments were state-of-the-art for 1699, but it took decades longer for such observations to became accurate and plentiful enough for climate reconstructions. He did set a precedent though – possibly as the first person to go to sea with a barometer and a thermometer – and we’re still following his example more than 300 years later.
When scientists talk about pressure, they measure it in Pascals (Pa: the SI unit for pressure). For atmospheric pressure, 1Pa is an inconveniently small number, so we lump them together in groups of 100 and talk about hectopascals (hPa: 1hPa=100Pa). The atmospheric pressure at sea level is usually given as 101325 Pa, which is approximately 1000 hPa; so 1 hectopascal is also referred to as 1 millibar – when you hear your weather forecaster talking about millibars, hectopascals are what he’s really using. The ships, however, don’t measure pressure in hectopascals or even millibars; they measure it in inches. This is an artefact of the way they measure the pressure – with a mercury barometer.
Back in the early 17th century there was much discussion among the scientists of the day about why it was impossible to pump water more than about 10m upwards. It was Evangelista Torricelli, in 1643, who realised not only that the height to which the water rose was determined by the weight of the surrounding air, but also that you could use this effect to measure changes in the air pressure. A 10m column of water is a nuisance to work with, so he switched to the much heavier mercury as his working medium, and made the first ever barometer measurement.
We’ve been measuring air pressure in the same way ever since – balance the weight of a column of mercury against the weight of the surrounding atmosphere, and the taller the column the higher the atmospheric pressure. At sea-level, the column will be about 76cm (29 inches) high, and the changes in atmospheric pressure as the weather changes cause fluctuations of up to a few inches. The pressure is proportional to the height, so we can get the pressure in hPa by multiplying the height in inches by 33.86389.
Of course, making precise measurements requires great care (very pure mercury, no air in the tube, careful calibration, …) but by our period (1914) barometer manufacturers were making very good instruments. There are, unfortunately, still a few complicating factors which we need to be aware of:
- The weight of a column of mercury changes with temperature – the weight of 760mm of mercury is less when it’s hot than when it’s cold, so we need to adjust for this when calculating pressure from height. A further complication is that the column height is usually measured using brass measuring rods, and the length of brass rods also changes with temperature. So we apply a correction from a table or an empirical formula – these tables vary slightly depending on the barometer design, but in OldWeather we don’t usually know the make of barometer in use so we use a generic table. To make this temperature correction we need, of course, to know the temperature of the barometer: Almost all mercury barometers have a thermometer attached and it is usual to record the barometer height and attached thermometer temperature together – as is done in many of our logs. Moving from 0C to 35C (Arctic to the tropics or February to July in the UK) would introduce a change of about 0.5% (2 tenths of an inch).
- The weight of a column of mercury changes with latitude. We launch satellites from French Guiana, rather than Europe, because satellites weigh less in French Guiana than they do in Europe. Moving from Plymouth to Singapore would reduce your weight by about 0.2% (about 8 hundredths of an inch)
- We generally want the pressure at sea-level. We usually keep the barometer above sea-level, so we need to add a little to the pressure to adjust for this. Every 80 or 90 feet above sea-level reduces the pressure by 1 tenth of an inch.
- It’s usual to measure the position of the top of the mercury column. As the mercury rises in the tube, the level of the mercury in the cistern at the bottom of the tube will fall. Because the mercury column balancing the atmosphere runs from the top of the level in the tube to the level in the cistern, we need to add a little to measured height changes to allow for this.
- If the glass tube containing the mercury column is narrow (to reduce weight and to damp oscillations) the height of the mercury will be reduced by capillary action. We need to add a little to the measured height to allow for this.
We call these, respectively, the temperature correction, the gravity correction, the height correction, the capacity correction and the capillary correction. By 1914, with a good barometer, the last two should have been allowed for in the instrument’s calibration and operation, and the third is small for ships, but we still need to make the first two corrections. The changes involved are small compared with the changes associated with short term weather, but they are important for correctly representing the more subtle, longer-term changes.
Mercury barometers are great for fixed, stable, weather stations. They are however expensive, difficult to read accurately in a ship in motion, a terrible nuisance to carry around, and really too fragile for service in a warship. So much ingenuity has been spent on devising cheap, portable, alternatives. The aneroid barometer is essentially a sealed metal bellows that grows and shrinks as the air pressure rises and falls, coupled to machinery to amplify its movements and display them on a scale. These first appeared in 1843, but it took a long time to make them accurate and reliable enough for serious use. By 1914, however, they were coming into use, and it’s clear from the logs that our ships used both mercury and aneroid barometers. Aneroids don’t require gravity, capacity, or capillary correction – and are mostly deliberately designed to be insensitive to temperature changes, so they don’t need an attached thermometer measurement. Nowadays aneroid barometers report pressure in hPa, but back in 1914 most gave readings in inches of mercury. So far I’ve only seen one ship reporting pressures in hPa – HMS Glowworm.
Were the aneroids on our ships less accurate than mercury barometers? more accurate? different in some subtle way? I don’t know – but I look forward to finding out. So if you see any reference in the log to the type or make of barometer in use, please transcribe it. We don’t need to know what they were using, as we can guess with good accuracy, but it does help. A few ships record both mercury and aneroid barometer readings – if you see this, please transcribe both of them; the comparison between them helps us estimate the accuracy of the measurements.
Remember the fog of ignorance – the uncertainties in global weather reconstructions that our new observations will help to clear away? Here’s another view of the problem (click on the image to see the movie version).
The skill with which we can reconstruct past weather depends critically on how many observations of it we have, and for the period we’re investigating in Oldweather, it varies a lot from place to place: in the UK and US we can do reconstructions precisely, but for much of the rest of the world – the southern hemisphere in particular – we’re still very uncertain.
So how much improvement do we expect when we add our new observations to these reconstructions? Obviously this depends on where the new observations were made, and on how good they were, so it’s interesting to compare a few, from a range of times and places. I chose to follow the battlecruiser HMS New Zealand on her circumnavigation in 1919: comparing, at every point in the voyage, her observations of the weather (air pressure) with the existing reconstruction – our best estimate of the weather before Oldweather.
The New Zealand started her trip in Plymouth in February, where we already knew that the weather was miserable – the thinness of the blue line means that we already had enough nearby observations to be sure of the weather, and the spikes in the pressure series are depressions blowing through. The good news is that the ship’s observations agree almost exactly with the reconstructions using other records, which means that the New Zealand was making good observations – they’d calibrated the barometer correctly and were careful in their measurements.
Almost as soon as the ship leaves the UK, the blue line widens – our reconstructions are less certain of the weather. it also gets less variable, as they are sailing in the more stable weather of the tropics. The best illustration of the value of the new data, however, comes in the southern hemisphere: for Australia we already have some observations, so our reconstruction was fairly well constrained already, but New Zealand (the country) was deep in the fog of ignorance, and the wide blue band at that point shows the huge uncertainty in the local weather – an uncertainty that we’re now able to remove using the new observations from New Zealand (the ship).
It will be a while before we can make another global weather reconstruction that includes our new Oldweather observations (that’s a major project taking lots of supercomputer time), but plans for doing it are well advanced. When we’ve done this, and I’m able to repeat this analysis using the resulting reconstruction, then the weather will be precisely known all along the route of the ship (the blue band will be thin at all points) and New Zealand (the country) will have emerged from the fog of ignorance – it’s weather conditions will be clearly known.
Which is only fair, as the New Zealanders paid for the construction of the eponymous battlecruiser in the first place.
Working with the logbooks has done wonders for my knowledge of global geography. If it’s at sea level, one of our ships has probably been there, or at least mentioned sighting it on the way past, and we can travel, vicariously, with them; from Abadan to Zanzibar by way of Cockatoo Island, Fernando Po, Nuku’alofa, Surabaya, and Wuhu (with assistance from lighthouses on Mwana Mwana, Muckle Roe, and Makatumbe).
We’d expect the Royal Navy to spend most of their time in British ports, but we deliberately chose the logs we’re looking at to include those going foreign, and omit the stay-at-homes, because this gives us better information on global weather. This choice means that foreign ports are the most frequently mentioned in our logs. In the 300,000 or so log-pages we’ve looked at so far, Hong Kong tops the ‘most visited’ table (with 23,000 mentions), followed by Bermuda and Shanghai. The first UK port comes in fourth: Devonport (6000 mentions) and though most of these are for the UK naval base near Plymouth, its statistics are boosted by the existence of another base of the same name in Auckland.
The existence of two Devonports highlights a difficulty we run into in using the port names. When the ship is in port, and sometimes when it is operating close to land, the port name or landmark is the only information we have on the ship’s location. So we have to convert the name into a latitude and longitude, and this can be challenging. For many ports a position is not hard to find: Gibraltar, Bombay, Glasgow and Aden are all well known. Many more are only a quick web search away: Esquimalt is on Vancouver Island, Thursday Island is in the Torres strait, and Walvis Bay is in Namibia.
After that it gets harder – East London is nowhere near East London, St Vincent usually means Cape Verde, rather than the identically named place in the West Indies or the Portuguese headland made famous by the battle of 1797. ‘No. 10 dock’, ‘No. 5 buoy’, and ‘No. 7 warf’ are all in Plymouth, but ‘on patrol’, ‘southern base’, and ‘on surveying ground’ could be anywhere.
The Navy are renowned for their courage and seamanship. Their orthography and penmanship are a little more variable, so we have Wei Hai Wei (2345 entries), Wei-hai-wei (1357), Wei hai Wei (633), Wei hai wei (314), Wei-Hai-Wei (231), wei hai wei (91), wei lai wei (69), Weihai wei (57), wei-hai-wei (53), Wei hei wei (33), Wei-hei-wei (32), WEI HAI WEI (30), and even W.H.W (26) – all of which are references to the same place.
With the technology of 1914-22, sorting all this out into a set of positions would have been a terrible job; but modern internet search engines, atlases, encyclopaedias and gazetteers are very powerful tools for tracking down obscure and badly spelt place-names. Today I’m particularly grateful that I live in the future.
My desk in the Met Office is some way from a window, but if I peer across the heads of a few colleagues I can see that the weather outside is, well, disappointing: A gloomy day, with the sky filled with mottled grey clouds from horizon to horizon (though at least it’s stopped raining). Here in the UK we’re famously obsessed with talking about the weather, but sailors would have no time for such waffle: Following an example set by the famous Admiral Beaufort they record the current weather in a terse code, and today’s weather in Exeter would be simply ‘o’ (overcast), or perhaps ‘oc’ (overcast cloudy) if they were feeling extravagant.
The weather code system has evolved quite a bit since Beaufort’s day, and it’s a powerful and concise way of recording notable weather events. The basic code records the amount of cloud in the sky, and ranges from ‘b’ (clear sky or mostly so), through ‘bc, and ‘c’ to ‘o’ (overcast). These are by far the most common codes, but you can add to them to record many of the various nastys the atmosphere can inflict on you – there are codes for rain, snow, hail, gales, squalls, fog etc.
This means that the longer the code recorded in a logbook, the worse the weather was (or at least the more exciting it was). The longest code I’ve found in the logs completed so far is ‘ocpqrlt’ (overcast, clouds, showers, squalls, rain, thunder and lightning) from HMS Bacchante, at Dakar at midnight on 31st August 1917. (Thanks to captain richbr15, lieutenant dazedandconfused, and the crew for patiently typing all that in). This sort of detail, however, is rarely necessary, and, on average, the logs only need 1.85 characters to record the current weather.
I’m excited by the weather codes because they offer a new opportunity to test our climate models. In principle, if we know the surface pressure and temperature (also in the logs, of course) our models should tell us where it’s clear, where it’s cloudy, where it’s raining, and even about thunderstorms and squalls. In practice it’s not quite as easy as that, partly because our computers are not yet powerful enough to run atmosphere models that are detailed enough to resolve small features like thunderstorms and squalls; but even so I look forward to learning more about the accuracy of our cloud and rainfall models. So please keep entering the weather codes – we need the ordinary records of cloud cover as well as the unusual events.
Since I started writing this the rain has come back, so I should modify my current weather report to ‘or’; but improvement is in sight – the forecast for this weekend is for ‘bc’ (broken cloud), maybe even ‘b’ (little or no cloud) at times. The designers of the weather codes were uninterested in particularly fine weather, so there’s no way of encoding ‘glorious sunshine’ for example (‘gs’ would be gales and snow). Still I wish you all as much ‘b’ as you care for, except for a dose of ‘r’ (rain) for anybody praying for it.
One question I’m asked again and again by people encountering OldWeather for the first time is ‘How accurate are the transcriptions?’. We’ve known for a while that the answer is ‘very accurate’, but it’s always nice to be precise about such things, so just how accurate are we?
To find out, let’s look at HMS Defence, which we followed through much of 1914 and 1915, on a voyage from the Dardanelles, to Montevideo, to South Africa, and then back to the UK and patrol in the North Sea. The figure shows the air temperature and pressure recorded during this voyage.
We can see clearly in this image the date when they stopped cruising in tropical and sub-tropical oceans, and returned to the colder and stormier seas around Great Britain – around the beginning of 1915 the air temperature fell by around 30F and the pressure became much more variable. But looking closely at the image, we can also see some errors, both ours and those of the mariners writing the logs in the first place.
We can spot our own errors because each log page is transcribed by at least three people, and when those three people disagree, someone has made a mistake. The logs of the Defence yielded 1119 pressure observations (six a day for about 6 months). For 997 of those observations (89%) everyone who transcribed the observation agreed what it was; for 107 of the observations (10%) two or more of the transcribers agreed on a value, but 1 person disagreed; and for the remaining 15 observations (1.3%) the transcribers did not agree, there was no value with a clear majority of the inputs. (The values entered by individuals that did not agree with the majority are shown in the figure as small red points.)
From the first two categories we can estimate the transcription error rate: in 997*3+107*2=3205 cases the value entered is correct, and in 107 cases it is incorrect, so the error rate is 107/3312 – about 3%. So transcriptions are about 97% accurate – in other words, about 97% of the time the value entered by an individual transcriber is the value that most people would agree is written in the logs – an excellent individual accuracy rate.
If you are familiar with statistics, you may have spotted an inconsistency here: if one person makes a mistake 3% of the time, at least two out of three people should make a mistake on the same observation only about 0.3% of the time (3%*3%*3), while actually this happens much more often than that (1.3% of the time). The reason for this excess of cases where all the transcribers disagree, is that some of the entries are illegible. For example, consider the barometer height at 4am in the log for Thursday 10th September 1914; this was variously transcribed as ‘30.18’, ‘30.10’, and ‘30.12’ – all of which are plausible readings. In this case there is no one answer we can agree on and the disagreement is not a transcription error but a success – we have flagged an entry which cannot be transcribed with confidence. (This is why we encourage you to guess when entering hard-to-read values, when everybody guesses a different answer we know the entry is illegible.)
Even when we have transcribed a value with certainty it may not be correct – sometimes the log-keepers wrote the wrong value in the log: There is no doubt that the barometer height entered for midnight on Wednesday 7th October 1914 is ‘28.80’ inches, but there is also no doubt that the actual pressure was much higher than this (possibly ‘29.80’), and this error can be seen as the first of the three spikes in the figure above. So there are three errors in the log big enough to be obvious in the plot, and probably others with a smaller effect.
This post has turned out much longer and more complicated than I planned – mostly because the definition of ‘transcription error’ from a logbook containing erroneous and illegible entries is not simple – so, in summary:
- Individual transcriptions are about 97% accurate
- Of 1000 transcribed logbook entries:
- 3 will be lost because of transcription errors
- 10 will be illegible
- At least 3 will be errors in the logs
So for every 16 errors in the transcribed data (which we pass to the science team), only 3 are the responsibility of those of us reading the logs; the other 13 are the problems in the logs themselves. We can say with some confidence that we are better at reading the logs than the original log-keepers were at writing them.
Congratulations to captain ebaldwin and the crew for an excellent job on HMS Defence; and to all the oldWeather participants, as the accuracy of transcription is similarly high on all the ships I’ve looked at.
It probably won’t surprise many of you to hear that hear that the Earth is generally warmer at the equator, and colder towards the poles. I base my holiday plans heavily on latitude: going north (from England) for snow, and south for sunbathing. We all know this, but OldWeather has now completed enough log pages that we can prove it just from the logbook observations – the image below shows how air temperature changes with latitude, using the 120,000 temperature observations from pages that have already been examined by the three people we need to provide reliable results.
So it’s warmer (on average) in Singapore, and colder in Scandinavia; we didn’t need the logbook records to tell us that, but that doesn’t mean that this way of looking at the data is not interesting – partly because comparing the temperature records with others made at the same latitude is a good way of finding outliers: values that are likely to be errors in either recording or transcription.
One thing we can immediately see from the figure is the spikes at locations associated with ports. The spikes go both up and down, meaning there more of both high and low temperatures at these locations. Partly this will be a a physical effect – temperatures over land do vary more than those over the ocean – but it’s also partly an artefact of the way I’ve made the plot: The Navy ships spend a lot of time in port, so we have many more observations from those locations, and so more unusually high or low values. Even in the ports, however, there are very few really way out values, but some are suspicious: are there really marine temperatures below 0F at about 45N? (Seawater freezes at 29F) Those values come from HMS Bayano, off the Canadian coast in December, (thanks captain spudman and lieutenant Dinsdale, among others) so very low temperatures can’t instantly be ruled out, but they will need further investigation.
The variation of barometer height (air pressure) with latitude is less well known, but just as interesting: this picture is dominated by the low pressure variability in the tropics (steady weather) and the much more variable pressure in the higher latitudes (anticyclones, depressions and storms). We can see very nicely the transition, in the southern hemisphere, from the steady trade-wind regions to the famous ‘roaring forties’ and ‘furious fifties’.
Captains care about the air pressure because it warns them of changes in the wind. This sort of plot isn’t ideal for showing winds, because the wind measurements are restricted to the Beaufort scale categories, but we can still see where the strong winds are to be found. Cruising in the North Atlantic, the Royal Navy’s main stamping ground, was clearly no picnic: with temperatures down to freezing, variable weather and strong winds.
The Beaufort scale only goes up to 12; extensions are sometimes used for severe tropical storms, but the value of 15 recorded by HMS Cambrian in Rosyth dockyard in March 1919 is not credible. (Though I congratulate captain MamaLizard and the crew on correctly entering the value in the log – we always want the value written, even when it’s obviously an error). If we disregard the Cambrian’s exaggerations, there are four reports of wind force 12 so far, but they are all typographical errors – it’s not much of a slip of the pen to turn ‘1-2′ into ’12’. We’re still waiting for our first real hurricane.
To use the weather records in the OldWeather logbooks, we need to know not only what the observed temperature and pressure were and the date and time of the observation, but also the position of the ship at the time the observation was made (its latitude and longitude).
We are collecting quite a bit of position information in the logs: if the ship is in port, we get the port name; if at sea, the latitude and longitude at noon (and sometimes at 8am and 8pm as well). From this information it’s relatively straightforward to estimate the ship position at any time of the day (we just draw a line between the noon positions and use positions along this line). But, as with all the data we collect, this doesn’t give us exact positions – any of various problems might cause the positions to be inaccurate, and in using the observations we have to make allowances for these inaccuracies.
What might go wrong:
- The port name is not always enough to uniquely identify the ship’s location. There is a Devonport in Plymouth in the UK – but there is another one in Auckland, New Zealand. HMS New Zealand was in Devonport on valentine’s day 1919, but which?
- The latitude and longitude might be wrong, either because of transcription error, or because of an error in the log. A common such error is confusion between east and west longitudes (or between north and south latitude).
- The noon observations are not perfect: they rely on the accuracy of the chronometers and observations used to calculate them. Dead-reckoning and observed noon positions are commonly a few minutes (maybe 10 miles) apart.
- When in port, we only have the name of the port, not the precise position of the ship in the associated harbour or anchorage. On December 16th 1919 the New Zealand’s position was given as ‘in Panama Canal’: The Panama Canal is 48 miles long , where were they exactly?
- Estimating a position at midnight (say) by assuming it’s half way between the preceding and following noon positions assumes the ship is sailing all the time on the same course at constant speed. This is rarely true, so interpolating positions at times of observation from noon positions will introduce an error. In theory, this could produce a big error – if a ship travelled at full speed (say 30 knots) in a straight line between noon and midnight, and then turned around and returned to its starting point for noon the next day, our estimate of the position at midnight would be out by about 400 miles; but ships almost never actually behave like this: as we’ve seen in the routes of the New Zealand, and the Goliath, Gloucester, and Glowworm, ships generally either hang around in one place, or move fairly directly from one point to another. This estimation does introduce an error into positions, but it’s usually modest, rarely more than 30 or 50 miles.
The first two of these will produce big errors in the positions (chosing the wrong Devonport is close to being the biggest error possible), so they are serious, but also easy to spot. These are a nuisance (because we have to correct them), but hardly ever a problem. The last three points are hard to correct, and do introduce modest errors in the observations positions.
The figure shows the noon positions of HMS Pegasus on her Voyage from Plymouth to Arkhangelsk and back to Dundee in 1919 – ably digitised by captain Uldis Ohaks, lieutenants Manock and elizabeth, and their crew.
One position is clearly in error – the visit to Baffin Bay (marked in red) is impossible. Checking the log page for that day shows that, most unusually, both the latitude and longitude of this point are wrong. The longitude has been incorrectly captured by our system as 59 degrees west, while it is actually 59 minutes (i.e. 1 degree) west. We have correctly captured the latitude given in the log (68 degrees north) but this must be an error in the log, because that page also refers to sighting landmarks in Orkney and Caithness, so the ship must be around 10 degrees south of this. We have to guess the actual position, and, in this case it seems likely that the ship was actually at 58 degrees north, and the entry in the log is a typo.
The other positions clearly don’t contain big errors, but it’s clear that the ship didn’t always travel in a straight line between her noon positions – the lines linking them on the map often cross land, in England, northern Norway, and the Kola Penninsula. In these cases, the actual midnight position of the ship is probably about 100 miles from the straight-line-guess.
It’s tempting to fix these problems: we could improve our ship position estimates noticeably by using more sophisticated methods of tracking them. For example, the logs often contain hourly course and speed information, so it would be possible to digitise this and make hourly position estimates by dead reckoning; when within sight of land the logs often contain bearings to points on shore, which could also, in principle, be used to derive a more precise route for the ship. So we could certainly do better (at the cost of a great deal of work) but we’ll never have perfect information on the ship positions, so it’s worth asking first, how precisely we need to know them.
The weather shows itself both in very local effects (showers, contrails, frost hollows) and in very big effects (such as the drought, and now flooding, in Australia). A good illustration of this is the excellent Oldweather authors poster – if you wanted to paint this image you’d need a large brush for North Africa and the tropical oceans, and a very small brush to capture the fine detail. We can capture this detail with modern satellites, but to do the same from ship observations we’d need millions of ships, densely packed over all the oceans – an obvious impossibility. So when doing historical weather reconstructions from ship data we have to forget about the small-scale effects, but it’s still really important to get the large-scale effects correct. Throw away the small brush, and concentrate on using the big one to best effect.
It turns out that the combination of the number of weather observations we can collect, the accuracy of their pressure and temperature measurement, and the power of the computers available to us, mean that we can’t reconstruct weather effects at smaller scales than about 200 miles. (For the Oldweather period – we can do better in the present day). And that’s with the new observations we’re producing; without them in many areas we can’t reconstruct the weather at all.
This means that, in practice, a ship position error of a few 10s of miles is not a big problem, and doing all the work necessary to get more precise positions would hardly help us at all. It’s much more useful to spend our effort on entering more data, and that’s what we’re doing.
So if your ship seems to have taken the M1 (road) out of London instead of the more conventional choice of using the Thames and North Sea, don’t worry – the observations are still useful. Unauthorised excursions to Greenland, Baffin Bay, and other far-flung locations are unacceptable, however, and will have to be corrected.
Quite a few people have asked why we don’t have to input the time of each weather observation. It’s a sensible question, and we do need the observation times, particularly for tracking fast-changing weather events like moving fronts. But one of the clever features of the the oldweather website is that we don’t have to enter the times – they are automatically collected through the process of entering the weather data.
To do this we take advantage of a symmetry between space and time in the logs (scientists love symmetries). The top of each log page corresponds to the beginning of the day, and the bottom of the page corresponds to the end of the day. So the further down the page an entry is, the later in the day it was taken. We record the position of the push-pin for each entry digitised from the page and from that push-pin position, we can find the time associated with that entry. The image below shows the positions of all the weather observations entered from the logs of HMS Bacchante (thanks captain richbr15, lieutenants dazedandconfused and davemcg, and all the crew).
As with most books, there are two sorts of pages: left-hand (red dots) and right-hand (blue dots). They have different margins in our images, so they don’t line up precisely horizontally, but their vertical position is the same, and that’s what gives us the time.
The Bacchante recorded the weather at the end of each watch: so at 4, 8 and 12 a.m., and the same times in the afternoon. They also recorded it at the end of the first dog watch (6 p.m.) – so we should expect to see three equally-spaced groups of points in the top half of the figure (the morning), and four, more irregularly spaced, groups in the bottom half (the afternoon). This is exactly what we see, and it’s clear that for the vast majority of the observations, we can easily say which watch they are associated with, and so when they were taken.
There are a few observations that are not quite so easy – we can see some smaller clusters of observations above and to the left of the main clusters; but again, it’s easy to see which watches there observations correspond to. There are also a few observations in irregular positions – lost in time and space – but these are only a tiny fraction of the total.
So it’s going to be easy to find the time of observation in the usual case where the observations are 2 or 4 hours apart. For the diligent few log-keepers who recorded observations every hour or even more frequently, we will have to be a bit cleverer; and use the differences between the observed weather values, as well as the position on the page, to group the observations into clusters and assign them to times.
All this, of course, relies on having accurate positions on the page for each observation, which means lining up the entry box with the observation text carefully each time when entering the data. So far, we’ve done well at this (as the figure shows); I’ve come to expect no less from oldweather, but it’s still a pleasure to see.