The Earth: Shape, Coordinates and Great Circles

Every navigation calculation starts from a model of the Earth. A position is a pair of angles, latitude and longitude; a distance is an arc of the Earth's surface; a direction is an angle measured from a meridian. Because the Earth is round and its meridians meet at the poles, the geometry is not that of a flat map: the shortest route between two points is a curve whose direction keeps changing, and a line of constant direction is not the shortest.

This article covers the shape of the Earth and the geodetic datums that fix coordinates, the grid of meridians and parallels, the arithmetic of change of latitude, change of longitude and departure, great circles and rhumb lines, convergency and conversion angle, and the units of distance and speed. These are the foundations of EASA subject 061 General Navigation and of the navigation chapters of the FAA knowledge tests; how the curved Earth is flattened onto paper is covered in chart projections.

Shape of the Earth and the geoid

The Earth is not a perfect sphere but an oblate spheroid: a sphere flattened at the poles and bulging at the equator. Its equatorial radius is about 6,378 km and its polar radius about 6,357 km, a flattening of about 1 part in 298. For most navigation problems, and throughout the exam formulas below, the difference is ignored and the Earth is treated as a sphere.

Two further surfaces matter once positions and heights must be exact:

• The geoid is the surface that mean sea level would follow if it were extended under the continents. Gravity varies from place to place with the distribution of mass inside the Earth, so the geoid is gently irregular.
• A reference ellipsoid is a smooth mathematical surface, defined by its radii and flattening, chosen to fit the geoid as closely as possible. Coordinates are computed on it.

The height of the geoid above or below the ellipsoid at a given place is the geoid separation, also called geoid undulation. In the EGM96 geoid model it ranges from about −105 m in the Indian Ocean to about +85 m near New Guinea. A GNSS receiver computes height above the ellipsoid and converts it to height above mean sea level with a geoid model. The result is a geometric altitude, unaffected by pressure or temperature, but terrain clearance and separation are still flown on the barometric altimeter (see altimeter settings, altitude and height).

Geodetic datums and WGS-84

A geodetic datum is a reference ellipsoid together with its position and orientation relative to the Earth. The same point on the ground has slightly different coordinates in different datums, which is why a navigation database, a chart and a satellite receiver must all use the same one.

Aviation uses the World Geodetic System 1984 (WGS-84). Published geographical coordinates in aviation have been expressed in WGS-84 since 1 January 1998, and since 5 November 1998 the geoid undulation at surveyed positions has also been published in the aerodrome section of the AIP. The precision now demanded is high: the French AIP defines each fix of a PBN approach by WGS-84 coordinates to a tenth of a second of arc, and runway thresholds to a hundredth. Where an item has not been surveyed to WGS-84 the AIP says so; the Chambéry entry, for example, flags its glide path and DME coordinates as "non WGS-84".

Other satellite systems use their own frames. GPS works in WGS-84 and GLONASS in PZ-90 (Parameters of the Earth 1990), a Russian model close to WGS-84. The transformations between such frames are well known and are applied inside the receiver (see GNSS).

Latitude, longitude and the graticule

• A meridian is half of a great circle running from the North Pole to the South Pole. The prime meridian, through Greenwich, is 0°.
• Longitude is the angle, measured at the Earth's axis in the plane of the equator, between the prime meridian and the meridian through a place: 0° to 180° east or west.
• A parallel of latitude is a circle on the surface parallel to the equator.
• Latitude, treating the Earth as a sphere, is the angle at its centre between the plane of the equator and a line to the place: 0° at the equator to 90° north or south at the poles.

The network of parallels and meridians drawn on a globe or a chart is the graticule. Latitude and longitude together fix a position, written in degrees, minutes and decimal minutes or seconds; a Jeppesen VFR chart gives the Stuttgart NDB as N4842.7 E00920.1, for example.

The antipodal point of a position is the point diametrically opposite it through the centre of the Earth: the same latitude in the other hemisphere, and a longitude differing by 180°.

Change of latitude, longitude and departure

The change of latitude (ch.lat) and change of longitude (ch.long) between two positions are found by subtracting the values if both lie in the same hemisphere and adding them if they lie on opposite sides of the equator or of the Greenwich meridian. Flying south along 030°W from 25°N to 12°S is a change of latitude of 37°, not 13°. Two aerodromes on 017°W and 026°E are 43° of longitude apart. Across the 180° meridian the change of longitude is 360° minus the sum of the two longitudes.

Distance along a meridian follows directly from the definition of the nautical mile: one minute of latitude is one nautical mile, so a degree is 60 NM, and the 37° flown above is 2,220 NM.

Distance along a parallel is different, because the meridians converge. The east-west distance along a parallel is called departure:

On the equator a minute of longitude is a nautical mile; at 60° it is half a mile. Two points on 45°N at 010°W and 020°W are 600 minutes of longitude apart, and 600 × 0.707 gives about 424 NM. Two aerodromes on 48°N at 010°W and 004°E are 840 minutes apart, which becomes about 562 NM. The formula also works backwards: an aeroplane flying due east along 30°N for 240 NM changes its longitude by 240 ÷ 0.866, about 277 minutes or 4.6°.

Great circles and small circles

A great circle is a circle on the Earth's surface whose centre, and whose plane, pass through the centre of the Earth. Its radius is the Earth's radius, and the shorter arc of the great circle through two points is the shortest distance between them. Only one great circle passes through any two points, unless they are antipodal, in which case an unlimited number do. The equator is a great circle, and every meridian together with its anti-meridian forms one.

A small circle is any circle on the surface whose plane does not pass through the Earth's centre. Every parallel of latitude except the equator is a small circle.

Radio waves also follow great circles, so a bearing measured by a VOR, an NDB or a VDF station is a great-circle bearing. Great-circle routes save distance on long flights, which is why computer flight plans and FMS legs are built on them.

Rhumb lines versus great circles

A rhumb line, or loxodrome, is a line that cuts every meridian at the same angle: a line of constant true track. It is easy to fly with a compass, but except along a meridian or the equator it is longer than the great circle joining the same points, and extended far enough it spirals towards the pole. The meridians and the equator are the only lines that are both great circles and rhumb lines; every other parallel is a rhumb line and a small circle.

On a great circle vs rhumb line comparison, three facts carry most exam questions:

1. A great circle crosses each meridian at a different angle, so its true track changes along the route. The track at the start is the initial great-circle track, and at the end the final track.
2. The great circle always lies on the polar side of the rhumb line, north of it in the northern hemisphere and south of it in the southern.
3. Halfway along, at the mid-meridian, the great circle runs very nearly parallel to the rhumb line. Its track there, the mean great-circle track, is approximately the rhumb-line track, which is why tracks are measured at the mid-meridian of a leg.

Convergency and conversion angle

Meridians are parallel where they cross the equator and meet at the poles. The angle between two meridians at a given latitude is called convergency, convergence of meridians or Earth convergency:

It is zero at the equator and equal to the full change of longitude at the pole. Because a great circle crosses the two end meridians at different angles, the difference between its initial and final tracks equals the convergency between them.

The conversion angle is the angle between the great circle and the rhumb line at either end of the route, and it is half the convergency:

Worked example: from 50°N 010°W to 50°N 040°W the rhumb line runs along the parallel, 270°(T). Convergency is 30° × sin 50°, about 23°, so the conversion angle is about 11.5°. The great circle starts on the polar side, north of west, at about 281.5°(T), and arrives at about 258.5°(T); the two differ by the convergency. In the southern hemisphere the great circle bows south, so an easterly great-circle track starts at a figure greater than the rhumb-line track.

The same geometry explains why radio bearings need correcting before they are plotted on some charts, and why grid navigation replaces true north near the poles (see polar and grid navigation).

Nautical miles, statute miles and knots

The nautical mile (NM) was originally one minute of arc along a meridian. Because the Earth is flattened, that arc varies from about 6,046 ft near the equator to about 6,108 ft near the poles, so the international nautical mile is fixed at exactly 1852 m, about 6,076 ft. The older United Kingdom nautical mile of 6,080 ft is obsolete, though 6,080 survives as a working round figure.

The statute mile (SM) is 5,280 ft, about 1,609 m or 0.868 NM. It survives in United States practice, most visibly in the visibilities reported and charted in statute miles. The knot (kt) is a speed of one nautical mile per hour.

These equivalents are used constantly, from converting a chart measurement in centimetres to nautical miles to turning 300 ft per NM into a descent path of about 3° (see aeronautical charts and dead reckoning and visual navigation).

Library articles are written for study and exam preparation. They do not replace your aircraft's approved documentation, your operator's procedures or the regulations themselves.