Thursday, 26 May 2011

PASing Around with the "Numbers": A Week in the Record

.
The country I live in used to have a government that loved statistics. Every year wheat and sugar beet production was up, the number of new houses built was up, production of tractors and cars was up. Over in the "rotten west" the only thing that the newspapers told us was going up was the number of unemployed and alcoholics, and the number of people shot in the streets by gangsters or killed in some catastrophe or other. Listening to the state media (the only media), one could believe one lived in a paradise on earth, except there were food shortages in the shops, there was a waiting list for new cars and somehow those new houses and flats seemed not to have been built in the citizen's own town. We lived in a world of statistical myth. Knowing that, nobody listened to the numbers and dismissed the claims that things were improving as propaganda. Which is a pity, because although life was bad (really bad) for most of us, things were happening which were to have great significance when the country threw out those communists and began to live under a new system. Many other nations in the post-Soviet Bloc countries found themselves with a far worse social and economic infrastructure with the results we see.

Like the countries of eastern Europe before 1989, the PAS loves broadcasting its propaganda of success. From press releases and conference programmes it can be seen that it knows no other type. The PAS webpage gives a running account of how the numbers keep getting bigger and bigger. So yesterday during the launch of the 2008 report, instead of hearing about them, we heard of the bigger numbers achieved two years later in 2010. We were treated to effusive accounts of how well "the numbers" show the PAS is doing in its struggle to inform citizens about the wealth of archaeological treasures that lies beneath their feet just waiting for them to "have a go" themselves at digging them up and showcase them on the PAS database. Now the whole world and its aunt is learning that the British Museum reports a large increase in archaeological finds found by the public and that "archaeological finds are up by 36%". Yesterday this "massive increase in archaeological finds found by the public" was being trumpeted around the press. The figures for 2010 were 139502 records referring to 233273 objects recorded through the PAS. In addition it was announced yesterday that in 2010, there were " 859 Treasure cases, up 10%":
the British Museum manages the PAS, and also administers the Treasure Act (sic) 1996. This increase in finds is mostly due to a rebuild of the PAS finds database in early 2010, which has made it easier to use for recorders and the public, and interns employed to record finds, generously funded by the Headley Trust and Institute for Archaeologists.
Well, that is not the whole truth is it? Let's have a look at those statistics for 2010, using the search the database facilities of the PAS. I have no special access to PAS records, I see as much as the average member of public who pays for the Scheme, so readers - and culture ministers - can check this out themselves from the 'statistics' sidebar in the database, the rest is very user-friendly so everybody can use it.

Using this facility to look at the 2010 Average per month records, we get these results:

Month

Objects

Records

January

12277

4289

February

5584

3509

March

95560

93774

April

57874

4183

May

5258

3948

June

14509

4476

July

5506

4574

August

11299

3459

September

4688

3844

October

6445

4641

November

9017

5062

December

5256

3743


233273

139502


Well, first of all let us look at that massive total, c. 139500 records (referring to 233200 objects) - wow, eh? The figures for 2009 are 39874 records (mentioning 67074 objects), so that is a big increase...

Looking in more detail however we can see that the monthly recording figures for 2010 are indeed a little up on the corresponding values for 2009. But there is one highly significant anomaly. March 2010 (March 22nd 2010 in particular) is interesting. On one day there was a huge leap in the numbers. In fact this is a huge leap which is largely responsible for the increase reported for the whole of 2010. Let us take a look at the PAS recording going on that third week in March 2010 using the PAS database search facility. Have a look at this: Statistical analysis of the database for Friday 19th March 2010 until Saturday 27th March 2010

Number of records: 91388 (Number of objects to which they refer: 92140 - an uncharacteristically low ratio this week, see why below)

Number due to Responsible Metal Detectorists reporting their Finds:
14118 records overall (15.4%) referring to 14864 objects.

Quantities recorded per Officer and assistants:
Only 36 FLOs recorded anything at all during that week (most recorded between 3 and 40 objects - average 17.1) That is 645 records referring to 1397 objects
Hero recorders that week:
Adam Daubney (Lincoln): submitted 66 records referring to 101 objects
Andrew Brown (Suffolk): submitted 58 records of 58 finds
and Ciorstaidh Hayward Trevarthen (Dorset) boosting the "number of objects" statistics with just one record, but referring to 662 objects.

What is significant in the records made in that week is that two additional bodies of data are present:
Peter Guest (not a PAS employee) submitted 52,812 records (listed as referring to 52812 objects) inserted from Cardiff University's Iron Age and Roman Coins of Wales project database.
and Celtic Coin Index (not a PAS employee) submitted 37,931 records (also listed as containing records of the same number of objects) inserted from Oxford University's Celtic Coin Index database.

So of March 2010's total of 93,774 records 90,743 are from these two extraneous (and independently funded and operated) sources alone added to the PAS database on March 22nd. That means only 3031 objects were recorded from outreach to both metal detectorists and non metal detecting members of the public that month (about seventy per FLO per month). It also means that c. 91,000 records of the total of 139,502 quoted for 2010 are in fact inserted from these other sources, and therefore only 48,759 come from the outreach work of the FLOs. So in 2010 there were just 8885 more records created on the PAS database as a result of outreach among the public in 2010 by the PAS than in 2009. That is not an increase of "36%" is it?

This alleged "36%" increase in statistics about PAS outreach and the overall totals have however been lauded by metal detecting Minister Ed Vaisey and others as showing the contribution of artefact hunting to the work of the PAS - which in turn shows how it is "working". But let us look at the actual date of finding of the objects entered onto the PAS database from those two external sources and now being included in the PAS statistics for 2010 (again from the PAS database). The source is the same statistical analysis of the database for Friday 19th March 2010 until Saturday 27th March 2010: Let us see how many of those data counted as "finds recording in 2010" come from finds made in and around 2010, or even in the same decade or century:




Year of discovery

Year Objects Records
No year recorded 64296 64244
year zer0 5945 5945
1720 3 3
1736 1 1
1746 1 1
1749 19 19
1750 1 1
1761 1 1
1762 5 5
1764 1 1
1775 1 1
1781 3 3
1786 1 1
1788 1 1
1796 1 1
1800 11 11
1801 1 1
1803 6 6
1805 1 1
1806 2 2
1813 1 1
1816 1 1
1821 1 1
1824 1 1
1825 9 9
1827 22 22
1829 21 21
1830 17 17
1832 6 6
1835 2 2
1836 1 1
1837 3 3
1838 9 9
1839 2 2
1840 9 9
1841 4 4
1842 19 19
1843 7 7
1844 4 4
1845 2 2
1846 1 1
1847 5 5
1848 31 31
1849 65 65
1850 5 5
1851 5 5
1852 1 1
1853 86 86
1854 11 11
1855 7 7
1856 5 5
1857 12 12
1858 5 5
1859 8 8
1860 167 167
1861 3 3
1862 9 9
1863 6 6
1864 60 60
1865 9 9
1866 6 6
1867 17 17
1868 7 7
1869 17 17
1870 18 18
1871 6 6
1872 5 5
1873 59 59
1874 4 4
1875 18 18
1876 4 4
1877 7 7
1878 99 99
1879 6 6
1880 29 29
1881 5 5
1882 11 11
1883 4 4
1884 3 3
1885 2 2
1886 5 5
1887 8 8
1888 14 14
1889 21 21
1890 30 30
1891 10 10
1892 32 32
1893 15 15
1894 5 5
1895 14 14
1896 7 7
1897 2 2
1898 115 115
1899 9 9
1900 22 22
1901 5 5
1902 6 6
1903 15 15
1904 33 33
1905 611 611
1906 4 4
1907 53 53
1908 118 118
1909 6 6
1910 6 6
1911 331 331
1912 31 31
1913 76 76
1914 21 21
1915 8 8
1916 5 5
1917 8 8
1918 18 18
1919 132 132
1920 9 9
1921 3 3
1922 4 4
1923 7 7
1924 14 14
1925 14 14
1926 4 4
1927 69 69
1928 14 14
1929 6 6
1930 34 34
1931 46 46
1932 30 30
1933 8 8
1934 35 35
1935 42 42
1936 16 16
1937 23 23
1938 16 16
1939 15 15
1940 12 12
1941 6 6
1942 3 3
1943 2 2
1944 2 2
1945 3 3
1946 3 3
1947 4 4
1948 122 122
1949 6 6
1950 17 17
1951 5 5
1952 12 12
1953 10 10
1954 40 40
1955 30 30
1956 9 9
1957 25 25
1958 23 23
1959 14 14
1960 69 69
1961 26 26
1962 56 56
1963 51 51
1964 35 35
1965 68 68
1966 110 110
1967 111 111
1968 84 84
1969 38 38
1970 150 150
1971 99 99
1972 183 183
1973 127 127
1974 33 33
1975 43 43
1976 209 209
1977 283 283
1978 128 128
1979 146 146
1980 131 131
1981 122 122
1982 267 267
1983 274 274
1984 464 464
1985 862 862
1986 455 455
1987 869 869
1988 255 255
1989 199 199
1990 386 386
1991 300 300
1992 602 602
1993 426 426
1994 828 828
1995 658 658
1996 330 330
1997 129 129
1998 90 90
1999 336 336
2000 398 398
2001 286 286
2002 463 463
2003 489 489
2004 254 254
2005 35 35
2006 34 34
2007 4 4
2008 23 22
2009 194 182
2010 6105 5418

92140 91388

Of the actual total of records on the PAS database made in that week for 2010 as we have seen, at a maximum 645 were records (referring to 1397 objects) created by the FLOs as a result of the PAS' public outreach. The rest seem clearly to be inserted data from these two other external and independent sources.

The CCI and IARCW data are also responsible for a large peak in the number of findspots recorded in March 2010 with 8 figure National Grid References (888) and 10 figure one (52870). Normally PAS data do not contain as many records of findspot location with such precision.

I think the figures speak for themselves. Far from the figures indicating that in 2010, a substantial number more members of the public (including metal detectorists) came forward with finds they had recently made is an illusion. What is happening instead is that in 2010 the PAS gained access to data compiled by others quite unrelated to the Scheme, some of it recorded nearly 300 years ago and it is these data which are being presented on the PAS webpage as part of its own "achievement" in recording finds. This is at best misleading, but - given the fact that there is not a word of this in the BM press release, as reported on the Scheme's own website - could also be construed as dishonest.

It might be suggested that it is legitimate to count these figures in the PAS database as they are date about "finds made by the public" - which is what, in broad terms, the PAS database records. This is false on two counts. Firstly the CCI and IARCW databases both contained considerable numbers of records of finds made during archaeological excavations and archaeological surveys. They were both funded by outside sources as academic research projects, not public outreach. Secondly, and more importantly, both of them already existed in the form of standalone databases, and could comfortably have continued to do so. Incorporating them in the PAS database is unnecessary and seems primarily an idea seized upon cynically to bump up the Scheme's statistics to make it look as if - in its outreach to members of the public, and the "metal-detecting" community in particular, the PAS is "working" much better than it actually is.

I really do not see the need for this subterfuge (for there really can be no other name for it). In fact it hides the real achievement of the Scheme which has been a genuine rise in the number of records being made by genuine outreach. I think this is largely due to the reconstruction of the database which streamlines the manner in which FLOs enter these data and is enabling them to get through their recording backlog more rapidly. The PAS can tell us themselves the percentage I am sure. The question is however, given their propensity for 'spinning' their statistics in a form which both flatters and defends their metal-detectorist-partners instead of in a form which gives realistic basis for assessment of policy effectiveness, whether anyone apart from DCMS ministers and superficial journalists are really interested in listening to their "look how well we are doing!" bleating any more.

Vignette: Polish propaganda poster from the Old Days: "Better results from working together", might be a good slogan for the PAS .

No comments:

 
Creative Commons License
Ten utw├│r jest dost─Öpny na licencji Creative Commons Uznanie autorstwa-Bez utwor├│w zale┼╝nych 3.0 Unported.