Focus on UK Metal Detecting: More Misunderstanding About "the Counter"

Over on a metal detectorists' blog near you, somebody calling herself "Lisa" decides to have a go at dissecting the Heritage Action Artefact Erosion Counter. Quite who she is remains unclear , when you click on the name, you get a message "No-such-user: We’re sorry, we couldn't find that profile". Probably a sock puppet. The blog owner however seems to know her: "It’s nice to know that there are few archaeologists who can look at things objectively and without any pre-conceived bias"). An archaeologist, eh? [Update 23.3.13: the blog owner clarifies that the individual concerned is fresh archaeology graduate Lisa Hume MacIntyre who apparently hangs out a lot with metal detectorists]. Archaeologist "Lisa" has her own philosophy of science ("Science provides us with valuable tools for finding theories. The value of these theories lies in the use of proper methodology"). She reckons that in both theory and methodology, the HA model falls short of her own criteria for true Science (capitalisation as in original), because in her opinion it does not correspond to the typical hypothetico-deductive scheme which she assumes (since she assumes its aims) it should have been applying:

We MUST follow the scientific method across the board!
1. Question…how many unreported artifacts do detectorist take a year?
2. Do the background research….any statistics anywhere else for comparison? what methods were used to collect the data? Sample size? Any bias in the data collection?
3. Construct a hypothesis…well, based on X I think Y must be an accurate number.
4. Test the hypothesis…[...] The test must be “fair”. In other words, all variables must remain the same throughout with changes occurring with only one variable at a time.
5. Analyze and draw the conclusion…Based on these FACTS, the data shows that detectorists take Y amount of unreported artifacts a year.
6. Report your findings
They conducted the first step, of course. We can all ask a question. Their research is based on asking a sample and using the answers in a logarithm to draw a conclusion. Many problems occur right here, which as any scientist knows, makes the rest of the study flawed.

I really do think the lady has not quite got a grasp of what it is we are talking about. One really however questions "Lisa's" actual understanding of what it is she is writing about when she blurts out "Their research is based on asking a sample and using the answers in a logarithm to draw a conclusion [...] The testing involved a logarithm and assumptions". First of all she apparently has not read the introduction to the presentation with any care. The algorithm is not the "test" but the presentation of the "hypothesis" (resulting from an analysis) of Lisa's schematic "how to do Science-with-a-capital-S" presentation. Secondly an algorithm is not the same as a logarithm. Neither is the Counter as presented on the website a "report" of research. The report and the results of research on this - and much else - are appearing in book form. Instead of a proper discussion of the implications of the figures if they are right (or half-right) we see yet another example of the "hostility" (of vested interests?) mentioned by Mike Heyworth in his private letter to John Howland.

To come back to her list, to what "comparable statistics" does "Lisa" refer in stage two? Comparing what with what?  What "FACTS" does she think there are in the UK when the hobby keeps its doings so close to its chest and resents any attempt to look over their shoulders? That's the problem. Indeed it is precisely this problem the Counter is intended to highlight - as Mike Heyworth correctly spotted. Surely the only way for Heritage Action to "test" the "hypothesis" (in other words the results of the analysis of the gathered data) is going to be the use of the same data. I'd like to ask the new "expert" on how to crack this problem where (given the nature of the problem discussed) she sees the possibilities of gaining another kind against which to "test" the first. I'd then ask here why - if they exist - they would not be used in the initial analysis)? There are ways to falsify the "hypothesis" - and Heritage Action urge anyone with real data capable of doing that to demonstrate it. So far, no alternative data which falsify theirs have been produced. Please, Lisa, be our guest.

The sample size is not "small" we've ten thousand or more people out there doing it, hoiking out the finds. What we do not have is them telling anyone about what it is they are doing, that's what we are trying to produce an estimate of. It's a similar problem to finding out how many children suffer sexual or physical abuse in their homes in Florida, there will be no direct data - few parents will admit to assaulting their own children I guess. Does that mean that the question cannot be asked? Does that mean the Florida social services (or a child abuse action group trying to raise public awareness) should be criticised for trying to provide an estimate of the scale of the problem? Of course not. Surely the authorities would not be doing their job if they were not at the same time attempting to determine the scale of the problem in order to identify the resources needed to deal with it.  Who would be the most vocal critics of such an attempt and why? 

As for 'sample size', let us note that just one of the sources of information used was a survey conducted by UK metal detectorists themselves and the basic data from which were published on the UKDN Forum (they were still there last time I looked, but has "Lisa"?). This was intended  to "prove" that critics of the hobby were "wrong" in asserting how much material is being hoiked out. They did a controlled experiment and hoiked away, kept everything and then catalogued what they'd found, giving the details of the number of hours where, and what they found. They were perfectly satisfied by the sample size, and were sure that they had proven their critics "wrong". Nobody in the detecting community was suggesting that their data were "tosh" because of the small sample size (and nota bene because the hypothesis - not a question - was formulated before the data were gathered to "illustrate" it - so quite the opposite of the hypothetico-deductive method urged by our American critic). Funny that she does not examine their results with the same eye to detail. But we have. Now, it just so happens (as we demonstrate in a forthcoming book) that those data in fact show the opposite to what the (incomplete) tekkie analysis of the information was taken to mean. This was the only attempt ever made to quantify this phenomenon by UK tekkies, and its results in fact correspond particularly well with those of the HA Artefact Erosion Counter.

"Lisa" obviously knows nothing of the actual sample size. The Counter's blurb (which I presume she never troubled to look up for herself before dashing off a naive knee-jerk approach), talks of the existing surveys drawn on in looking at finds rate. These were checked against the information candidly volunteered in the accounts of tekkies themselves on UK forums, detailing what they'd found in known periods of time, coupled with an analysis of accounts of various people attending the same rallies, the number boasting "wotta-lotta-stuff-I-got-yesterday [clever me]!"  compared with the number of "got-nuffing-waste-of-time" posts about the same rallies. "Lisa" knows nothing of many months trawling forums for similar accounts of the results of individual searches, or the manner in which they were statistically processed (by Nigel Swift whose professional experience was invaluable here).  I really do not think "Lisa" is in any position to criticise the methodology  when she has no idea what it was, and on what basis the data were collected (from the reports of UK detectorists themselves). The woman even blithely admits that before she saw the blog post, she'd never even heard of the Counter, and presumably does not even know who its author was, or what its background is. 

And if she does not know that, then she will know nothing of the activities and interactions with the UK detecting community in a variety of venues which led up to the Counter's creation, so she is hardly in any position to state: "I also see no evidence of background research except assumptions". All she is showing is her own ignorance of the background to the Counter, from whence it came. She asserts that Heritage Action "did construct a hypothesis, however as mentioned before, it is based on flawed evidence so it is moot", but a moment ago states she knows nothing of what evidence was used. It is HER ASSUMPTION that these data were "flawed" because she wants to be able to pronounce the the model built on them to be flawed to impress her new friends.  Hardly a very objective approach. Or - since she criticises the Counter's author of this -  a very professional one.
 
The Counter is a presentation of what the analysis of data gathered from a number of sources suggest. The Counter is clearly presented as a conservative estimate and as supplied on an as-is basis. This is very clear from the accompanying blurb. As for "testing" it, that really is up to the PAS and those who have direct access to data which can be used to do that - and whose job it is to assess the progress being made with public money to mitigate information loss due to these 'finders' making off with the stuff without reporting it (PAS aim five from 2003). Heritage Action has not ceased calling for this. I have time and time again on this blog pointed out the necessity of testing this model and replacing it with a proper official survey by the authorities responsible for overseeing the management of the archaeological heritage of England and Wales. That was the whole point of posting the Counter up all those years ago. Why are they not posing the question and resolving to find out the answer? Is it "not an important question" when millions of pounds are being spent trying to deal with a problem, when we have - after fifteen years - not the foggiest idea how effective in overall terms that has been.   

But yes, we desperately need a test of the model. Let the PAS do it immediately. That is the whole point of having it up there, and as the Director of the CBA wrote to John Howland "I think it serves its purpose in this regard". Somehow the PAS seems to be dragging its feet in producing comparable figures. Why is "Lisa" not suggesting this? Would she not support us in calling for this to happen?

Finally, I will however add one remark to "Lisa's" comment about "testing". It is an interesting circumstance that, time and time again on this blog I have pointed (with a working hyperlink) to yet another tekkie "look what I have found" post on various tekkie forums as being indicative that the HA Counter's finds rate is by no means an inflated one. In most cases, the moment I do that, the photo, post, video or sometimes whole thread has a habit of 'disappearing'. That suggests to me that I am not alone in thinking that the results of repeated public showing of the results of just a few hours' searching can provide credence for the Counter's "conservative estimate". The UK's metal detectorists themselves are very much aware of the scale of this problem, the trouble is - they don't want to admit it, and [the Jacksonville archaeology graduate] is encouraging them in this and undermining the public awareness efforts of others. Why?


UPDATE (Burble-Burble - March 22, 2013 at 5:37 pm)
The train wreck goes on. "Bill of Lachline" then explained to "Lisa" what the PAS is (getting it round his neck as he usually does)... but she's grateful for the clarification, apparently she knows little about that too. It's not at all clear what she is on about. I am finding it increasingly difficult to understand what she means when she uses the word "theory" (research design?):
 

 if we eliminate the objects that fall into PAS criteria, we are in fact eliminating data [...]  However if you are going to attempt a theory, which they attempted to do with this “study”, and you are eliminating variables, then in essence you have eliminated data. Again, maybe not a problem unless you choose not to reveal such a fact and account for it in your study. [...] And TY for the additional info! It tends to back up what we have all been saying. This study is flawed at best.

What? What objects which "fall into the PAS criteria" does she think the HA Counter "eliminates"? What proof does she adduce that "objects" are "eliminated" in the calculations? More to the point, given the tack of her earlier argument, in what way is she using the word "theory" now? Has she understood anything at all of what the Counter concerns? She says it is "flawed" but quite clearly has no grasp of what it is she is attempting to discuss, so her ability to identify the alleged flaws becomes increasingly questionable. And it seemed like such a simple concept. I think I'll have to ask Nigel to produce a version of the accompanying blurb written in Simple English, reading age 8-11 years old. That should do the trick.


I am sure the topic of my support for the ideas behind the Counter, and maybe the methods by which it was arrived at, will come up at the seminar on the 10th. I'll let you know how I get along.