Evidence based approach?

Senator Conroy is on record as saying

The government intends to take an evidence based approach to this issue. The results of the live pilot will inform the government’s policy in this area. (Senate, 13 November, 2008)

A number of questions arise from this statement.

1. What is an evidenced based approach?
2. What evidence will the live pilot produce?

I’m not an expert in Evidence Based Medicine from which Evidence Based Policy (and I’m assuming, an evidence based approach to policy) arises. I do remember reading a critique of EBM somewhere online, but as it’s only a side issue for me, I did not bookmark nor record that particular piece of ‘evidence’. I do, however, have some skills in searching out evidence, skills I used to find some critiques of evidence based methodologies.

One of the most interesting finds was a paper on Evidence Based Policy which looks specifically at the Australian context. It was noted in that paper there are many forms of evidence including

Expert knowledge; published research, existing research; stakeholder consultations; previous policy evaluations; the Internet; outcomes from consultations; costings of policy options; output from economic and statistical modelling (UK Cabinet Office, 1999 quoted in Marston and Watts, 2003, p145)

This is not an exhaustive list, but gives some idea of the kinds of ‘knowledge’ that can be used within the process of making policy. Marston and Watts make the claim that “the meaning and practice of ‘evidence-based policy’ are contested” (2203, p143). Knowledge is also highly contested[1] and hierarchical, with scientific, objective forms of research contributing more strongly than other forms of empirical data (including expert knowledge derived from practice – that is, that developed by Systems Administrators who daily engage with the concepts under contestation).

Marston and Watts (2003, p150-1) give the following list as the “core features of all evidence-based arguments“:

• an implied or identified question
• a claim or proposition that such and such is the case or that such and such explains or renders intelligible
• the evidence adduced in support of that claim or proposition
• a set of assumptions that have assisted in a) shaping the question, b) selecting what will count as the relevant evidence and c) which then link the evidence to the claim by means of conceptual processes that warrant the proposed link between the claim and the evidence actually advanced.

Let’s apply these to the current debate.

The implied or identified question

The identified question seems to be “Can we protect children from viewing unsavoury material?” NOTE: This is not the same question as “Can we protect children from abuse?” The latter question is often asked and to my knowledge, as a society, no, we can’t. We try. But we are not as successful as we could be.

Working off my own set of assumptions, I believe the question is “Can we control the content that the Australian People access on the Internet?” This is definitely an implied question, but if I wade through all of the reports to date, I cannot see a more fundamental question than this. It seems to lie at the very foundation of the current discourse.

The claim

Here, I go back to the statement by Senator Conroy made in “Labor’s Plan for Cyber-safety“.  Senator Conroy states that “Cyber-safety skills are important” and “Children today are ‘digital natives’”[2]. So, the assumption, then, is that teaching children about cyber-safety is important. As an educator of school leavers, who possibly make up more than 50% of my first year class (the remainder include international students and mature aged students), I can safely say that the current evidence is that young people do not really understand the full implications of a hyper-connected environment. They have little idea about identity and trust; very little insight into the kinds of scams prevalent on the Web or in email, and very little understanding of the potential of the technology that they will be engaged with on a daily basis. Senator Conroy states that young people “arguably primarily spend hours on MySpace or MSN or writing blogs”. Sorry to disagree with you Mr Conroy, but my students generally do not even know what a blog is.

Also in the Cyber-safety plan is that “Labor recognises that cyber-safety today is an important part of children’s overall health and well-being” followed shortly thereafter, and emphasised by its own box is

Just as we teach Australian children about the risks of drink driving, a Rudd Labor Government will teach Australian children how to be responsible cyber-citizens and about the importance of cyber safety.

There’s a bit problem with that statement. Teaching about the risks of drink driving is to highlight the potential for the causing the death or serious injury of themselves or another person, not to mention the ruination of property. A better analogy for Cyber-safety would be teaching children to cross roads safely, not to ban them totally from roads and not to close of freeways because they’re dangerous and a child might wander onto one. But I get ahead of myself.

We can see that the claim that the internet is a big bad scary place is firmly entrenched by people who are not in the “Digital native” category. They seem to fear what they cannot understand and so we have the proposal for a clean feed based on a set of less than appropriate ideas and almost coming unannounced in Labor’s Cyber-safety plan. Let’s turn our attention to the evidence.

What evidence have we to date?

Looking over the information available to most Australians, there seems to be some reason to question Senator Conroy’s statement above. There is plenty of evidence about providing a “clean feed” within Australia.

1. EXPERTS: SAGE-AU: Systems Administrators Guild of Australia: Press Releases. Systems Administrators are the people who build and maintain networks. They deal with the issues of ensuring users get what they want when they get on a network. SAGE-AU is “the network behind the people behind the network“. So if this network of people believe that there are fundamental flaws with the proposal, then, as the people who build and design networks, you’d think their concerns would be seriously considered. No, they are ignored. The most vocal systems administrator, Mark Newton, has described quite clearly the problems inherent in the proposed plans. This is evidence.
2. RESEARCH: The initial trial demonstrated the inadequacies of all of the current solutions. Overblocking and underblocking seem to be huge problems without considering the suggested slowing of the network. I refer here, to the documentation straight from ACMA. I, however, defer to an expert’s reading of this. This is evidence.
3. STAKEHOLDER CONSULTATION: …[3]
4. PREVIOUS POLICY EVALUATIONS: NetAlert, the government-sponsored, computer-based filter solution has been downloaded approximately 150000 times according to this report. Of these approximately 20% are still in use. The current government calls this a failure and evidence FOR ISP level filters. I call this a success and almost a MANDATE AGAINST ISP level filters. These results could also be considered under STAKEHOLDER CONSULTATION, but that’s really outside the evidence that Senator Conroy wants to consider.
5. The Internet: This one speaks volumes of evidence.
6. OUTCOMES FROM CONSULTATIONS: Senator Conroy has been very selective of who he consults. Obviously STAKEHOLDERS are not included. I have no EVIDENCE for this, except a complete lack of evidence.
7. COSTINGS OF POLICY OPTIONS: This was excluded from the ACMA trial considerations and given that the previous option offered to the Australian Public is deemed a failure by the current Government, that cost should also be included, but I’m still not sure of the cost to ISPs nor the cost to the Australian People, nor the cost to Australian Business. There are analyses out there, Google them, while you still can.
8. OUTCOME FROM economic and STATISTICAL MODELLING: This has contributions from ACMA’s study, but I believe the real evidence that Senator Conroy is looking for is the so-called Live Trial. This will probably be deemed a success and evidence FOR filtering, regardless of the actual findings.

As I stated above, this is not an exhaustive list, but there is plenty of evidence. So I have to ask, with all this evidence, what more could be needed?  So let’s turn to what assumptions underlie the current push for an ISP level trial of filtering software.

Assumptions

From above, we have the Internet is a scary place, children cannot protect themselves, children don’t understand the implications of the Internet and moral panic (actually, I think I missed that, but it’s out there and oh, be careful!).

The following is from Labour’s Cyber-safety policy:

Australian children must not simply be taught how to use a computer. Australian children must have such skills as:

• how to be responsible cyber-citizens;
• understanding how their online actions may affect others;
• how to protect their identity online; and
• how to respond to cyber incidents such as cyber-bulling or breaches of privacy.

This I actually agree with. I really do see the need for these things. I teach them in my courses (Business Informatics and Mobile Workforce Technologies). I teach them because students haven’t learnt them. I teach them that and much more (see the Course Outline link on each of the course pages). These are important issues for ALL people to learn, but they won’t learn them in a protected environment. What they will learn is complacency and superficial attention to the things they are doing[4]. The big assumption here is that we can learn things in a protected (simulated) environment which is refuted by the evidence that the Government rejected the outcomes of ACMA trial in a controlled environment.

Another assumption is that “Father knows best”. Sorry, but I reject your reality and substitute my own [5]). Mr Conroy, as has been stated elsewhere, children are more likely to be the victim of someone they know, not some online stranger. Your assumption of paternalistic protection is unwarranted and unfounded. You fear what you do not know and deny what you do.

I think the most telling assumption is that eventually, there will be some evidence to support the mandatory filtering of the Australian Internet. I suppose you could use the results that the earlier filter tests were worse than the current tests to bolster the claim that it is possible. But, this aside, the other assumptions do not hold up under scrutiny, scrutiny which is being denied to the Australian Public.

These are not necessarily the assumptions that underpin the Senator’s position.  I’m sure there are other assumptions that I cannot fathom as I have a different understanding of the Internet and the World Wide Web.  It would be useful to know the assumptions that do lead to the contestable position that mandatory filters are necessary for the well-being of children as this may lead to a different understanding of the position of filtering advocates.

Evidence based analysis?

On an analysis of the evidence presented here and elsewhere, I think the balance of that evidence is that Senator Conroy lacks an understanding of what EVIDENCE BASED means.  The evidence is overwhelmingly against mandatory filtering of content on the World Wide Web and the Internet.

As my daughter has just suggested to me:

I don’t get why the government doesn’t realise that this internet filter is like going through a desert and checking each piece of sand for granules that may one day become a rock[6]

It’s Security Theatre!

References:

Australian Senate, 2008, Debates, 13 November

Marston, G. and Watts, R. 2003, Tampering With the Evidence: A Critical Appraisal of Evidence-Based Policy-Making, The Drawing Board: An Australian Review of Public Affairs, 3 (3) 143–163.

Edited 5/12/08 to add link to Darcy’s Blog

1. and if you contest that you prove my point [↩]
2. Don’t get me started on the whole notion of digital natives. I am a 40+ year old digital native. [↩]
3. This seems to be evidence that Senator Conroy wants to ignore, and succeeds. And that is evidence, too! [↩]
4. There will be students whose parents have taught them these things and these students often don’t get why I teach it, but I digress. [↩]
5. Mythbusters reference, FTW! [↩]
6. That the general pattern is that rocks break down into smaller parts, rather than the other way around, was not lost on either of us! [↩]

This entry was posted on Thursday, December 4th, 2008 at 7:26 pm and is filed under thoughts. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.