Innumerate or deceptive: @adropex dissects @TheIPA ‘report’ into ABC coal coverage #leardblockade

IPA-Media-Release-ABC-Fossil-Fuel

Innumerate Ignorance or Calculated Deception

It is unfortunate that the IPA, and in particular James Paterson, did not take note of the ‘scholarship’ referred to in their recent report of the ABC’s coverage of ‘Australia’s energy choices’ and apply it to their own publication. It would undoubtedly have saved them the embarrassment of making public a deeply flawed piece of political rhetoric loosely dressed up as a quasi research effort that would never pass the most basic academic peer review.

A mere glance at the report’s title is sufficient to sound warning bells that the proffered document may not tick all the ethical research boxes. Whilst ‘Public Broadcaster or Green Activist?’ smacks of a political journalistic headlining it is the follow up line ‘How the ABC spins Australia’s energy choices’ that defines what is to come which is certainly not the objective approach real scientific research aspires to.

Does it matter? Well yes, it does. The IPA makes some serious allegations against the ABC in their report and supports the claims on the basis of their landmark scientific research.

“This report marks the first time a comprehensive, independent and scientific assessment of ABC bias has been undertaken.” 

Furthermore, the IPA is an Approved Research Institute and hence qualifies as a deductible gift recipient whereby any donations to the IPA are tax deductible. The tax laws pertaining to this status are clearly defined. The IPA may only use the tax-deductible donations for the purpose of scientific research.

The IPA states on its website that it relies solely on the financial support of its members and donors. If the IPA has indeed funded this report from tax-deductible donations, to be legally compliant proper scientific research methodology and reporting does matter, a lot.

So, to the report itself. A legitimate basis for the IPA research goes like this. Every form of communication is ‘framed’ in some way so that a desired message is imparted. Framing may be implicit or explicit, objective or subjective. Media informs citizens and this facilitates decision-making. Consequently the media is a significant and powerful factor in the formation of public policy. Media stories that are consistently framed in a way that favors one particular view may therefore unduly influence the formation of public policy. Whilst this naturally pertains to the ABC as a media outlet the ABC is also specifically required to ‘broadcast in the interests of all Australian citizens’ as it is publically funded by the taxpayer. The IPA has reservations about the ABC’s impartiality and consequently wished to conduct a scientific assessment of the ABC’s broadcasts in order to determine whether there was a basis to their concerns.

Sounds good? Reasonable, considered, objective? Unfortunately this is not how the IPA conducted itself. I wrote this research proposal drawing upon loose threads scattered throughout the report and moderated emotive language.

What in fact the IPA proposed was:

“The ABC has a persistent and systematic left wing bias in its coverage of contemporary Australian political issues…The ABC is required by law to provide an impartial news service… the Australia Network [failed] to broadcast Australian values…[the energy mix has] been one of the most contentious political issues over the last decade…the government should adopt an agnostic approach to energy sources, allowing the best technology to prevail without distortionary [sic] subsidies…biases are of course inevitable in any communication of issue…one reason that the ABC should be privatized…[the ABC] is not a welcome home for conservatives or classical liberals…” 

The IPA’s ‘independent review’ is characterized by pre-existing assumptions: the ABC is biased and has a political agenda, the only solution is privatization, and the renewable energy industry is unfairly advantaged over the coal and CSG industries. The agenda is set. Now to the gathering of the data.

The IPA commissioned media analysis firm, iSentia, ‘to conduct a third party assessment of ABC bias’ using a media measurement methodology called CARMA. Media measurement evaluates the effectiveness companies, organizations or associations communicate their message to their target audience through various media channels.

 “To develop the best possible measurement program for our clients CARMA begins each assignment with a consultation to study [its] goals and objectives…[we work] with our clients to determine Key Performance Indicators (KPIs) that will gauge our clients’ success against key objectives…” 

The IPA, not the ABC, is the client in this case. Consequently, the CARMA framework used in this study necessarily reflects the IPA’s position and measures the ABC’s reports against the IPA’s key performance indicators. The IPA’s viewpoint is that the ABC has a persistent left wing bias, and that economic growth and environmental protection are conflicting political policies. IPA’s goals and objectives include privatizing the ABC and the removal of renewable energy subsidies. Economic benefits, jobs, royalties, tax contributions are ‘beneficial’, environment, public health, economic and social impacts are ‘harmful’.

The report does not reference any of the ABC stories used in the assessment, nor does it clarify what defines a story as being about a particular energy industry. Does a story about the prosecution of a hoax emailer that resulted in a short-term drop in Whitehaven Coal’s share price qualify as a coal industry story? If so, would it be deemed an unfavorable story because it reported a negative economic impact? James Paterson did not respond to my emailed request for a sample of the ABC reports iSentia selected.

For these reasons alone the value of the data collected in this study could be regarded as compromised. However, James Paterson, author of the IPA report, really destroys the data’s creditability with this acknowledgement:

“The choice of interviewees reflects the fact that the coverage on coal mining was dominated in the sample period by the Abbot Point controversy.” 

The sample of stories can hardly be considered representative of coal industry coverage when a single inflammatory issue with enormous environmental implications predominates. What this highlights is that six-months is too short a time period for a study of this type. Events such as elections, natural disasters, celebrity and controversial decisions attract intense nationwide media coverage. A much longer timeframe than the six months used in the IPA study would be necessary to ensure a truly representative cross section of stories can be sampled. What Paterson’s comment also identifies is that he is ignorant of, or chooses to ignore, basic research sampling methodology.

So what did the IPA and Paterson do with the figures they arrived at, such as they were? Pages were covered with reams and reams of tables and bar charts, interspersed with the author’s subjective commentary before concluding with a summary table and a repetition of the IPA viewpoint. A third of all articles referenced were the IPA’s own publications and a table of relative energy costs was unreferenced. Statistical analysis was non-existent and large chunks of data were ignored. It would require a thesis length article to comment on all the ‘findings’ and it would be an exercise in the unnecessary.

At times Paterson demonstrates he clearly doesn’t have the mathematical skills to understand his own figures or he is being deliberately misleading, or both. He repeatedly compares the categories ‘favorable’ to ‘unfavorable’ without defining what benchmark would, in the IPA’s opinion, constitute impartial representation, a must given that that is the point of the research. Furthermore, the figures for ‘neutral’, which is a measurement of impartiality in itself, are largely ignored or given a negative connotation.

For example, of coverage specifically of the economic impact of the coal industry 57% of stories were identified as neutral and 43% framed, 59% of which were favorable and 41% unfavorable. Favorable representation and impartial framing are the winners here but the IPA does not appear to agree saying that:

“…a high proportion of stories focused upon the negative [economic] impact of the coal industry. Stories which recognized the positive benefit of the industry were significantly out numbered by those which treated the issue neutrally.” 

The use of the term ‘significantly’ in a quantitative research study suggests statistical analysis has been used to support a conclusion. This clearly was not the case as such an analysis would have found that the proportion of stories that treated the economic benefits of the coal industry neutrally was not significantly different from those that treated it favorably (p > 0.1).

As part of the CARMA analysis the tone of the language used in each story is evaluated as being of one of five types. ‘Reason’ and ‘pragmatism’ use rational argument and constructive terminology, ‘fear’ and ‘hope’ are characterized by negative or positive emotions and ‘cynicism’ by doubt. Pragmatism and reason would have to be considered admirable traits in any media story implying a common sense logical presentation of information. Paterson consistently ignores the majority proportion of stories that fall into these two categories and focuses solely upon the stories using emotive language. Once again he does not articulate the benchmark by which he judges impartiality, and once again he has misrepresented some outcomes to suit the IPA agenda.

The IPA contends that the ABC highly favors the renewable energy industry and this is reflected in the language used in the stories. Paterson does not elaborate as to why the ‘overwhelming’ number of reasoned and pragmatic stories is a criticism of the ABC reporting style but he certainly highlights his own bias in his representation of the emotive language indicators. The 93 stories of hope were deemed a ‘significant’ number but ‘stories which deployed the language of fear and cynicism were extremely rare’. From the graphical representation in the report it would appear 98 stories used the language of fear and cynicism.

The choice and mixture of interviewees a story quotes is regarded as ‘one key mechanism by which frames are applied’. Comments of the most frequently quoted spokespeople were collected and categorized as to whether they were favorable, neutral or unfavorable to the industry of interest. The IPA report found the ‘analysis demonstrates that those who are interviewed is [sic] strongly correlated with the overall favorability rating of the story.’ The IPA did not reference any test of correlation applied to the data to support this conclusion. Nor would a correlation have been found.

Greg Hunt, the Federal Environment Minister, was quoted in 29 stories about the coal industry and Green’s Senator, Larissa Waters, in 25. The Greg Hunt quotes were evenly spread; 10 favorable, 9 neutral and 10 unfavorable, and Larissa Waters’ were all unfavorable. Stories that quoted Greg Hunt were found to have an average favorability rating of 51.7 and those that quoted Larissa Waters had an average favorability rating of 46. Statistical analysis finds that these two ratings are not significantly different from each other or from 50, the neutral midpoint (p > 0.1). Even though all Larissa Waters’ quotes were unfavorable, on average the framing of the containing stories was neutral suggesting the ABC counterbalanced Larissa Waters’ views with favorable comments about the coal industry.

The IPA report is fundamentally flawed on many levels. The subjective and inadequate sampling procedure, the lack of any analytical methodology and the inappropriateness, deliberate or otherwise, of some of the ‘findings’ leaves an enormous question mark over the entire report. It is in the conclusion, however, that the question mark is removed. The conclusion provides the definitive example of the IPA and Paterson’s deliberate misrepresentation of figures to fit a preordained objective.

In summing up, Paterson presents the CARMA Favorability Rating, as produced by iSentia. The rating is an ultimate summary figure produced by the CARMA methodology to describe the ABC framing of stories about the coal, coal seam gas and renewable energy industries. It is:

“A measure of quality that takes into account the tone, sources, messages and prominence of a story. As an overall metric, it provides the “average rating per story” across all analyzed stories.” 

And these are those figures as per the IPA report.

Favorability Rating  (where 50 is neutral)

Industry Rating(0 – 100)
Coal industry 48.8
Renewable energy 54.1
Coal seam gas 47.7

 

Paterson once again expresses a finding that is entirely unsupported and is incorrect. Unlike Paterson, I can support my position using a high school level statistical test.

 

Industry IPA Finding Statistical Test(p > 0.1)
Coal industry Slightly unfavorable Neutral
Renewable energy Highly favorable Neutral
Coal seam gas Highly unfavorable Neutral

 

On a scale that measures from 0 to100 differences of 1.2 and 2.3 from the midpoint are miniscule and 4.1 is insignificant. It doesn’t require any mathematical or statistical expertise to recognize that Paterson is clutching at straws when he relies upon such minute variations in figures to justify his, and IPA’s view that:

“The ABC’s coverage is so biased it amounts to campaigning by the taxpayer funded broadcaster…The only way for the government to ensure that it is not subsidizing biased media is to privatize the ABC.” 

On the basis of the IPA report presented, I don’t think so. I could go on, there is so much more, but there is no need.

Of course, free speech applies and being unconstrained by any obligation to the taxpayer the IPA is quite at liberty to broadcast any rubbish it likes.

Unless, of course, that broadcast was funded from the tax-deductible donations of its undisclosed benefactors.


Support an independent media voice. Support No Fibs Citizen Journalism.
Monthly Donation



Comments

  1. Colin Broadfoot says

    “Lies, damned lies and statistics” Once again we see how statistics are misrepresented to suit a political agenda. Well done Lesley, for exposing the IPA tactic.

  2. Neil Harris says

    Excellent analysis. It seems that the IPA has more questions to answer than the ABC.


    • I’m sure that if you made a suitable donation they’d be happy to produce a report answering any questions you have.

      There may be minor methodological mistakes, loose logic, sloppy statistics and a bit of bald bias, but that’s presumably what the donors are paying for. The main thing is that the conclusions they start with will definitely be reflected in the final summary.


  3. Young James writes a lot about the ABC, but is more often than not actually on it. Drum etc.


  4. This analysis of the IPA article only gives strength to the reasons we must have independent journalism. Stop subsidies to all forms of energy James.

  5. Laurie Forde says

    The ABC should sue the IPA and Paterson


  6. The IPA should be recognised for what it is – and far right political organisation, happily prepared to lie its way into government. Isn’t this just another right wing training college? I am offended that they think we are this stupid.

    As for privatisation of the ABC. The IPA will look you straight in the eye witihout blinking and say that Rupert Murdoch would run the ABC much better! With the ‘correct’ politics. AND THAT IS JUST AN INSULT TO ANYONE WITH A BRAINCELL.