Political Reform

Subscribe to Political Reform feed Political Reform
A PSAI blog for discussion of politics, policy and political reform
Updated: 1 hour 4 min ago

PSAI 2017: ‘Reliability in political science data: a cautionary tale’

12 October 2017 - 9:34am

This post is based on the author’s paper to PSAI Annual Conference 2017, titled ‘Reliable text-analysis coding of emotive campaign rhetoric in referendums’, due to be presented Sunday 15 October.

‘There never was a good knife made of bad steel,’ according to Benjamin Franklin in his Poor Richard’s Almanack of 1755. It is intuitive to suggest that for anything worthwhile to be produced from primary or basic materials, those materials must be sound. This applies as much to the analysis of data as it does to knife-making. If the data we have collected are not valid and reliable, then any time spent attempting to analyse and derive meaning from them is, ultimately, time wasted.

However, despite its importance, studies that require the use of coded material do not always report or discuss the intercoder reliability of their data. According to one meta-study of communications research (Lombard, Snyder-Duch, & Bracken, 2002), examining Communication Abstracts as a bimonthly index of communication literature in over 75 journals during the period 1994-1998, less than 70% of reports considered gave any indication of intercoder agreement or reliability; of those that did provide a such an indication, less than half indicated the measure of intercoder reliability being presented. Almost 40% of these articles omitted the size of the sample used in the reliability testing, while almost 60% failed to provide the intercoder reliability for individual variables (ibid.).

Such omissions are hardly unusual in social scientific research. Another meta-study, in this case taking in 25 years’ worth of publications in the journal Journalism & Mass Communications Quarterly, in the period 1971-1995, found that only 56% of those articles presented a reliability assessment (Riffe & Freitag, 1997). The rate of reporting reliability assessments in that journal has improved since then, but has not reached anything approaching full compliance; an unpublished study of those articles published in Journalism & Mass Communications Quarterly since 1998 found that less than three-quarters published reliability assessments (Riffe, Lacy, & Fico, 2014, p. 122). More worryingly, less than a third conducted their assessments on randomly-selected content, while less than half accounted for chance-agreement (ibid.).

So, while it’s easy to make the argument in favour of a rigorous approach to social science data––and particularly political science data––that doesn’t seem to mean social scientists are taking it as seriously as it deserves. There seems to be a tendency to trust data on face value, which may be perfectly safe when using popular datasets with a wide and responsive user base, but much less so when using a novel dataset.

The example provided in this paper works its way through two attempts at creating a novel dataset of values for emotional rhetoric during the 2015 Irish Marriage Equality Referendum. Discussing two different phases of coding, and reporting the intercoder reliability scores as a proxy for data reliability, the paper shows just how unreliable novel data can be (phase 1) and suggests some ways to improve reliability in future attempts (phase 2).

While the paper sadly does not have any hard and fast answers for guaranteeing reliable social science data collection, it will hopefully get people thinking critically about the data we use.

This article is part of a series by authors presenting at this year’s PSAI Annual Conference. If you are a participant at this year’s conference and would like to have your work featured on the blog, please contact us and let us know.


Riffe, D., & Freitag, A. (1997). A content analysis of content analyses: Twenty-five years of Journalism Quarterly. Journalism & Mass Communication Quarterly, 74, pp. 873–882.
Riffe, D., Lacy, S., & Fico, F. (2014). Analysing Media Messages: Using quantitative content analysis in research (3rd ed.). New York, NY, United States of America: Routledge.

PSAI 2017: ‘PM Theresa May––appealing to the few and not the many?’

11 October 2017 - 9:00am

Guest post by Joe MacDonagh, lecturer at the School of Business and Humanities, IT Tallaght. This post is based on his paper to PSAI Annual Conference 2017, titled ‘PM Theresa May––appealing to the few and not the many?’, due to be presented Saturday 14 October.

Senior politicians’ speeches fit into a special category of reported speech––usually not written by the politician but by a number of professional speechwriters/advisors. When former Taoiseach Enda Kenny was asked about having authored his speeches, he or his spokespersons would say he had been the ‘architect’ of them. Nevertheless, though we know that Theodore Sorenson composed the words ‘ask not what your country can do for you, but what you can do for your country’, most people are happy to attribute it to U.S. President John F Kennedy. Robert Kennedy’s speeches also featured ‘inversions’ such as the one above, this one from George Bernard Shaw: ‘There are those who look at things the way they are, and ask why… I dream of things that never were, and ask why not?

In these instances there is the sense of referring to others and to abstract ideals in order to appropriate them for one’s cause. The lexicon of political speech tropes is a wide one and is used skillfully by accomplished politicians to clarify, elaborate or even to obfuscate. The professional discourse of politicians is frequently abstract and aspirational, with the use of highly stylised and formulaic tropes. Thus the former Governor of New York Mario Cuomo said that ‘you campaign in poetry and govern in prose.’

An example of this is British Prime Minister Theresa May’s speech outside 10 Downing Street on 13 July 2016 setting a vision for her tenure as British Prime Minister upon succeeding David Cameron after the latter had resigned upon losing the Brexit referendum. She used quite novel––for the Conservative Party––rhetorical structures and devices to try and establish herself as a credible successor to Mr Cameron. She attempted to do this by endorsing fundamental Conservative party values of fiscal prudence whilst reminding her listeners that the United Kingdom must also care for those who can’t care for themselves, in the manner of a ‘one nation’ Tory. In doing this she implied agreement with traditional Tory beliefs whilst also seeking to appeal to as many parts of the electorate as possible.

Mrs May’s speech was a radical departure from her usual formal and stilted mode of delivery––frequently high on detail and low on emotion. When someone is speaking they are not limited to a particular speech genre and can move between multiple identities in their discourse by means of different ‘voices’. In her speech Mrs May used voices of care, concern and solidarity; not traditional Tory perspectives nor ones used frequently when she was Home Secretary.

As a newly appointed Prime Minister, of an uncertain nation and fractious political party, Theresa May was trying to:

  1. Establish herself as a worthy leader of the UK and as a successor to her predecessor David Cameron:
    David Cameron has led a one-nation government, and it is in that spirit that I also plan to lead.
  2. Instill unity in the country post Brexit.
    …we believe in a union not just between the nations of the United Kingdom but between all of our citizens, every one of us, whoever we are and wherever we’re from.
  3. Set a social solidarity path for her new government, by using a “seven part list” incorporating the phrase ‘if you’re’ followed by different marginalised social groups and then the plight affecting them. This was rounded off with the phrases ‘I want to talk to you’ and:
    the government I lead will be driven not by the interests of the privileged few, but by yours…we will make Britain a country that works not for a privileged few, but for every one of us.

Thus this was a well-constructed and sophisticated introduction of her to the British public which was delivered in a relaxed and passionate manner. It can be useful to see such speeches not just in terms of themselves, at a particular moment in time, but longitudinally as a series of linked speeches that explicate the vision of a politician. When politicians do this it can form a type of roman fleuve of that politician’s beliefs, even if written by others.

Given Theresa May’s current difficulties with her party, and with her perception in the recent General Election as an emotionless and aloof individual who was afraid to debate her counterparts, it is interesting to see how fluently and intricately she stated her vision for Britain in this speech. After that, however, when she moved to the ‘prose’ of governance it seems that the electorate thought she was really working for the privileged few rather than for everyone in the UK.

This article is part of a series by authors presenting at this year’s PSAI Annual Conference. If you are a participant at this year’s conference and would like to have your work featured on the blog, please contact us and let us know.

PSAI 2017: ‘The DUP-Conservative deal as a path to a Soft Brexit: A Two-Level Game Analysis’

10 October 2017 - 9:00am

Guest post by Anthony Costello, lecturer in EU Politics and Comparative Government at University College Cork. This post is based on his paper to PSAI Annual Conference 2017, titled ‘The DUP-Conservative deal as a path to a Soft Brexit: A Two-Level Game Analysis’, due to be presented Saturday 14 October.

On 26 June 2017, Northern Ireland’s Democratic Unionist Party (DUP) and the UK’s Conservative Party forged a most unprecedented inter-party arrangement. Arguably one of the most controversial political developments in recent months, the deal cemented the relationship between the DUP and the Conservatives; an arrangement meant to leverage the Conservative Party’s minority government. In propping up the Conservative party, the DUP effectively endorse the Conservatives in fulfilling their election promises regarding Brexit. An examination of the deal suggests this. The deal has generated much concern, with many believing that it guarantees a Hard Brexit; one which will see the UK not only leave the single market, but also leave the customs union. These fears are not unfounded. However, if they came to represent a reality, this would place the island of Ireland into a most difficult position, politically, economically and socially.

Albeit reasonable to suggest that the DUP (on account of their political nature) actively endorse a Hard Brexit––one suggests that there are several influential regional opportunity structures in Northern Ireland (NI) which could divert the DUP from taking an ideological, rather than pragmatic stance in their relationship with the Conservative party. From an ideological point of view, a Hard Brexit arguably fits in-line with the DUP’s long term political and socio-cultural goals, but from a pragmatic standpoint, it is politically unfeasible and arguably discouraged.

A political party’s main objective is to achieve power and hold onto that power; ensured by re-election. In liberal-democracy, this requires a keen sense of responsiveness and regard for the complex web of liberal domestic forces at play––factors with greatly influence the electoral success of political parties. For this reason, one looks toward the theory of Two-level Games for a potential explanation. Initially developed by Robert Putnam as an International Relations theory of negotiation and bargaining, it attempts to explain how domestic forces in a nation-state can alter the national preferences of governments at the international level.

Using the Two-level Games theory, but substituting the relationship between the domestic level (nation-state) and the international level (EU etc.) with the regional level (Northern Ireland) and the national level (United Kingdom), one argues that the DUP-Conservative deal could be a path to a Soft Brexit providing that the DUP act pragmatically by drawing upon regional opportunity structures that could inherently constrain the preference formation of the Conservative Party. This is made possible by the DUP’s influence over the fate of British minority government via the established deal. Indeed, for long term domestic and party-based benefits, the DUP will require to sacrifice their own agenda; an agenda which itself is fundamentally constraint by the very same opportunity structures.

Four opportunity structures have been currently identified as having potential influence from a two-level game model. These are: power-sharing constraints, the NI Brexit Referendum outcome, growing momentum in the United Ireland debate, and civil society/private sphere preferences and interests. The DUP would likely desire to refrain from utilising these structures in inter-party negotiations with the Conservatives. However, it would be politically nonsensical for the party to do; at least from a standpoint of party-credibility and sustainability. Additionally, channelling these opportunity structures effectively (albeit reluctantly) would be beneficial for assuring confidence in long-term devolution in the region, socio-political stability, and ensuring the best possible Brexit outcome for Northern Ireland in light of the majority interests.

This article is part of a series by authors presenting at this year’s PSAI Annual Conference. If you are a participant at this year’s conference and would like to have your work featured on the blog, please contact us and let us know.

PSAI 2017: ‘Using accounting numbers in political debate: The Irish Water case’

9 October 2017 - 9:00am

Guest post by Desmond Gibney, lecturer in accounting at National College of Ireland. This post is based on his paper to PSAI Annual Conference 2017, titled ‘Using account numbers in political debate: the Irish Water case’, due to be presented Saturday 14 October.

Theodore Porter wrote in 1995 that ‘numbers are the medium through which dissimilar desires, needs, and expectations are somehow made commensurable.’ Consider the following news headlines in relation to the potential cost of abolishing the State-owned utility called Irish Water: ‘It would cost €900 million to abolish Irish Water…’ (TheJournal.ie, April 2015); ‘Irish Water abolition “would cost State up to €7 billion”‘ (The Irish Times, March 2016); ‘Scrapping Irish Water could cost €1.6 billion’ (Irish edition of The Times, May 2016 (£)). This range of figures illustrates the challenges in making sense of accounting numbers used in political debate. But this is not a new phenomenon, as illustrated by the case of Lockheed and its ‘Tri Star’ jet in the 1960s and early 1970s. Lockheed spent $1 billion on developing the jet, and then sought $250 million in federal government loan guarantees to complete the programme. Lockheed was accused of using “highly misleading information”, and during lengthy congressional hearings in 1971 on the loan guarantee, nobody mentioned that the $1 billion already spent was, in accounting terms, a ‘sunk cost’ and therefore was irrelevant to the decision about future spending.

The accounting issues related to Irish Water include cost classification, off-balance sheet financing, budget enlargement and accounting arbitrage. The backdrop to these issues is the impact of a period of austerity. Some of these have featured in the news headlines mentioned above and have also impacted on the political debates surrounding Irish Water and in particular whether or not the utility should be abolished. The news story featuring the €7 billion cost of closing Irish Water was published on the front page of The Irish Times in the period between the inconclusive results from February 2016’s general election and the start of talks between various political groupings about formation of a new government. The article dealt with cost classification and correctly categorised certain items such as the cost of the water meters and systems for procurement and billing as ‘sunk costs’, but failed to explain why such ‘sunk costs’ should be excluded from decision-making. The article also failed to show a full breakdown of the €7 billion figure.

The accounting issue of off-balance sheet financing received considerable prominence during the lengthy process that eventually led Eurostat to conclude in 2015 that Irish Water’s borrowings should be included in the government’s balance sheet. Strictly speaking, the Irish government does not have a balance sheet in the sense that some other governments do have, as instead it relates to whether particular liabilities are included or excluded from the general government balance (GGB), which is the difference between government revenues and expenditure, and the balance is either a surplus or (more usually) a deficit. The focus is on liabilities rather than liabilities, because accounting rules dictate that if assets are included, then so must the corresponding liabilities.

Off-balance sheet financing is significant because of the EU’s Maastricht Treaty of 1992, which requires compliance with targets for ratios of debt to GDP and deficit to GDP. If a government is trying to reduce its deficit, it may use a strict application of EU rules on GGB to keep liabilities out of the GGB, and this can encourage governments to consider making expenditure commitments that may otherwise not have been considered, and this acts as a form of budget enlargement. In relation to the debate surrounding Irish Water, this could mean that if the government had been successful in keeping Irish Water debt off-balance sheet, then that would have enlarged the overall budget of the government because it would have freed up the expenditure limits for extra spending on public services such as schools and hospitals.

The period of austerity has posed particular challenges for public sector accounting in the EU, with a focus on the reduction of expenditure, debt and deficit. The greater emphasis placed on the condition of a country’s public finances and the balance sheet classification of government expenditure is particularly important for countries (such as Ireland) who received financial assistance from the IMF, EU and ECB ‘troika’, because regular reporting of financial and accounting information was a condition of receiving such assistance. One of the items contained in the Irish government’s December 2010 memorandum of understanding with the ‘troika’ was a plan for moving towards full cost-recovery in the provision of water services, with a view to start charging in 2012/2013. However, austerity also provides an incentive for EU member states to practice accounting arbitrage, whereby weaknesses in reporting of government statistics are employed, and use of off-balance sheet devices can be contrary to the declared objectives of increased transparency.

The paper attempts the use of Actor-Network Theory and the work of Bruno Latour to offer an interpretation of the relevant issues mentioned above. The methodology used includes interviews, analysis of published information and parliamentary debates, and a FOI request granted to a third party. There has been little research on the use of accounting information by politicians, and it is not always clear in advance what accounting information would be helpful to politicians. The use of accounting numbers in political debate has been somewhat neglected in political studies, but its importance in accounting and finance can be demonstrated by a special issue of Public Money & Management devoted to the topic in 2016, as well as coverage in political economy journals.

This article is part of a series by authors presenting at this year’s PSAI Annual Conference. If you are a participant at this year’s conference and would like to have your work featured on the blog, please contact us and let us know.

PSAI Conference 2017: Paper Previews

9 October 2017 - 7:06am

With PSAI Conference 2017 due to take place this coming weekend at Dublin City University, the Irish Politics Forum will be hosting a series of posts this week from participants at the conference.

Each post offers a preview of the paper and presentation that will be available in full through the conference. All authors presenting at the conference were invited to submit to this series.

To find all of the posts in this series, check out the ‘PSAI 2017’ category on this site and don’t forget to follow @psaitweets and #psai17 on Twitter.