Seminar
Bringing the ‘credibility revolution’ to archaeological field research
- Shawn Ross
- Date
- Friday 27 September 2024
- Time
- Location
-
Willem Einthoven
Kolffpad 1
2333 BN Leiden - Room
- CWTS Common Rooms
Abstract
Over the past two decades, researchers have recognised a ‘reproducibility crisis’ in the natural and social sciences, with indications that perhaps one-third to two-thirds of published research is false. A ‘credibility revolution’ has arisen in response, introducing new approaches and seeking to change research culture and incentives (Vazire 2018). Adoption of open science principles, approaches, and practices, however, varies significantly by domain. The credibility revolution has been most pronounced in the experimental sciences, but it extends to field research disciplines. Amongst these, ecology and geoscience have been leaders. Archaeology, however, has been slower to engage the credibility revolution. In this regard, practice in the discipline fails to align with national and international standards, leaving archaeologists unprepared for future developments such as a growing emphasis on producing machine-actionable data (Peng et al. 2024). Without well-documented and machine-readable data, furthermore, archaeologists cannot exploit existing and emerging tools like traditional machine learning (ML) or large language models (LLMs), which are transforming the discipline.
In this context, my colleagues and I proposed a large-scale, comprehensive evaluation of research credibility in archaeology and subsequent articulation of approaches and practices to improve its openness and reliability. An Expression of Interest (EOI) was submitted in the 2024 round of the Australian Research Council (ARC) Discovery Projects scheme. The proposal was, however, soundly rejected. Our Assessors on the Humanities and Creative Arts Panel ranked us in the 'Bottom 25%' on both 'Investigators(s)/Capability' (despite the international stature of the investigators) and 'Project Quality and Innovation'. Overall, we were ranked 303 of 386 EOIs. Interestingly, 'normalisation' above the level of the Panel raised our final rank by two quartiles. We interpret these results as indicating a particular bias against metascience applications in the Humanities and Creative Arts Panel - but we recognise that the quality of the application or the composition of the research team may be inadequate. With this in mind, I would like to ask participants in this seminar to read the Expression of Interest (two pages with citations) so that we can discuss (a) the viability of the proposal, (b) the make-up of the team, (c) ways to improve the project, and (d) avenues for funding the project more likely to meet with success.
Presenter
Shawn Ross is Professor of History and Archaeology at Macquarie University, Australia.