This workshop aims to help participants become familiar with what is necessary to judge and improve the quality of test items in a test with right/wrong answers such as tests of language knowledge and some reading/listening tests. Key concepts such as item analysis, discrimination, difficulty levels and reliability are defined and explained. There will also be practical demonstrations of how to carry out item analysis, calculate overall test reliability and interpret the results. No prior statistical knowledge is required.
Files from the session:
by Dr Olena Rossi.
Wednesday 23rd February 1:00-2:30pm (UK time).
Care taken during test development directly impacts on the test’s validity because poorly-constructed tests weaken inferences based on test scores. An important but often overlooked stage in the test development process is item moderation. Moderation is “the scrutiny of proposed items by (ideally) at least two colleagues, neither of whom is the author of the items being examined” (Hughes & Hughes, 2020, p.68). Careful item moderation makes item trialling more efficient. This is because moderation allows the identification and subsequent rejection (or revision) of problematic items early into the test development process so that trialling efforts are not wasted on poorly-constructed items.
In this workshop, I suggest ways of setting up an efficient item moderation system at your institution. In particular, I provide guidance on establishing item-writer working groups, producing item moderation checklists, and giving effective feedback on items.
Below, you can view the recording, moderation checklist, link to item writing for language testing website, and slides used during the workshop.
Dr Olena Rossi & TAFSIG committee
About Olena Rossi
Olena Rossi has an MA and a PhD in Language Testing from Lancaster University, UK. Her main research interests are in test development, item writing, and assessment literacy for test stakeholders. Olena’s doctoral thesis looked into the nature of item-writing skills and their development. Olena also has a lot of practical language testing experience as an item writer, item reviewer, and consultant for language testing organisations. She has facilitated assessment-related workshops in different countries and has taught item-writing courses.
BALEAP TAFSIG Webinar
led by Dr Emma Bruce and Heléna Stakounis
Emergency Remote Assessment in the UK EAP Sector: Challenges, Innovations and Opportunities
RECORDED ON FRIDAY, 12th NOVEMBER 2021, 1:00pm-1:45pm
Language assessment is an area which has demanded timely action and innovative contingency planning in response to the Covid-19 pandemic. To enable students and test takers to continue to access tests while complying with government safety restrictions, test providers, assessment developers, teachers and administrators were urgently tasked with adapting methods and processes. In this talk we share findings from a BALEAP-funded project investigating the immediate response to the Covid-19 pandemic across the UK EAP sector. Drawing on both quantitative and qualitative data from this large-scale research study we will share insights into the challenges, innovative strategies and in many cases, opportunities which arose as assessment teams in a variety of higher education contexts tackled the shifting landscape of the pandemic.
Dear BALEAP Members,
As the Testing Assessment and Feedback SIG (TAFSIG) approached our 2nd birthday, we invited you to our Annual General Meeting (AGM) on Wednesday 13th October 1pm, which was followed by an informal Coffee Club event to get together and chat.
In the AGM we recapped our activities of the past year and our plans for the future.
We also elected new members to our committee, Eddie Cowling (Special Projects Officer) and Paul Dickinson (Secretary).
You can view the recording below.
Emma, Rob, Sam, Wayne, Jo, Eddie, Paul (TAFSIG Committee)
EAP assessment for Social Justice
Wednesday, 29th September 2021, 12:45-14:00 (BST)
This event was inspired by TAFSIG and EAP4SJ recent collaboration with Dr Jan McArthur when we talked about higher education assessment for social justice, and particularly the role of values of trust, honesty, responsibility, forgiveness, and responsiveness (McArthur, 2020).
In this world café event, we invited you to join us in this discussion, with the focus on EAP assessment in terms of the current state of the affairs, as well as the ways forward for our field in embracing assessment for social justice. Those discussions were then followed by a short Q&A with Dr McArthur, when participants had an opportunity to ask questions about her work on assessment for social justice.
Relevant literature on the topic includes the below:
To view and contribute to our discussions on the day, click on the dark 'View Discussion Padlet Here' button below.
To watch the recording of the Q&A, click on the light 'Watch Q&A with Dr Jan McArthur Here' button below.
Thank you very much.
TAFSIG & EAP4SJ
by Dr Carolyn Westbrook, SFHEA (Test Development Researcher, British Council)
FRIDAY, 24th September 2021, 12pm-1pm (BST)
While many teachers use both audio and audiovisual materials when teaching listening skills, many tests still use audio only for reasons of practicality and cost. However, this may be to the detriment of the authenticity of the task. In an EAP environment, for example, much real-life listening involves visual input, e.g. lectures, which may be live or recorded. Moreover, academic tasks often require lecture listening to be integrated with writing or speaking skills, and possibly reading, as students listen and take notes on the lecture content or discuss the content in a seminar (Westbrook, 2011). Thus, creating authentic tests requires integrated tasks which replicate the TLU domain.
This presentation will discuss the results of a PhD study which investigated the impact of input medium on performance on an EAP listening-into-writing test. In the study 116 Russian and Ukrainian university students watched a lecture which was divided into two halves: one half being audio only and the other half being a video showing the speaker and PowerPoint visuals. Test takers took notes on the lecture and then wrote a summary of the content of the lecture. The summaries were split according to the two input formats and performances were rated by trained raters and then analysed quantitatively and qualitatively. A Hotelling’s T2 analysis revealed that there was a significant difference between the two groups on the combined group means. Post-hoc independent samples t-tests revealed no statistically significant difference for mean video scores for the two groups yet there was a statistically significant difference for the mean audio scores for the two groups. The presentation will also present some key findings from the qualitative linguistic analysis and discuss the implications of the study for EAP assessment design.
Dr Carolyn Westbrook is a Test Development Researcher at the British Council. She has been involved in Language Testing and Assessment for 15 years and has been involved in a number of test development and language assessment literacy projects. Her main research interests are in EAP and ESP assessment.
Formerly an Associate Professor in EFL, she is a Senior Fellow of the UK Higher Education Academy (Advance HE) and has also worked as a teacher, teacher trainer and materials writer for over 25 years. She has a wide range of experience teaching and assessing General English, Business English, ESP and EAP in Spain, Austria, Germany and the UK.
Carolyn has written books and other materials including promotional presentations and online teacher training materials for Cambridge University Press. She has also co-authored a chapter on ESP testing (Lanteigne, Coombe & Brown (Eds.), 2001) and has published in several journals.
An inter-institutional TAFSIG Piloting Network - a case study and discussion, Eddie Cowling and Maddy Hall (University of York) and Rob Playfair and Wil Hardman (University of Liverpool)
This session responded to the high level of interest amongst TAFSIG members about establishing a test piloting network. The first part of the webinar illustrated the role of piloting in the test development cycle. We then reported a case study of a recent online test pilot between the University of York and the University of Liverpool, and offered advice for developers who would like to do something similar. Following this we proposed ideas for a TAFSIG Test Piloting Network - an inter-institutional resource for test developers to connect.
We hope this proposal will lead to discussion of some of the ethical and logistical issues this may present. The event took place on Wed 16th June 2021 and was free to attend.
You can view the recording by clicking on the WATCH RECORDING HERE button below.
You can view and contribute to the discussion we began by clicking on the DISCUSSION button below.
Friday 21st May 2021
What can eye-tracking tell us
about academic reading on reading-into-writing tasks?
(Nicola Latimer, CRELLA)
This presentation gives an overview of a study which used eye-tracking and retrospective think aloud (RTA) to investigate students’ reading on an academic reading-into-writing diagnostic test task. Eye-tracking measures were triangulated with RTA data to reveal the types of reading used by participants and reflect on the rationale behind differences in participants’ approach to reading on the task. The presentation concludes by considering the implications for those supporting students to develop their academic reading skills.