Recording Speaking Tests for Oral Assessment

Iain Lambert
iain [at] cck.dendai.ac.jp
Tokyo Denki University (Tokyo, Japan)

Oral Communication courses are a common feature of English Programmes at Japanese Universities; however it can be difficult to provide a record of how they were assessed. In my workplace this has become an issue as the Japan Accreditation Board for Engineering Education (JABEE) now require records to be kept of all academic results throughout a student's university career. In this article I describe how I prepared my students for oral examinations that were recorded on mini disc and discuss some of the positive and negative aspects of the process.

Introduction

Class Profile

The tests were given at the end of term to nine classes of between 26-31 First year Japanese university students majoring in electrical and mechanical engineering, predominantly male, upper elementary to pre-intermediate level. Students were streamed at the start of the academic year into seven bands according to the results of a combined reading/listening placement test. The students described here were in the first (intermediate level) and fifth (beginner/elementary) bands.

Aims

To gauge students' ability to initiate and develop a conversation and check uptake of lexis presented during the course.
To provide evidence of standards of pronunciation and communicative ability.
To provide practice in a test format that students may encounter in the future e.g. Cambridge Main Suite exams.

Approach to Assessment

Students received a final course grade out of 100, of which the speaking test counted for 35 marks. Given the disparate levels of the students I decided to adopt a criterion-referenced as opposed to a norm-referenced approach. Criterion-referenced testing is defined as, "a test which measures a test-taker's performance according to a particular standard or criterion that has been agreed on. The test-taker must reach this level of performance to pass the test, and a test-taker's score is interpreted with reference to the criterion score, rather than to the scores of other test-takers, which is the case with a norm-referenced test." (Longman Dictionary of Language Teaching and Applied Linguistics 3 rd Ed, Richards & Schmidt, Longman 2002). Thus, it was necessary to have a clear set of statements describing what the learners can and cannot do at each level. These, together with recordings of the student-student interviews, would provide a clear justification for the marks awarded. I chose to use the descriptors for the International English Language Testing System (IELTS) examination. These give a mark in bands from 0 to a maximum of 9 for the following four categories: Fluency & Coherence, Pronunciation, Lexical resource and Grammar. I decided to add a fifth category based on the Scoring Rubric given for Conversation tests (see appendix below) on p50 of the Teacher's Book for the set text, J-Talk by Linda Lee, Kensaku Yoshida & Steve Ziolkowski (OUP 2000). My reasoning was that a mark in this category would specifically reflect their uptake and use of language and conversational strategies presented in the text. I was not interested in comparing individual students or classes (as in a norm-referenced test) as I specifically wanted to concentrate on measuring what they had absorbed from the term's lessons. Thus, students were assessed on five well-defined criteria, each of which was marked out of 7 (based on an assumption that 7 would be the maximum likely score for my students in each category, were they to actually do the Speaking section of the IELTS exam).

Procedure

The lack of available space and constraints on time meant that interviews had to be carried out in class in the penultimate lesson of term while students were doing a written test. As this was the first time both the students and I had attempted anything like this on such a scale, I decided to give the students as much support as possible without compromising the validity of the test. Thus, in the lesson before the test each class was given a series of revision tasks covering each unit they were to be tested on (see Lesson plan 1) and then went around the classroom in pairs, using (unknown to them) the actual cue cards from the test, which I had stuck on the wall around the room, to have conversations. Finally, following a general feedback session, I announced that the final test role cards would be similar to those they had just looked at and that I would be putting them on the Departmental intranet a week before the test. Students would thus have a chance of getting a good mark if they prepared for each of the roles, knowing that they had a one in five chance.

On the day of the test, after taking the register I explained the procedure to students before handing out the written test papers and an end-of-term questionnaire to be completed if they finished the test early. I went through the written paper section by section and pointed out the area set aside at the back of the class for the speaking tests, telling the class that once they had started the written test I would be calling them out in random pairs and giving them a role card. They would have a minute to read through the card together and then two or three minutes to have a conversation, which would be recorded. Students were not permitted to leave the room until all the speaking tests were finished and the time allotted for the written test (one hour) had elapsed. I used the back of the ID cards I'd had the students make at the start of term (with a photo and short biographical notes) to note down their performance based on the IELTS and J-Talk assessment criteria.

The main problems arising from the rather cramped conditions were twofold. Firstly, despite having had time to prepare and rehearse, the students naturally felt some pressure being forced to speak in front of their peers and, as a result, some chose to speak in a low voice, which made it difficult for their partners to respond and also meant that they were not recorded properly. Secondly, I was worried that students who were called later in the session might have had an advantage in that they could hear what other students had said and use some of that language in their own interview. This didn't actually end up being much of a problem due to having five different role cards and drawing them at random. Also, as noted above, some students tended to speak in quite a low voice owing to the pressure of the situation which meant they couldn't be heard by others.

Giving students the actual role cards in advance raised the possibility that they might simply memorise large "parts" of the text and end up using them inappropriately. From my point of view this was not necessarily a bad thing, given that the idea of the test was to see if they could use and respond appropriately to such language.

Conclusions

While I felt that the format of the speaking tests was fundamentally good, the actual test conditions proved more detrimental to the process than anticipated, and there was some negative feedback to this effect in end of term questionnaires. Students were put under pressure by the lack of space and the recording process, which wasn't entirely successful anyway due to problems with equipment, including the microphone being too sensitive to outside noises. There were some unanticipated problems with another class having a listening test in a neighbouring classroom. This certainly distracted my class and sound leaking into our room turned up on the student recordings. On the positive side, however, the use of role cards provided a good way to initiate interaction (in fact the Cambridge main suite exams use this method) and it was a good idea to give the students a chance to not only simulate test conditions (to a certain extent) in the previous lesson, but also to think about what they would say by putting the actual test roles on the Intranet. I think that the problems mentioned above concerning the test conditions could be significantly reduced by having a mock test simulating actual conditions as a follow up to the activities given below in lesson plan 1.

Appendix

Materials/Aids:

Mini disc player with main cord and extension cable
External microphone. Important to check settings for the recording level.
Spare batteries for microphone
Descriptors (IELTS & J-Talk)
Student role cards (set of 5)
Student ID cards

Lesson Plan 1 (for review/preview session) 90 minute lesson

Stage 1

Passive

Review

After taking the roll and going through a board plan of the day's lesson, the teacher assigns students to five groups and explains that they will work together to review the course in preparation for the speaking test.

The teacher assigns each group one Unit from the material covered from the course book.

Students work together and write down questions based on the language/topics of their respective units.

The teacher monitors and helps where necessary.

5 minutes

15 minutes

(20)

Stage 2

Active

Review

The teacher now assigns students to new groups, each containing at least one student from those in stage 1.

Students ask and answer each others' questions using Answer Plus strategies.

The teacher monitors and notes difficulties.

During this stage, the teacher puts two sets of the role cards (below) on the wall around the classroom.

25 minutes

(45)

Stage 3

Practice for Speaking Test

The teacher gives feedback on the activity, especially referring to groups which used the Answer Plus strategy effectively.

The teacher points out the role cards on the wall and explains the new activity. The students will work in pairs in a non-threatening environment (i.e. "protected&quo; by the surrounding conversations of other students) in simulation of the speaking test.

(The teacher and a student can demonstrate the activity if necessary.)

The teacher assigns the students to pairs.

The students go around the room having short conversations. There is no writing.

The teacher monitors, providing assistance and noting problem areas for students or difficulties with the wording of the role cards.

(The teacher nominates some pairs if appropriate.)

5 minutes

20 minutes

(75)

Stage 4

Give general feedback and administrative information for next week's test.

10 minutes

(85)