The 21st Language Testing Research Colloquium
Poster Session Two, July 30, 1999
Tsukuba International Congress Center
"A comprehensive survey of English proficiency and attitudes toward learning English among college and university students in China, South Korea, and Japan" (The JACET Kyushu-Okinawa Chapter Project Committee, 1997) showed that Japanese university students evaluated their English proficiency level to be much lower than that of the other two other countries. This result seemed to correspond to the recent TOEFL's scores. At the high school stage, to our knowledge, there has not been any kind of comparison studies carried out within the region. Therefore, it is our intention to explore the similarities and differences that may occur in English language education between two neighboring countries in the East Asian region: South Korea and Japan. For example, one difference between the two countries is that since 1993, a nationwide listening comprehension test has been introduced as an entrance qualification test for universities in South Korea. On the other hand, this has not happened yet in Japan (although some universities have included the listening component in entrance tests for their schools). Consequently, listening comprehension has been given a higher priority in South Korea. As a result of the difference of South Korea introducing a listening comprehension test in their university entrance exams and Japan not yet requiring one, we have decided to do a comparison study on listening comprehension abilities of Korean and Japanese high school students. Since both countries have a reading component in their tests, we have included it in our test for comparison as well.
There are two purposes of this research. One is to explore whether or not the existence of a listening comprehension test in the entrance examination of one country can be seen as a factor of the score results. The other is to determine if there are any differences in reading comprehension abilities between the test takers of both countries. Overall in investigating the two purposes of the study we will consider the 'washback effect'. In our case this refers to the effects a university entrance test has on the high school curriculums and the students. For example, the contents of the test may affect high school students' English performance, their attitude toward English itself, and other interests brought on by the difference of the entrance examination for universities in both Korea and Japan.
The test consists of two parts, listening and reading comprehension. There are two variables to consider. The listening component is the 'marked' aspect of the test since it is required only in Korean universities' entrance exams. The reading component is required in the examinations of both countries and therefore offers a different variable. However, within this variable there are some differences. For example, one reading section included 4 shorter readings, which were designed to meet the type of reading tests used in South Korea. Another longer reading item was designed to represent the Japanese style of reading test items, and in fact was taken from the 1992 nationwide examination. The results of the test and variables are analyzed and discussed.
The test included 23 test items aimed at assessing general, selective and interactive listening and reading skills. The listening component consists of 15 questions and the reading component consists of 8 questions. The following points are the formal features of the test:
1) For each question the test takers are expected to choose 1 out of 4 choices except for Question 16-19 in which 5 choices are given. For the data analysis, the least chosen answers were excluded in Question 16-19 in order to calculate the actual equivalent number of options (AENO) .
2) All the choices are written on the test sheet except for Question 1-5 in which the choices are given in pictures.
3) Conversations and oral descriptions are read only one time except for Question 11-15 in which the questions are given twice.
1) Sound quality: The tape contains little noise and its volume is most appropriate.
2) Length: 10 minutes.
3) The listening test component was taken from a private proficiency test called STEP. The level of the test was a pre-second grade level, suitable for average high school students.
Question 1 - 5: After listening to an explanation, the test takers are to choose a picture that best fits the description. Selective listening skills are tested. The speech rate is 170-180wpm.
Question 6 - 10: After listening to a short dialogue, the test takers should take the role of one speaker and choose the most appropriate response to the other. Interactive listening skills are tested. The speech rate is 150-160wpm.
Question 11-15: After listening to short dialogues and a question, the test takers are to choose the most appropriate answer. General listening skills are tested. The speech rate is 170-180wpm.
Question 16 -19: After reading a short reading, the test takers are to choose the most appropriate answer to the question given in each reading. This reading section was designed to meet the type of reading tests used in South Korea.
Question 20 - 23: After reading a longer reading text consisting of four paragraphs, the test takers are to choose the most appropriate answer to the question given. This section was designed to represent the type of reading test items in Japan
According to the work on speech rates presented by Tauroza and Allison (1990), the speech rates of the utterances of the monologues (Question 1-5) were moderately fast and on the dialogues (Question 6-15) they were slower than normal or moderately slow.
The tests were administered to 362 Japanese and 388 Korean high school students, and the tests were implemented in both countries. The subjects in each country consisted of about 150 students in the second year and 240 students in the third year of high school. We divided the two levels as a way to better identify where influences of the test may occur.
As shown in Table 1, the mean score of the total score of Korean students was higher than that of Japanese students. The difference was statistically significant (p< .001).
Table 1. The total scores
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Table 2 shows that Korean students' scores of Listening, Reading (I) and Reading (II) were better than Japanese students' scores and the differences were statistically significant (p< .001).
Table 2. The scores of each component
|
|
|
|
| |||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Table 3 and 4 show that the scores of Japanese 3rd year students were lower than those of Korean 2nd year students. The differences were found significant in Listening (p<.05) and Reading (II) (p<.05).
Table 3. The total scores in each grade
|
|
|
|
|
|
|
|
|
|
2nd year Japanese |
|
|
|
|
|
|
|
|
2nd year Korean |
|
|
|
|
|
|
|
|
3rd year Japanese |
|
|
|
|
|
|
|
|
3rd year Korean |
|
|
|
|
|
|
|
|
|
|
|
| |||
|
|
|
|
|
|
|
|
|
2nd year Japanese |
|
|
|
|
|
|
|
2nd year Korean |
|
|
|
|
|
|
|
3rd year Japanese |
|
|
|
|
|
|
|
3rd year Korean |
|
|
|
|
|
|
Three years ago, Kinoshita et al. (1998a) did a similar research on listening abilities and reported no significant differences between the scores of Japanese and Korean high school students. It seems that the difference we found this time is due to the backwash effect of the introduction of listening test into the entrance examination of Korea.
The following questionnaire was given to our subjects after the test.
Survey on Item No.1- No. 15 (Listening component)
|
Q1 |
Tape Speed? |
Fast <-1 2 3 4 5 -> Slow |
|
Q2 |
Sound Quality? |
Not clear <-1 2 3 4 5 -> Clear |
|
Q3 |
Vocabulary? |
Difficult <-1 2 3 4 5 -> Easy |
|
Q4 |
Grammar? |
Difficult <-1 2 3 4 5 -> Easy |
|
Q5 |
Time for marking answers? |
Short <-1 2 3 4 5 -> Long |
|
Q6 |
Content? |
Difficult <-1 2 3 4 5 -> Easy |
|
Q7 |
Vocabulary? |
Difficult <-1 2 3 4 5 -> Easy |
|
Q8 |
Grammar? |
Difficult <-1 2 3 4 5 -> Easy |
|
Q9 |
Content? |
Difficult <-1 2 3 4 5 -> Easy |
|
Q10 |
Vocabulary? |
Difficult <-1 2 3 4 5 -> Easy |
|
Q11 |
Grammar? |
Difficult <-1 2 3 4 5 -> Easy |
|
Q12 |
Content? |
Difficult <-1 2 3 4 5 -> Easy |
|
Q13 |
Directions of questions? |
Not clear <-1 2 3 4 5 -> Clear |
|
Q14 |
Time for reading questions? |
Short <-1 2 3 4 5 -> Long |
|
Q15 |
Test time? |
Short <-1 2 3 4 5 -> Long |
Table 5. Comparison of the scores of the survey given to 3rd year students
|
|
|
|
|
| ||||
|
|
|
|
|
|
|
|
|
|
|
Q1 |
219 |
2.863 |
.896 |
226 |
3.208 |
.895 |
4.06 |
P<.001 |
|
Q2 |
219 |
3.288 |
1.079 |
228 |
1.781 |
.925 |
15.82 |
P<.001 |
|
Q3 |
219 |
3.155 |
1.078 |
228 |
3.763 |
1.007 |
6.16 |
P<.001 |
|
Q4 |
218 |
3.078 |
1.169 |
228 |
3.689 |
.984 |
5.96 |
P<.001 |
|
Q5 |
219 |
2.954 |
.875 |
228 |
3.096 |
.946 |
1.65 |
NS |
|
Q6 |
218 |
3.083 |
1.042 |
228 |
3.719 |
.982 |
6.63 |
P<.001 |
|
Q7 |
218 |
2.615 |
1.157 |
228 |
3.154 |
1.059 |
5.13 |
P<.001 |
|
Q8 |
218 |
2.670 |
1.185 |
227 |
3.159 |
1.042 |
4.62 |
P<.001 |
|
Q9 |
218 |
2.610 |
1.117 |
227 |
3.141 |
1.061 |
5.14 |
P<.001 |
|
Q10 |
218 |
2.606 |
1.169 |
226 |
2.872 |
1.050 |
2.52 |
P<.05 |
|
Q11 |
218 |
2.606 |
1.189 |
226 |
2.712 |
1.077 |
.98 |
NS |
|
Q12 |
218 |
2.468 |
1.142 |
226 |
2.845 |
1.008 |
3.68 |
P<.001 |
|
Q13 |
217 |
3.028 |
1.140 |
227 |
2.921 |
1.063 |
1.02 |
NS |
|
Q14 |
217 |
3.018 |
.911 |
226 |
3.066 |
1.026 |
.52 |
NS |
|
Q15 |
217 |
3.078 |
.969 |
225 |
3.036 |
1.099 |
.43 |
NS |
Table 6. Comparison of the scores of the survey given to 2nd year students
|
|
|
|
|
| ||||
|
|
|
|
|
|
|
|
|
|
|
Q1 |
124 |
2.331 |
1.030 |
149 |
3.114 |
.756 |
7.03 |
P<.001 |
|
Q2 |
125 |
2.664 |
1.019 |
149 |
2.275 |
1.225 |
2.87 |
P<.01 |
|
Q3 |
125 |
2.712 |
1.225 |
149 |
3.765 |
1.108 |
7.40 |
P<.001 |
|
Q4 |
125 |
2.552 |
1.135 |
149 |
3.658 |
1.022 |
8.40 |
P<.001 |
|
Q5 |
125 |
2.656 |
1.052 |
149 |
2.940 |
.907 |
2.37 |
P<.05 |
|
Q6 |
125 |
2.568 |
1.007 |
149 |
3.604 |
.933 |
8.77 |
P<.001 |
|
Q7 |
125 |
1.936 |
.953 |
149 |
2.919 |
.923 |
8.63 |
P<.001 |
|
Q8 |
125 |
2.104 |
1.034 |
149 |
2.913 |
.859 |
6.96 |
P<.001 |
|
Q9 |
125 |
2.192 |
1.017 |
149 |
2.946 |
.888 |
6.47 |
P<.001 |
|
Q10 |
125 |
1.704 |
.904 |
149 |
2.577 |
.991 |
7.62 |
P<.001 |
|
Q11 |
125 |
1.816 |
.941 |
149 |
2.497 |
.953 |
5.93 |
P<.001 |
|
Q12 |
125 |
1.856 |
.953 |
149 |
2.503 |
.931 |
5.66 |
P<.001 |
|
Q13 |
125 |
2.896 |
1.101 |
149 |
2.926 |
.963 |
.24 |
NS |
|
Q14 |
125 |
2.960 |
.991 |
149 |
2.886 |
.993 |
.62 |
NS |
|
Q15 |
125 |
2.976 |
1.039 |
148 |
2.980 |
1.049 |
.03 |
NS |
Alderson, J. C., C. Clapham and D. Wall. (1995). Language Test Construction and Evaluation. Cambridge: Cambridge University Press.
Bachman, L. F. (1990). Fundamental Considerations in Language Testing. Oxford: Oxford University Press.
Canale, M. (1983). From communicative competence to communicative language pedagogy. In J. C. Richards and R. Schmidt (Eds.), Language and Communication (pp. 2-27). London: Longman.
Henning, G. (1987). A Guide to Language Testing: Development, Evaluation, Research. New York: Newbury House Publishers.
河合忠仁 1994 「韓国の英語教科書ー中学校英語教科書を中心に」『現代英語教育』, 30(11), 12-15.
Kinoshita, M., Y. Ikari, K. Ishii, T. Imamura, A. Ohtsu, M. Kawajiri, H. Shimatani, Y. Takanashi, and T. Laskowski. (1996). Developing a test of listening comprehension abilities focusing on oral communication. Language Laboratory, 33, 81-96.
Kinoshita, M., K. Ishii, A. Ohtsu, M. Kawajiri, H. Shimatani, Y. Takanashi, and T. Laskowski. (1998a). The formation and analysis of a test of listening comprehension abilities for high school students in Japan and South Korea. Language Laboratory, 35, 35-51.
Kinoshita, M., K. Ishii, A. Ohtsu, M. Kawajiri, H. Shimatani, Y. Takanashi, and T. Laskowski. (1998b). Listening comprehension test focusing on oral communication ability: Effects of the testing conditions on the test results. JLTA Journal, 1, 1-14.
Monbusho (the Ministry of Education, Science, and Culture). (1989). The Guidelines for Course of Study in the Senior High School. Tokyo: Kairyudo Shuppan.
中村洋一 1998a 「テストデータ分析のコンピュータプログラムについて Test Data Analysis Program: TDAP Ver. 1.0」『上越教育大学大学院学校教育研究科言語系コース（英語）研究論集』, 13, 31-46.
中村洋一 1998b 「マルチメディア・データを含むリスニングテスト・アイテムバンクの実現可能性」STEP Bulletin, 10, 27-37.
Omaggio, A. (1986). Teaching Language in Context. Boston, Mass.: Heinle and Heinle.
大谷泰照 1997 「韓国の外国語教育事情ーなぜ第二外国語の中学導入に踏み切ったのか」『英語教育』, 46(9), 12-15.
大友賢二 1996 『項目応答理論入門』東京：大修館書店.
大友賢二・中村洋一 1996 『Test Data Analysis Program: TDAP Ver. 1.0. 』 （大友 1996に添付）
Richards, J. C. (1983). Listening comprehension: Approach, design, procedure. TESOL Quarterly, 17(2), 219-239.
Rost, M. (1990). Listening in Language Learning. London: Longman.
Seo, S. Ok. (1994). ELT in Korea and proposals for its improvement. Proceedings of the Japan-Korea International Symposium on Foreign Language Education in Japan and Korea: Prospects for the 21st Century (pp. 19-29). Fukuoka: Kyushu University.
島谷 浩・木下 正義・Terry Laskowski・高梨 芳郎・大津 敦史・石井 和仁・川尻 徳 (In press) 「日本と韓国の高校生に対する英語リスニング・テストに見られたテスト・バイアス：古典的テスト理論と項目応答理論に基づくデータ分析」
Ur, P. (1984). Teaching Listening Comprehension. Cambridge: Cambridge University Press.
Weir, C. J. (1990). Communicative Language Testing. Englewood Cliffs, NJ: Prentice-Hall Regent.