Eighteen months of TELLCAP-test use have revealed the weakest elements of Russian controllers and pilots in language proficiency. Improvement of some elements will require essential efforts. On the other hand, inadequate aviation context may result in unreliable test results.
One electronic dictionary translates the word “rater” as “swearer”, and offers the following synonym for it: a "habitual user of bad language". The translation of the word "rater" as an "evaluator of proficiency in a language” is missing both from this dictionary and from many others as well. Regrettably, participation in the process of organizing the testing of pilots and controllers demonstrates the danger of changing “rater” to its synonym.

It looks like we are now at the stage of implementing ICAO language requirements when it is too late to start developing a testing system from scratch. Certainly, there is the possibility of compiling a test based on well-known tests in General and Aviation English, but the time left is too short for test developers to care for its validity and reliability. So, it is most likely that existing testing systems, or those that might appear in the near future, will be used.
Testing, and moreover, proficiency testing, when the careers of thousands and thousands of people and their families depend on test results, should evaluate the language proficiency with the highest possible accuracy. Unfortunately, a one-hundred-percent accuracy is unattainable. This has been proved by a native speaker of English who tried to pass a computerized test and was awarded Level 5 proficiency. This native speaker of English has been an Aviation English teacher for decades, and his professionalism has never been called into question at the international level.
One of the language schools in an English-speaking country evaluates the language abilities of a non-native speaking teacher, a person with an MA in English Philology, as Level 4 at all elements of the ICAO Language Proficiency Rating Scale. Another language school in the same country gave a Level 5 throughout the Scale for a person who was studying English from time to time and was unable to translate the word “proficiency” into his native language correctly.
Probably, these blunders are not worth recalling here, but they are not isolated instances. It is easy to foresee a situation when in the near future the majority of candidates will try to pass a test at those places where test results are higher. Last March, ways of protecting air navigation service providers, aircraft operators and developers of valid testing systems were discussed at a meeting of the CoG ATM Training Task Force in Paris. We realize that it will not be easy to find answers to such questions. However, the absence of answers could jeopardize the proper implementation of the ICAO language requirements. While syllabi of training programs and training management issues are generally discussed at a national level, determining test validity and reliability is becoming an international issue. Hopefully, our Symposium and the ICAO Headquarters personnel will be effective in finding an acceptable solution.

Testing of air traffic controllers for benchmarking purposes at a major ATC Center in Russia in 2004-2006 has allowed us to:

More than nine hundred benchmarking tests administered at that time, have allowed to paint a portrait of an average air traffic controller during the periods between 2004 and 2005, and between 2005 and 2006. In 2004/05 it was a specialist at the following level of language proficiency:

Pronunciation	Structure	Vocabulary	Fluency	Comprehension	Interactions
3	2	2	2	2	2

A year after, the picture has changed, but not dramatically:

Pronunciation	Structure	Vocabulary	Fluency	Comprehension	Interactions
3	2	3	3	2	3

It should be noted that between the tests the majority of specialists were undergoing training at a General English course of 150 classroom hours, and moreover, the employer paid for the full course. (There could have been more significant results after such a course, but attendance was rather low because of the odd certainty among controllers that ICAO language requirements would never be implemented. Some frequent visitors to Russian aviation web fori advised controllers not to study the language because those 10-15% who would be unable to demonstrate Level 4, could be fired. But if the figure rises to 80%, “they’d never close the skies, and there’s no way they could bypass Russia”).
Statistical data obtained as a result of testing have allowed us to draw the conclusion that the most vulnerable elements in personnel training are Structure and Comprehension. Weakness in Structure is explained by the difficult heritage that the Russian training system acquired from the times when it was thought that knowledge of standard ICAO phraseology was sufficient to operate and serve international flights. The training system when a person who has never in his life pressed a mike button is teaching others is still more alive than dead.
That is why there has been nobody to explain to pilots and controllers why structures used in standard phraseology were so different from structures used in General English. Moreover, absence of such understanding has led to drilling grammar on standard phrases. After this students failed to understand where and which grammar is used, and what is standard phraseology and what is not.
Regrettably, for many educational institutions such a situation is typical even today. Attempts to explain the essence of grammatical structures used in standard phraseology, absence of modal verbs and forms of the expression of modality, general use of elliptic grammatical forms etc, encounter a complete inability to understand it and lead to mistrust – how come, when everything has been so different for decades…
Well no, it was not different, just the whole system of language training was based on wrong principles. Today, when there is a requirement to demonstrate Level 4 proficiency in Structures as well it is difficult to re-teach personnel to stick to certain grammar rules while using standard phraseology, and switch to conventional grammatical structures when changing from formulaic to spontaneous speech.
Certain experience gained during the last few years indicates that this task will probably become the main and most difficult problem in aviation language training. And, by the way, these problems are completely alike independently of the place where the staff have been recently trained - in Russia or in an English-speaking country. In spite of incomparable environmental factors, different approaches and methods, this problem has not been solved once and for all, though this year marks a 20-year period since air traffic controllers began to get training abroad, mainly in Great Britain and USA. Even a multi-level approach in teaching, in particular, English grammar, which has been used over decades in various textbooks published in UK and other English-speaking countries, is unable to solve this problem.
Obviously, we are going to hear its echo until those specialists who have been and are trained in accordance with these imperfect programs, stop their professional activities. Probably only when we clearly divide training personnel into teachers of General English and RTF instructors, will we be able to speak about any qualitative improvement of aviation language training in my country.
However, identification of structure as a weak element will enable training personnel to introduce corrective changes into syllabi, and hopefully, it will be done.
The second weakest element is comprehension. The majority of aviation training centers do have audio materials that go together with textbooks, but they are not intensely used. But real ATC audio recordings are almost always missing, and in those training centers where ATC recordings are available, teachers are scared to use them due to their lack of RTF knowledge, misunderstanding of aviation context and inability to understand the complete contents of recordings.
However, improvement in Comprehension can be assured somewhat easier than in Structure. In the process of teaching radiotelephony and aviation English at CompLang aviation training center, where I work, audio materials make up a rather weighty part of the course. We use authentic texts on aviation themes recorded by native speakers from Great Britain, USA and Canada, as well as non-native speakers from various European, Asian and African countries whose accent is comprehensible to international aviation community.
It is a well-known fact that in aviation industry (and not only) any test, direct or indirect, solves two tasks: firstly, it reveals the level of skills and abilities of an individual in the tested area of expertise, and secondly, it has washback effect on training programs. Tests and testing systems that evaluate the level of proficiency in General and Aviation English, are not exceptions.
With less and less time remaining, a new situation arises - when tests that only cover themselves with the ICAO Scale as if it were a shield, pop up on the market, though in reality they do not correspond to the task set by this international organization.
Let us imagine a situation when test developers just declare that their test is valid, though this test has never passed the certification procedure at the national certification authority. The probability that the test used will have a negative influence on training program will be very high. It is easy to judge about the quality of a test being prepared now in one of the States, when its developers translate "language proficiency" into their mother tongue as "professional language".
Any test of General and Aviation English used by a State, Air Navigation Service Provider, Aircraft Operator or educational institution, must pass a validity check and certification at a national or international certification authority. It would be preferable that ICAO or another existing or newly created organization acting on ICAO’s behalf could have this done.
Only within the latest six months the TELLCAP test, certified in Russian Federation in 2005, was taken by more than 170 persons, mainly pilots from twelve Russian and three foreign airlines. The test was taken by representatives of four states. Regrettably, the percentage of those who on no grounds consider they have already reached Level 4, is rather high among candidates, and instead of a benchmarking test a person does for a proficiency test. That is why those who take a proficiency test sometimes get Level 2 only.
Eighteen months of TELLCAP use have proved the appropriateness of the majority of Doc 9835 provisions. In particular, complete interdependence has been tracked between language proficiency and motivation of the candidate, his/her cultural level, ability to acquire new knowledge, etc. However, in Russia it is impossible to agree with a suggestion to consider gross amount of hours spent on the language study as a factor in final evaluation of proficiency. We repeatedly face candidates who had spent hundreds and hundreds of class hours in class before they took a proficiency exam, however during this time they "were passing the course", rather then studied the language. Feel the difference.
Preliminary analysis of test results highlights the aviation context as the weakest part of aviation language training programs. The problem is that aviation context is interpreted differently by various teams of language specialists. Due to this it often happens that students do not acquire linguistic tools required for reliable communication in non-routine situations. In other words, the linguistic element of flight safety is either inadequate or entirely missing.
In one of the existing testing systems a situation is offered when during the flight a female passenger calls the captain into the cabin and tells him she has forgotten her child at the airport. So the captain is supposed to discuss the situation and sooth her. It is unclear how the candidate will behave in the absurd situation suggested by the test, but it is easy with a high share of probability to expect what he will think about the examiner and test developers.
Some training systems consider it sufficient to satiate vocabulary with aviation technical terms. Of course it is more impressive to use “HT wiring conduit ferrule” instead of “electrical connector”, but this knowledge will most probably not help the pilot if electrical system fails.
Since 2004, when Doc 9835 was published, it has become rule of conduct to mention its lexical domains in every modification of existing training programs. Citing the same document teachers draw different conclusions, though Doc 9835 clearly defines General English, Aviation English, and Radiotelephony.
Test analysis shows that many controllers and pilots are unable to communicate on work related issues: they mispronounce types of aircraft, they do not know types of flights, they are unable to explain what cargo they are transporting or provide details on the intruder aircraft, and they fail to identify ground reference points or describe sounds that accompany failures or malfunctions…
It is a real pity, as this lexis is listed in detail in Doc 9835. Conclusion: Quotation of the document in training programs does not guarantee that 9835 has been really used by program developers.
Information on the test, its format, contents, concrete questions and tasks, becomes a matter of intensive discussions among candidates directly before passing the test and immediately after it. To certain extent, the interest in the test tasks is warmed by the teacher who is a participant in education but not in testing. They are It is necessary too eager "to cram" their students for higher test results.
With limited number of test versions, those candidates who pass the test at later periods than their colleagues, will be in a privileged position. To mitigate the extent of familiarity with the test contents, it is necessary to develop quite a few variants and organize the test passing in such a way that people become to understand – there is no sense in getting ready for a certain test, this is the language learning that counts.
The washback effect of the test spreads, in the first place, on candidates, in the second - on the teachers, and only in the third - on the program.
In our situation, the washback effect of testing systems on training programs has some particularities - when there are only 10 months left before the date of implementation of the ICAO language requirements, it is difficult to expect that real effect from amending the training program will be reached.
In Russia, Aviation English teachers usually do not have aviation background. The problem is the same when NNS controllers attend Aviation English courses in NS countries. Suggestion: ICAO should recommend strict division between courses in (1) General English and Aviation English, and (2) Standard Phraseology and Radiotelephony English; to be taught accordingly by linguistic (1) and operational (2) personnel.
It is clear that personnel testing for different purposes - as benchmarking, diagnostic, progress, course, proficiency, etc. – will continue after March 5, 2008.
A desire to get a higher level in testing can become an additional motivation for a candidate who wants to continue hi/her professional activity after this date confident in their knowledge.
Testing also becomes a motivation for teachers and training centres in their secret struggle for a higher number of those who has passed the proficiency test.