Ongoing and detailed monitoring of State implementation plans and compliance with the ICAO Language Proficiency Requirements is essential to ensure that the full safety and efficiency objectives at the heart of the new Level 4 Proficiency Standard are effectively achieved.

The Journal spoke recently with Charles Alderson, a Member of the Lancaster Language Testing Research Group and Professor of Linguistics and English Language Education at Lancaster University, about his team’s new research findings regarding ICAO FSIX Web site data and the pressing need to ensure that States are taking valid and measurable steps to ensure the new Level 4 Standards are being properly met.

In 2003, ICAO introduced new Language Proficiency Requirements (LPRs) which defined six different levels of aviation English competency. Pilots, air traffic controllers and aeronautical station operators are required to achieve at least Level 4 on this scale, initially by March 2008, in order to be licensed to fly aircraft or control air traffic on international (cross-border) flights, or to work in international operations.

By 2007, it had become apparent that many ICAO Member States were not going to be in a position to comply with the Level 4 Requirement by the 2008 deadline. As a result, the Council of ICAO proposed, and its 36th Assembly adopted, Resolution A36-11—Proficiency in the English language used for radiotelephony communications. In that resolution, the ICAO Council recognized that:
“…some Contracting States encounter considerable difficulties in implementing the language proficiency requirements including the establishment of language training and testing capabilities; … (and) that some Contracting States will require additional time to implement the language proficiency provisions beyond the applicability date.”

It was agreed, therefore, that the March 2008 deadline could be extended to March 2011. Contracting States that were not able to comply with the language proficiency requirement were urged:

“to post their language proficiency implementation plans including their interim measures to mitigate risk, as required, for pilots, air traffic controllers and aeronautical station operators involved in international operations on the ICAO website as outlined in accordance with the Associated Practices below and ICAO guidance material.”

In 2006, two years before the ICAO LPRs were originally due to be implemented, the European Organisation for the Safety of Air Navigation (EUROCONTROL) commissioned the Lancaster Language Testing Research Group (LLTRG) to conduct a validation study of the development of the ELPAC (English Language Proficiency for Aeronautical Communication) test, which was intended to assess the language proficiency of air traffic controllers. The ELPAC validation study would require the LLTRG to develop a cross-sample of aviation English tests currently being employed in order for them to develop a proper analysis.

ICAO’S LEVEL 4 PROFICIENCY REQUIREMENTS AT A GLANCE

ICAO’s Language proficiency Requirements establish six levels of skill in six areas of English language usage: pronunciation, structure, vocabulary, fluency, comprehension and interactions. The criteria for achieving Level 4 Proficiency in each of these areas are as follows: Pronunciation Assumes a dialect and/or accent intelligible to the aeronautical community. Pronunciation, stress, rhythm, and intonation may be influenced by the first language or regional variation, but only sometimes interfere with ease of understanding. Structure Relevant grammatical structures and sentence patterns are determined by language functions appropriate to the task. Basic grammatical structures and sentence patterns are used creatively and are usually well-controlled. Errors may occur, particularly in unusual or unexpected circumstances, but rarely interfere with meaning. Vocabulary Vocabulary range and accuracy should be sufficient to communicate effectively on common, concrete, and work-related topics. Subjects may paraphrase successfully when lacking vocabulary in unusual or unexpected circumstances. Fluency Produces stretches of language at an appropriate tempo. There may be occasional loss of fluency on transition from rehearsed or formulaic speech to spontaneous interaction, but this should not prevent effective communication. Can make limited use of discourse markers or connectors. Fillers are not distracting. Comprehension Comprehension is to be mostly accurate on common, concrete, and work related-topics when the accent or variety used is sufficiently intelligible for an international community of users. When the speaker is confronted with a linguistic or situational complication, or an unexpected turn of events, comprehension may be slower or require clarification strategies. Interactions Responses are usually immediate, appropriate, and informative. Subject initiates and maintains exchanges even when dealing with an unexpected turn of events, and deals adequately with apparent misunderstandings by checking, confirming, or clarifying.

Though it wasn’t obvious from the outset, this general investigation by the LLTRG into the number and quality of English language proficiency tests in the aviation sector would eventually reveal important data on the measurable state of compliance by ICAO Member States with the Organization’s new global Language Proficiency Requirements.

“The impetus for the FSIX study can actually be traced back to when we were originally invited to tender on the validation study for the ELPAC test in Europe,” began Charles Alderson, LLTRG Member and Professor of Linguistics and English Language Education at Lancaster University, England. “At that point we didn’t have any particular aviation English experience under our belts but that wasn’t important in this case and didn’t affect the fact that we were eventually awarded the study. The validation of a language test is more concerned with the objective analysis of the various studies and data that are already available to comparatively gauge a test’s overall effectiveness.”

As part of the ELPAC Validation Study, Internet searches for evidence of other tests of air traffic controller English proficiency found very little evidence available to attest to the quality of the tests being employed at that time for licensure of either pilots or air traffic controllers. It was therefore decided to conduct an independent survey of all providers of tests intended for air traffic control. Alderson (2008) reports on the methodology and findings of that survey.

Since the Executive Summary of the ELPAC Validation Study had been framed by the Guidelines for Good Practice of the European Association for Language Testing and Assessment (EALTA), it was decided to base the new LLTRG survey questionnaire on the same Guidelines, which consist of the following headings:

The LLTRG ELPAC survey therefore consisted of two stages: a filter questionnaire which elicited opinions about the ICAO Requirements and general information about aviation English; and secondly some detailed follow-up questions sent to 74 organisations and individuals whose tests were thought to be used for licensure of pilots and air traffic controllers.

Only 22 relevant responses to this survey were received, and these varied considerably in quantity and quality. While this disappointing response rate may be indicative of the lack of quality of the tests being used and an associated reluctance to admit this in public, non-response cannot absolutely be taken to indicate lack of quality, although it may well indicate lack of public accountability of such providers.

We were rather shocked to discover that, although some tests in this area did exist, there was actually no independent data available on the quality of current aviation English examinations. We decided to conduct an independent study to provide the language testing community with some much-needed and objective data on the user-experience and general quality of the aviation English tests that were out there and being employed.

Charles Alderson,
Professor of Linguistics and English Language Education, Lancaster University

The LLTRG ELPAC survey therefore concluded that:

“We can have little confidence in the meaningfulness, reliability, and validity of several of the aviation language tests currently available for licensure. Monitoring is required of the quality of language tests used in aviation to ensure they follow accepted professional standards for language tests and assessment procedures.”

“We were rather shocked to discover that, although some tests in this area did exist, there was actually no independent data available on the quality of current aviation English examinations,” Alderson commented. “It was decided therefore, after we’d submitted our ELPAC Report, that we’d conduct an independent study to provide the language testing community with some much-needed and objective data on the user-experience and general quality of the aviation English tests that were out there and being employed. It was this independent research which led us to the ICAO Flight Safety Information Exchange (FSIX) Web site and some of the surprises that we found there.”

LLTRG research into the ICAO FSIX

On the ICAO FSIX, the LLTRG researchers could access update reports from ICAO State Civil Aviation Authorities (CAAs) that reflected the extent of the various States’ compliance with the ICAO LPRs. The researchers could also access details of the reporting States’ implemen tation plans. These online documents included the names, addresses and, in most cases, the e-mail addresses of those responsible within the various CAAs for compliance with the ICAO Requirements and for the implemen tation of pertinent national plans.

Since the LLTRG ELPAC survey had identified a relatively small number of aviation English examinations that were being used at that time for licensure of pilots, air traffic controllers and others, the Group felt that it would be important to find out which tests were already recognized by ICAO Member State CAAs, and furthermore any additional tests that non-compliant States may have been planning to recognize.

Unlike the ELPAC process, where theLLTRG had developed an online questionnaire also available as an MS-Word document, it decided in the first round of the FSIX Reporting Study simply to address an e-mail query to the person named on the site as being responsible for implementation of the LPRs in each CAA. This letter requested the names of the tests that the authority had recognized or approved, as well as the contact details of those organizations providing the tests.

At the time of data collection for the first round of the study, 55 States had still not supplied information to ICAO regarding their compliance with the regulations. Of the remaining 140, four States had responded but had not provided any email contact address and 25 had replied solely in languages other than English—specifically French or Spanish—as compared with other countries that supplied two versions of their responses (one in English). Due to time pressures the LLTRG focused its efforts on those States who had supplied responses in English.

RESPONSE RATES TO THE FIRST ROUND SURVEY OF ICAO STATES REPORTING ON THE ICAO FLIGHT SAFETY INFORMATION EXCHANGE (FSIX) WEB SITE
Contracting States contacted by ICAO	195 (190 States + 5 territories)	% of 195
Round 1 (May 2008): Survey of CAAs
Countries which did not reply to ICAO request for compliance details	55	28%
Countries which replied to ICAO	140	72%
Of those who did reply to ICAO we were unable to make contact with some because:	140
a) no (usable) e-mail for an official contact person	4	3% (of 140)
b) replies only in languages other than English	25	18% (of 140)
TOTAL	29	21% (of 140)
Of the 140 replies to ICAO which we did contact
We did not receive reply	87	78% (of 111)
We received reply	24	21% (of 111)
TOTAL	111	79% (of 140)
Round 2 (October 2008): Analysis of ICAO data
Claimed compliance	53	27%
Stated non-compliance	89	46%
Not known – had not replied to ICAO	53	28%
TOTAL	195	100%

In response to its request for information to the 140 or so e-mail addresses available to it based on these sample criteria, 11 representatives were unreachable due to an incorrect e-mail address having been supplied, while a further 87 for which the LLTRG did have functional e-mail addresses failed to respond to its survey query. Table 1 (above) summarizes these figures.

Out of the 24 responses to the LLTRG survey query, a total of only ten different tests or assessment procedures was reported, seven of which were reported by a single respondent. The Mayflower College procedure (for pilots and controllers) was reportedly in use by seven CAAs, and the ELPAC test for air-traffic controllers was being used by six CAAs.

These responses did not reassure the LLTRG that appropriate tests or assessment procedures had been recognized by applicable CAAs. Indeed, Alderson 2008/1 had concluded, on the basis of the 17 responses received by the time that report was compiled, that: “…the Survey of Aviation English Tests indicates that it is unclear whether national civil aviation authorities have the knowledge to judge the quality of tests.”

The LLTRG’s research in this regard was carried out in two rounds. The 12 percent response rate to round 1 did not reassure it that appropriate tests or assessment procedures had been recognized by the CAAs, and the Group concluded that it could have little confidence in the meaningfulness, reliability and validity of several of the aviation English language tests currently being employed for licensure.

The second round involved analyzing responses, not to a survey, but to the ICAO request for information on implementation plans and responding.

State details of the assessment procedures they had in place or which were in the process of being developed.

Although the response rate for this second round was greater than to the first, only 53 respondents (27 percent of the ICAO Member States) noted that they had complied with regulations requiring the assessment of the English language proficiency of pilots and air traffic control staff. A further 27 percent had failed to supply ICAO with compliance details and the remaining 89, or almost half of ICAO’s Member States, stated that they were non--compliant.

It should be noted, however, that of those states which claimed they were already compliant with the ICAO LPRs, different countries replied with varying degrees of information that rarelyconstituted proper evidence of compliance. 16 States merely asserted that they were compliant but included no further details to support this claim.

31 out of the 53 States claiming compliance indicated the regulations they were complying with and gave references to their own legislation. Nevertheless, only 14 States out of the compliant 53 States provide estimates of the language proficiency levels of pilots and ATCs—and it is far from clear how accurate these estimates are.

Furthermore, details of the assessments used were very scant and not obviously relevant in many cases. Descriptions of “test methods” used included:

For the LLTRG researchers, these descriptive labels did notinspire confidence in the ability of applicable CAAs to identify, develop or commission suitable instruments to measure language proficiency in such a high-stakes setting. Indeed, it is unclear whether many CAAs have the knowledge necessary to judge the quality of tests, or whether they take seriously the need to ensure that the language proficiency of aviation personnel is assessed professionally and meets standards that the public has a right to expect.

“I was very surprised at the number of CAAs who either hadn’t replied to the FSIX yet or else had replied but didn’t have any particular plans in place,” noted Alderson. “The other thing that leapt out from our research was how vague a lot of State plans were with respect to the measures they had in place or would be putting in place to achieve compliance in the future.”

The lack of response to the request for information on measures to mitigate risk, and the lack of detail in the few responses which were received did not provide a basis for reassurance in the LLTRG team that CAAs are taking seriously the risks associated with inadequate measures of language proficiency.

The LLTRG determined that response rates to ICAO’s requests were disappointing and suggestive that not all Contracting States heeded either ICAO’s request or even the agreed need to comply with ICAO’s Language Proficiency Requirements. The lack of evidence for compliance gave the researchers further cause for concern. The LLTRG concluded that ongoing and detailed monitoring of implementation plans and compliance with the ICAO LPRs is essential, preferably by ICAO or by an independent organization, and that full details of the result of such monitoring should be made publicly available.

The LLTRG’s second conclusion is that appropriate test accreditation services are clearly required and the Group strongly recommends that these be established. Alderson will be visiting ICAO later in 2009 to discuss the types of measures that might be able to resolve some of the shortcomings in this regard. The LLTRG further concluded that it is of the utmost importance that CAAs supply full details to ICAO of the assessment procedures and tests they recognize, and that those who develop such tests and assessment procedures be required to provide objective evidence of the quality, the reliability and the validity of the instruments and procedures in use.

“The CAAs themselves need to be made much more aware of what is meant by “quality” in the area of language tests,” Alderson noted. “It’s now under discussion with testing quality stakeholders such as the International Civil Aviation English Association (ICAEA, see related article on page 4) and the International Language Testing Association (ILTA) to establish a type of voluntary accre ditation system for aviation English tests. ILTA, also a Montreal-based organization, now has a Working Group under proposal to look at the ins-and-outs of this process.”

In Alderson’s view this type of accreditation programme should be voluntary and could be paid for by the test development organizations themselves to help support their quality assurance claims. In a parallel development, he noted that arecent Teaching English as a Second Language (TESOL) conference also held a symposium on aviation English that led to discussions and conclusions similarly highlighting the need for a test accreditation process to be established.

ICAO would likely play a role in encouraging and recognizing the envisaged accreditation process now coming under discussion, as well as possibly expanding the scope of the FSIX Web site to include accreditation information of the aviation English tests being employed by its Member States.

“These are precisely the types of preliminary suggestions and details I hope to be discussing with ICAO when I visit there this summer,” concluded Alderson. The Journal will publish an update in a later 2009 issue that reports on the substance and any results of his meetings.