Eliana Kobayashi
Federal Institute of Education, Science and Technology
of São Paulo, Brazil
E-mail: likobayashi@ifsp.edu.br
Submission: 01/16/2019
Accept: 02/10/2019
ABSTRACT
Although
English language is considered an essential requirement to find a position in
the Brazilian work market, little do we know about what proficiency level is
expected from workers by companies and how language assessment occurs at work.
This investigation was developed in an automation company located in São Paulo
metropolitan area. It is a case study which aimed to better understand the
language use expectations and analyze the assessment process to identify: the
assessment tools used in the company, its language view and if the assessment
process meets the company’s language use expectations. This study uses
questionnaires, interviews and documents analysis for data collection and
triangulation of sources. The company is a place where English is used in daily
tasks due to international partnerships and taught in an in-company course. The
results show a mismatch between the proficiency and achievement tests applied
in the company and its language view. In addition English language is a
gatekeeper in recruitment process only for some positions. The language use
expectations for workers vary according to their position and department but
are not assessed likewise.
Keywords: English language assessment; English for
Specific Purposes; business context
1. INTRODUCTION
In the Brazilian job market, English
language is considered an essential requirement to find a position. Although it
seems to be common knowledge, little do we know about what proficiency level is
expected from workers by companies and how language assessment occurs at work.
On the other hand these questions can be considered relevant aspects especially
in a globalized world where English is the main language in fields like
business, technology and science (GRADDOL, 2000). While companies search for
skilled professionals, applicants struggle to get a position and step up in
their careers.
Thus apart from the two sides
involved in such scenario: professionals and companies or employees and
employers, there are also language schools and English teachers who are hired
by companies to work on screening process or to teach their staff.
Professionals regardless of the area also usually enroll in English courses to
get prepared for job selections and meet work demands. Therefore, the language
seems to be considered a gatekeeper in hiring and promoting processes.
Having said it, language assessment
plays an important role in companies as eventually it is through tests that
skills are measured and applicants and employees future may be influenced.
Understanding how and why tests are applied and when and who takes them is
essential to infer what language and proficiency views underline such
assessment process.
Therefore it is possible to see that
companies are important places to investigate and better understand English
influence as it is fair to say that language impacts the work of many
professionals. Such relation between English and employability shows the relevance
of assessment research in the area.
This investigation was developed in
an automation company located in São Paulo metropolitan area. It is a case
study which aimed to better understand the language uses expectations and
analyze the English assessment process conducted in the company to answer the
following research questions: 1) Which assessment tools are used?; 2) which
language and proficiency views can be inferred?; 3) Does the assessment process
meets the company’s language use expectations?
This context was chosen due to three
main reasons. First, English was the foreign language used in the company by
directors and some employees. Besides, there was an in-company English course
to better train employees and the application of proficiency tests in hiring
process. Thus, it is possible to see how language played an important role in
the company in terms of learning and assessment being for these reasons a
relevant context to be investigated.
2. LITERATURE REVIEW
Language tests can be used for
several reasons and situations in society. In schools they are regularly
applied to check if learning has been effective, companies test applicants
before hiring and promoting, immigration applications also require passing tests,
among other uses. Thus in practical terms, language assessment occurs in
situations where the language is the main subject to be assessed or it is the
means which allows the applicants to show their skills like in job interviews.
Therefore some tests can be
considered high-stakes since their results can affect candidates’ future
(MADAUS, 1998). For this reason, investigating their uses is of paramount
importance in many contexts like in companies.
There are different types of
language tests like proficiency, achievement, placement (HUGHES, 1989) each one
used for specific purpose and objective. Proficiency tests are applied to see
how candidates can use the language in future situations regardless of their
learning or the language program they have studied. As Vallete
(1977, p. 6) claimed “the aim of proficiency test is to determine whether their
(the students’) language ability corresponds to specific requirements”.
Besides proficiency tests are based
on job analysis which focusses on such future use of the language in specific
contexts such as academic and business ones. In the latter, the objective is to
assess candidates’ ability to perform tasks required for the position using the
target language, for example work as a secretary in an English company. Thus
there is no reference to the language courses that the applicants may or may
not have taken (HUGHES, 1988).
On the other hand, achievement tests
are related to the past since their purpose is to collect evidence during and
at the end of a course to assess learning progress. According to Anastasi
(1976, p. 398), “they are designed to measure the effects of a specific programme of instruction or training”.
However, Hughes (1988) points out
that successful results in achievement tests do not mean necessarily a certain
proficiency acquisition since the course may be deficient even when it was
designed to achieve such proficiency. Wiseman (1961) had already criticized the
syllabus content approach because it tends to perpetuate ineffective educational
practices, proposing the goal-oriented test which evaluates learning according
to the curriculum.
Hughes (1988) argues the problem is
the different ways in which the contents of the two tests, proficiency and
achievement, are determined while they should be essentially the same. The
course objectives should be according to what students are expected to perform
by the end of the course reflecting the needs analysis.
Such gap between proficiency and
achievement tests become even more problematic in investigation contexts like
this one where the former are applied in screening processes and the latter is
used in the English course for the employees.
On the other hand, proficiency tests
when designed based on the expected uses of the language have proved to be
effective in business contexts as discussed by McNamara (1990), Cole and
Neufeld (1991) and Sajavaara (1992).
According to Douglas (2008), there
are two aspects which distinguish specific purpose testing from more general
purpose language testing, authenticity of task and the interaction between
language knowledge and specific purpose content knowledge. Authenticity means
that the tasks should be similar to the ones in the target language use. Thus
it would increase the likelihood that test takers will do the test task in the
same way as they would probably carry out in the actual target situation. The
author also emphasizes that the interaction between language knowledge and
content knowledge is the clearest defining feature of language for specific
purpose testing.
Taking it into account, proficiency
tests in business contexts should assess English for Specific Purposes and
reflect as much as possible language use situations likely to happen in real
life. As a consequence content knowledge has an important role. Due to these
reasons, performance assessment can be considered the most adequate way of
testing applicants’ skills for a company’s position.
In performance based tests, language
skills are assessed in an act of communication and are most commonly tests of
speaking and writing. A sample of speech in the context of simulations of
real-world tasks is elicited from the test-taker and judged by raters who
follow rating procedure (MCNAMARA, 2000).
There are two traditions in second
language performance assessment: the work sample and the psychological and
cognitive tradition. This investigation follows the former since this approach
is illustrated in language testing for specific occupational purpose. As the
key for such approach is the realistic representation of relevant real world
task in the test setting, McNamara (1999) emphasizes the difference between the
criterion (relevant communicative behavior in the target situation; a series of
performances subsequent to the test) and the test (a performance or series of
performances simulating, representing or sampled from the criterion).
Therefore, test performances are used as basis for making inferences about
criterion performances.
Other testing foundation is its
validity which has many aspects like content, criterion-related, construct and
face. On the other hand, for this investigation the content validity is seen to
be the most important. For Hughes (1989), a test is said to have content
validity if it has a representative sample of the language skills and relevant
structures with which is meant to be concerned. Therefore, in order to judge a
test validity there needs to be a specification of the skills or structures
that the test is meant to cover. Thus a comparison of the test content and its
specification is the basis for judgments as to content validity (HUGHES,
1989).
As a conclusion, it is possible to
see that building and using English language tests involve many foundation
aspects which if not present can end up causing negative effects to the
assessment process and affect both test users and applicants.
3. METHODOLOGY
This research is a qualitative case
study developed in a medium-sized automation company located in São Paulo
metropolitan area where English language was used in some of their daily tasks,
taught to employees and was also tested in recruitment process. According to
Adelman et al (1976), a case study is strong in reality and for this reason it
appeals to practitioners who can identify with the raised issues. In addition,
the insights from a case study can be put into use for a variety of purposes
like staff development, within-institution feedback, formative evaluation and
educational policy-making.
Yin (1984, p. 23) also states it is
“an empirical enquiry that investigates a contemporary phenomenon within its
real-life context; when the boundaries between phenomenon and context are not
clearly evident; and in which multiple sources of evidence are used”.
For this reason, the participants
were selected for having a meaningful role in language use situations: the
employees (students) who were taking English course; the Human Resources
supervisor for being responsible for recruitments; and the director as he
interviewed the candidates in English in order to assess their language level.
The data were collected using
questionnaires with the students and the HR supervisor and an interview was
conducted with the director. The practicability was the main factor for
selecting the questionnaire as a tool since it could be sent to the
participants. As stated by Nunan (1992) questionnaire
enables the researcher to collect data in field settings and the data are more
possible to be quantified than discursive data. In a context where employees
face hard and busy routine, data collection, apart from being accurate, must
also be feasible and practical. The students’ questionnaire items were mostly
closed while the items for the HR questionnaire were open ended.
On the other hand, a semi-structured
interview was chosen as a tool to be applied with the director. This tool was
chosen due to its possibilities of interaction which can allow findings which
may not happen with others (DOWSETT, 1986). According to Seidman (1991 p. 3),
“at the root of in-depth interviewing is an interest in understanding the
experience of other people and the meaning they make of that experience”.
Concerning language use, this
participant has a key role in the company since he is the professional who
speaks the most with foreign suppliers and interviews applicants in English to
assess their level. Thus, analyzing his language view is essential to
understand the company expectations related to language learning and how it
assessed.
In addition, language tests used in
the in company English course and the ones applied in the screening process
were also analyzed.
The data collected from different
sources were then triangulated. Triangulation is a technique very used in
qualitative research based on the use of multiple sources of data aiming at
increasing the reliability of the conclusions. As Bryman (1988) claimed it is
an attempt to draw conclusions with more confidence.
According to Denzin (1978), there
are four types of triangulation techniques: sources, methods, researchers and
theories. The most used ones pointed by the authors were also applied in this
investigation the sources and methods. The objective of using different sources
of data and methods is to decrease the bias of one technique to compensate with
another (LINCOLN, GUBA, 1985).
4. RESULTS
The collected data involved
different sources such as the company’s director, the Human Resources
supervisor, the workers and the proficiency test applied in recruitment process
as well as the achievement test used in the in-company English course. With the
triangulation results it was possible to gather evidence to clarify this
research questions.
Concerning the first and the second
questions about the assessment tools used in the company and the inferences
that could be made about the underlying language views, the results showed some
problems in the written and oral tests applied in the recruitment process and
in the achievement tests. However in order to better understand them it is
necessary to discuss the language use needs in the company.
English interviews with job
applicants are conducted by the director when the recruitment process is for
some specific positions such as: managers, engineers and technicians who will
work in technical areas of the company. Although there are other departments in
the company and positions ranging from production line workers to directors,
not all of them must use the language and go through testing process in case of
hiring and promoting. What justifies such need is related directly to the
company’s business and strategies.
As they must use a specific
technology provided by an American company, all workers involved directly to it
are expected to know English at a level set according to their duties. For this
reason, an engineer, for example, who works in an area that is not related to
such supplier has different language demands from those who belong to
automation area. The table below shows the positions and the main tasks
expected to be performed in the language.
Table 1:
Positions and expected language performance in English
|
Tasks |
Director |
Project Manager |
Engineers,
technicians (automation area) |
Engineers,
technicians (other areas) |
Purchasing area |
Export Import area |
|
Write and read emails |
X |
X |
X |
X |
X |
X |
|
Read manuals |
X |
X |
X |
- |
- |
- |
|
Talk on the phone with foreigners |
X |
X |
X |
- |
X |
X |
|
Receive English
speaking visitors |
X |
X |
X |
X |
- |
- |
|
Take part as a listener in conference calls |
X |
X |
X |
X |
- |
- |
|
Take part and give opinions in conference calls |
X |
X |
X |
- |
X |
X |
|
Write reports |
X |
X |
X |
- |
- |
- |
|
Other |
Travel and negotiate |
Travel and negotiate |
- |
- |
- |
- |
Reading and writing are skills
expected from all workers while aural skills are necessary in some areas
showing that demand for the use of English in the company is not
homogeneous.
There are not standardized
procedures in interviews according to the director but the usual topics are
applicants’ working experience, reasons for leaving their previous job and
expectations about the company.
The director claimed he focuses on
general subjects rather than technical ones since getting to the interview
phase means candidates have already passed the technical written test.
Therefore it seems that the company wants candidates able to express themselves
orally about general topics rather than technical ones. At this point it is
possible to see that the company differs technical knowledge testing and
English testing.
However talking about experience,
duties and responsibilities involve technical knowledge and showing it in
English interview requires specific lexis, typical expressions and register
which characterize business English (PICKET, 1986). In addition, according to
the director, the most important aspects considered in this oral test are:
vocabulary use, interaction and pronunciation, which confirm the need for
specific lexis.
Thus such situation shows a problem
between the criterion, which according to McNamara (2000) is related to
relevant communicative behavior in the analyzed context, and its representation
in assessment. For the company, the criterion means only general subjects and
not technical ones, so the oral test does not reflect such criterion.
Non linguistics skills were also
mentioned like confidence, clarity, gestures and conviction since in corporate
environment workers are expected to convey a confident and reliable image. “I
think you have to be confident regardless of the mistakes that can happen in a
presentation or even in an interview”. By “mistakes”, the director mentions
prepositions and time tense that he usually notices in emails exchanged between
his employees and the American suppliers. However he believes that such
mistakes are acceptable. On the other hand, it was clearly claimed that if
communication occurs at a managerial level involving some business decisions,
there should not be as many mistakes.
English can be considered a
gatekeeper depending on the positions. Managers and those above technicians are
not hired if they fail the language tests even passing others. However, if it
happens at technician level, they can still be hired but are advised to study
English since the work demands will prevent them from developing and
progressing in the company.
The Human Resource supervisor is
responsible for recruiting applicants and managing the English course offered
to the employees, whose fee is 70% paid by the company. According to him, the
course main objective is to allow workers to better communicate with the
overseas suppliers, which is the same reason mentioned by the director.
The company controls the workers
progress by checking the monthly report sent by the school in which attendance
and marks are informed. However, the assessment report is based on a scale with
“very good”, “good” and “regular” but without any criteria specifications. When
tests were applied, the school sent the marks that could be from zero to 10 but
there was not a description of what students were supposed to be able to do
after being promoted to another level. On the other side, the company
regulation was to cancel the benefit in case of attendance lower than 70% and
marks below six.
It is not clearly stated what
language skills are being assessed and what workers can do using the language
with detailed descriptions of the criteria. There was not a course program
designed according to the company’s needs analysis but a regular program offered
to all students who wanted to learn general English at that language school.
Thus instead of having a proficiency test mark, the company received
achievement tests marks which even being satisfactory would not mean
necessarily the language use skills expected for the tasks workers were
supposed to perform. English for Specific Purpose, in this case business
English instead of general English could have been a more adequate language
choice.
The data collected among workers
ranked reading as the most important English skill to be used in daily tasks
followed by email writing as they have to request for information and also
provide information to American and German suppliers and partners. Speaking and
listening skills are needed to receive visitors and take part in trainings. The
collected data also showed that technical areas are those which need language
the most followed by export import area. There are also departments that have
little or no need to use English like finance and human resources. The latter
usually resorts to the director, interviewed in this study, when recruiting
applicants.
The written test applied to
candidates consists of seven questions which assess language structure
knowledge in isolated sentences and reading comprehension in two texts, one of
them business related. Cloze was the
main format of the questions but
although it was believed in the 70’s that integrative tests could assess
several aspects of the language, more recent research has showed cloze tests
assessed the same aspects as discrete items (MCNAMARA, 2000).
Therefore it can be inferred that
the company’s language view is more concerned with form, which can be a
consequence of such tests popularity in the past and nowadays as they are still
used in some language schools. However this research data points to a different
view since the company main objective is to select and train workers to be able
to use the language in real communication. For these reason, performance tests
should be applied instead. In order to insure content validity the test would
have to present authentic materials from the company like manuals and exchanged
emails with overseas partners with tasks likely to be done in the company’s
daily routine.
Considering
the third research question “Does the assessment process meets the company’s
language use expectations?”, the results indicated a mismatch between the
company’s expectations in terms of language use and the applied tests.
According to the director’s interview, the Human Resource supervisor and
workers’ data, their language view is related to business English language use
skills in specific situations. On the other hand, the language view underlying
the assessment tests applied in the recruitment process and by the English
school focused more on structure and form of general English language
knowledge indicating a more traditional approach. Such situation reflects the
dichotomy use and usage (WIDDOWSON, 1978). It is important to emphasize that
the problem is not related to having one view or another in a test but the
mismatch between the necessary language use in a given context and the tests
applied in order to attend the expectation in it.
Concerning the achievement tests,
the main focus is measuring the knowledge of grammatical topics presented in
the textbook, although the purpose and syllabus of the book focused on the four
skills: reading, writing, listening and speaking development through
communication activities. The grammar was one of the components to be studied.
However, the design of the achievement test prioritized such aspect which can
mean either what has been focused in class or a choice of what should be
assessed. As lessons were not observed for this research, only the textbook and
the tests, it can be inferred that the language view underlying the tests
reflect language knowledge whereas the company’s need indicates the development
of language use. In addition, since achievement tests are expected to focus on
the course content and objectives, such result may point to possible a
traditional approach to language teaching.
In addition, as workers progress in
English language is followed by monthly reports based also on achievement
tests, marks may have different meanings and performance expectations in the
language for company and school. For that reason, Hughes (1989) pointed out the
disadvantages of achievements tests in “syllabus-content approach” like badly
designed program, badly chosen books or other material because the results
could be misleading. In this research, such situation may even cause problems
if workers are delegated tasks in the language which are beyond their actual
proficiency.
5. CONCLUSIONS
This paper discussed how English
assessment occurred in a company where the language was used, taught by an
English school and required from some workers and job applicants. The results
showed mismatches concerning language views between the company and the school
which affected assessment process integrated by proficiency and achievement
tests. As a consequence it could influence how the company analyzed its
workers’ English proficiency.
There were also problems related to
the tests used in recruitments process which analyzed language knowledge
instead of language use. In addition, English for Specific Purposes, in this
case Business English would be more adequate to the company’s needs than
General English. It was also found that the language can be a gatekeeper only
in some positions and for certain departments. As this research was a case
study it would be important if other investigations were developed in similar
business contexts to better understand how English is assessed due to the
increasing relevancy of this language in society.
REFERENCES
ADELMAN, C.; JENKINS, D.; KEMMIS, S. (1976) Rethinking
case study: notes from the second Cambridge conference. Cambridge Journal of Education,
v. 6, n. 3, p. 139-150.
ANASTASI, A.
(1976) Psychological testing. New
York: Macmillan.
BRYMAN, A.
(1988) Quantity and Quality in Social Research.
London: Unwin Hyman Publications.
COLE, G.; NEUFELD, D. (1991) Les tests d’evaluation de langue second de la function publique du Canada. Actes du Colloque Bulletin: Association quebequoise des enseignants du francais langue seconde. Ottawa:
University of Ottawa.
DENZIN, N. K.
(1978) Sociological methods.
Nova York: McGraw-Hill.
DOUGLAS, D. (2000) Assessing Language for
Specific Purpose. Cambridge:
Cambridge University Press.
DOWSETT, G.
(1986) Interaction in semi-structured interview. In: M. Emery: Qualitative Research. Canberra:
Australian Association of Adult Education.
GRADDOL, D. (2000) The future of English? The British Council.
HUGHES, A. (1988) Achievement and Proficiency: The
Missing Link? In Hughes, A. (ed) Testing English for University Study. ELT Documents: 127. Modern English
HUGHES, A. (1989) Testing for language teachers.
Cambridge: Cambridge University Press.
LINCOLN, Y. S.;
GUBA, E. G. (1985) Naturalistic Inquiry.
Beverly Hills: Sage.
MADAUS, G. (1988) The influence of testing on the
curriculum. In: TANNER, L. N. (ed). Critical
issues in curriculum. Chicago: Chicago University Press, p. 83-11.
MCNAMARA, T. (1990) Item Response Theory and the
validation of an ESP test for health professionals. Language Testing, v. 7, p. 52-77.
MCNAMARA, T.
(2000) Language Testing. Oxford: Oxford University Press.
MCNAMARA, T. (1999) Performance Testing. Encyclopedia
of Language and Education, v. 7, p. 132-137.
NUNAN, D. (1992)
Research
Methods in Language Learning.
Cambridge: Cambridge University Press.
PICKET, D. (1986)
Business English: falling between two
styles. Comlon,
n. 26.
SAJAAVARA, K. Designing tests to match the needs of
the workplace. In: SEIDMAN, I. E.
(1991). Interviewing as qualitative
research: A guide for researchers in education and the social sciences. New
York, NY, US: Teachers College Press.
VALETTE, R. M.
(1977) Modern Language Testing. New
York: Harcourt Brace Jovanovich.
WIDDOWSON, H. G.(1978)
Teaching Language as Communication.
London: Oxford University Press.
YIN, R. K. (1984) Case study
research: design and methods. California: Sage publications.