TRICC Logo
Lifelong Learning Programme
Lifelong Learning Programme, Grundtvig - Multilateral Project
Project no.: 142235-2008-LLP-NL-GRUNDTVIG-GMP
Project title: TRICC - Training Intercultural and Bilingual Competencies in Health and Social Care
Interdisciplinary Social Science Department
Utrecht University

Utrecht, the Netherlands
Go
Boğaziçi University
Istanbul, Turkey
Go
COOSS Marche
Ancona, Italy
Go
dock europe e.V.
Hamburg, Germany
Go
PPRE Limited
London, Great Britain
Go

Evaluation Methods

What effect do the trainings have on the daily routine of the different target groups? To what degree were the aims of the trainings achieved? Were the trainings successful in reaching out to the target groups? To find the answers to these questions, the trainings conducted were evaluated in individual and group discussions and with the aid of questionnaires.
Featured here is an overview of the different evaluation methods and results.

 

Evaluation of the Training ”Do we speak the same language?”
Bridging language barriers in general practice (Netherlands)

After identifying the needs of general practitioners (GP’s) on working with (in)formal interpreters, a training was designed for GP’s to enhance their knowledge, attitude and skills on this subject. A description of the evaluation methods and results of this training is given below.

Evaluation
Quantitative Evaluation
a) a pre and post test with a questionnaire consisting of 17 items about knowledge, attitude and skills; b) after the training, a question form was filled out by the participants about the organization of the day and the contribution of the trainers.

Qualitative Evaluation
a) after the training day the participants performed a ‘free association’ of the day on post-it notes; b) furthermore, they were asked to deliver cases on working with interpreters for the feedback evening. These case descriptions contain reflections on the daily practice after the perceived knowledge and skills during the training day; c) after the feedback evening there was an evaluation on organization, content and trainers through the ‘bull’s eye method’. This method is based on a visualization technique in which the opinions of the participants are immediately visible and can be discussed; d) three months after the total training semi structured interviews were conducted with a few participants.

Qualitative results
Trainings day: The quotes from the ‘free association’ showed that the participants were satisfied about the organization, the content and the trainers. Participants appreciated the combination of theory and practice and the approach of the HoutenBeenTheater (Image and Forum theatre) was highly valued. Some participants have their doubts though about working with formal interpreters in daily practice due to time and organizational constraints.
Feedback evening: Six participants provided cases in which the awareness and change of behavior was clearly visible. An example was given of a Moroccan patient who visits the GP practice on a regular basis with diverse somatic complaints. After the training, the GP who participated in the training decided to call for a formal interpreter to facilitate the next consult. By doing so, the true cause of the patient’s complaints became clear in a highly efficient way, which resulted immediately in a different diagnosis and treatment strategy.
Bull’s eye: nearly all participants were satisfied about the atmosphere in the group. The majority was satisfied about the organization. It was suggested to bring in more diversity in the evening programme.
Interviews: Five participants were interviewed. These GP’s stated that since the training they are more aware of the role of the (in) formal interpreter, they have gained more knowledge about the official interpreting policy and they try to work with a formal interpreter more often.

Quantitative results
Training day: the information services about the training were mainly regarded as sufficient, the location was fine, the training material was good, the presentations were inspiring and informative and the HoutenBeenTheater was very inspiring and recognizable. Total training: On all three dimensions a substantial learning effect was reached as a result of the training (see Table 1), especially regarding knowledge, but also regarding attitude and behavior.

De GP’s now know more about the law and regulations regarding the use of formal interpreters. Moreover, they are aware of the difficulties in interpreted consultations and 1/3 of them tends to adjust the way in which they coordinate conversations with an interpreter.

Conclusion
The training was evaluated positively and the aims were largely accomplished. The GP’s have gained more knowledge about (in)formal interpreters, are more aware of language and culture barriers and in some cases this also leads to a change in behavior like working with a formal interpreter more often.

Download nederlandstalige versie (PDF)

 

Evaluation of 2-day training for ad hoc interpreting
for multilingual health care providers in the hospital (Germany)

Friday, June 25, 2010, Albertinenschule in Hamburg
Five months after the training, a 1-hour evaluation session was conducted with 12 participants. It included a review of the training in the form of a questionnaire, as well as evaluating if and how the training needs were met, i.e.

  • Development of the role of the ad hoc interpreter;
  • Interpreting in a triad setting;
  • So-called distance or delayed interpreting;
  • Interpreting for patients with limited command of the German language;
  • Responsibility and liability.

Although only three participants had actually interpreted since taking part in the training, all confirmed they had become more self-assured after the training. They had learned about being empathetic and clear at the same time. Several said they needed a lot more practice and had not had much opportunity to apply their new knowledge.

This was also the conclusion that could be drawn from the Bulls-Eye conducted at the end of the session (see image). The four sectors evaluated were:

  1. Learning atmosphere & organisation
  2. Learning success/results
  3. Practical relevance/applicability
  4. Personal activity

While participants assessed the learning atmosphere and organisation as well as their personal activity as strong, they attributed less points to learning success/results and practical relevance/applicability – because the majority had not had the chance to implement what they had learned.
One important statement made during this session was that the training was not only useful for ad hoc interpreting but that the communication skills (e.g. transparency) practiced in the training were applicable in many other work situations.

 

Evaluation in Italy – methods, tools and results

Great attention was devoted to the evaluation of the training initiative, choosing methods and tools allowing qualitative and quantitative analysis of the results. The main evaluation objective consisted in measuring the impact of the training in the participants, both in the short and in the medium term. Evaluation also aimed at assessing the participants satisfaction towards the initiative.

Evaluation of the training impact
A structured Likert scale questionnaire was designed, meant to allow a quantitative analysis. The questionnaire included 4 macro-areas, each corresponding to a training objective:

Macro-areaTraining objective
KnowledgeIncreasing the learners’ knowledge in terms of cultural mediation, rules and related services
CompetenceImprove the learners’ competence in the interaction with migrant patients
AwarenessSensitize the learners on the difficulties the migrant patients meet because of language and cultural barriers
BehaviourMake the learner reflect on the behaviours they adopt towards their migrant patients, and on the possibility to correct them.

 

The questionnaire included a number of questions common for all the partners, and some other specific questions each partner included because of their relevance to the territorial area and/or the specific target.

The evaluation aimed to measure any changes in the learners’ knowledge, attitudes and behaviours after the training course. The questionnaire was therefore submitted to the learners twice: once before the training and once at the end of it. The differences in the values of the answers would have given the measure of the training impact. The results were initially analysed by groups, i.e. comparing the average value for each question between the first and the second questionnaire. Still, some of the values emerged from the comparison of data lent themselves to interpretative ambiguities. Further measurements were therefore carried out to verify their reliability (Fig. 1). In the last column, the “p-value” tells you if the difference is just incidental, or based on a meaningful difference. If the p value is <.05, then you can attach meaning to the differences. It is evident the significant increase in knowledge, skills, behaviour (tendentially), but not so with regard to awareness. Nonetheless, the awareness score was higher then the other ones in the first questionnaire: this might be interpreted that the learners were aware of the problems already, but lacked knowledge and skills, which increased significantly after the training.

Fig. 1 –Comparison of values, per individual data, between the two questionnaires (in Italian)

Pre-testPost-testt-value p-value
Knowledge (5 questions)2.83.4 - 2.483.016
Competence (2 questions)2.73.3 -2.642.011
Behaviour (3 questions)2.83.2 -1.640.11
Awareness (7 questions)3.53.8 -1.394.17
Total (17 questions)3.13.5 -2.497.016

 

Evaluation of the participants satisfaction
This kind of evaluation aimed to assess whether the learners had liked the proposed methods and contents. As the training initiative was officially accredited, a standard structured questionnaire provided by the Ministry had to be used, and was submitted to the participants at the end of each training day. The analysis of data revealed an extremely high score with respect to organisation, content, methods and trainers for both the days. As for the Forum Theatre, the evaluation reached the highest scores in the Likert scale.

Follow-up
The evaluation activities ended with a follow-up phase: 6 months after the end of the training, the components of the pilot group were contacted to be interviewed again. The guidelines for the interviews were based on the previous questionnaire items. The objective consisted in verifying the training long-term impact, specifically if it had affected the participants’ working practices and behaviours towards their migrant patients.

14 out of 30 participants were available for the interviews. Their answers revealed that:

  • the communication problems with the migrant patients are still a reality, mainly due to the difficulty to understand their family, social and cultural context, which gives easily way to misunderstandings;
  • the importance of working with a formal mediator/interpreter to make the communication more reliable and effective is acknowledged. Nonetheless, the use of informal interpreters persists, especially when urgencies occur and there is no time to look for a formal interpreter;
  • the use of the mediation service has increased in those departments less burdened by urgencies;
  • the informal interpreters can offer a partial and distorted translation, but their emotional closeness to the patient continues to be seen as a value;
  • their prejudices and behaviours have been analysed after the course, and their working modalities sometimes modified. Nonetheless, the enthusiasm aroused by the course soon faded because of the problems making their daily work with the migrant patients difficult, even frustrating;
  • further training is suggested, with diversified contents according to the departments needs and characteristics;
  • training should be provided to GPs, as they often send improperly their patients to the hospital;
  • the appreciation for the course was confirmed, especially for its innovative and experiential aspect.

Download italian evaluation (PDF)

 

Evaluation of the trainings in Turkey

Since the Turkish partner in the TRICC project was primarily responsible for cultural mirroring and was focusing on conducting innovative research into the extent and characteristics of informal interpreting in Turkey, it was not until late on in the project that we considered appraising the training we provided anyway in such a structured manner as our other partners were doing. Having said this, we learned a lot from the methods used by our partners, from the discussions of these methods at the various TRICC meetings, and from the feedback we received about our program when we presented it at the TRICC meeting in Hamburg (22-24 January 2010). Because the training was not the central component of our contribution to TRICC, we did not conduct any pre-training testing or elicit any evaluation right at the end of the training. A feedback form was prepared and distributed to all the students in the class, asking them to list things they had learned during the module and things they would have liked to learn more about, to identify aspects of the module from which they think they had or had not benefited, and to suggest ways of improving the module. Unfortunately, very few (4 out of 26) students returned their forms, presumably because the form was emailed out immediately following the training, which happened to be the end of term, when students were busy with exams. All the same, the four responses were generally positive about the training, and the things students claimed to have learnt concurred with the aspects of medical interpreting we had been wanting to emphasise, such as medical terminology in different registers in both languages―an element students still wanted to learn more about―, communication-crisis-management skills, and the need for the interpreter to explain their role to the patient and doctor before the consultation.

If we incorporate the overall exam performance of students into our evaluation of the training, this performance can be said to reflect well on both the students and the module. All students in both courses passed the exam, with the average grade in the undergraduate course being 78.5% (a grade judged “Fair to Good” according to Boğaziçi’s official grade descriptors) and in the postgraduate course 86%. Students mostly managed to maintain effective communication between the doctor and the patient, and only in the case of a few students did we witness poor interpreting seriously affecting the flow of information between the two. Impressively, almost all the students started their exam-slot by briefing the patient and the doctor about the ground rules for the consultation (e.g. by warning both that he/she would translate everything that was said, by asking both parties to speak in short manageable chunks and to address each other directly), and by and large students displayed very professional and ethical behaviour, such as correcting themselves once they had misinterpreted and making an effort to uphold transparency, by informing one party about what they had been talking about with the other.