Evaluation
From SCOME Wiki
Nancy's Bladder Sling Recall Blog Updates
Evaluation is a tool of quality improvement. Hip Pain - YahooAnswers
Contents |
Summary
Apart from accreditation, evaluation is the central activity to assure quality in higher education. To evaluate means to assess teaching and academic studies in a subject or department and the related degree programmes. Strengths and weaknesses of education and training should be demonstrated by stocktaking and analysis and proposals should be formulated to promote its quality as well. Evaluation is carried out through internal or external procedures. The process of internal evaluation is comprised of the systematic collection of administrative data, questioning of students and graduates, as well as moderated conversations with lecturers and students. As part of the process of external evaluation a review team visits the department in order to review the quality of the academic studies and teaching. External peers are lecturers or persons from vocational practice who discuss with students and young scientists and present a final report. The evaluation of academic studies and teaching has to be followed by an account of how effective the measures of quality assurance are. Besides academic studies and teaching, the performance of research is evaluated at different levels: with reference to national research systems, individual institutions, research programmes or individual projects. In the field of research evaluation internal and external evaluations are also employed.
Definition
A systematic and critical analysis leading to judgements and/or recommendations regarding the quality of a higher education institution or a programme.
UNESCO CEPES
An ability to perform well or to achieve a result without wasted resources, effort, time, or money (using the smallest quantity of resources possible). Educational efficiency can be measured in physical terms (technical efficiency) or in terms of cost (economic efficiency). Greater educational efficiency is achieved when the same amount and standard of educational services are produced at a lower cost, if a more useful educational activity is substituted for a less useful one at the same cost, or if unnecessary educational activities are eliminated. A programme or a higher education institution may be efficiently managed, but not effective in achieving its mission, goals, or objectives.
External Evaluation
The process whereby a specialized agency collects data, information, and evidence about an institution, a particular unit of a given institution, or a core activity of an institution, in order to make a statement about its quality. External evaluation is carried out by a team of external experts, peers, or inspectors, and usually requires three distinct operations:
- analysis of the self-study report;
- a site visit;
- the drafting of an evaluation report.
Internal Evaluation/Self-Evaluation
The process of self-evaluation consists of the systematic collection of administrative data, the questioning of students and graduates, and the holding of moderated interviews with lecturers and students, resulting in a self-study report. Self-evaluation is basically a collective institutional reflection and an opportunity for quality enhancement. The resulting report further serves as a provider of information for the review team in charge of the external evaluation.
Student Evalutaion of Teachers
The process of using student inputs concerning the general activity and attitude of teachers. These observations allow the overall assessors to determine the degree of conformability between student expectations and the actual teaching approaches of teachers. Student evaluations are expected to offer insights regarding the attitude in class of a teacher (approachable, open-minded, entertaining, creative, patient, etc.), and/or the abilities of a teacher (to explain things, to motivate students, to help students think, to correct mistakes in a friendly manner, to offer information efficiently, etc.).
What is evaluation all about? Is it worth the work at all?
| "If you don’t know where you’re going, it doesn’t matter which way you take." |
Well first of all it is an instrument to assess any outcome of any project. It helps you to improve the quality of the project and to optimise processes when running the project again. It helps you to get feedback from all the participants and to create a follow-up.
But used in an inadequate way it may not help you at all to follow your outline.
It is important to state your goals and objectives before you start thinking about the questionnaire itself. Then the evaluation should be outcome-based, asking the right questions concerning the outline.
How can I start a proper evaluation?
There are eight steps to take setting up an evaluation questionnaire.
- Define your goals and objectives!
To assess an outcome you might prefer a summative evaluation (it can be an exam), to improve a project formative evaluation might be the one that should be used.
Be aware that the time-point of the evaluation in relation to the project is important. Evaluating every single lecture of a long course several weeks later is difficult if you ask to specific questions. On the other hand having an evaluation after each lecture leads to tiredness. Motivating participants to evaluate regularly may be a problem.
- Brainstorm on the different items you want to evaluate referring to your goals and objectives.
The OECD DAC (Organization for Economic Cooperation and Development – Development Assistance Committee) set up certain standard evaluation criteria, which also might fit into your goals and objectives. These are:
- Efficiency,
- Effectiveness,
- Impact,
- Relevance and
- Sustainability.
- Then you should choose the appropriate instrument for your evaluation.
Select the tools for descriptive purpose (leading to a summative evaluation) as well as for diagnostic purpose (leading to a formative evaluation). Different evaluation tools are for example assessment, self-assessment, fast-feedback-questionnaire, evaluation-form-questionnaire, interviews, peer-reviews and feedback-groups, but also tests and exams are a kind of evaluation evaluating each student. - This step is an optional one, but taking it can prevent serious problems: you should consider potential sources of error.
There might be cultural differences among those who fill the questionnaire. Having language problems and understanding problems may also lead to failure of the evaluation. So be aware of evaluating a heterogeneous group! - The next step is a pre-test of the evaluation among a small collective.
This helps you find and adjust some coarse mistakes and to apply improvements on answer or question format. - Step six is the evaluation itself.
Participants are motivated by personal addressing them. The aims of the evaluation should also be explained. - The seventh step is the analysis and interpretation.
The question forms should be recollected directly after the evaluation. It does not make sense to evaluate to late after a project. The participants won’t remember the details then, even if you ask only global questions. Then you collect all the data, interpret it and summarize the outcome with regard to your goals and objectives.
The outcome must then be published. Publishing it creates transparency and motivates both sides of the project. The evaluation is then more accepted among all the envolved persons. Maybe some suggestions and changes can also be published at that time. - The last step is also the most important one: draw consequences from the outcome!
You should design a follow-up and make another plan of action. After all the changes are applied, they also need to be evaluated.
Before I go into some examples of different question formulation, rating scales, validity and items, I would like to summarize some “take-home-messages”:
- Clear goals and objectives are essential for evaluation
- Evaluation without follow-up is useless.
- Transparency and involvement significantly improve the efficiency of the evaluation
- Evaluation can definitely improve your project!
Question formulation
| Question | Is the food at the restaurant good? |
| Statement | The food at the restaurant is good. |
| Positive formulation | The food at the restaurant is good. |
| Negative formulation | The food at the restaurant is not good. |
| Subjective formulation | I like the food at the restaurant. |
| Objective formulation | The food at the restaurant is good. |
Descriptive formulation is better understandable.
You should avoid double negation (Isn’t the food at the restaurant not good?) as it’s difficult to understand.
Using rating scales (i.e. not open answer), also double questions (Do you like breakfast and/or diner at the restaurant?) should be avoided, as one doesn’t know then, to which part of the question the answer refers to.
The questions and statements should be kept as short and simple as possible.
Rating scales
The scale size has an influence on the result. You can have more precise results having 10 possibilities. But on the other hand even Yes-No may also be enough. It’s important at this point, that you keep an eye on your goals and objectives!
The answer format must fit to the question itself, of course. Agree/Disagree-Scales do not fit to any kind of question. The opportunities to answer the question must differ clearly.
It makes a difference weather you use an even (2, 4, 6…) or an odd (3, 5, 7…) scale. Odd scales always give the opportunity to check the middle, which in an extreme way may lead to an evaluation where all middle items are crossed and for this reason don’t lead to any conclusion.
The “left-right-placement” also influences the results. In country’s using Latin letters there is a tendency to choose options on the left side of the scale. If you put “negative” on the left side the results of course differ compared to having “positive” on the left side.
Valid evaluation
In order to find out if an evaluation is valid, the ratio of participants vs. filled questionnaires must be taken into consideration. You should always give the possibility NOT to participate in the whole evaluation or not to answer certain questions. Then it is improbable that somebody just answers the questions without reading them at all.
You must stick to your original outline and should prepare the questions before the project started. Otherwise mistakes made or successes lead to different kind of questions in the questionnaire.
For example, if a sessions starts delayed or equipment is missing these things would definitively appear in the questionnaire and then easily a predominance either to the positive or the negative things exists. Therefore it is important to add open questions, so that things like missing equipment can still be mentioned by the participants.
Before the evaluation starts it should be reviewed to find mistakes in question and answer format and to eliminate complicated items.
Items
The answer format must be appropriate to the question and the question format. Open questions are important, because you can figure out all the possible things that might happen during the project in advance when setting up the questionnaire.
