Guidelines - intended learning outcomes

Added by Simon Grant, last edited by Simon Grant on Nov 26, 2009  (view change)

Please send comments or suggested changes to the document editor, Simon Grant.

Guidelines on European learner mobility: intended learning outcomes

This section is a separate supplement to the core Guidelines on European Learner Mobility.

1. Useful information not represented in current mobility documents

CEDEFOP has researched and produced weighty documents on "The shift to learning outcomes" [CEDEFOP 2008], [CEDEFOP 2009], which lead the way in terms of pointing out what is missing from more traditional approaches to learner mobility. CEDEFOP is just one body in what seems to be a consensus that something like learning outcomes (other people refer to competencies, etc.) are the single most important feature not currently represented in institutionally-derived learner mobility documents, and represented rather inflexibly and thinly even in learner-generated documents.

The [CEDEFOP 2009] study argues that

...learning based uniquely on input will not respond adequately to future challenges for individuals, society or the economy. The trend is to rely, increasingly, on the identification of learning outcomes. This trend is recognised as critical in many different contexts across education and training systems.

The CEDEFOP work does not explicitly suggest ways of recording learning outcomes in transcripts, but it does refer to the ENQA report on Standards and Guidelines [ESG], which emphasises that

The realisation of the EHEA depends crucially on a commitment at all levels of an institution to ensuring that its programmes have clear and explicit intended outcomes; that its staff are ready, willing and able to provide teaching and learner support that will help its students achieve those outcomes ...

The introduction in the most recent templates for the DS provided by CEDEFOP [TEMPLATES] includes the explicit mention that the purpose is

... to provide sufficient independent data to improve the international 'transparency' and fair academic and professional recognition of qualifications ...

The intention to go beyond the academic and into professional recognition would be greatly helped by the ‌inclusion of intended learning outcomes, and evidence of their attainment, including, where appropriate, competences.

2. Motives for representing intended learning outcomes

There are several indications that the outputs or outcomes of learning processes are seen as being of substantial significance, and that they may be increasingly seen as of comparable importance to the inputs to educational processes — that is, the topics and methods of study.

  • The strategic framework for European cooperation in education and training [ET 2020] emphasises the importance of learning outcomes in its strategic objective 1.
  • The ECTS Users' Guide has plenty of material and discussion on the topic of learning outcomes.
  • The EQF brochure distinguishes knowledge, skills, and competence as three possible kinds of learning outcome.
  • In 2007, Aviana Bulgarelli [Bulgarelli] spoke on the "shift to learning outcomes in European education and training policies and practises".
  • Many arguments for the use of learning outcomes are given in detail by two CEDEFOP publications [CEDEFOP 2008], [CEDEFOP 2009] under the common title "The shift to learning outcomes" (though with different subtitles).
  • The ENIC-NARIC / ENQA joint project, "Study on the Diploma Supplement as seen by its users" [ENQA] has, among its recommendations (p.33) "There should be greater emphasis on learning outcomes and the provision of personalised information which should distinguish the graduate from others. The use of clearly described learning outcomes facilitates the understanding of the document."
  • As an example of EU funded work, the ICOPER best practice network [ICOPER] is addressing such issues as "exchange of competency models and learning outcomes".

Clearly, if learning outcomes corresponded to entry requirements, either for further study or for employment, they would have the potential to contribute greatly to European learner mobility. However, intended learning outcomes are not directly represented in European learner mobility documents. Competences are recorded only informally in the Europass CV, under "personal skills and competences", while only very specific areas of linguistic competence are self-assessed in the Europass Language Passport.

3. What are intended learning outcomes, and how can they be used?

To base this discussion on current official European thinking, a good place to start is the EQF recommendation [EQF-REC] and brochure [EQF brochure].

3.1 EQF definitions and discussion

The EQF [EQF brochure] defines "learning outcomes" as meaning "statements of what a learner knows, understands and is able to do on completion of a learning process, which are defined in terms of knowledge, skills and competence". Similar definitions are used by CEDEFOP [CEDEFOP 2008], [CEDEFOP 2009] and the European Credit Transfer System Users' Guide [ECTS Users' Guide].

These definitions make good sense, but it is difficult to be sure of exactly what the influence of a learning process may have been on "what a learner knows, understands or is able to do". It makes more sense to focus on intended learning outcomes. The ENQA use the term "intended learning outcomes" and not just "learning outcomes" in their Standards and Guidelines document [ESG]. Given intended learning outcomes, it is possible to make judgements about whether learners have achieved any particular intended outcome, much more easily than finding out just what a particular learner knows, or can do.

More detailed understanding of the EQF definition comes from following through to their definitions of knowledge, skills and competence. To understand in practice what these mean, it is helpful also to consider how they are assessed.

Knowledge

The EQF definition of knowledge,

"the outcome of the assimilation of information through learning. Knowledge is the body of facts, principles, theories and practices that is related to a field of work or study ... knowledge is described as theoretical and/or factual"

is enough to indicate that familiar kinds of test and examination questions, including e-assessment, are a reasonable way of approaching assessment of this area. The materials to be assessed are the answers to the questions. The results of tests or examinations are used to present evidence to others of the attainment of a learning outcome in this area.

Skill

The EQF defines skill as

"the ability to apply knowledge and use know-how to complete tasks and solve problems ... skills are described as cognitive (involving the use of logical, intuitive and creative thinking) or practical (involving manual dexterity and the use of methods, materials, tools and instruments)"

Of these, the logical "cognitive skills" might be assessed through traditional examinations and tests, including automatically processed ones, where there are questions and well-defined answers. Evidence here would be treated in a way similarly to evidence of knowledge.

Practical skills may be assessed through tasks of various kinds, some perhaps involving computer games or simulations, and others involving other "materials, tools and instruments". Performance on computer mediated tasks can often be recorded and judged automatically. However, other skills may not be suitable for automatic marking.

The fact that the EQF definition makes no reference to real-life contexts of application suggests that, for the EQF, skill is not defined by context, beyond the presence of the necessary materials, and perhaps suggests also that skills generalise across contexts.

Competence

The EQF definition of competence

"the proven ability to use knowledge, skills and personal, social and/or methodological abilities, in work or study situations and in professional and personal development ... described in terms of responsibility and autonomy"

is much more challenging. The reference to work or study situations suggests that competence is understood as requiring the use of knowledge, and the practice of skills, in real life, rather than in a context designed specifically for learning education or training. The term "proven" may suggest that there is some standard of performance which has been proved, or quality standard achieved. The words "responsibility" and "autonomy" also suggest that there are some rules, typically tied to outcomes, involved in the display of competence.

As competence, according to the EQF, "uses" knowledge, skills and other abilities, logically it cannot be equated with them. This goes against the imprecise common usage, which tends to conceive of skills and competences as similar things. Rather, the careful study of this definition indicates that, for the EQF, skill and competence must be quite different. Perhaps what is holding people back from this understanding is not having a clearer concept of what this difference is, and how competence "uses" knowledge and skills.

One way of thinking about this is that competence also requires or involves choices, which need to be adequate to the situation. The first kind of choice is about knowledge: which facts to attend to, and what theory to apply. The second kind of choice is about skills: which skilled actions to take, according to rules and conventions governing the appropriate choice of actions, when there is more than one course of action is available to the competent actor. Thirdly, competence may involve social and/or methodological abilities to determine outcome criteria by which to judge the competence or incompetence of a particular performance.

The sense of this can be seen in its boundary conditions. In situations where there is only one course of action available, all the facts are known, and there is at most one theory to apply, it seems likely that people would not judge performance as "competent", but rather as routine or mechanical, and it might well be suitable for automation. Other boundary conditions are suggested by common usage of the term "incompetent". It is fairly clear that incompetence can result from either a lack of knowledge, or a lack of skill, or a disposition to make inadequate choices in terms of attention, action and performance criteria.

In short, competence involves knowledge, skill, and the disposition of a person to choose which rules to follow, what information to take into consideration, and which courses of (skilled) action to take, with the outcome of meeting agreed performance standards, in a natural appropriate context of application of the competence. The assessment of competence must therefore go beyond the assessment of knowledge and skill, and needs to be in the context of application, or in a sufficiently similar context that the same choices would be made as in the real context.

3.2 Use of intended learning outcomes

Expressions of intended learning outcomes, whether relating to knowledge, skills, competence, or a combination, can be used in several ways, most obviously in conjunction with learning opportunities.

  • They are naturally used in the design of learning opportunities, to express their intended outcomes.
  • They can be used in the design of the evaluation or assessment that is part of, or follows, the learning opportunity with those intended learning outcomes.

When used separately from the learning opportunities that intend to result in them, they may be easier to think of in terms of definitions of knowledge, skill or competence.

  • They can be listed as pre-requisites for acceptance into learning opportunities.
  • They can be used in person specifications in the course of recruitment.
  • Individuals can claim to have attained them.
  • Evidence can be assembled by or about individuals to support a claim to their attainment.
  • They can be used by employers or professional bodies as the basis for review processes that tie in with career progression.

In wider personal and professional development, action planning and target setting, individuals can plan to attain knowledge, skill, competence, etc., for example as defined by some professional body. This may be a requirement for career progression. Continued certification by the professional body may depend on adequate evidence being presented to assessors. E-portfolio tools may help in these processes.

4. Positioning intended learning outcomes in learner mobility models

There are these clear distinctions between knowledge, skill, and the other aspects of competence, but that does not seem to affect how different types of learning outcome relate to learner mobility models. Whereas it may be that employers often look for fully-developed competences, educational processes may require only knowledge or skills acquired from earlier stages of education. Employers may also require knowledge or skill outcomes, on the understanding that competence will develop with training and experience. Thus, no distinction is made for these models between different kinds of intended learning outcomes.

A similar lack of operational distinctions can be seen reflected in the definition, from 2004, of "competency" in HR-XML 2.3 [HR-XML]:

A specific, identifiable, definable, and measurable knowledge, skill, ability and/or other deployment-related characteristic (e.g. attitude, behavior, physical ability) which a human resource may possess and which is necessary for, or material to, the performance of an activity within a specific business context.

4.1 The conceptual model

The conceptual model, given in the main Guidelines [GUIDELINES], has been expressly designed to included intended learning outcomes, which are related by the conceptual model to learning opportunities. It would make little sense for two instances of the same learning opportunity to have different intended learning outcomes: in that case they should be regarded as instances of different learning opportunities.

As suggested by the proposed conceptual model, evidence generated by a learner can be taken as evidence of the attainment of a relevant learning outcome. How that evidence is to be evaluated may be specified somewhere — this would correspond to an evaluation specification in the conceptual model.

4.2 A possible domain model including intended learning outcomes

Intended learning outcomes may be specified for each component learning opportunity, but, particularly for generic, transferable or key skills, any one of those outcomes may be an objective of several components (courses, modules, units, etc.). What follows is a first attempt to describe how the related information could be represented.

Intended learning outcomes could be associated with each learning opportunity, potentially at any level. This association is already provided for in MLO with the "objective" property. If the attainment of an intended learning outcome has been directly assessed, this could be represented as a separate result of the learning opportunity instance in which it was assessed. Whether or not it has been assessed as such, some circumstantial evidence for the attainment of a learning outcome is in assessment results relating to any learning opportunity instance that has the intended learning outcome as an objective.

Here is one possible view of a possible abstract model, based on the abstract model for European Learner Mobility documents in the European Learner Mobility specification, and envisaged as applied to a programme where all the learning outcomes recorded were the objectives of formally specified learning opportunities.

Figure 1: A possible abstract model including intended learning outcomes

In general, intended learning outcomes could be described either internally within a learner mobility achievement information report, or externally at a stable location on the web. If they were described internally, the highest-level reference (i.e., for the broadest learning opportunity with which that outcome was linked) would include an internal ID and full description; to avoid duplication, and to ensure the ability to link them together, other representations of the same outcome would be represented just by an internal link.

Due to the different purposes of domain models and conceptual models, this model does not correspond exactly with the conceptual model in the core Guidelines. The "hasResultAchieved" and "hasResultPredicted" correspond between them to the "results in" dotted line relationship in the core conceptual model.

The relationships "isGivenBy" and "isPredictedBy" are not shown in the core conceptual model, but they are implicit in the facts that the assessing body manages the process which "has result" of the assessment result. Particularly in the case of predictions, but also for formal assessments, the assessing body can be any agent, including the learner, or another individual, and the process may be informal in nature. The assessment process itself is not usually represented in information models, so for an information model it is reasonable to elide it.

The "hasEvidence" relationships can be seen in two ways on the conceptual model. First, it could be an elision of the fact that evidence is assessed in the assessment process, which results in the assessment result. Second, it could be an elision involving the fact that an assessment result contributes to a wider assessment process — the wider process in this case would be the assessment of the intended learning outcome, and the narrower process would be the assessment related to the learning opportunity with the intended learning outcome as objective. Thus the narrower assessment result could be taken as evidence in the wider assessment process, resulting in the assessment of the attainment of the intended learning outcome.

It is when the result is about the attainment of an intended learning outcome that it references the outcome. Otherwise, that reference is unnecessary, as the connection between result and outcome can be traced through the learning opportunity instance and specification.

5. How learning outcomes (including competences) could be included in Europass documents

5.1 As an enhancement to the current Europass Diploma Supplement

Current Diploma Supplement documents tend to record the results of formal learning opportunities in the form of a transcript, in section 4.3, where marks or grades associated with each module, and credits for the module, are given against the module title. Section 4.3 would therefore also be the natural place to record the attainment of learning outcomes that have been officially assessed. Section 4.3 could including learning outcomes in various ways.

  • Intended learning outcomes could be represented, once primarily, at an appropriate level of learning opportunity specification. If an outcome was relevant only to a particular module, it would make sense to represent it as an objective of the module. If to several modules, but only within (e.g.) a single year, then it could be represented primarily as an objective of the year. If several modules across more than one year have the same intended learning outcome, it should be primarily represented as an objective of the programme as a whole.
  • Wherever they are represented in an electronic document, an intended learning outcome should have an ID to allow cross-reference from other places. Ideally, this should be a universal URI that is able to be referenced from anywhere in the Web, and which resolves to useful information including the outcome definition. However, where this is not appropriate, it could alternatively be an internal ID.
  • Where an intended learning outcome is an objective of more than one module, then each module it is an objective of should refer to the outcome by reference to its internal ID, not by duplicating the definition of the outcome. This reference would be alongside the normal results and credit values for the module.
  • If an outcome has been assessed, attaining it could be represented as a standalone module with no other content. If transcript practice changes in the future, to allow standalone assessments, then it should be represented there.

This reflects the abstract model given above. However, it must be stressed that this is no more than a first attempt to suggest a way of including intended learning outcomes, and there may be ones that are seen to be better.

Cross-referencing the unit results, to the learning outcomes they are evidence for, can easily be envisaged in an electronic document, using the familiar hyperlink approach. This would not be difficult to represent using any common technological approach. To envisage the same on paper would perhaps involve some cross-reference codes, much as in footnotes, endnotes, or bibliographic references.

5.2 The current Europass Certificate Supplement

For the current Certificate Supplement, the position is substantially easier. As the Certificate Supplement typically does not attempt to represent individual assessments or results, there would be no need to include any evidence relationships, and the intended learning outcomes may be listed, without assessment results, alongside the relevant learning opportunity components. The abstract model can be a restricted version of the one for the Diploma Supplement, where the restriction is that the results and associated relationships are left out.

5.3 The current Europass CV

The Europass CV has a section for self-assessed "personal skills and competences". This is clearly not expected to be the same as institutionally-assessed competences. The Europass CV template only allows learners to document their skills under certain predefined headings:

  • Languages (as in the Europass Language Passport)
  • Social skills and competences
  • Organisational skills and competences
  • Technical skills and competences
  • Artistic skills and competences
  • Other skills and competences
  • Driving licences

The problem here is that there is no common basis for these predefined headings. What would the dividing lines be? Most people can imagine particular examples of a typical skill or competence belonging to each heading, but what exactly, for instance, is the dividing line between social and organisational skills? The Europass CV documentation does not offer any clear definitions of what should go in each section, and while this is not a particular problem for a human reading an individual Europass CV, it means that there can be no reliable machine processing relying on the different headings. Thus the value of these headings for electronic purposes is questionable.

Moreover, there is no explicit, separate place for the intended learning outcomes of educational opportunities that are taken by a learner, and this implies that the current Europass CV structure is not well-adapted to representing learning outcomes in a way that is coherent with the currently represented skills and competences. In any case, it could be expected that the Europass CV would be better represented by an e-portfolio specification such as LEAP2A [LEAP2A]. When using skill and competence information from a EuroLMAI report to create a Europass CV, the learner would need to connect the intended learning outcome information from EuroLMAI to the Europass CV categories.

5.4 The current Europass Language Passport

Similar considerations apply here as with the Europass CV. The Europass Language Passport relies on a self-assessment grid, which contains descriptions of different levels of proficiency in different areas of language. These descriptions could be intended as descriptions of learning outcomes of learning opportunities. No other learning outcomes are immediately relevant.

5.5 Possible future mobility documents

If representing learning outcomes became a high enough priority that it was seen as an essential part of learner mobility documentation, it might make sense to represent those learning outcomes as a primary part of learner mobility documents, and not just as dependent on having taken particular learning opportunities with intended learning outcomes. This would assume that the assessing body — whether the educational institution or some other provider of assessment services — had methods of assessing whether a learning outcome had been attained, and that these methods went beyond the evaluation of the intended learning outcomes of regular learning opportunities. This is not difficult to imagine.

Currently, with an electronic version of a Diploma Supplement, one can envisage the names of component learning opportunities acting as links to detailed definitions of those opportunities (held by the institution in MLO format). Similarly, in the future, one could envisage the names of intended learning outcomes being linked to detailed definitions of those outcomes. These definitions could be held either by the educational institutions, or by industry bodies (e.g., in the UK, Sector Skills Councils) with the remit to define occupational standards for their industry. In each case, the linkage would be by URI. The URI would be the identifier for the learning outcome, but following W3C recommendations, may not be the address of a web page, but instead may redirect to a web page. In any case, the result of entering the URI in a browser should be that some useful information concerning the intended learning outcome would be displayed, at least containing a definition, and possibly with links to more information.

There are various possibilities for changing the structure of an electronic version of the Europass CV to represent learning outcomes, including knowledge, skills, and competences. The simplest way to do this might be to replace the predefined headings with an open structure allowing learners to give their own headings for skills that they believe are relevant to the opportunities to which they are aiming in writing their CV. This has the potential to result in a more focused document, not attempting to cover every kind of skill.

If the European Language Portfolio (of which the Europass Language Passport is one of three constituent parts), or something similar, was given a formal structure, one could envisage other language related learning outcomes being represented as well as the standard ones as defined by the Council of Europe.

6. Other structures needed for effective deployment of services based on learning outcomes

Defining intended learning outcomes is a vital step, but does not, of itself, ensure that they can be effectively deployed. If learning outcomes are to play their role in effectively linking the outcomes of education to the requirements of employment, the intended learning outcomes assessed by HEIs need to be identifiable as the same as those required by employers. This goes against the tendency of everyone to invent their own definitions for everything. To be useful, definitions of intended learning outcomes need to be widely shared and recognised. In principle, this could be achieved through the use of a common reference framework for learning outcomes, including skill and competence. However, experience suggests that this is difficult.

As an alternative, or while awaiting a commonly adopted reference framework, individual definitions will have to be cross-mapped to each other so that common meanings can be recognised in varied contexts. Mechanisms for doing this should be explored for a future CWA.

References

  • "Available at" means that the URL given is the URL of the document itself, or a version of it.
  • "Available through" means that a link to the document appears on the page with the given URL, where other related material and documents may also be found.
  • "See" introduces a web site or sub-site, that is, several relevant web pages which may be browsed.

All web references were accessed successfully in November 2009.

Documents

[Bulgarelli] Rhetoric or reality? The shift to learning outcomes in European education and training policies and practises. Aviana Bulgarelli. 15 October 2007. Available at http://www.cedefop.europa.eu/etv/upload/etvnews/news/2924-att1-2-speech_lo_conference_bulgarelli_final.doc.

[CEDEFOP 2008] The shift to learning outcomes: Conceptual, political and practical developments in Europe. Available through http://www.cedefop.europa.eu/etv/Information_Resources/Bookshop/publication_details.asp?pub_id=494.

[CEDEFOP 2009] The shift to learning outcomes: Policies and practices in Europe. Available through http://www.cedefop.europa.eu/etv/Information_resources/Bookshop/publication_details.asp?pub_id=525.

[ECTS Users' Guide]. Available at http://ec.europa.eu/education/lifelong-learning-policy/doc/ects/guide_en.pdf.

[ENQA] "Study on the Diploma Supplement as seen by its users" – A project organized in joint cooperation between ENIC-NARIC and ENQA (November 2008). Available at http://www.enqa.eu/files/Diploma%20Supplement%20Study_Edit%20MS.pdf.

  • See in particular: Conclusions and Recommendations (p. 33).

[EQF brochure] The European Qualifications Framework for Lifelong Learning (brochure), April 2008. ISBN 978-92-79-08474-4. Available through [EQF].

[EQF-REC] RECOMMENDATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 23 April 2008 on the establishment of the European Qualifications Framework for lifelong learning. Available through [EQF]. English version available at http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:C:2008:111:0001:0007:EN:PDF.

[ESG] Standards and Guidelines for Quality Assurance in the European Higher Education Area 3rd Edition. ENQA, Helsinki, Finland, 2009. Available through http://www.enqa.eu/pubs_esg.lasso.

[ET 2020] Strategic framework for European cooperation in education and training. Available at http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:C:2009:119:0002:0010:EN:PDF or through http://ec.europa.eu/education/lifelong-learning-policy/doc1120_en.htm.

[GUIDELINES] Guidelines on a European Learner Mobility model. Core document, to which this is a supplement. A wiki version is available at http://wiki.teria.no/confluence/display/EuropeanLearnerMobility/Guidelines+on+European+Learner+Mobility.

[HR-XML] Competencies (Measurable Characteristics) Recommendation, 2004-08-02. Available at http://ns.hr-xml.org/2_3/HR-XML-2_3/CPO/Competencies.html.

Web sites

[EQF] The European Qualifications Framework. See http://ec.europa.eu/education/lifelong-learning-policy/doc44_en.htm.

[ICOPER] ICOPER Best Practice Network. See http://www.icoper.org/.

[LEAP2A] The LEAP2A specification for portability and interoperability of e-portfolio information. Version of March 2009. See http://wiki.cetis.ac.uk/2009-03/LEAP2A_specification.

[TEMPLATES] Europass Diploma Supplement templates. Web page. See http://europass.cedefop.europa.eu/europass/home/vernav/InformationOn/EuropassDiplomaSupplement/Restricted/DIPDownload/DSTemplate.csp.

Labels

 
(None)