Proposal review
We have received 4 proposals which all seem to be interesting and in line with the workshops goals as well as the ICT standards work programme by the EC. For all reviewers:
- Please read carefully through the proposals
- Please answer the questions below
- Please provide a short statement 10 days days before the WS April meetingThanks for contributing to the workshops's work to assure the quality of the proposals.
Please see the ICT standardization work program 2011 here: 2010-2013_ict_standardisation_work_programme_1st_update_en.pdf
Comments from Erlend in blue
Comments from Simon in Green
Comments from Adam in purple.
Comments from Cleo in orange.
Comments from Joris in grey
Comments from Tore in light green
Comments from Christian in yellow
Proposal 1: A specification to describe CONTEXT for competence-based learning
Draft_FPA_2011_Context for Competencies_20110228final.pdf
Please answer the following questions:
- Relevance and strategic alignment
- Is the proposal relevant? Yes
Yes
Yes - Does the proposal clearly relate to the ICT work programme? Yes - it is related to metadata, and should focus on that
Yes, it clearly falls within the areas of interest of European Learner Mobility (related to Learning Opportunities, Learning Outcomes and Competences), supporting the development of context-aware adaptive/adaptable, interoperable systems.
Yes, as it is focusing a base data schema to enable context-related systems - Does the proposal build on mature experiences (e.g., validated specifications, project developments)Specifying context metadata is new work, however the proposal is building on existing mature standards such as MLO, LOM, ELM, INLOC etc.
The proposal builds on existing mature standards (LOM, MLO, ELM) which have dealt with and include context-related metadata. In addition, there is significant experience with specific application profiles of these standards (e.g. ECTS refinements of MLO, UK XCRI) that have proposed extensions of a context-related nature. Finally, the ICOPER European project experience and outcomes also constitute an important basis for developments in the proposed field.
[Note: INLOC is not a mature specification – mention close collaboration and coordinated activities of the two initiatives]
The proposal highlights many other specifications: some mature (ELM, MLO, LOM) but not all of them can be called mature experience (e.g., ICoper and its PALO model as well as the InLOC project that is starting only now)
- Is the proposal relevant? Yes
Feasibility
- Target group: Does the proposal address a sufficiently large audience Yes
Yes
Yes - Stakeholders' interest and involvement; does the proposal provide suggestions for experts and organizations to be involved; have they expressed their interest? The proposal have identified a sufficient number of organisations willing to participate and contribute.
A number of stakeholder organisations have expressed their interest in contributing and/or liaising with the proposed project. Explicit categorisation could be made in order to highlight industry involvement, liaison with other standardization organizations, interaction with European policy making entities /initiatives and collaboration with other projects.
A list of stakeholders is included that have been contacted and are considered as potential partners and liaisons - Resources: Is the planning of the proposal feasible in terms of resources and management The deliverables and different identified activities in the proposal should be enhanced. And it is important that the project focuses on context-metadata.
The workplan, deliverables list and titles should be further elaborated. The conceptual model and information model could be the two main deliverables. Reporting is made through Progress Reports (e.g. half-year), an Interim (at half time of the project) and a Final Report. The suggested effort in man days is appropriate; however proposers could consider shortening the project time span.
The workplan with its work descriptions as well as the dliverables are only described very briefly: The planning seems to be feasible but it is hardly to judge whether the resources are too little, sufficient or overestimated. - Additional Resources: Does the proposal provide evidence that voluntary, non-paid experts will contribute? Participation from non-paid experts is not evident, however relevant projects are identified, how resources from these projects will be facilitated should be elaborated. Based on earlier experience from WSLT non-paid experts are contributing to the development of proposals.
Non-paid experts’ contribution is not explicitly mentioned. Proposers could add that the PT will be supported by a group of unpaid experts from different stakeholder communities, among which experts from collaborating projects, WSLT experts, industry representatives etc.
There is no explicit reference to non-paid experts.
Clarity
- Is the proposal justified, is the argumentation / reasoning clear? Yes - however how this work will influence and contribute to other standardisation activities such as SC36/WG7 and WG4 standards should be mentioned. References and harmonisation with ISO/IEC 19788-1 (MLR) should also be emphasised.
The proposal is justified. The argumentation could be enhanced with regard to the project’s impact on the existing mature specifications of CEN (MLO, ELM) and the related INLOC initiative.
Yes, the proposal and its background and objective are justified but the relationship to the other mentioned standards and initiatives could be improved and enlarged. - Are the deliverables clear, are the indicators appropriate? Tasks and deliverables should be further elaborated.
Workplan should be further analysed to include tasks, adequate descriptions and associated deliverables, as well as estimated effort (mas days) for each task.
The workplan with its work descriptions as well as the dliverables could be described more in details, there are no indicators for stakeholder engagement and impact.
Recommendations
- Please provde a short summary, This is a sound project, addressing a "white space" on our standardisation map. To clearly meet the ICT work programme, the proposal should make it clear that it is about describing different LET Contexts using metadata. That this is a LET metadata work, providing a general context description that could be facilitated by many other standardisation projects such as MLO, ELM, InLOC etc.
Existing mature specifications related to learning opportunities and European Learner Mobility would benefit from the development of a specification on context information. Such a proposal is consistent with the ICT work programme required standardization actions for e-learning, as well as with the Workshop’s aim to continuously enhance the LET standardization landscape with new specifications that build upon and further enhance the expressiveness of existing mature standards.
The topic of the proposal is very interesting and relevant, it needs more elaboration concerning the tasks and steps to achieve the objectives and relations to other standardization intiatives inside CEN (in particular InLOC, but also ELM) and outside CEN (in particular within SC36: the competence standards ISO/IEC 20006-x but also MLR). - State clear Improvement recommendations, what should be done to make the proposal better?
- Elaborate deliverables, and tasks
- Focus proposal on metadata
- Demonstrate how this proposal fits within existing standards such as MLR, MLO, ELM etc.
- Clearly state how related mature and under development specs will benefit from the outcomes of the project
- Enhance Workplan and deliverables description
- Further elaborate the stakeholder engagement part
- Use term competence(s) instead of competency(ies)
- See above concerning workplan, deliverables, relations to other standards (InLOC, ISO/IEC 2006-x)
- Focus more on concept than on metadata
- Use competence as key term consistently
- Do not use the term unit of learning but learning opportunity (as it is key term within CEN WS-LT and its specification like MLO, ELM and InLOC)
Proposal 2: Recommendations for the Implementation of Outcome-oriented Higher Education Outcome-Orientation-in-HE_v2011-02-28b.pdfPlease answer the following questions:
- Relevance and strategic alignment
- Is the proposal relevant? Yes
- Does the proposal clearly relate to the ICT work programme? The relationship to the ICT work programme is rather weak, not helped by the fact that the quotations seem not to be exact.
- Does the proposal build on mature experiences (e.g., validated specifications, project developments) It builds solely on the ICOPER project, which may not be as strong a basis as needed.
Feasibility
- Target group: Does the proposal address a sufficiently large audience Yes
- Stakeholders' interest and involvement; does the proposal provide suggestions for experts and organizations to be involved; have they expressed their interest? This is difficult to assess. Many partners/stakeholders are named, but one cannot see the level of interest.
- Resources: Is the planning of the proposal feasible in terms of resources and management Yes
- Additional Resources: Does the proposal provide evidence that voluntary, non-paid experts will contribute? Yes
Clarity
- Is the proposal justified, is the argumentation 7 reasoning clear? Some justification is provided; however it is unclear how much agreement there is about this justification.
- Are the deliverables clear, are the indicators appropriate? No. Reference models seem to be very much "flavour of the month", but what exactly are they, and how are they to be evaluated? These questions have not been addressed at the level where it would be clear that the proposed deliverables would be of genuine value. The indicators promise only that a reasonable process will be followed. As it is stated in the indicators section, the "problem with measuring the impact in the community is that the required data will not be available until some considerable time after the completion..." This is, indeed, highly problematic, and needs to be addressed within the proposal.
Recommendations
- Please provde a short summary, In the proposal's words, "The main objective is to harmonize a validated reference model for processes and data to enable competence-based learning." "The main focus will not be model development but the improvement..."
- State clear Improvement recommendations, what should be done to make the proposal better?
- What exactly is a reference model? It is a model that its authors want other people to refer to, but what is not so clear, in this case as well, is the basis on which the appeal to refer is going to be made. The word "validated" is extremely slippery: what is one person's validation, another person may reject as irrelevant. Again, here, what would help would be some way of assuring the reader that what is delivered matches what people really want. Simply "agreeing" something between a set of stakeholders by itself isn't any guarantee that it will be useful (in any way).
- The proposal seems to belong to the "big integrated system" school of thinking, and perhaps that was one of the problems with ICOPER itself. This project would surely be better off if it focused on one narrower outcome, more clearly needed, and carefully justified, than the ICOPER reference model as a whole.
Proposal 3: An Information Model for capturing Social Data in Technology Enhanced Learning
Proposal_Social_data_FINAL.pdf
Please answer the following questions:
- Relevance and strategic alignment
- Is the proposal relevant?
Yes, the proposal is relevant. The range of affordances of social data in TEL described in the proposal are relevant and known to be of quite wide interest. - Does the proposal clearly relate to the ICT work programme?
Yes, "social metadata" is identified in the "eskills and elearning" part of the EC work programme. - Does the proposal build on mature experiences (e.g., validated specifications, project developments)
The proposal builds on some projects and small-scale prior work. The proposal states "as shown with CAM... and afterwards in a matured state, also by industry" but does not provide evidence for the claim of industrial uptake or maturity. A greater emphasis on prior experiences with social data on the web at large would be preferable to demonstrate wider validity.
- Is the proposal relevant?
Feasibility
- Target group: Does the proposal address a sufficiently large audience
Partially. Viability of a social data approach requires a large scale of adoption and it is felt that the approach taken would not be appealing to a sufficiently large audience to maximimise impact and benefits because it is adopting a TEL-community-specific approach rather than building out from from a "universal" social web approach. - Stakeholders' interest and involvement; does the proposal provide suggestions for experts and organizations to be involved; have they expressed their interest?
There is evidence of interest from a community of European projects and LO repository groups. This is adequate for the proposal as written but could usefully be extended to match the approach suggested below. - Resources: Is the planning of the proposal feasible in terms of resources and management
The resourcing is reasonable to undertake useful work in this area but seems rather small to accomplish all elements of the plan with sufficient attention to detail. For example, 1/ the objective "will investigate privacy issues" risks dealing with rather an extensive field if it considers "issues" rather than established models for social data on the web and 2/ the objective "will build at least two reference implementations" seems unfeasible given effort quoted. - Additional Resources: Does the proposal provide evidence that voluntary, non-paid experts will contribute?
There is some evidence (second set of bullets under 5.3) that some stakeholders will be involved. The proposal may assume unpaid expert contributions, given the scale of the objectives, but may be omitting clear statements about them.
Clarity
- Is the proposal justified, is the argumentation 7 reasoning clear?
The arguments are clear on some points but a number of key questions should be addressed for a more robust justification:
1. It is claimed that "the main way to find learning objects is searching in such [LO repository] metadata catalogues" yet this does not seem to match general practice in educational institutions. The emphasis on LO repositories in the proposal is an unjustified restriction that should be justified or dropped in favour of a more "open web" attitude.
2. That privacy and security issues "need to be solved" within the project is not justified. An alternative (more pragmatic and practicable) approach would identify current approaches to such issues that are in use on the open web.
3. That a TEL-specific information/data model is both necessary or viable is not justified. Why is it necessary not to start from general open web protocols etc? Can it be shown that a TEL-specific approach is viable from an economic point of view (compare the "tooling" cost of TEL-specific vs use of common libraries and expertise for general web). Is is reasonable to expect international suppliers to adopt an idiosyncratic TEL-Europe approach. - Are the deliverables clear, are the indicators appropriate?
The deliverables are clear and the indicators appropriate for the work as described.
Recommendations
- Please provde a short summary,
The overall benefits to TEL are desirable but the approach adopted, with an emphasis on LO repositories and neglect of existing and emerging social web protocols/standards for the general web is believed by this reviewer to reduce the potential for impact. A better RoI for EC funds could be achieved by a stronger base in the work of the W3C Social Web incubator groups. - State clear Improvement recommendations, what should be done to make the proposal better?
1. Remove emphasis on LO repositories
2. Adopt a web-scale approach, for example, building on and referencing W3C Incubator Group Report "A Standards-based, Open and Privacy-aware Social Web", http://www.w3.org/2005/Incubator/socialweb/XGR-socialweb-20101206/
3. Consider alignment of the work with the W3C Federated Social Web Incubator Group (charter http://www.w3.org/2005/Incubator/federatedsocialweb/charter). This group ends in December 2011. Consider re-working the CEN PT deliverables to align with the three deliverables in this charter (section 2), with a TEL-specific slant.
It is recongised that not all of the ideas expressed by the proposers can be accommodated with general web approaches but it is recommended that this should be the starting point, especially in the absence of a clear case that a TEL-specific approach is viable. TEL-specific extensions to ubiquitous technologies and TEL-contextualised "reference models" based on ubiquitous technologies offer a prima facie more believable approach, even if the "solution" is less functionally-rich, because of the benefit that scale of use will bring.
Proposal 4: Framework for Information Related to and Managed by Learners
Since this is a proposal in development I would provide general comments:
* How this proposal is related to and different from ePortfolio, Leap2A and other relevant specifications should be made clear.
* Areas of privacy, lifetime of information etc. should be mentioned
* How this work will fit with other existing projects such as ELM, MLO, InLOC etc. should be more evident.
* Contribution/relationship to ongoing relevant SC36 activities should be mentioned
Please answer the following questions:
- Relevance and strategic alignment
- Is the proposal relevant?
- Yes, this proposal relates strongly to agenda for new skills and jobs, innovation, etc.
- Does the proposal clearly relate to the ICT work programme?
- yes
- yes
- Does the proposal build on mature experiences (e.g., validated specifications, project developments)
- it lists the appropriate some existing specs should mention other existing projects as well (e.g in iTec they are working on such framework i think).
- Yes, the proposal builds on a very solid eportfolio practice, however, this should be better outlined in the proposal. The benefits are described for the individual users and society at large. There is a need to see how the standard will answer to industry needs and the development of a market for eportfolio systems.*** Yes, the proposal builds on a very solid eportfolio practice, however, this should be better outlined in the proposal. The benefits are described for the individual users and society at large. There is a need to see how the standard will answer to industry needs and the development of a market for eportfolio systems.
- Is the proposal relevant?
Feasibility
- Target group: Does the proposal address a sufficiently large audience
- Quite general audience, could be specified further
- The potential market for this proposal is huge.
- Stakeholders' interest and involvement; does the proposal provide suggestions for experts and organizations to be involved; have they expressed their interest?
- N/A (proposal not finished)
- The proposal should identify a number of eportfolio systems developers willing to take part in the work in one way or another.
- Resources: Is the planning of the proposal feasible in terms of resources and management
- N/A (proposal not finished)
- N/A
- Additional Resources: Does the proposal provide evidence that voluntary, non-paid experts will contribute?
- N/A (proposal not finished)
- Indirectly it does, but needs to be clarified
Clarity
- Is the proposal justified, is the argumentation & reasoning clear?
- Quite short but to the point. Certainly a topic that has a bit of interest
- Is should be made more clear what is the current state of affairs, e.g., the shortcomings of the IMS ePortfolio and other specs that this work will improve.
- Are the deliverables clear, are the indicators appropriate?
- N/A (proposal not finished)
- N/A
Recommendations
- Please provde a short summary,
- The proposal aims at developing a conceptual model of learner-owned and learner-related objects; and an information model based on the objects that the market will need to implement in systems in the foreseeable future
- State clear Improvement recommendations, what should be done to make the proposal better?
- The proposal should be worked out completely, interest from a number of EU projects should be added that would use this format to be able to validate the harmonised format and to ensure impact. There could be added an implementation task to validate the scheme.
- The proposal is based on a sound methodological approach, described in the section on objectives and impact. This approach should be detailed in a work plan that makes it possible to see if the project could be split into different stages. This would make it easier to see if we could go ahead with the first part this year, knowing that we will follow up the work that needs to be specified more in detail later.