In 2008, Kelly and Tangney’s discussion of adaptive learning systems notes that they ‘can bring different benefits such as increased learning performance, greater enjoyment, enhanced motivation and reduced learning time’. In outlining this quartet, they hint at just how useful adaptation could be in an online distance teaching context. However, to date, the realisation of properly adaptive teaching in distance learning and the planning for adaptation in their design remains arguably quite limited.

If there is indeed the potential to make improvements to workload, to better engage and empower students in their own learning, to increase retention and to help students achieve more, then it is imperative that online and blended learning understand when and how adaptation can become an effective teaching tool could be used.

Of course, designing adaptation requires a departure from the linear, single path a learner is taken through a course, be this across a lecture series, a distance learning guide or a series of online learning tasks. Reigleuth makes the point that ‘one size does not fit all’ and indeed, in 1999, the materials for the OU’s own course on Teaching in Higher Education stressed how students learn differently and proposed as good teaching practice that the teacher understand and respect their student needs, approaches to learning, stage of learning development, interests, motivations, backgrounds, prior knowledge, etc. Doing this, it was explained, will allow teachers to best adapt the learning experience to their students including planning for their support needs.

According to Moreno-Ger et al. (2009) explain that ‘adaptation is a wide and rich field [which has evolved] from the early 1980s where computer-based training was used to fully control the flow of a learning process.’ Adaptive systems provide an alternative to the ‘traditional just-put-it-on-the-web’ or ‘one size fits all’ approach to the development of Wed-based education courseware. Put simply, they suggest, ‘adaptation is the ability to modify lessons using different parameters and a set of predefined rules, in order to try to cater for the needs of all kind of learners and thus maximise the effectiveness of the learning experience. Perhaps unsurprisingly, terms such as ‘adapting’ and ‘student needs’ feature prominently and advocates are united by a desire to enhance the student learning experience. I have created working definitions of adaptation, customisation and personalisation (see end of post).

In the eight staff interviews undertaken, a number of potential benefits were identified. These included:

• Increase the sense of student involvement in the course
• Increase student satisfaction with the course
• Enhance experience for work based learners
• Better integrate the support for student and provide only if they need it
• Better monitoring data of student study and preferences
• Help students manage the array of resources and activity prepared
• Better estimate the student workload and integrate optional material
• Offer clear potential in integrating ICT and IL sessions more directly in to the study planner

Adaptive systems can be categorised into a number of types. The following table, whilst not intended as a definitive representation, summaries these. The main focus of the report was adaptive course delivery and teaching and, although haven’t the space to go in to detail here, the full report looks more closely at aspects of this (navigation, presentation, assessment, sequencing, feedback etc.), gathers a couple of exemplars of institutional examples and begin the task of reflecting on the issues of implementation within a teaching institution.

Looking back to when I started this project, I was surprised that there was already some, albeit limited, functionality within Moodle to build adaptation in to the online study planner (a tool that students are encouraged to use and which takes up a prominent position on their student home page).

Simone and I have taken the opportunity presented by this project to think about how to develop the Moodle functionality further. One obvious omission in the adaptive arsenal is a ‘decision’ plug-in – something that presents a choice to the learner and which can trigger adaptation of content, navigation, links, sequences etc. within that week of learning or later in the course. We have so far completed a specification and development work has begun, and hope that resourcing can be found to complete this.

I also believe that there would be great benefit in building and evaluation of a demonstration unit which includes the various adaptive options available (so if you are working on a course at the OU (or indeed elsewhere) please do drop me a line!)

Selected References

Kelly, D. & Tangney, B. (2006) Adapting to intelligence profile in an adaptive educational system, Interacting with Computers, 18, 385-409

Karakostas, A. and Demetriadis, S. (2011) Enhancing collaborative learning through dynamic forms of support: the impact of an adaptive domain-specific support strategy, Journal of Computer Assisted Learning, 27, 243-258

Moreno-Ger, P., Burgos, D. and Torrente, J. (2009) Digital Games in eLearning Environments: Current uses and emerging trends, Simulation and Gaming, 40, 5, 669-687

At present the full report is only avialable to OU staff, however, I hope to rework this for wider dissemination shortly.

My working definitions:

Adaptation is broadly taken to refer to any change to a unit of learning or teaching made with the intention of enhancing the learning experience whilst the module is in presentation and in response, either directly or indirectly, to student activity seen, witnessed, recorded or logged whilst they were taking the module. Presenting students with choices, developing decision-making logics and conditionality, or responding to learner behavior are examples of adaptation. Given the very nature how adaptation works, there will always be more than one potential route through an adaptive learning element.

Customisation is taken to refer to change made to a unit of learning or teaching with the intention of enhancing the learning experience before the learner starts studying it. These changes are made in response to specified criteria and may represent a deviation in some way from an existing original module design. The resulting module may be no more adaptive or interactive than the original one (i.e. it may be just as linear).

Personalisation is defined in respect to student perception. For personalisation to have taken place, an individual student must perceive that the learning and teaching has been made more appropriate to their personal needs – that it has been made ‘more personal’ (whether they acknowledged those needs in advance or not). Adaptation and customisation are methods by which we can seek to make a module more personal.

For this round, I’m also written a new additional module titled An Introduction to how to learn with open distance courses such as openED.  The aim of this module is to help the learner understand how to study better using open online material and should last 1-2 hours. This module is based on the most common questions asked by openED participants which I have identified from a detailed review of the Round 1 and Round 2 forums and over fifteen hours of text-chat session transcripts. The module uses examples from the openED course to help illustrate ‘real-world’ examples and includes three short reflective exercises (5-10 minutes) aimed at thinking how using online social spaces such as forums can enrich learning experiences. The module includes short sections on:

• What is open distance learning?
• When to start?
• How to study an open module
• Using forums to meet others, share work and resolve questions
• Live text-chat sessions
• Group working
• Getting feedback and recognition for your work

To see the module in situ check out the openED website. I have also made the original text document (3,600 words) available under a limited creative commons licence for reuse and repurposing. You can download it using this link: Introduction To How to Learn With Open Distance Courses_Cross_2011_v6  

One exciting feature of this module is my sharing with the learners a basic visual representation of the recommended learning route (learning design). I have of course had to abstract the design to suit any of the 10 mondules (each differs in content) so it does not convey the detail or nuance that I would often put in such designs. 

The intention is that participants work through the Introduction module first. This will better prepare them for the course modules and we will monitor how this contributes to greater or more prolonged engagement.

In case this format of representation is unfamiliar: you can see the seuqence of tasks for the learner running down the centre (white boxes). The resources to be used are shown to the left and right of the tasks. The left hand-side refers to the section of the module that would be relevant and the right-hand side to the online tools that could support them.

Another form of representaiton I’ve used in the module has been comparison tables. These tables aim to help the learner understand the choices available to them - the potential options. This one contrasts the option to start the course any time with studying it during the ‘supported’ times offered by the openED team:

For more information about the openED course please visit the website. Also for this third round we are also trialling a twitter stream (#openEDr3) which will feature module start/end reminders, flag up interesting discussions, times for the chat sessions etc. And of course, if you do find a use for the Introduction module outlined below, feel free to post your experience (and adapted material) below.

The idea of sharing refers to what could be termed ‘reciprocal altruism’ – a form of altruism where there is anticipated to be exchange of things, often of those judged of broadly similar value. This notion of reciprocal altruism, often with an emphasis on the interaction that takes place not reasons for it, forms a cornerstone of how many people imagine how open education systems will work and should be configured; designers exchange designs, students review each others work, teachers share their resources. Ask why they think such interaction will take place and either the fall-back position of reciprocal altruism is assumed accepts begrudgingly accepts conscious self-interest.

Yet psychological, sociological and economic research tells us that there is more to altruism than reciprocity or sharing. Altruism for example can also take the form of kin selection, or nepotism, with individuals supporting others seen as similar to themselves thereby binding communities closer together. It can also take the form of charity – the act of giving to a stranger; the understanding of this is fundamental to entire sectors!

It seems to me that focusing on the act of ‘giving’ – looking at individual and structural motivations, intents, behaviours and social norms – can enrich our models of OER use. What if we look to interpret contributions to OER repositories or other open learning and design repositories as acts of altruism such as charity or kin-selection? Can we use such perspectives to build a complete picture of what openness is or could be like?

So why do people ‘give’? There are many overlapping theories. One perspective holds that people give to feel good or better about themselves (even to remove a sense of guilt or obligation). This ‘warm glow’ comes from  feeling the individual has made some altruistic or self-sacrificial gesture that will benefit the wider community. Of course, this doesn’t mean that such an act will benefit anyone, but that’s not the point, the point is that it might do and that’s enough. The implication for education is that a contributor may be more focused more on fulfilling self-need than the value of the contribution to the consumer or recipient. And indeed, this model of charity is often used to encourage educators to contribute (give) for those trying to teach in poorer resourced countries.

A second perspective argues that people give to boost social status, to be recognised as having given and for enhancement of self-image. Coupled with this are the ‘private benefits’ that giving may yield. As one study on charitable donations found ‘people who may have greater incentive to seek occasions for social networking place a much higher value on private benefits associated with their giving levels.’ It seems the power of the charity’s status in the community is also a central influence on contributions – people give more to things that are more valued by the community (for us this may be those teaching and learning and (open) education communities with higher status or visibility). Conferring social approval rewards conformity to, and replication of, particular norms and it is through this enforcement that social norms associated with types of altruism with be maintained. However, unlike charity, sharing expects external reward. Reward mechanisms such as ranking or stars would seem to be aimed at offering private benefit to individual contributors yet may conversely discourage charitable contribution.

A third explanation for altruism focuses on the perceived amount of benefit or cost to the individual. This means that often people give that which is of low value, marginal cost or give ‘away excess’. When we look at contributions to an OER repository or teaching and learning sharing website are we actually seeing people give away artifacts of relatively low value? (although this is not to say that OER repositories run the risk of becoming the equivalent of a second-hand charity shop clothes rail!). And, when posting on public websites there is the potential professional cost (such as others rating a contribution as of poor academic or scholarly quality). Of course, value is relative and what one group values highly another may not. For example, the author of a journal paper sees value in having it published and read by colleagues, whereas a publisher would, I guess at least from a commercial perspective, be concerned with how many copies are sold and if it appeals to its readership. An author no more readily invest the time writing it but then see that time wasted by it being published under someone’s name with no acknowledgement, than a publisher would want to see the time invested in editing, printing, publicity, subscription services and building reputation wasted by it being given away for free.

Other research on altruism suggests that people give because of empathic or emotional attachment. Behind this is presumably is a process of building an imagined beneficiary; the better the recipient (be that another teacher or learner) is defined, the greater the potential empathic response. Others talk of the importance of obligation, attribution, role identity and fairness.

I’ve also come across two other interesting ideas whilst writing this post. The first is that ‘discrimination is a necessary part of persisting altruism’ (Hardin). This ideas holds that in the real world everyone cannot be altruistic all of the time (or in fact some even part of the time). Consequently, for altruism to work as a strategy, groups evolve who gain benefit from mutual sharing which help them relative to other groups. The idea that altruistic activity must necessarily involve setting boundaries challenge a fundamental vision of open systems being places for everyone.

The second idea relates to the creation or management of public goods and the free-rider phenomenon (and I suggest we could consider open learning repositories and resources as public goods). I understand this idea comes from economic theory. It suggests that the ‘free-rider’ (in OER terms the equivalent of those who look at/take from sites but do contribute) will cause a group to provide itself with no more than a minimal level of public goods, even when they could all benefit much more if the goods were provided at higher levels (Kim & Walker). So, far from the public spaces at which sharing takes place maximising public goods, behaviour in this space may work against promoting significant quality or quantity of contributions.

In my work on the openED project, such perspectives may be quite enlightening. Why, for example, has there been almost no peer review of submitted assignments? Why have few beyond the project team offered additional support for free? Do participants in different countries have different attitudes towards the act of giving?

In conclusion, it seems to me that developers of open education resources repositories or sites for contributing learning and teaching ideas would find it useful to construct multiple models of altruism – for example ones that could sit alongside and complement a existing ‘model of sharing’. Whilst this may require a tempering of the more utopian hope that simply opening something up will be enough, or indeed the best solution, if theories of giving can help frame the strategy of fundraisers and charities, then they are certainly worth our attention also.

Bibliography for this post available on request. This post was inspired by a seminar presented by Rob Farrow last week about OER and its philosophical foundations.

The theme of the conference was learning and sustainability and it was interesting to see the present state of thinking around how e-learning was seeking to frame and understand what ‘sustainable learning’ means. The two uses of the concept most prevalent were: 1. The planning to sustain project accomplishments or innovation, and 2. The teaching of themes associated with sustainability within a course. The first is clearly important to the longevity of a project but I find this a rather limited and lacking use of the term; it priorities the project interests instead of asking how, or even if, the project can truly contribute to a broader socially, culturally, environmentally and economically sustainable learning. So wouldn’t it be great if in the sustainability section of project reports, authors were asked to review the projects social, environmental and economic impact too? The second use – teaching ‘about’ sustainability – was also evidence at Eden and this figures education as a having a role in setting foundations to more sustainable development. Of course, it is important to build an understanding of what works in doing this, and, although the examples of this teaching most often come from courses about sustainable development or related disciplines, there seem to be many excellent examples and pockets of innovation across all disciplines.

Yet there are also at least a third and fourth way to approach the concept of sustainability. The third is that of an organisations role in achieving sustainable development – a practice of sustainability review becoming common for all businesses yet, perhaps surprisingly, universities seem to lag behind other sectors. In a quick review of recent papers in the International Journal of Sustainability in Higher Education, I found reference to several frameworks of indicators for institutional assessment with GRI (Global Reporting Initiative) guidelines mentioned quite often. However, use of these appears fragmented with Fonseca (2011) found just 30% of Canadian universities had done this, whilst Lozano (2011) seems to have been able to identify just 12 which used GRI. Furthermore, both these authors found institutional reporting focused on eco-performance and economic indicators with proportionally (or actually) far less consideration of sustainability indicators relating to curricula, teaching and research and society (and it would also seem that even these indicators are limited in what they cover). I think this should be a concern yet perhaps this is partly a symptom of the level of reporting being too broad and abstract. There can be great variation in teaching and learning within a university so why not localise the issue of sustainability and build it in to the course and curriculum design process. It seems a missed opportunity that we don’t do Course Sustainability Reviews which asks course authors should think about how their course contributes to sustainability (in addition to issues around sustaining the course as a viable business service or teaching sustainability issues within it). As a starting point for such CSRs one may find useful indicators in the frameworks for institutional sustainability auditing and in the essential characteristics of education for sustainable development as defined by UNESCO in 2005 (limitations etc of these aside):

UNESCO stated that education for sustainable development:
• is based on the principles and values that underlie SD;
• deals with the well being of all three realms of sustainability – environment, society and economy;
• promotes life-long learning;
• is locally relevant and culturally appropriate;
• is based on local needs, perceptions and conditions, but acknowledges that fulfilling local needs often has international effects and consequences;
• engages formal, non-formal and informal education;
• accommodates the evolving nature of the concept of sustainability;
• addresses content, taking into account context, global issues and local priorities;
• builds civil capacity for community-based decision-making, social tolerance, environmental stewardship, adaptable workforce and quality of life;
• is interdisciplinary. No one discipline can claim ESD for its own, but all disciplines can contribute to ESD; and
• uses a variety of pedagogical techniques that promote participatory learning and higher-order thinking skills.

Lists such as this, offer some ideas on how open and ‘free’ courses, such as the openEd course mentioned earlier, could be evaluated against sustainable development. On the one hand there are clear benefits such as promoting life-long learning, engaging in (and indeed combining in the same place) formal, non-formal and informal education, promoting participatory learning and building capacity for social tolerance. For example, the sharing of business management experience was the most frequently mentioned benefits to group working described by those participants who responded to our Learner Reflection Forms. For example one participant commented ‘[I like] work[ing] in groups because it’s a great way to share and improve ideas, and we still have time to share some cultural differences’. Yet on the other hand other stipulations may present a greater to challenge to open learning. For example, that: courses account for context and local priorities; that they are locally relevant and culturally appropriate; and based on local needs may present challenges for open learning instruction and courses. The provision of local services, as encouraged on the openED website, may help here.

And, finally, I find a fourth angle from which to approach the concept of sustainability? Let us forget for a moment about education’s role in facilitating the accomplishment of sustainable development by more informed individuals (i.e. by promoting better understanding and skills around it) and instead constitute education as a resource of knowledge, practice and artefacts in its own right. Now, not only is it a means to an end, but is itself an object of potential alteration, change or damage. This would include the social sustainability of education professions such as academics, teaching and learning staff etc. – and a good example of this is the interesting paper by Hammond & Churchman (2008) which, using Barron and Gauntlett’s five principles of social sustainability, identifies worrying counter-sustainable trends in Australian academia and highlights a real danger of ‘the extinction of the academic personality’ and its subsequent ability to itself affect through education a transformation towards sustainable development.

In the Brundtland Commission’s 1987 definition of sustainable development – that it is ‘development that meets the needs of the present without compromising the ability of future generations to meet their own needs‘ – we can also see an essential balance between the optimism for making positive change and the caution of being mindful of the impacts this change may have. Educationalists seem to tend to focus on the former and perhaps there needs to be greater sensitive and reverence to this undercurrent of caution also – of being aware of our potential to damage or alter something valuable in the act of creating something new.

Turning back to the theme of open or free courses, it is clear these do to represent a change in educational structures and have, potentially, an impact on their sustainability. For example, there may be changes to the control structures used in the traditional course and the pedagogies that authors will find most effective. Participants may, at least on some level, now: not pay or make some other financial exchange for the course; may be asked to have a very different relationship from the text (and facilitation) than that of the traditional teacher-student; may not be driven by the demands of formal assessment; may approach learning with more informal notions of the social structures of reciprocity and community they anticipate.

For example, it is widely acknowledged that around 10% at most contribute in sustained fashions to forums or similar social media yet forums are increasingly used as an essential activity in learning teaching designs (with the problem of low engagement solved by using assessment to compel student to participate). In an open learning situation such control structures may exert less, or no, influence and so the pedagogies which accompany such structures may be less effective. Indeed, we see this in the participation rates for the early openEd modules: preliminary participant survey results indicate that less than half who ‘studied’ a module posted on the forum; that only half who were active on the forum and subscribed to a study group completed the group activities; and that almost no one made the final step of undertaking peer assessment for another participant. So, whilst in a formal course each of these activities could be enforced, in the open and free openEd course each activity was voluntary with no formal structure to compel participation (although obviously social or cultural factors may have applied etc).

So in conclusion, does this mean that in transferring from a controlled formal context to an open context the very pedagogies themselves (those used to derive activities and influence construction of learning resources) may no longer be appropriate or effective? Just as we ask which pedagogies travel well from face-to-face to online learning, how should we also ask which travel well from online formal learning to online open learning? How can courses better understand their responsibility to sustainability by considering both positive and potentially negative impacts both in, of and through themselves?

Aczel, J., Cross, S.,  Meiszner, A., Hardy, P., McAndrew, P. & Clow, D. (2011) Some Issues Affecting the Sustainability of Open Learning Courses, EDEN conference, Dublin, June 2011.

Fonseca, A., Macdonald, A., Dandy, E., & Valentri, P. (2011) The state of sustainability reporting at Canadian universities. International Journal of Sustainability in Higher Education, 12, 1, pp22-40

Hammond, C. And Chruchman, D. (2008) Sustaining academic life: a case for applying principles of social sustainability to the academic profession, International Journal of Sustainability in Higher Education, 9, 3, pp235-245

Lozano, R., (2011) The state of sustinablity reporting in universities, International Journal of Sustainability in Higher Education, 12, 1, pp67-78

UNESCO (2005) United Nations Decade of Education for Sustainable Development (2005-2014): International Implementation Scheme

Please do remember to reference online articles and blog posts if using within your own work

The Stage-Gate concept was developed by Robert Cooper as an innovation management method for businesses. It is now a widely used mechanism to manage the product development process – in the case of a University this may be the development and production of a course. The object of the model is to support the making of a key decision at each stage – the question: should development continue?– or as Cooper puts it; to ‘Go’ with or ‘Kill’ it. For Cooper, the period between Gates consists of ‘information gathering activities’ followed by an analysis of the result which is then entered in to the Gate for decision.

Over the last five years some work has begun to assess and revise the Stage-Gate model. Indeed, Cooper (2008) himself acknowledges the need for the‘Next Generation Stage-Gate’. Reading through this work, I was encouraged to find comments and observations that aligned with themes discussed in my last blog post. Cooper (2008) offers a range of ways to revise or add to his model. I’ve picked out seven below:

1. A Flexible Process: we observed (see last post) that learning design activities should be planned in advance of the stage commencing and here we find Cooper too  recommending that project teams propose a ‘go-forward’ plan. Furthermore, for Cooper there is a suggestion that the project team ‘has much discretion over which activities it executes and which it chooses not to do’ which is a key aspect to our notion of ‘sufficiency’ and the underlying principle of the toolbox approach to learning design.
2. Simultaneous Stage Execution: This is the key message behind the diagrams in my last post and something Cooper acknowledges when he suggests activities or complete stages may overlap.
3. A Spiralling Process: This idea of cyclical rather than linear activity, is clearly shown in the second diagram in my last post and for Cooper the notion of spiral or agile development should be fused to the new Stage-Gate model. One aspect of Cooper’s idea about this is for a ‘full proposition concept test’ to take place before State-gate 2 (what some may consider a business appraisal stage-Gate) where ‘the project team present a representation of the proposed product.’ He clearly has in mind something visual which, so far as teaching and learning goes, suggests visual course representations have value. It is also clear that in moving proto-typing and development further ‘up’ the Stage-Gate process there is an intention for production teams to be involved earlier on.  
4. Setting Next Gate Success Criteria: Cooper cites successful examples of setting the success criteria for the next stage at the preceding Gate.
5. Self-Reflection by Project Team: although Cooper fails to really develop this idea, (he suggests that a project team could also score themselves before a Stage-Gate meeting and in small projects manage this themselves) he is identifying the value of self-reflection in the design process. This aligns well with the notion of ‘sufficiency’ / ‘sufficient design work’ we mention: that it is at least in part a responsiblity of a project team to know when they have done enough and to be involved in deciding what ‘enough’ means.
6. Virtual scoring of Gates: here Cooper is beginning to explore the role of distributed access to Stage-Gate information. I believe this can be taken further, for example, by using virtual portfolios of design work that can be accessed remotely by all stakeholders (as is being implemented by Birmingham City University and shown in our representations as folders).
7. Accountability: I mentioned accountability in my post, and Cooper too suggests a new Stage-Gate builds a ‘tough post-launch review to instil accountability for the results.’ In general, I would imagine this is an under-achieving part of many product developments so good to see comparison of results against success criteria and continuous improvement being highlighted.

Overall, we have the impression of what Ettlie and Elsenbach (2007) call a ‘modified’ Stage-Gate process. Their work (which compares responses from managers involved in either an ‘original’ or ‘modified’ industrial design Stage-Gate process) found those with modified processes were more likely to be: using more formalised strategies to guide development processes; have more structured processes; adopt collaborative development software support tools and use virtual teams. For learning design projects seeking to achieve more structured guidance and processes, use of support tools and online team working this may be of interest. 

Openly innovating

I’ve also found reference to ‘open innovation’ alongside this ‘Next-Generation’ modified Stage-Gate. The term was coined Henry Chesbrough and a paper by Grönlund et al (2010) gives an example of how the two ideas are being fused. Their definition of open innovation is ‘…that firms should use external as well as internal ideas, and both internal and external paths to market, when they seek to maximise returns from new product development.’

Or put another way to ‘connect and develop’ rather than ‘research and develop’. This of course is a fundamental practice in academic research, however, product innovation activities are perhaps a little different. Grönlund et al. see this as a solution to reinventing-the-wheel and talk about the importance of making ad-hoc practices more formal. This could be achieved by: the seeking and evaluating of opportunities – especially for ‘import’ or ‘export’ of practices; the adding of ‘open innovation evaluation criteria’ to the Stage-gate process; and the continuous evaluation and change of business models. From an educational perspective this underscores the OULDI-JISC project interest in sharing practice (so others can import) and so many other pursuits such as the reusing of learning material.

So what questions could be asked to help clarify the nature of the innovations being managed? Here are a couple of suggestions:

What is the innovation to be managed?

For universities, the ‘product innovation’ is often seen to be creating a new course. And inevitably, and rightly, the initial focus is on the market potential of a topic/content (either new for the university or sector). However, even once the decision has been made to proceed on financial/market grounds, this is no longer the end to the ‘innovation’. Today, the greater choice in pedagogy, technology, production options, and presentation practice means that there are a range of other areas to innovate – many of which may or may not be known or explored until after the market/financial decision has been made.

Examples of such secondary innovations would be the development of a new assessment structure or online collaborative working environment – innovations that ‘emerge’ as the production team better understand the design problem space they face (see earlier posts). The question is, are these to still be considered ‘part’ of the original course product innovation or separate to it? If a course could be developed without, say, a new innovative assessment structure then strictly speaking there are two independent innovations going on here: the course product and the assessment strategy. Of course, if the assessment strategy is essential to (a dependent part of) the course product it must remain part of it.

There is, therefore, at least a potential for a multiplicity or nesting of innovative activity and a broadening of what should be considered and managed as a learning and teaching ‘innovation product’. At what point does a ‘product innovation’ that is not a course become important enough in its own right to warrant a separate Stage-Gate process? Is some sort of Dual-Path process required? How should innovations emerging after Stage-Gate 2 or 3 by sufficiently recognised in the innovation management process?

This may require a clarification of the nature of the innovations being proposed at both the second and third Stage-Gate. For example, criteria could ask how much innovation is expected in: (a) market position of course / course topic or content outcomes (b) pedagogy (c) technology being used (d) teaching provision (e) assessment strategy (f) resource use. Such factors could be reviewed at Stage-Gate 3 and, if they change, action taken as necessary.

 For who is this an innovation?

By this I mean; on whose terms is the product/method of achieving the product/method of delivering the  product considered to be ‘innovative’? Is the idea innovative for the sector, the university, the faculty, the course, or the individual? Being clear about this will surely fundamentally affect the terms on which innovation is managed. So, for example, if innovation is being exported from elsewhere then the Stage-Gate decision-makers should be told from where and about the evidence that this has/will be successful. Or, in another case, if the innovation is believed to be entirely new, then information relating to how well the sector has been scoped (to support the ‘claim’ that it’s new) should be provided. A question about the ‘scale’ of the innovation could be asked at a Stage-Gate.

A second question on this theme relates to the student perspective: ‘is this innovative for the student’? This would require consideration of what the student expects and has experienced to date (indeed, ensuring that this is not only innovate but at least on par).

What is the cost of innovation?

Again, the cost of the innovation is likely to be approached differently depending on what type of ‘innovation’ is being considered: if the innovation is truly ‘sector-wide’ then the cost will need working out specifically (as there are no, or only partial, prior cases to go on). However, if it is a ‘university-wide’ innovation then there must be some cost data from elsewhere although this may be difficult to get (especially when case studies of innovation seldom adequately report the cost of it). Moving down the levels again; if the innovation is just ‘new’ for a particular course then there are likely to be examples within the organisation to draw on (although each case may be different).

So in summary, there is a growing awareness of a need to revise the Stage-Gate process and explore how ‘open innovation’ can add a second ‘type’ of innovation (that of ‘importing innovation’) to a more traditional approach associated with internal research and development. Coupled with this, the explosion of teaching and learning choices available to those devising courses today means innovation may occur on several fronts simultaneously. And indeed, with fewer constants, the management and costing of this innovation may become more complex. This may necessitate asking for more information about different types of innovation at different times and accepting some innovations (such as in pedagogy or technology use) may be a response to, as much as a component of, the original course product innovation specification.

Becker B. (2006) ‘Re-thinking the Stage-Gate® Process – A reply to the Critics, Management Roundtable Inc. Link (http://www.pd-advantage.com/images/RethinkingtheStage-Gate_Process_AReplytotheCritics.pdf)

Cooper, R. (2008) ‘Perspective: the Stage-Gate® Idea-To-Launch Process – Update, What’s New, and NexGen Systems, Journal of Product Innovation Management, 25: pp213-232.

Ettlie, J. & Elsenbach, J. (2007) Modified Stage-Gate Regimes in New Product Development, Journal of Production Innovation Management, 24, 1, pp20-33.

Grönlund, J., Sjödin, D., Frishammar, J. (2010) Open Innovation and the Stage-gate Process: a revised model for new product development, California Management Review, 52, 3, pp106-131.

The above comments were written as I prepare to co-present a session on Innovation in Learning and Assessment Design at the university’s annual Module Chair Training event later this month.

Here are two different draft representations to help communicate our interpretation of recent work. I think it important that such models seek to weave together the less formal practices and processes (such as design activities) with more formal and necessary demands:

If you click on the images an enlarged version should load.

By way of explanation, these representations were the outcome of our thinking about a range of questions. Some of these were:

• Does a business case, or any related business approval stage, need any pedagogic information or design? – Possibly not. Could learning design be more damaging than enabling at this stage? – Very possibly (after all the design process has not yet properly started and decisions made here may constrain and limit design options
• Where in the process is the learning design problem space considered? And how much planning of learning design activities (both formal in the design team, and semi-formal individual work)? The business case may set some constraints but little in respect to important pedagogical aspects associated with the design problem. Recent work in the wider design literature has highlighted the importance of fully understanding the ‘design problem’ before developing a design solution. A number of advantages are cited from promoting a more creative, innovative approach to the problem; to better understanding of the core purpose of the design process and more effective quality assurance and design validation.
• How important is it to recognise that design stages are not discrete units that ‘end’ once a particular form or document has been submitted. It is not a linear process for each overlaps and may continue to be worked later in the process with more porous links to it’s neighbours (shown by the dotted lines above). Furthermore, it would now appear course development involves a greater range of people throughout the entire process (rather than each just involved in a discrete portion of the process).
• Is too great an emphasis placed on the submission of a form or specification before work moves on? Instead why not focus on the process of creating these forms and specifications and use the cumulative amount of ‘work’ put in to preparing these documents as indicators of ‘sufficient’ design? Here the notion of doing a ‘sufficient’ amount of design work replaces that of producing a document/forum ‘sufficiently’. Yet how could we record and store documents and design activity ‘evidence’? The most obvious is some form of portfolio/s (e.g. folders on a local or shared area). The diagrams indicate some of these portfolios and each module would need to decide, upfront, what sufficiency may mean for them.
• Is there enough formal review of the meso- and micro- levels of design? Module design currently takes place at several inter-connected levels, however often formal institutional process fade as design work in earnest begins.
• Is there sufficient recognition of the micro-design taking place in the production of course materials and services even once courses have been written or operational documents prepared? Maybe not, and a consequence does this mean that micro-design decisions may not be properly supported or articulated?
• Module/course evaluation (after delivery of the course) is often well formalised, however, can this be enhanced by better definition of the design problem space and planning of activity for the design process at the beginning? There is certainly potential for further quality assurance measures here and this may enable better tracking of decisions and therefore of accountability.

Cloudworks aims to support the sharing of teaching and learning and my original aim was to get an indication of how many were using the site to really ‘share’ and discuss ideas; that is to say, the proportion who posted on the site (rather than just using it as a resource to read). So how did the 2008-09 cohort do in 2010?

• In 2008-9, my original review found 8% (20 people) had posted two or more contributions over a period greater than 28 days from their first post (what I termed a ‘prolonged’ contribution). In 2010,  just over half of this group (13 our of 20 people) continued to make a contribution to the site. The remaining 7 did not post anything in 2010. However, this decrease is almost balanced by the fact that in 2010, 5 new people (from the original sample) made posts in a period of more than 28 days. This may be an indication of an emergence of a core group (5-7%) of sustained users. Scaling this 10% sample up to the entire Cloudworks usership this suggests there may be some 150 to 250 users making prolonged contributions.
• In 2008-9, some 39% of registered users demonstrated some ‘activity’ on the site (such as posting a comment or link). In 2010, however,  just 14% of the sample had made a contribution to the site.

Taken together, these data seem to point to a core group of contributors posting to the site.

So what are they contributing to? I took the original 8% of core contributors (n=20) and recorded details of every post they made since 2008; including the topic, whether they had set-up the cloud,and the number of contributions made to the cloud. In total these 20 users contributed to 103 clouds. Of these clouds:

• 40% were associated with workshops or events that had been held in a face-to-face context
• 20% were posed questions – the Cloud asking the community for comment or feedback
• 10% were about or refered to ’real things’ such as reports, tools, objects or publications
• 7.5% were about virtual media (online newsletters, re-tweets etc)
• 7.5% focused on presentations that had been made (e.g. at a conference)

Although only a few Clouds specifically focused on sharing designs or talking about the theory of learning design explicitly, in some way all could be considered to be ‘talking around’ the subject of designing learning. Of interest to me is the apparently high proportion of Clouds formed as supports or ‘on-the-back-of’ real face-to-face workshops, events and meetings.

This is evidence that, mixed-in with the process of creating a new virtual community, the site is recreating, consolidating and reinforcing existing real-world relationships taking place in real spaces. This is intriguing and perhaps unsurprising but it does suggest we need to pay attention to how people move between spaces and how SOVC (sense of virtual community) and SOC (sense of community) overlap. Blanchard, Welbourne & Boughton (2011) talk of the connection between trust, norms and sense of virtual community and suggest that as members adhere more closely to the community norms their bond to it increases and the risk in participating increases. Galley et al. (forthcoming) also review a range of indicators of community. From a quite different field we find Raymond (2010) attempting to bring together different dimensions of attachment to place. If one were to see Cloudworks as a virtual place there are perhaps some parallels with this work – not just in the indicators of community, but also in personal contexts (place identity and rootedness) and online environmental contexts (affinity to the virtual world, online identity etc).

I also looked at these 103 clouds from a second perspective: the average number of contributions made to Clouds started by one of the core (20) users and by the project team. In the case below we can see that a Cloud started by a ‘core’ user attracted fewer comments from other (non-project team) users and from the project team than Clouds started by the project team. Of course, the sample is relatively small yet I use this merely to suggest that the experience of using Cloudworks to elicit contributions from others may vary between groups.

Moving on, I also wondered how new people joining the site in 2010 got on. Are they making a greater or fewer number of contributions? To do this, I made a second sample, this time of 93 new users, who had registered between March and June 2010. I’ll call this the 2010 Group. Here’s how they compare to the 2008-09 group:

In would appear that rates of user contribution in the first year are similar for both groups (the original 2008-09 sample and the new 2010 sample). If this trend were to hold, then it may offer a good guide as to the attrition rate for new members of the site.

The analysis above has tended to focus on contributions and it is clear from the strong viewing figures for the site that many registered users, and non-registered users, are reading/consuming the information on the site without ever contributing. Certainly reading alone is important for building norms, encouraging new contributions, helping individuals assess the risk of contributing and forming a perception of trust across the group. However, the question remains, how do we interpret their silence?

• Do those who simply read the content on Cloudworks agree with and acquires to it?
• Can we take their absence of discussion or challenge to indicate agreement and consensus? – That is to say, does what is said on Cloudworks represent the consensus of those who subscribe to the site – the consensus of the community?
• If not, and we take seriously the intent  to create/represent a community, on whose behalf does the content speak?
• Who is exerting the greatest voice?
• Does it matter that there is an apparent inequality in contributions?

Such questions become increasingly important for Cloudworks as it continues to grow both in terms of the number and range of contributions.

Blanchard, A., Welbourne, J. and Boughton, M. (2011). A Model of online trust, Information, Communication & Society, 14, 1, pp76-106.

Galley, R., Conole, G. and Alevizou, P. (forthcoming) Community Indicators: A framework for building and evaluating community on Cloudworks, Interactive Learning Environments.

Raymond, C., Brown, G. and Weber, D. (2010). The measurement of place attachment: Personal, community, and environmental connections, Journal of Environmental Psychology, 30, 4, pp422-434.

‘Design can be viewed as a mutual learning process among designers (Beguin, 2003) and can be described as a reflective conversation between designers and the designs they create… Traditional architectural studies teaching is based on the notion that successful design solutions and learning are a direct outcome of the extent to which the design problem space is explored’ (p434, my emphasis) 

Wang W., Shih, S. & Chien, S, (2010). A ‘Knowledge Trading Game’ for collaborative design learning in an architectural design studio, International Journal of Technology Design Education, 20, pp433-451.

In some recent research that Denise Whitelock and I have been doing within the OU we have been examining the concept and practice of authentic assessment – and in particular, how to make visible those assessment approaches associated with concept but not understood as such. It’s been great to have the opportunity to explore the term a little and in this post I hope to outline a few of my initial impressions.

The current use of the term emerges from a discourse around ‘authentic’ and ‘genuine’ testing that had become established by the mid-1980s and which combined with the broader social constructivist project to become part of what Serafini considers to be the most recent of three assessment paradigms: ‘assessment for enquiry’.

The notion of ‘authentic’ certainly appealed to those interested in getting closer to ‘real’, ‘meaningful’ learning and represented an idea with an innate capacity to help problematise traditional assessment practice. Early definitions include Wiggins who defines it as ‘[the extent to which] student experience questions and tasks under constraints as they typically and ‘naturally’ occur, with access to the tools that are usually available for solving such problems’, Newmann et al. (1996), and Torrance who suggests ‘[it is the] assessment tasks designed for students should be more practical, realistic and challenging than what one might call ‘traditional’’ and that it is ‘a generic term… to describe a range of new approaches to assessment.’

By 2000, overlapping interpretations of what authentic assessment meant and authentic assessment tasks comprised of were emerging (e.g. McTighe and Wiggins (1999) and the review by Cummings et al.). There were also efforts to instantiate these in to guidance or advice on designing authentic assessment tasks (Darling-Hammond & Snyder; Williams; Hughes) or integrating the idea of authenticity in to principles for instruction (for example, Merrill).

The upshot of this is a range of emphasis and interpretations about what ‘authentic’ means (sometimes in respect to particular disciplines) and what constitutes an ‘authentic’ assessment task. In the itemised paragraphs below I attempt to identify some of the components of the authentic assessment discourse. However, before I do it’s important to acknowledge that this is to a degree a contested term and one that perhaps has as much utility as a driver for debate and reflection, as it does for directing practice. We must bear in mind for example: the argument that assessment itself, in any form, may be considered inauthentic (Spolsky; Spence-Brown); that there may be concerns over reliability or the relationship between the criteria used to judge assessed performance and ‘real-world’ judgements; the view that tasks are only authentic is students themselves treat them and behave as such; that students and teachers perceptions of what authentic means may differ (Gulikers, et al.); that this may not necessarily lead to reductions in cost; and that, as is often the case, the empirical base to support the ideas is developing more slowly (Kleinfled).

So how has ‘authentic’ been understood?

• Authentic intellectual work: a recurring theme seems to be the need to assess how students construct and use knowledge, skills and judgement, not just their ability to memorise and repeat. This is about ‘students ability to efficiently and effectively use a repertoire of knowledge and skills to negotiate a complex task’
• Authentic situations: this draws on theories of situated learning that insist the physical context is important underlie much writing on authentic assessment. The notion of a ‘real-world’ is often evoked here and it ‘requires [a] fidelity of context to reflect the conditions under which the performance will occur – rather than contrived, artificial or de-contextualised conditions’ (Harrington & Herrington). Brothwick et al. (2007) describe three types of authentic learning associated with situated learning: the apprenticeship model where students become part of a workplace and are mentored by a relevant professional thereby providing an authentic work experience in the ‘real world’; the simulated reality model where the ‘real world’ moves in the classroom and learning activities aim to simulate aspects of authentic work conditions; and the enminding model where authenticity comes from the connection between a student’s experiences and the disciplinary ‘mind’. Here the authentic activity is the student ‘engagement with the discipline through a connection with themselves, which may encourage a desire to continue learning beyond the classroom’. In developing the ‘fidelity’ of the situated space a number of considerations may be required; for example the authentic social context.
• Authentic competence or performance: this is about expecting students to ‘demonstrate methods and procedures used by experts in the field’ and is drawn from vocational education and training where there is a need for satisfactory performance of skills or activities. Unlike situating assessment, this is more about the tasks themselves and, indeed, the terms ‘authentic’ and ‘performance-based’ assessment have been used interchangeably. Performance may not just mean ‘performing’ a task, but also using modes of ‘performance’ for assessment (such as presentations, portfolios, video diaries, and other multimedia) and the potential issues associated with students ‘performing’ to the assessor
• Authentic Learning Experience: this focus on the experience of learning seems to be a common theme, be it concerned with integrating learning and assessment or providing for ‘worthwhile, ‘meaningful’ and ‘significant’ experience to the student. Learning experience is considered by Cunningham et al. as the second of two major considerations resulting from the notion of authentic assessment. Underlying much of this is the assumption that students will become more motivated, interested and inspired if assessment seems to them to be more authentic (although whether this is achieved by teachers is another matter). This overlaps with a view of authenticity as an emergent process that ‘lies in the learner-perceived relations between the practices they are carrying out and the use value of these practices’. Other aspects of an authentic learning experience may be considered to be: Authentic problems: drawing squarely on the problem-solving approach or enquiry based learning this view focuses on the authenticity of the problem (Segers et al.) and promoting higher-order thinking and complexity; Authentic materials and resources: often teachers and learning designers will talk of how materials and resources (including computer resources) have been constructed from material derived from an authentic place or time – such as raw data from scientific study (e.g. Meyers); and Authentic collaboration: another proposed characteristic of authentic assessment is associated with interaction; that it ‘requires significant student time and effort in collaboration with others’ (Herrington and Herrington).
• Authentic measurement and criteria
• Authentic audiences: this focuses not on the student or teacher but the role and position of the assessor (be this teacher, peers or a computer) and the relationship between them.
• Authentic personal trajectories: this marks the association between ‘authenticity’ and the promotion of life-long learning, competency, professional and even cultural practice. This appears to have been developed further within the concept of ‘sustainable assessment’ and the importance of offering learners multiple pathways – what I could perhaps call providing authentic options.
• Authentic design structures: Linked to the notion of authentic ‘experience of activities’, this looks wider to the broader issue of authenticity of course or curriculum design. Frick et al. argue for an authentic structuring of the tasks through a unit of learning or course from the simple to complex. Such work I think may be considered to refer to an authentic learning journey, rather than focus on authenticity of a specific tasks, practice or place (as above). This is perhaps more about authentic pedagogy and learning and aligns with Leung and Lewkowicz’s view that since schooling takes up a meaningful portion of ones life, school assessment tasks are authentic in their own right.
• Authentic reporting of teaching and learning: although perhaps more tangentially related to assessment, I’ve included this point anyway! This is about particular descriptions of learning and assessment that are given by teachers which, whilst perhaps not rigorous or research-led, are seen to give an ‘authentic’ account of the experience and process from a teaching perspective.

So there we go… a few initial observations which have helped inform our writing of a staff survey about assessment approaches. This questionnaire survey has been open during September and Denise and I looking forward to beginning to analyse the responses. As in earlier posts, I’ve given a few references; the rest are available on request.

Falchikov, N. (2005). Improving Assessment Through Student Involvement, RoutledgeFalmer, USA.

Grant, S.G., Gradwell, J.M., Cimbricz, S.K. (2004) A question of authenticity: the document-based question as an assessment of students’ knowledge of history, Journal of Curriculum and Supervision, 19, 4, 309-337

Meyers, N. (2009) How to use 5 curriculum design principles to align authentic learning environments, assessment, students’ approaches to thinking and learning outcomes, Assessment and Evaluation in Higher Education, 34 (5), 565-577

Newmann, F., Marks, H. and Gamoran, A. (1996) Authentic Pedagogy and Student Performance, American Journal of Education, 104, 286

Spence-Brown, R. (2001) The eye of the beholder: authenticity in an embedded assessment task, Language Testing, 18 (4), 463-481

As part of this work, I am presenting on our behalf a paper at the Computer Assisted Assessment Conference in Southampton this week. In the abstract we make the point that benchmarking offers a comprehensive way of measuring current practice in an institution per se; whilst also gauging achievement against external competitors. It would appear that although e-learning has been benchmarked with a number of universities in the UK and abroad no one to date has tackled the area of assessment; which is now becoming of more concern with the advent of e-assessment. 

Our paper describes the construction of a set of benchmarking measures/indicators and the outcome of early pilots which combine a survey instrument and semi-structured interview methodologies. The findings suggest that building a comprehensive and robust core of benchmark measures would have great utility and value to institutions; not just in external benchmarking but in internal benchmarks and reviews, setting baselines, exploring the student experience, providing staff with data meaningful to their role and professional development and supporting continuous improvement.

The paper is accompanied by our current working draft of the benchamarking measures. [Since writing this post the measures have been revised further. A new version of the measures is available at: Assessment Benchmarking Criteria v17_A]. We will be very interested to hear your comments or feedback on the paper and the draft.