Assessing the potential for adaptation and personalisation of teaching and learning

Over the last few months I’ve been working with Simone Davies at the OU’s Learning and Teaching Solutions unit on (a) an online guide to help course authors use the study planner in Moodle to build adaptive elements to their learning content and designed activities (b) a review of current use of the study planner in selected courses (c) developing a specification for a new Moodle Study Planner ‘Decision’ plug-in (d) a review of the nature, benefits and barriers associated with adaptive learning and teaching (adaptive learning systems) based on staff interviews and a literature search. I touch upon (d) and a little of (c) in this post.

In 2008, Kelly and Tangney’s discussion of adaptive learning systems notes that they ‘can bring different benefits such as increased learning performance, greater enjoyment, enhanced motivation and reduced learning time’. In outlining this quartet, they hint at just how useful adaptation could be in an online distance teaching context. However, to date, the realisation of properly adaptive teaching in distance learning and the planning for adaptation in their design remains arguably quite limited.

If there is indeed the potential to make improvements to workload, to better engage and empower students in their own learning, to increase retention and to help students achieve more, then it is imperative that online and blended learning understand when and how adaptation can become an effective teaching tool could be used.

Of course, designing adaptation requires a departure from the linear, single path a learner is taken through a course, be this across a lecture series, a distance learning guide or a series of online learning tasks. Reigleuth makes the point that ‘one size does not fit all’ and indeed, in 1999, the materials for the OU’s own course on Teaching in Higher Education stressed how students learn differently and proposed as good teaching practice that the teacher understand and respect their student needs, approaches to learning, stage of learning development, interests, motivations, backgrounds, prior knowledge, etc.

Introducing learners to open distance learning: a new 1-2 hour study module

The openED online open distance course titled ‘Business and Management Competencies in a Web2.0 world’ commences its third presentation next week. As before, I’ll be involved in the research and evaluation of the course and the use by distance open learners of the website.

For this round, I’m also written a new additional module titled An Introduction to how to learn with open distance courses such as openED.  The aim of this module is to help the learner understand how to study better using open online material and should last 1-2 hours. This module is based on the most common questions asked by openED participants which I have identified from a detailed review of the Round 1 and Round 2 forums and over fifteen hours of text-chat session transcripts. The module uses examples from the openED course to help illustrate ‘real-world’ examples and includes three short reflective exercises (5-10 minutes) aimed at thinking how using online social spaces such as forums can enrich learning experiences. The module includes short sections on:

• What is open distance learning?
• When to start?
• How to study an open module
• Using forums to meet others, share work and resolve questions
• Live text-chat sessions
• Group working
• Getting feedback and recognition for your work

To see the module in situ check out the openED website. I have also made the original text document (3,600 words) available under a limited creative commons licence for reuse and repurposing. You can download it using this link: Introduction To How to Learn With Open Distance Courses_Cross_2011_v6  

One exciting feature of this module is my sharing with the learners a basic visual representation of the recommended learning route (learning design). I have of course had to abstract the design to suit any of the 10 mondules (each differs in content) so it does not convey the detail or nuance that I would often put in such designs. 

 

The intention is that participants work through the Introduction module first. This will better prepare them for the course modules and we will monitor how this contributes to greater or more prolonged engagement.

Altruism and OER /OED: sharing, charity and the process of giving

Read any article about open educational resources or designs and the term ‘sharing’ pops up time and again. Whilst an apparently innocuous, self-explanatory term, for something so closely bound to notions of openness of why we think people want and are open and of how we imagine the mechanism of interaction and co-creation of common resources, it is often evoked with scarily little critical reflection. I propose here that it can be a misnomer and, as such, is operating as much to obstruct our analysis and critical understanding of the mechanisms for enacting openness.

The idea of sharing refers to what could be termed ‘reciprocal altruism’ – a form of altruism where there is anticipated to be exchange of things, often of those judged of broadly similar value. This notion of reciprocal altruism, often with an emphasis on the interaction that takes place not reasons for it, forms a cornerstone of how many people imagine how open education systems will work and should be configured; designers exchange designs, students review each others work, teachers share their resources. Ask why they think such interaction will take place and either the fall-back position of reciprocal altruism is assumed accepts begrudgingly accepts conscious self-interest.

Yet psychological, sociological and economic research tells us that there is more to altruism than reciprocity or sharing. Altruism for example can also take the form of kin selection, or nepotism, with individuals supporting others seen as similar to themselves thereby binding communities closer together. It can also take the form of charity – the act of giving to a stranger; the understanding of this is fundamental to entire sectors!

It seems to me that focusing on the act of ‘giving’ – looking at individual and structural motivations, intents, behaviours and social norms – can enrich our models of OER use. What if we look to interpret contributions to OER repositories or other open learning and design repositories as acts of altruism such as charity or kin-selection? Can we use such perspectives to build a complete picture of what openness is or could be like?

So why do people ‘give’? There are many overlapping theories. One perspective holds that people give to feel good or better about themselves (even to remove a sense of guilt or obligation). This ‘warm glow’ comes from

Pedagogies that travel and those that don’t: Sustainability in open courses and learning design

Last week I presented a paper titled ‘Some Issues Affecting the Sustainability of Open Learning Courses’ written by James Aczel, myself and four other colleagues (2011) working on the EU funded openED project at the EDEN conference in Dublin. The aim of the project is to explore participatory learning in open educational environments by producing and delivering an online open/’free’ course is business management. In so doing, the objective is to not only to produce an OER, but to use open technology, open Web 2.0 methods of communication, open opportunity to facilitate, open peer assessment, and open opportunity to build a resource for reuse. So a key question for the project is to study the effect and issues around ‘opening’ up many aspects of a course together. And one of my roles in the project is to develop and deploy a complex of evaluation methodologies before, during and after each of our three rounds.

The theme of the conference was learning and sustainability and it was interesting to see the present state of thinking around how e-learning was seeking to frame and understand what ‘sustainable learning’ means. The two uses of the concept most prevalent were: 1. The planning to sustain project accomplishments or innovation, and 2. The teaching of themes associated with sustainability within a course. The first is clearly important to the longevity of a project but I find this

Open Innovation and the ‘Next Generation Stage-Gate’ process: Reflections on Learning Design processes 2

Next-Generation Stage-Gating

The Stage-Gate concept was developed by Robert Cooper as an innovation management method for businesses. It is now a widely used mechanism to manage the product development process – in the case of a University this may be the development and production of a course. The object of the model is to support the making of a key decision at each stage – the question: should development continue?– or as Cooper puts it; to ‘Go’ with or ‘Kill’ it. For Cooper, the period between Gates consists of ‘information gathering activities’ followed by an analysis of the result which is then entered in to the Gate for decision.

Over the last five years some work has begun to assess and revise the Stage-Gate model. Indeed, Cooper (2008) himself acknowledges the need for the‘Next Generation Stage-Gate’. Reading through this work, I was encouraged to find comments and observations that aligned with themes discussed in my last blog post. Cooper (2008) offers a range of ways to revise or add to his model. I’ve picked out seven below:

1. A Flexible Process: we observed (see last post) that learning design activities should be planned in advance of the stage commencing and here

Reflections on Learning Design Process Models 1

Over the last month, Rebecca Galley, Paul Mundin and I have been reflecting on a range of our recent meetings and project related activities (including meeting with colleagues from Birmingham City and Gloucestershire, with our external partners, and OU colleagues). This post shares some of this ‘reflection-in-progress’ and as such uses languages familiar to those exploring new structures for curriculum/learning design processes.  

Here are two different draft representations to help communicate our interpretation of recent work. I think it important that such models seek to weave together the less formal practices and processes (such as design activities) with more formal and necessary demands:

If you click on the images an enlarged version should load.

By way of explanation, these representations were the outcome of our thinking about a range of questions. Some of these were:

• Does a business case, or any related business approval stage, need any pedagogic information or design? – Possibly not. Could learning design be more damaging than enabling at this stage? – Very possibly (after all the design process has not yet properly started and decisions made here may constrain and limit design options
• Where in the process is the learning design problem space considered? And how much planning of learning design activities (both formal in the design team, and semi-formal individual work)? The business case may set some constraints but little in respect to important pedagogical aspects associated with the design problem. Recent work in the wider design literature has highlighted the importance of fully understanding the ‘design problem’ before developing a design solution. A number of advantages are cited from promoting a more creative, innovative approach to the problem; to better understanding of the core purpose of the design process and more effective quality assurance and design validation.
• How important is it to recognise that design stages are not discrete units that ‘end’ once a particular form or document has been submitted. It is not a linear process for each overlaps and may continue to be worked later in the process with more porous links to it’s neighbours (shown by the dotted lines above). Furthermore, it would now appear course development involves a greater range of people throughout the entire process (rather than each just involved in a discrete portion of the process).
• Is too great an emphasis placed on the submission of a form or specification before work moves on? Instead why not focus on the process of creating these forms and specifications and use the cumulative amount of ‘work’ put in to preparing these documents as indicators of ‘sufficient’ design? Here the notion of doing a ‘sufficient’ amount of design work replaces that of producing a document/forum ‘sufficiently’. Yet how could we record and store documents and design activity ‘evidence’? The most obvious is some form of portfolio/s (e.g. folders on a local or shared area). The diagrams indicate some of these portfolios and each module would need to decide, upfront, what sufficiency may mean for them.
• Is there enough formal review of the meso- and micro- levels of design? Module design currently takes place at several inter-connected levels, however often formal institutional process fade as design work in earnest begins.
• Is there sufficient recognition of the micro-design taking place in the production of course materials and services even once courses have been written or operational documents prepared? Maybe not, and a consequence does this mean that micro-design decisions may not be properly supported or articulated?
• Module/course evaluation (after delivery of the course) is often well formalised, however, can this be enhanced by better definition of the design problem space and planning of activity for the design process at the beginning? There is certainly potential for further quality assurance measures here and this may enable better tracking of decisions and therefore of accountability.

Exploring Spheres of Sharing: Analysis of Cloudworks Part 3

In March last year I looked at the extent and longevity of contributions made to the Cloudworks website by a sample of 250 users who registered in 2008 and 2009 (approximately 10% of registered users on the site). In this post I’ll ask what pattern of use, especially continued use, has this sample demonstrated over 2010? And is this similar to those who registered in 2010 year?

Cloudworks aims to support the sharing of teaching and learning and my original aim was to get an indication of how many were using the site to really ‘share’ and discuss ideas; that is to say, the proportion who posted on the site (rather than just using it as a resource to read). So how did the 2008-09 cohort do in 2010?

• In 2008-9, my original review found 8% (20 people) had posted two or more contributions over a period greater than 28 days from their first post (what I termed a ‘prolonged’ contribution). In 2010,  just over half of this group (13 our of 20 people) continued to make a contribution to the site. The remaining 7 did not post anything in 2010. However, this decrease is almost balanced by the fact that in 2010, 5 new people (from the original sample) made posts in a period of more than 28 days. This may be an indication of an emergence of a core group (5-7%) of sustained users. Scaling this 10% sample up to the entire Cloudworks usership this suggests there may be some 150 to 250 users making prolonged contributions.
• In 2008-9, some 39% of registered users demonstrated some ‘activity’ on the site (such as posting a comment or link). In 2010, however,  just 14% of the sample had made a contribution to the site.

Taken together, these data seem to point to a core group of contributors posting to the site.

Exploring the Design Problem Space

Last year I posted my first attempt at mapping the Learning Design problem space arguing a sound understanding is a precursor to any good developed of designed learning. The following quote, taken from a recent paper about teaching architectural design,  I think furthers this argument and captures what could also be the essence of designed learning:

‘Design can be viewed as a mutual learning process among designers (Beguin, 2003) and can be described as a reflective conversation between designers and the designs they create… Traditional architectural studies teaching is based on the notion that successful design solutions and learning are a direct outcome of the extent to which the design problem space is explored’ (p434, my emphasis) 

Wang W., Shih, S. & Chien, S, (2010). A ‘Knowledge Trading Game’ for collaborative design learning in an architectural design studio, International Journal of Technology Design Education, 20, pp433-451.

Authentic Assessment: approaches and practices

The term ‘authentic assessment’ is likely to unfamiliar to many reading this blog, however, it is a concept that Falchikov (2005) observed ‘appears to be increasingly used in further and higher education’. So what explains this discrepancy? Falchikov herself offers one reason, explaining that ‘my own work… has involved my students in all of the activities [I regard as authentic]. However, I have not used the term ‘authentic’ to describe the type of assessment being carried out.’

In some recent research that Denise Whitelock and I have been doing within the OU we have been examining the concept and practice of authentic assessment – and in particular, how to make visible those assessment approaches associated with concept but not understood as such. It’s been great to have the opportunity to explore the term a little and in this post I hope to outline a few of my initial impressions.

The current use of the term emerges from a discourse around ‘authentic’ and ‘genuine’ testing that had become established by the mid-1980s and which combined with the broader social constructivist project to become part of what Serafini considers to be the most recent of three assessment paradigms: ‘assessment for enquiry’.

The notion of ‘authentic’ certainly appealed to those interested in getting closer to ‘real’, ‘meaningful’ learning and represented an idea with an innate capacity to help problematise traditional assessment practice. Early definitions include Wiggins who defines it as ‘[the extent to which] student experience questions and tasks under constraints as they typically and ‘naturally’ occur, with access to the tools that are usually available for solving such problems’, Newmann et al. (1996), and Torrance who suggests ‘[it is the] assessment tasks designed for students should be more practical, realistic and challenging than what one might call ‘traditional’’ and that it is ‘a generic term… to describe a range of new approaches to assessment.’

By 2000, overlapping interpretations of what authentic assessment meant and authentic assessment tasks comprised of were emerging (e.g. McTighe and Wiggins (1999) and the review by Cummings et al.). There were also efforts to instantiate these in to guidance or advice on designing authentic assessment tasks (Darling-Hammond & Snyder; Williams; Hughes) or integrating the idea of authenticity in to principles for instruction (for example, Merrill).

The upshot of this is a range of emphasis and interpretations about what ‘authentic’ means (sometimes in respect to particular disciplines) and what constitutes an ‘authentic’ assessment task. In the itemised paragraphs below I attempt to identify some of the components of the authentic assessment discourse.

Benchmarking Assessment (conference paper): breaking down barriers and building institutional understanding

Denise Whitelock and myself have recnetly been working on a project to identfy key meaures (or criteria) of assessment processes and practice. The aim to to develop a benchmark instrument for use in better assessing and understanding assessment practice in Higher Education.

As part of this work, I am presenting on our behalf a paper at the Computer Assisted Assessment Conference in Southampton this week. In the abstract we make the point that benchmarking offers a comprehensive way of measuring current practice in an institution per se; whilst also gauging achievement against external competitors. It would appear that although e-learning has been benchmarked with a number of universities in the UK and abroad no one to date has tackled the area of assessment; which is now becoming of more concern with the advent of e-assessment. 

Our paper describes the construction of a set of benchmarking measures/indicators and the outcome of early pilots which combine a survey instrument and semi-structured interview methodologies. The findings suggest that building a comprehensive and robust core of benchmark measures would have great utility and value to institutions; not just in external benchmarking but in internal benchmarks and reviews, setting baselines, exploring the student experience, providing staff with data meaningful to their role and professional development and supporting continuous improvement.

The paper is accompanied by our current working draft of the benchamarking measures. [Since writing this post the measures have been revised further. A new version of the measures is available at: Assessment Benchmarking Criteria v17_A]. We will be very interested to hear your comments or feedback on the paper and the draft.