Pedagogies that travel and those that don’t: Sustainability in open courses and learning design

Last week I presented a paper titled ‘Some Issues Affecting the Sustainability of Open Learning Courses’ written by James Aczel, myself and four other colleagues (2011) working on the EU funded openED project at the EDEN conference in Dublin. The aim of the project is to explore participatory learning in open educational environments by producing and delivering an online open/’free’ course is business management. In so doing, the objective is to not only to produce an OER, but to use open technology, open Web 2.0 methods of communication, open opportunity to facilitate, open peer assessment, and open opportunity to build a resource for reuse. So a key question for the project is to study the effect and issues around ‘opening’ up many aspects of a course together. And one of my roles in the project is to develop and deploy a complex of evaluation methodologies before, during and after each of our three rounds.

The theme of the conference was learning and sustainability and it was interesting to see the present state of thinking around how e-learning was seeking to frame and understand what ‘sustainable learning’ means. The two uses of the concept most prevalent were: 1. The planning to sustain project accomplishments or innovation, and 2. The teaching of themes associated with sustainability within a course. The first is clearly important to the longevity of a project but I find this Read more of this post

Open Innovation and the ‘Next Generation Stage-Gate’ process: Reflections on Learning Design processes 2

Next-Generation Stage-Gating

The Stage-Gate concept was developed by Robert Cooper as an innovation management method for businesses. It is now a widely used mechanism to manage the product development process – in the case of a University this may be the development and production of a course. The object of the model is to support the making of a key decision at each stage – the question: should development continue?– or as Cooper puts it; to ‘Go’ with or ‘Kill’ it. For Cooper, the period between Gates consists of ‘information gathering activities’ followed by an analysis of the result which is then entered in to the Gate for decision.

Over the last five years some work has begun to assess and revise the Stage-Gate model. Indeed, Cooper (2008) himself acknowledges the need for the‘Next Generation Stage-Gate’. Reading through this work, I was encouraged to find comments and observations that aligned with themes discussed in my last blog post. Cooper (2008) offers a range of ways to revise or add to his model. I’ve picked out seven below:

  1. A Flexible Process: we observed (see last post) that learning design activities should be planned in advance of the stage commencing and here Read more of this post

Reflections on Learning Design Process Models 1

Over the last month, Rebecca Galley, Paul Mundin and I have been reflecting on a range of our recent meetings and project related activities (including meeting with colleagues from Birmingham City and Gloucestershire, with our external partners, and OU colleagues). This post shares some of this ‘reflection-in-progress’ and as such uses languages familiar to those exploring new structures for curriculum/learning design processes.  

Here are two different draft representations to help communicate our interpretation of recent work. I think it important that such models seek to weave together the less formal practices and processes (such as design activities) with more formal and necessary demands:

If you click on the images an enlarged version should load.

By way of explanation, these representations were the outcome of our thinking about a range of questions. Some of these were:

  • Does a business case, or any related business approval stage, need any pedagogic information or design? – Possibly not. Could learning design be more damaging than enabling at this stage? – Very possibly (after all the design process has not yet properly started and decisions made here may constrain and limit design options
  • Where in the process is the learning design problem space considered? And how much planning of learning design activities (both formal in the design team, and semi-formal individual work)? The business case may set some constraints but little in respect to important pedagogical aspects associated with the design problem. Recent work in the wider design literature has highlighted the importance of fully understanding the ‘design problem’ before developing a design solution. A number of advantages are cited from promoting a more creative, innovative approach to the problem; to better understanding of the core purpose of the design process and more effective quality assurance and design validation.
  • How important is it to recognise that design stages are not discrete units that ‘end’ once a particular form or document has been submitted. It is not a linear process for each overlaps and may continue to be worked later in the process with more porous links to it’s neighbours (shown by the dotted lines above). Furthermore, it would now appear course development involves a greater range of people throughout the entire process (rather than each just involved in a discrete portion of the process).
  • Is too great an emphasis placed on the submission of a form or specification before work moves on? Instead why not focus on the process of creating these forms and specifications and use the cumulative amount of ‘work’ put in to preparing these documents as indicators of ‘sufficient’ design? Here the notion of doing a ‘sufficient’ amount of design work replaces that of producing a document/forum ‘sufficiently’. Yet how could we record and store documents and design activity ‘evidence’? The most obvious is some form of portfolio/s (e.g. folders on a local or shared area). The diagrams indicate some of these portfolios and each module would need to decide, upfront, what sufficiency may mean for them.
  • Is there enough formal review of the meso- and micro- levels of design? Module design currently takes place at several inter-connected levels, however often formal institutional process fade as design work in earnest begins.
  • Is there sufficient recognition of the micro-design taking place in the production of course materials and services even once courses have been written or operational documents prepared? Maybe not, and a consequence does this mean that micro-design decisions may not be properly supported or articulated?
  • Module/course evaluation (after delivery of the course) is often well formalised, however, can this be enhanced by better definition of the design problem space and planning of activity for the design process at the beginning? There is certainly potential for further quality assurance measures here and this may enable better tracking of decisions and therefore of accountability.

Exploring Spheres of Sharing: Analysis of Cloudworks Part 3

In March last year I looked at the extent and longevity of contributions made to the Cloudworks website by a sample of 250 users who registered in 2008 and 2009 (approximately 10% of registered users on the site). In this post I’ll ask what pattern of use, especially continued use, has this sample demonstrated over 2010? And is this similar to those who registered in 2010 year?

Cloudworks aims to support the sharing of teaching and learning and my original aim was to get an indication of how many were using the site to really ‘share’ and discuss ideas; that is to say, the proportion who posted on the site (rather than just using it as a resource to read). So how did the 2008-09 cohort do in 2010?

  • In 2008-9, my original review found 8% (20 people) had posted two or more contributions over a period greater than 28 days from their first post (what I termed a ‘prolonged’ contribution). In 2010,  just over half of this group (13 our of 20 people) continued to make a contribution to the site. The remaining 7 did not post anything in 2010. However, this decrease is almost balanced by the fact that in 2010, 5 new people (from the original sample) made posts in a period of more than 28 days. This may be an indication of an emergence of a core group (5-7%) of sustained users. Scaling this 10% sample up to the entire Cloudworks usership this suggests there may be some 150 to 250 users making prolonged contributions.
  • In 2008-9, some 39% of registered users demonstrated some ‘activity’ on the site (such as posting a comment or link). In 2010, however,  just 14% of the sample had made a contribution to the site.

Taken together, these data seem to point to a core group of contributors posting to the site. Read more of this post

Exploring the Design Problem Space

Last year I posted my first attempt at mapping the Learning Design problem space arguing a sound understanding is a precursor to any good developed of designed learning. The following quote, taken from a recent paper about teaching architectural design,  I think furthers this argument and captures what could also be the essence of designed learning:

‘Design can be viewed as a mutual learning process among designers (Beguin, 2003) and can be described as a reflective conversation between designers and the designs they create… Traditional architectural studies teaching is based on the notion that successful design solutions and learning are a direct outcome of the extent to which the design problem space is explored’ (p434, my emphasis) 

Wang W., Shih, S. & Chien, S, (2010). A ‘Knowledge Trading Game’ for collaborative design learning in an architectural design studio, International Journal of Technology Design Education, 20, pp433-451.

Authentic Assessment: approaches and practices

The term ‘authentic assessment’ is likely to unfamiliar to many reading this blog, however, it is a concept that Falchikov (2005) observed ‘appears to be increasingly used in further and higher education’. So what explains this discrepancy? Falchikov herself offers one reason, explaining that ‘my own work… has involved my students in all of the activities [I regard as authentic]. However, I have not used the term ‘authentic’ to describe the type of assessment being carried out.’

In some recent research that Denise Whitelock and I have been doing within the OU we have been examining the concept and practice of authentic assessment – and in particular, how to make visible those assessment approaches associated with concept but not understood as such. It’s been great to have the opportunity to explore the term a little and in this post I hope to outline a few of my initial impressions.

The current use of the term emerges from a discourse around ‘authentic’ and ‘genuine’ testing that had become established by the mid-1980s and which combined with the broader social constructivist project to become part of what Serafini considers to be the most recent of three assessment paradigms: ‘assessment for enquiry’.

The notion of ‘authentic’ certainly appealed to those interested in getting closer to ‘real’, ‘meaningful’ learning and represented an idea with an innate capacity to help problematise traditional assessment practice. Early definitions include Wiggins who defines it as ‘[the extent to which] student experience questions and tasks under constraints as they typically and ‘naturally’ occur, with access to the tools that are usually available for solving such problems’, Newmann et al. (1996), and Torrance who suggests ‘[it is the] assessment tasks designed for students should be more practical, realistic and challenging than what one might call ‘traditional’’ and that it is ‘a generic term… to describe a range of new approaches to assessment.’

By 2000, overlapping interpretations of what authentic assessment meant and authentic assessment tasks comprised of were emerging (e.g. McTighe and Wiggins (1999) and the review by Cummings et al.). There were also efforts to instantiate these in to guidance or advice on designing authentic assessment tasks (Darling-Hammond & Snyder; Williams; Hughes) or integrating the idea of authenticity in to principles for instruction (for example, Merrill).

The upshot of this is a range of emphasis and interpretations about what ‘authentic’ means (sometimes in respect to particular disciplines) and what constitutes an ‘authentic’ assessment task. In the itemised paragraphs below I attempt to identify some of the components of the authentic assessment discourse. Read more of this post

Benchmarking Assessment (conference paper): breaking down barriers and building institutional understanding

Denise Whitelock and myself have recnetly been working on a project to identfy key meaures (or criteria) of assessment processes and practice. The aim to to develop a benchmark instrument for use in better assessing and understanding assessment practice in Higher Education.

As part of this work, I am presenting on our behalf a paper at the Computer Assisted Assessment Conference in Southampton this week. In the abstract we make the point that benchmarking offers a comprehensive way of measuring current practice in an institution per se; whilst also gauging achievement against external competitors. It would appear that although e-learning has been benchmarked with a number of universities in the UK and abroad no one to date has tackled the area of assessment; which is now becoming of more concern with the advent of e-assessment. 

Our paper describes the construction of a set of benchmarking measures/indicators and the outcome of early pilots which combine a survey instrument and semi-structured interview methodologies. The findings suggest that building a comprehensive and robust core of benchmark measures would have great utility and value to institutions; not just in external benchmarking but in internal benchmarks and reviews, setting baselines, exploring the student experience, providing staff with data meaningful to their role and professional development and supporting continuous improvement.

The paper is accompanied by our current working draft of the benchamarking measures. [Since writing this post the measures have been revised further. A new version of the measures is available at: Assessment Benchmarking Criteria v17_A]. We will be very interested to hear your comments or feedback on the paper and the draft.

The Lattice model for designing learning: defining the design problem space and guiding the design solutions

Sheila MacNeill recently suggested posting current work or ideas ahead of the JISC Design Bash (taking place later this month in Oxford). Since January I’ve scaled back my time with the OU Learning Design Initiative, however, I am still involved with project managing our JISC funded project. In addition, I have continued to retain my personal interest in how visual conceptualisations and cartographies of learning could benefit the design process, and in how these can enrich, even fundamentally change, the student experience of learning.

In this post I’ll address the first part of this interest – the design process. There are two related concerns; how to set-out and imagine a model of the design problem space (the first step to developing a solution), and how could this be used for a more practical tool to design learning. In this post you will notice that I talk about ‘designing learning’ or ‘designed learning’ rather than ‘learning design’ or ‘Learning Design’ and this is intentional as should become evident. 

The diagram below shows where I am currently in imagining what key dimensions exist in a design problem/solution space and how they link together (click on image to enlarge). I call this the Lattice model because of the inter-relational nature of the design elements. The purpose of laying this out as a network, rather than as a list or linear form, is to make explicit and explore the interconnectedness in designing learning. I would note that this is just a snap shot of a changing model.

The construction of this model has been framed by a number of observations and literatures. I’ll set out a few below but haven’t the space for an exhaustive account:

• Representations of learning designs tend to be concerned more with observable, performed elements of activity but we need to move much further beyond this. The sequence (or swim-lane) visualisation is a good example – a vertical line showing learning tasks with resources, support and sometimes learning outcomes connected to it. This layout was used in an early paper by Oliver and Hetherington in 2001 who showed the ‘three critical elements of learning design environments’ – learning task, learning resources and learning supports – with a basic notational system of rectangles, triangles and circles. Eight years later, these components are still important to learning design – for example Helen Beetham (2009) defines a learning activity as ‘a specific interaction of learner(s) with other(s) using specific tools and resources, orientated towards specific outcomes’ (marked A on the diagram below). Conole’s pedagogy profiler and the OU’s broader project to combine pedagogic and business visualisations of a course are examples of this moving forward with specific representations of aspects of designed and delivered learning. However, it remains uncertain how these descriptions connect together, how they help conceptualise the overall problem/solution space and how far they offer critical understanding. There are still many constraints and drivers to a design are undisclosed. A greater range of dimensions (elements relating to the design) are needed to fully map the design landscape.

• Use of the ‘sociocultural approach’ is an important perspective for Educational psychology in its attempts to theorise the role of culture and society. Although this is certainly not the only theoretical position from which to derive understanding (see later), given its key role any model should aim to accommodate (and yet also push?) this. In doing so we should acknowledge that the designer/teacher is not detached from the design process but implicated at a personal level with it. As the designer is both culturally and historically situated this makes their positionality and ‘intent’ (a term with echoes back to American pragmatism) important. Goodyear talks of the importance of representing intent, Strobel et al (2008) of capturing the design context and my experience at the OU working with Paul Clark and Andrew Brasher in trying to de-construct and visualise existing units revealed how important it is to know the thinking – and evidence supporting that thinking – ’behind’ a design (marked B below). Moving further, there is a need to situate learning as a social act – as Rogoff, for example, holds: learners engage in shared endeavours that are both structured and constituted by cultural and social norms (Rogoff, 1995). However, it difficult to find a language with which to label this dimension/box because traditions in social and cultural theory range widely on how this act could be interpreted and there is now an increasing interlacing between them. For now, I’ve borrowed from Giddens’ structuration theory the notion that there are structural rules and resources and added discourse to this principally as a nod to post-structuralism and hermeneutics (C). This label is therefore vague enough but drives us: to a more nuanced understanding of our students – be this a deeper psychological (Solomon, 2000), social, and cultural (Scheel & Branch, 1993) and the associated opportunity for and means of learning these enable or constrain; and to the intention of the designer and purpose of the activities (and ‘where’ they happen).

• From other design disciplines we learn the importance of first reflecting on and describing the design ‘problem’ space – from which the solution(s) will emerge (i.e. not just racing straight into developing the solution) – see earlier posts. Early IMS Learning Design had little to say about how one actually arrived at the design and whilst patterns outline aspects of the problem, the representation is designed to support someone looking for a solution rather than understanding the problem in the first place.

• The role of assessment in the design needs to be reconsidered – seeing it not as a product but as an activity itself. One option is to understand assessment as a process that ‘acts on’ student output (i.e. an object, action, spoken word, etc.). It would see this output as a resource produced for a specific audience that could be used again later in the learning activity or that could be transformed in to a new artefact/resource (i.e. through the activity of the teacher, student etc.). Irrespective of if, or how, this output is re-used in the learning activity, it will (or should) also constitute the evidence: to demonstrate the learning outcomes/objectives have been achieved (marked D below), to reveal other unanticipated outcomes (after Eisner, Polanyi, etc.) and to support other forms of evaluation (I’ve just jotted down Zakrzewski’s three on the diagram at present).

• There remain many other, often more pragmatic, perspectives to integrate in to the design problem space – thereby reflecting the heterogeneity of educational thought. For example: instructional designs interest in detailing what is to be learnt, learning tasks, student prior learning etc. (marked E); and the belief that a design should be built around key learning or conceptual ‘challenges’ (G). Clearly to appeal to a range of teachers the model should not be restricted to one individual theory of learning. This is in partly why I favour talking about ‘designing learning’ or ‘designed learning’ rather than ‘learning design.’

• Design of a unit of learning is influenced by practical constraints and conditions (H) defined at higher levels e.g. the block or the course (the issue of layers of design and fitting them together has been much discussed and something we’ve looked at in mapping courses), by other ‘evaluation’ demands from the institution or researchers (F), by previous units (for example, prior learning (I)) and by guidelines and training required of staff (J) . The temporal and multi-scale nature of the design problem needs representation (Grey-shaded boxes).

• Visual representation is a powerful means to communicate complex, non-linear, inter-connected relationships. It offers distinct advantages over linear descriptions and can support problem solving performance (for example, Baylor et al., 2005). This is supported by our small-scale studies at the OU (n=45-50) where we have found that a majority of staff said there were aspects of their work that would/do benefits from using visual representation and techniques (81%); they would like to improve their knowledge of visual representation and tools (81%); and that more use of visual representations (that show what is to be learnt and how) could help students better understand and plan their study’ (73%) (Cross et al. 2009)


As a practical design tool? 
Whilst the model itself can provide a framework for imaging the problem/solution space, of interest to many will be how this model can be translated directly in to a more practical application. The screenshot below shows an early attempt in Excel. Here, each dimension becomes a zone (a box) in which information about the design (be this text, lists or labelled mind-mapped objects) can be inserted. In a typical scenario, the design will evolve and mature as Read more of this post

Conference poster: How do ‘WP’ students differ from others in their engagement with e-learning activities?

Tomorrow evening I’ll be presenting a poster with Rita Tingle at the OU’s Widening Participation Conference. In it we look at evidence from a student survey (n=120) and weblogs of student access to our VLE (n=650,000) for possible differences in use of online course components by Widening Participation (WP) and young students. Click here to view the Poster (PDF 97k)

From the weblogs and survey we identify a gradual fall in use over the course despite the fact that the end of course surveys show no major issues with the technology (and indeed students apparently appreciated much of it). We find that, whilst there is little difference between WP students and other students, students under 25 used online quizzes and optional podcasts less than other students – the graph below is one of six on the poster (the labels ’09B’ and ’08J’ indicate the year in which the course was presented).

We also find from the survey that daily use of a computer for study is highest for 35-46 year olds and use for leisure highest for 25-35 year olds. This is apparently at odds with the discourse (/myth) that ‘Net’ Generation students are more likely to use such technologies and suggests potential for a larger study.

This data represents only part of our broader investigation which covers students’ initial reaction to online components such as course website, study planner, forums, podcasts, videos, quizzes, etc., how their use of them changed, reasons for skipping activities, use of computers and study preferences and practices. Together this may help explain why use falls and for differences such as those we find in this poster. We also aim to ask it there are trends in relation to student educational qualifications and completion/pass rates that are also evidenced in relation to engagement with online course resources.

Exploring spheres of sharing: Analysis of contributions to Cloudworks – Part 2

In my last post I begun an analysis of 250 subscribers to the teaching and learning sharing website Cloudworks – the post presented some headline data relating to size, rate and longevity of contributions. Of course, the next step is to get under the skin of these data to begin unpacking patterns of engagement. To do this, it would be useful to have a representational form capable of showing which Clouds (web-pages) the subscriber contributed to, what they contributed, how much, the time between contributions, and, importantly, how all this fits in to the wider sequence of contributions to these Clouds by others. 

Visualising these patterns should better equip us for interpreting subscriber activity. I’ve come up with a method for representing the contributions made by an individual subscriber (although this should work for representing contributions by one or many to any collection of Clouds). The approach aims to visualise the contributions made to a Cloud in columns running across the chart, and to show the alternating periods of activity (contributions) and inactivity in rows. Symbols represent the contributions made by different groups – in this case by the individual subscriber, members of the team developing the site, and other Cloudworks subscribers.

The following three images present: an example of one the more prolific contributors in my sample; a key to the diagram; and an annotated diagram explaining how to interpret the layout.

There is an interesting pattern of engagement shown in the above example with an intense spell of activity within the first two months (6 periods) although no contributions since. We see how the individual configures the Clouds they create (adding comments and links early on) and can see what the impact and interest in the ones they formed (such as columns 5 & 6) is compared to other Clouds they contributed to (for example, those established by a project team member given in columns 3, 4 & 7). It is also interesting to note that the subscriber often contributes to several Clouds in the same ‘period’ of activity. This may indicate that when they do log-on they look across several of the Clouds they are (or have been) interested in.

The next visualisation (below) shows the contributions made to Cloudworks by another subscriber in the sample. The diagram format certainly supports a quick comparison with others, such as the one above. It shows the Read more of this post