Evaluation of the Open Learning Design Studio MOOC: Participant perspectives and experiences

Over the last two months I have been evaluating the Open Learning Design Studio MOOC (OLDS MOOC). This was written by contributors from several UK universities and funded by JISC. The evaluation has analysed quantitative data captured before, during and after the courseand richer qualitative data provided by participants in pre- and post-course questionnaires and in blogs, posts and comments contributed to the public course space.

The report is available on the Open Research Online website or alternatively a copy of the report is available here. The length of the document has been kept to around 20 pages and includes sections on:

• Participant profile and expectations,
• Participation and contribution rates,
• Use of the course space and technologies,
• Impact and changes of practice,
• Effectiveness and challenges of collaborative group working,
• Awarding course badges,
• Conclusions

One challenge has been how to represent and balance the often hugely contrasting perspectives and experiences of the MOOC: how much weight do you give to the views of those who remained for the duration and followed the course instructions, those who started but undertook only part of the course, and those who watched but did not contribute? Another challenge was to determine on whose terms the MOOC should be evaluated. For example: participant compliance with design; participant deviation from design; meeting design goals; meeting participant goals; performance against established measures; and performance against participant defined evaluation criteria – the latter yielded particularly interesting responses in our pre-course survey.

40 Tips for running an Open Online Course or MOOC from those who have experienced them

To coincide with the announcement of a new Learning Design MOOC (now confirmed as starting in January 2013) some of those registering have been invited to tell us about their previous experiences of participating, or helping organise, open online courses. Information about MOOCs can be difficult to come by, so the thirty-five responses we have received so far (which cover 50 separate experiences of 25 different MOOCs) represents a really interesting dataset.

In the survey we asked people about what they had liked and disliked about the MOOCs they had been involved with, what the MOOC did well and not so well, how the issues they had encountered could be resolved and what advice they would give to others planning a MOOC. I’ve taken these responses and attempted to express these as a series of tips – each tip based directly on one or more pieces of advice, suggestions or experiences reported by the survey respondents.

If you would like to add your experiences to those already contributed, please visit the survey. Also, as this survey represents the first stage in a broader schedule of monitoring and evaluation for the MOOC, watch out for further reporting over the next six months.

So here are 40 Tips for running a MOOC (in no particular order)…

1. Be clear about what type of MOOC you are planning  – there is no one way of doing a MOOC for, just like any course, it may draw on constructivism, instructivism,  connectivism etc. (or any combination). Be clear about the pedagogy even if just within your design team.

2. Ensure a balance between the suggested activities (those you have designed) and time allowed for participants to develop/build from them

3. Ensure the course rewards participation – some will be motivated by the promise of awards such as badges or certificates however for others the benefit of an enriching experience, helping others, social interaction over a share interest, or better understanding themselves may be as, if not more, important.

4. Ensure the course mechanics/design focuses on achieving a coalescing or binding together of knowledge  – so as to avoid a spiralling fragmentation of posts, content and participation and generating what one respondent termed ‘too much noise’

5. Consider the expectations and skills of those who will be involved what are the prerequisite skills required? At what point might the degree of difference between participants become an issue?

 6. Provide ‘plenty of structure’ in terms of activities, group working and website organisation/navigation yet also ensure such structure is not a constraint for those who want to go further or to reorganise knowledge in different ways. Consider the negative impact on motivation some feel during scripted courses and, conversely, others feel during very ‘open’ courses

7. Consider future participants as well as current participants will the post, tweets, links etc make sense to someone new visiting the site in 6 months time? Will the course really be ‘open’ across time as well as open geographically at the time it takes place?

8. Expect a high-drop out rate low barriers to entry may also mean low barriers to withdrawal or lower tolerance thresholds). Take the opportunity to reflect on what a drop-out rate of 50% or 90% of registrations would mean for your course, for the participants registering on the course and your own educational mission/purpose in organising it.

9. Create a sense of closure at the end of the coursefor example, as one respondent suggests ‘an assessment, a designed activity or summary podcast’.

10. Build in the potential to scale-up (in terms of numbers), reach further (in terms of accessibility and portability) and be sustained (repeated in future)

11. Keep the pace and depth of topics higher than traditional face to face courses – and in making this change, ensure you recommend or estimate how long tasks should take

12. Determine and communicate a vision for how different forms of knowledge will be handled and privileged within the courseare some forms of knowledge more authoritative? Do anecdotal opinions carry equal weight to empirical published research?

Read more of this post

How are E-Readers and Tablets impacting study practices and learning? – Some references

I am currently working with Mike Sharples on a university funded project looking at how post-graduate students use e-readers and tablet devices such as Kindles and ipads for their learning. Are these devices having an impact on the efficiency and nature of student study and what they are learning? What implications does this have for our pedagogy?

We begun by casting a look at recent studies that had examined the changing role of e-reader devices. Whilst many of these are from the US and some focus more on the e-book rather than e-reader they reveal a rapidly evolving landscape of device use. The following aims to gather together and share references to some of the papers, reports and articles looked at. Please do add others if you have them in the comments box at the end of the post. The next stage of the project is analyse survey data from over 150 post-graduate students and complete a series of follow-on in-depth interviews.

1. JISC National e-Book Observatory Project

As the JISC project website notes, ‘back in 2007, there was very little known about how e-books were being used by students in universities and colleges across the UK and even worldwide.’ The JISC National observatory project perhaps marked a watershed in the UK; gathering a huge amount of data by working with over 120 universities and collecting survey responses from over 52,000 students and academics.

Hamid R. Jamali, David Nicholas, Ian Rowlands, (2009) ‘Scholarly e-books: the views of 16,000 academics: Results from the JISC National E-Book Observatory’, Aslib Proceedings, Vol. 61 Iss: 1, pp.33 – 47. The aim of this paper by Hamid et al. was to review the open-text responses given in the survey conducted as part of the UK National E-books Observatory and to use the 16,000 responses given to understand respondent perspectives on the advantages of e-books and thoughts about them and the context in which they are used. The paper looks at the responses to two open-text questions asked in the survey. Link

Rowlands, I., Nicholas, D., Jamali, H. and Huntington, P. (2007) What do faculty and students really think of e-books? (Report), CIBER University College London.

Nicholas, D et al. (2010) E-textbook use, information seeking behaviour and its impact: Case study business and management, Journal of Information Science, 36, 263-280.

2. Gathering Momentum 2007-2009

During 2008 and 2009 a number of investigations into the use of e-books were undertaken; some into student use of e-books and – to a more limited extent – e-readers, and others into the use by and response from faculty and lecturing staff. For example:

Noorhidawati, A. and Gibb, F. (2008) How students use e-books – reading or referring? Malaysian Journal of Library and Information Science, 13 (2), 1-14. This paper by Noorhidawati and Gibb presents the results of a web survey of students undertaken in 2005/06 at the University of Strathclyde. There were 1372 responses (around 10% of total university). The study found ‘that there were three different types of e-book use in an academic setting: (a) fact finding; (b) finding relevant content; and (c) extended reading.

Lam, P., Lam, S., Lam, J. and McNaught, C. (2009). Usability and usefulness of eBooks on PPCs: How students’ opinion vary over time, Australasian J. of Ed. Tech, 25(1), 30- 44. Like Biermans’, this is a relatively small-scale study and involved 12 students using PPCs (Pocket PCs). In their introduction, Lam et al. take the following position: mobile learning can extend learning spaces; ownership of mobile devices in Hong Kong is high; reading digital materials such as newspapers is valued but ‘the value of reading academic e-Materials in a mobile formal is less known;’ and the ‘phenomenon’ of e-books is ‘quite different’ from that of e-Journal use. They claim that rising cost of print books and the ability to link multimedia resources are seen as two key drivers to e-book up-take by libraries.

Bierman, J., Ortega, L. & Rupp-Serrano, K. (2010) E-Book Usage in Pure and Applied Sciences, Science and Technology Libraries, 29:1-2, 69-91.

Camacho, L. & Spackman, A. (2010) Transitioning to E-books: Usage and Attitudes Among Business Faculty, J. of Business & Finance Librarianship, 16, 1, 33-45.

Foster, G. and Remy, E. (2009) E-Books for Academe: A Study from GettysburgCollege,’ Educause Research Bulletin, 22.

Rickman, J., Von Holzen, R., Klute, P. & Tobin, T. (2009) A Campus-wide E-Textbook Initiative, Educause Quarterly, 32, 2 Link

3. Kindle e-Reader US pilots 2009-2010

In 2009, Amazon commissioned a number of pilots in US universities that sought to understand how the Kindle e-reader could be used in a Higher Education context. The focus of these pilots varied yet teaching and learning was usually considered to some degree.

Cliatt, C. (2010) Kindle pilot results highlight possibilities for paper reduction, News at Princeton, 22 February 2010. The intention of this pilot was to ascertain if using Kindle e-readers could help the institution to save paper whilst not adversely affecting the learning experience. The study begun in late 2009 and involved 50 students across 3 courses. Participating students and faculty members in the pilot courses received a free device that they could keep.

Darden School of Business (2010) Darden Shares Results of Kindle Experiment, University of Virginia News and Media. 5 November 2010. The purpose of this Kindle pilot, which like the others begun in late 2009, was to ‘explore how the Kindle DX can enhance teaching and learning, promote effective applications of technology in higher education, improve environmental sustainability and lead to potential cost savings to schools and students. In addition to saving students from lugging around heavy backpacks, the technology could substantially reduce the use of paper.’

Marmarelli, T. and Ringle, M. (2010) The Reed College Kindle Study. 26 February 2010.

4. iPads, e-Books and e-Readers 2010

The Kindle pilots appear to have been closely followed by two other pilots in the US, both of which appear to be widely cited. The first was undertaken by Oklahoma State University and involved use of the Apple iPad and the second by the University of California looking at e-book usage (sponsored by the book publisher Springer). Alongside these can be found many smaller studies in to e-readers and e-textbooks

 iPad Study Released by Oklahoma State University, Executive Summary, 3 May 2011. This study was undertaken during the autumn of 2010. Five sections of two courses across two colleges and two campuses participated in the Apple iPad Pilot Program.

Li, C., Poe, F., Potter, M, Quigley, B. and Wilson, J. (2011) UC Libraries Academic e-Book Usage Survey: Springer e-Book Pilot Project, University of California Libraries. The primary objectives of this study were to examine: respondents’ general preference for print books as compared to e-books; how respondents interact with e-books and barriers to e-book adoption and use; how users of Springer e-books discover their availability; and satisfaction level with Springer content and features, including the ‘MyCopy’ service.

Weisberg, M. (2011) Student Attitudes and Behaviors Towards Digital Textbooks, Pub Res Q, 27, 188-196. This paper takes the position that the textbook industry, like others, is being disrupted and transformed by digital technologies in the form of e-Readers and e-books which may impact economics and business models. The paper reports findings of a two year study at SawyerBusinessSchool, SuffolkUniversity, Boston. Students from the ‘cap-stone’ undergraduate course were divided in to groups and given either a Kindle, Sony e-Reader Touch, iPad, entourage eDGe or CourseSmart. All five groups also got apaper textbook and there was a sixth group who were only give a paper textbook.

Foasberg, N. (2011) Adoption of E-Book Readers among College Students: A Survey, Information Technology and Libraries, September 2011, 108-128. The aim of this study was to learn whether e-book readers have become widely popular among college students at one large, urban, four-year public college. A survey asked students whether they owned e-book readers and if so, how often they used them and for what purposes. This information, Foasberg argues, is important to help libraries to understand student use of eBooks and whether students use them ‘of their own volition.’

Pattuelli, M. & Rabina, D. (2010) Forms, effects, function: LIS students’ attitudes towards portable e-book readers, Aslib Proceedings, 62 (3), 228-244.

Reynolds, R. (2011) Trends Influencing he Growth of Digital Textbooks in US Higher Education, Pub. Res. Q., 27, 178-187.

Robinson, S. (2011) Student use of a Free Online Textbook, Academy of Educational Leadership Journal, 15 (3)

5. E-Reader Use and National Studies 2011

This year saw the publication of the ECAR 2011 survey results, 2011 Horizon Report and the Mobile Education Landscape Report. Of the three, E-readers appear to feature most in the ECAR survey. In the UK, the Open University took part in the HEA Post-graduate Taught Experience Student Survey.

Dahlstrom, E., de Boor, T., Grunwald, P. & Vockley, M. (2011) ECAR: National Study of Undergraduate Students and Information Technology, 2011 (Research Report). Boulder, CO: Educause Center for Applied Research. The ECAR Student Technology Study looked at student use and perceptions of technology. In the preface it is acknowledged that ‘while no single study can mirror results you might find on your campus, the 2011 study used a stratified random sample of students so you can extrapolate these results to higher education.’ The data presented in this study represents the responses of around 3,000 students drawn from the much larger population of US HE students.

Johnson, L., Smith, R., Willis, H., Levine, A., and Haywood, K., (2011). The 2011 Horizon Report. Austin, Texas: The New Media Consortium.

In addition

There have been plenty of articles and opinion pieces about the use and role of e-readers. Here are a few:

Murray, O. & Olcese, N. (2011) Teaching and Learning with iPads, Ready or Not? TechTrends, 55, 6, 42-48.

Nelson, M. (2008) E-Books in Higher Education: Nearing the End of an Era of Hype? Educause Review, 43 (2)

Nelson, M. (2010) E-books in Higher Education: Are we there yet? ECAR Research Bulletin, 2.

Slater, R. (2010) Why Aren’t E-books Gaining more Ground in Academic Libraries? EBook Use and Perceptions: A Review of Published Literate and Research, Journal of Web Librarianship, 4 (4) 305-331.

Young, J. (2009) This Could be the Year of E-Textbooks, if Students Accept Them: Many titles are available, but student are wary, The Chronicle of Higher Education, 7 September 2009.

OULDI Project: Evaluation and Final Project Reports now published – the impact of new curriculum design tools and approaches on institutional process and design cultures

Over the last month I’ve been working with the rest of the OULDI team on finishing our final Evaluation and Project reports. The focus of the OULDI project has been on developing and piloting tools, resources and approaches that support institutional curriculum design processes and design practice/cultures.

The Evaluation Report (91pp) pulls together research and experience from all five project strands and nine pilots. First the report reviews the tools we developed and trialled for sharing learning design expertise (Cloudworks), for visualising designs (CompendiumLD, Module Map, Activity Profile) and for supporting design and reflection in workshops (Facilitation Cards, workshop activities, etc.). The Report then presents evidence for how effective these have been and what this tells us. For example: how do communities of practice use a shared design space? How well do university processes map actual practice? What benefits and barriers can we discern? What discourses of resistance and case studies of personal success have we recorded?

One clear message is that it is possible to be successful in changing or improving the process, practice and perception of curriculum design but has found this requires a combination of elements working together: selection of effective design tools; well configured institutional and informal design processes; proper opportunity for collaboration; reflexive working and dedicated time away from the day-to-day to work on a design; positive and real management endorsement; staff with positive attitudes and adequate tacit knowledges of the art of teaching and the discipline of designing learning; and an expert consultant role to guide and advise teams. The success of embedding learning design is therefore dependent both on the internal quality of the particular design approach or tool and on the capability of the (institutional/professional skills) context to ‘receive’ it

Building on the Evaluation Report, the final Project Report presents a concise project summary. In this we offer an overview of the educational, learning design and technical context, and document benefits  to the design process, design communities, design practices, design software and visualisation, and the sector. The Project Report concludes with some ‘lessons to share.’ These may be summarised as:

1. There is often a great difference between the idealised process (that codified in guidance and formal organisation of staff) and the real process interactions that take place. It is in the practices and culture of the latter – the lived enacted process –that this project has particularly sought change.

2 The challenge of convincing staff that a design approach would be helpful can be hindered by the difficulty in demonstrating or ‘proving’ there is a need to change.

3. The project has provided insight into how interconnected the stages are in the design process. It has noted the problem of course teams moving too readily past the design in to the ‘embodiment’ phase of course creation and of designers evaluating tools in respect to how much the tool saves them time or effort rather than on the benefits to others in the process, most notably students.

4. Stakeholders differ in their understandings of the purpose and potential of curriculum design.

5. Tools or design activities that represent a minor step for staff are more likely to be accepted and used. This may be because they require little change of design practice/approach or the learning of new pedagogic knowledge.

6. There is a complex and nuanced relationship between course quality and the time expended in design. Providing even modest additional opportunities to reflect on or to discuss designs with others can lead to improvements in the design quality.

7. The open exchange of teaching and learning design experience remains limited to a small minority (between 5 and 10%) of staff although there is clear evidence of users being interested in reading others posts and observing the sharing of knowledge.

8. Communicating the breath of the project ambition – of working concurrently with and the overlap between multiple strands of work has as times proved a challenge.

9. The project has derived great benefit from being part of a twelve project national JISC programme and from gaining insight and experience of other institutional practice.

10. Project tools and resources use a variety of representational forms; from concept mapping to structured tables. Many of these are different to current methods of representing learning and curriculum designs. As such, in disrupting practice and accepted process, they have the capacity to offer new perspectives, to challenge, and to promote reflection. Whilst not necessarily conducive to making a process shorter, such disruption may help make it more efficient, effective and capable of delivering greater quality outcomes.

References:

Cross, S., Galley, R., Brasher, A. & Weller, M., (2012) OULDI-JISC Project Evaluation Report: the impact of new curriculum design tools and approaches on institutional process and design cultures (PDF)

Cross, S., Galley, R., Brasher, A. & Weller, M., (2012) Final Project Report of the OULDI-JISC Project: Challenge and Change in Curriculum Design Process, Communities, Visualisation and Practice (PDF)

There is a separate report for each of the project’s nine pilots and over twenty personal narratives/case  studies. For example see: Galley, R. (2012a) The impact of early learning design on the efficiency and effectiveness of curriculum design processes and practices: Faculty of Education and Languages Studies Case Study Final Report.

Lattice II Framework

Earlier this afternoon I presented a paper at the Blended Learning Conference 2012 titled ‘Defining the Learning Design Problem Space:Creating a Better Learning Solution. The slides will be posted shortly, but in the meantime, here is the revised Lattice Learning Design Problem Framework which was used during the session. 

Image

‘Preliminary findings from a series of staff surveys on perceptions, attitudes and practices of learning design’ paper now available online

Someone recently pointed out that the full text and data from my 2009 ALT-C paper titled ‘Preliminary findings from a series of staff surveys on perceptions, attitudes and practices of learning design’ (co-written with Paul Clark and Andrew Brasher) is difficult to find so I’ve just deposited a PDF version in the Open Research Online repository.

This short paper reported the findings of a series of connected staff surveys which looked at some attitudes and perceptions about issues associated with learning design. This included asking staff if they agree/disagree with the following eight statements:

• ‘The advent of elearning is making the process of creating courses more complex’
• ‘I sometimes feel overwhelmed by the challenge of how to effectively integrate ICT in a course’
• ‘There is a pedagogic need for better integration of learning technologies (e.g. VLE) into courses’
• ‘It is becoming harder to understand how all the parts/components of planned learning and teaching fit together’
• ‘I do not find it difficult imagining how online learning content fits with other course content’
• ‘Technologies available today offer great potential for enhancing the student learning experience’
• ‘Understanding the relationship between pedagogy and learners’ activity is a priority for me’
• ‘Do you believe that more use of visual representations (that show what is to be learnt and how) could help students better understand and plan their study’

These statements were derived from several claims made in the learning design literature about the potential role (and need) for more designed approaches to learning and teaching and ongoing OULDI work at the time (semi-structured interviews, workshops, design observation, and focus groups site). The survey was administered to selected staff in 2009 and 50 members of staff responded.

Whilst this data is now a couple of years old, it is an interesting record of how staff responded to some key questions that are still being asked today: Has the advent of new technologies has created a much more complex situation in the design and planning of teaching and learning? Is this perception widespread across staff? And what is the perceived role for visualisation in the design process?

For example, take the first of the eight statements

Read more of this post

Describing Teaching and Learning Innovation

In this post I want to propose three ‘lenses’ that may be useful for describing and understanding the nature of a teaching and/or learning innovation. Innovation is certainly a prized commodity; deemed important enough to use to structure formal university module development and delivery processes, and to trade across and between practitioners and instititions. We also know that the question of innovation is likely to be an important consideration early on in the process of creating, designing and teaching a module or learning activities (and hopefully later on in the process also!). 

So how do teachers talk about and represent their innovation? I have been involved in a couple of projects that sought to to capture innovation and, like similar attempts elsewhere, found it difficult to define what innovation was. The conclusion is, of course, that innovation is a nebulus term embracing everything from a paradigm shifting idea to borrowing one from a colleague and using in a class. Broadly speaking, one could suggest that innovation is anything that creates new possiblities by combining different knowledge sets that should result in positive change or added value. This expectation for there to be a positive result I think distinguishes innovation from other terms such ‘something different’ or ‘ something new’  and prevents it being abused to the extent that  ‘being innovative’ becomes an aim in itself.   

So if innovation can be many things, do we need to be more precise in how we talk about it? Perhaps, like for other aspects of learning design, it could be helpful to have some lenses with which to view, describe and define it. Fair enough, each lens will present only a partial representation, but together such views may help better understand the character of the innovation and to use and evaluate it effectively.

LENS 1: For who is the innovation innovative? Each individual, group or community who experiences the innovation will likely experience it differently: what is innovative for one module author may not be for their students who did it last year on a previous course; what is innovative for the general way a subject is taught may actually be ‘common practice’ for a ‘leading-edge’ faculty or university; what is innovative in one faculty may not be so across the univeristy.

So this first view represents seven key stakeholders in the educational innovation. When selecting these seven, I looked at the ways practitioners (in interviews or in written documents) spoke about who their innovation was innovative for. I also thought it important that the diagram be organised in terms of the scale/size of the group effected (note the progression from the individual (on the far left) to the entire sector (on the far right)). Arranging the representation in this way should make it is easier to see how extensive or local the innovation may be. 

 

One way of using this view would be to determine how innovative the innovation is for each stakeholder and assign a score from 0-5. These scores could be written in the blanks provied. Scoring may Read more of this post

Follow

Get every new post delivered to your Inbox.