E-learning Benchmarking evaluation day
26/06/08
Programme contexts and outcomes

The day started with a brief outline of the programme from Derek Morrison & Terry Mayes, who have been very active in managing the programme. Derek started by giving us a bit of history. The benchmarking programme started in 2005 amid some concern, and even anger about about centralised initiatives. HEFCE attempted to listen to this and the programme was one result and the feeling was that the programme had made something of a difference since 2005. The irony in the fact that the UKEU collapse had facilitated the pathfinder processes did not go unnoticed
77 institutions participated
37 pathfinder projects
27 in phase 2

The aim had been to make available opportunities for participation across the wider sector, and that was still going on. We want to move away from the idea that HEFCE does things for us – we are the people who do it. The sector offers itself advice and support. A key part of the programme had been the idea of critical friends, for example offering consultancy on the bemchmarking process, or in the projects They had been a little nervous about the idea of critical friends but in fact this had gone down extremely well in general. There had been rigorous institutional reflection and analysis of e-learning provision and practice across the sector and the emphasis on ownership by the institutions rather than prescription by the centre had guaranteed confidentiality and trust . A drawback though is that this makes it rather difficult to extract sector level messages that national bodies such as HEFCE and QAA can take on

Outcomes?

Reports will be published on the Pathfinder web site. Terry didn’t tell us where this was, and an admittedly superficial Google search didn’t find it either. (Nor could I find it by searching the HEFCE site!) He did say that reading them you could not help but be struck by how different they seem from ordinary project reports. They seemed to be genuinely about capacity building abd often built on weaknesses as strengths. Derek had noted that they had found the model of using critical friends for projects a bit worrying, but they had been able to bring a lot of critical support to the projects – and this is one of the most important outcomes

Finally a model of collaboration between institutions (like CAMEL – http://www.jiscinfonet.ac.uk/camel ) was another output and this too emerges from the reports.
Institutional Perspectives

Next up we had brief presentations from six of the institutions who did get pathfinder funding on the theme of how benchmarking helped in the preparation for subsequent institutional developments?
University of Chester – Jethro Newton

The approach at Chester was that they took quite a strategic approach. In 2005 at the start of the programme there was a low level of embededness. Some engagement with technology but not much on pedagogy. Jethro pointed out, quite rightly I think that context is important – you never start from a blank sheet. And you’ve got to remember that content and pedagogy more important than technology. Interestingly one of their outcomes was that they developed a learning technology unit, along with a group of E-learning coordinators. I wasn’t quite clear whether these were based in faculties or in the unit itself, and unfortunately there was very little time for questions at the end. Their actual project was around podcasting and rather than talk about it, here’s the web link. http://podcastingforpp.pbwiki.com

They felt that their outcomes were that they had achieve clearer targets in faculty business planning, and they were managing to offer better staff development through the Learning Technology Unit.

University of Glamorgan – Virendra Mistry

This project seemed to be at quite a strategic level too. They started with a statement from their vice chancellor that the university aimed for highest standard of e-learning, tutor facilitation and cutting edge learning facilities.

Their outcomes were based around engendering design, Measurement, data collection, collegial spirit, mapping, changing practice, informing policy and they now have a statement about what students can expect in terms of e-learning. They now have much more of a focus on learning and teaching, and are conducting an institutional review. Interestingly they were the only university who is talking about developing the scholarship of Benchmarking. For example they planned to take some of the data collection into journals

Barbara Newland
Bournemouth University

Bournemouth had produced an institutional Review document, which had helped them focus on where they were at that point in time, and this had helped them to produce actions , provided an opportunity to Benchmark with other institutions, developed and understanding of how they were using Blackboard and helped them understand senior management perspectives. They had found the timescale a bit of a challenge as well as the need to develop a single response.

The programme helped develop staff support and e-resources. They’d developed something called E-res. It was about E-learning with quality e-resources – using web 2.0

They felt that it would be useful to revisit pathfinder although a little bit of extra funding would be nice. One of their aims was to recognise the experience of implementing, supporting and researching e-learning within their central services

Maria Lee – Queens University Belfast

Pretty much what Barbara said. Their pathfinder had been mainly used to support campus based learning. Participation was timely for them, coinciding with development of new education strategy and assessment policy. It had confirmed the approach of embedding e-learning and provided an emerging vision of how e-learning will support their goals. Having said that blurring boundaries between campus based e-learning and distance learning are changing the concept of blended learning and they are planning to develop an e-learning policy in 2008-9

Sue Timmis
University of Bristol

This was a very interesting presentation in that Bristol was a university that had Very strong disciplnary cultures. They had had a central e-learning unit since early 1990s, a strong tradition of innovation ,and were an early adopter of Blackboard (Since 2000). But, they didn’t have an educational development unit. E-learning requires differentiated strategies based on cultural contexts and knowledge fields. They wanted to find a way of giving faculties more say in how e-learning is embedded and already had a framework called ELTI – Embedding learning technologies institutionally. They adapted this framework to look specifically about what was happening in each of their departments which provided a real drive towards faculty based support for e-learning, although it was probably true to say that faculties were not thinking about e-learning or teaching and learning in any strategic way. So their project was to support faculties in embedding e-learning. They had created a distributed model by providing satellite areas of expertise in all the faculties. Their pathfinder project had been based around a set of faculty focussed strategic projects? One, for example was using Tablet PCs and evaluating their use, another around maths support in the science faculty. Students were coming in with poor level of maths – so e-learning materials being developed to deal with this. The Faculty of Arts had been looking at use of Turnitin/Blackboard plug in at a strategic level. They had also just formed an Education Support Unit – includes e-learning. (This is something that may be worth following up for my own research)

Brian Sayer,
University of London External

They took a structured approach to reviewing teaching and learning in a research led institute. Already quite deeply immersed in mainstream e-learning. Rather than take an institutional perspective they went straight in at programme level. Interestingly they said they wanted to take an Open Source approach. They provided the opportunity for programmes to take a wholly owned perspective of their teaching and learning activity. Institutionally the exercise provided a useful reminder of their strengths in this area, and areas for improvement, helped them to clarify institutional responsibility for policy and processes, and to actively collaborate in dissemination of best practice, and to improve awareness and understanding of their e-learning strategy, planning and infrastructure. They were also considering minimum standards for eLearning.
HEFCE Perspectives

Next up we had a talk from Dr. John Selby from HEFCE. I couldn’t help noticing the onteresting equation of “Educational developer” with “people who do the technical development” at the beginning of the talk. Anyway he felt that the challenge is to get those people who were reaching out to academic staff (i.e. educational developers) realise that they had new ways to do it. But also there is a need for the developers to ask what their target audience needed and why.

He also had some reservations about the programme’s title. They were not pathfinders but trailblazers. However, they had had an important impact on HEFCE ‘s thinking. But that thinking is also influenced by the external context, with issues like differential fees, questions about value for money and different modes of delivery being important. It’s important to assess how the work that is going on in e-learning connects to the wider environment, and how what we are doing is perceived outside higher education. There is a risk that eLearning seen by outsiders as a way of cutting costs (and reducing contact hours)

He reminded us that we are a very diverse sector and suggested that benchmarking was one way of addressing this. There remains a need for a comprehensive view of the e-learning landscape in the sector with markers that enable institutions to position themselves and plan their development in particular directions. He qualified this by pointing out that HEFCE needs to be careful what it says and bear in mind how it will be heard. And the sector should not read too much into what they say. (In some circumstances anyway)

He then made an interesting observation about students use of technology. For all we talk about the Google Generation, there is actually quite a limited use of anything beyond the basic technology Virtually all students use Word processing and the Internet, but for other important technologies, the figures were something like Presentation development 65%, Spreadsheets 63%, Graphics 49%, Creating web pages 25%. Which raises the questions of whether investment yields interest and if it does what kind of return does it produce. The most fundamental point to come out of all the case studies is that the appropriate use of technology is leading to improved satisfaction, retention and achievement. It facilitate increases in the size of the operation without corresponding increases in the estate. He also pointed out that in fact we are e-mature in the sense that it is no longer possible to work as we do without technology. In a while we’ll be able to stop talking about e-learning (Arguably we’re already there) Technologies are embedded in social structures and systems and the technology needs to take account of them.

How do we connect the work we do into the senior management of institutions? Many quite senior people who don’t know what we do, and think of us as a cost rather than a benefit.
The afternoon consisted of breakout sessions and a panel discussion. The first one I attended was entitled “Learner experience and the student voice.”

First there was a presentation from the University of Bradford, who started from the premise that there are high levels of technology ownership and concomitant social expectations of technology. This raises all sorts of issues about for example staff training, rules and regulations, security, communication, establishing contacts, networking and student perceptions of e-assessment

Their project was about developing the extended student, and based around social networking, skills development activities. links with academic programmes and providing integrated support, around a social site called “Ning”. – you can see the site at http://Developme.ning.com and there is background information http://www.brad.ac.uk/development. Outcomes of this is that they have created a social network, improved their PDP processes, embedded student voice into their institutional Strategy. They’ve also provided a sort of digital storytelling area, building on current you tube content – Bradford students will be telling their story – what’s it like to be a student in the C21

Next we heard about Pathfinder at Wolverhampton

This was about embedding the concept of the e-portfolio at level 1. They were using PebblePad, to do this, but their challenge was how to move past “champions” and getting staff to support e-learning and specifically e-portfolios. They created teams of mentors, and used the e-portfolio through the system. But they did acknowledge that it was important to assess the question of whether PDP was culturally desirable, or feasible? They built in 3 retreats for staff involved and used them to explain the desirability and feasibility but they also felt it was very important to get in with the students and talk to them.

Staff and students see things as chunks of learning – modularity tends to work against students and staff seeing a holistic experience. So modularity can be a bit of a problem

1) Change takes time.
2) Listen to the learner voice
3) Listen to the staff too.

But be careful “I don’t like Pebble Pad” might actually mean that “The member of staff didn’t tell me what to do” Big measure of success was an “improved student experience” I asked how they knew how they’d improved the student experience and they said that they had conducted a variety of evaluations which had had very positive feedback, but that they also had quantitative data that suggested a considerable improvement in student grades.
Then we heard from Rhona Sharpe from Oxford Brookes whose project was also based around evaluation of the learner experience, looking particularly at patterns and preferences in online media use and at experiences of social software as part of the curriculum. They found that local data was much more powerful in communicating with colleagues (Interesting that most on-campus students study at home with their own laptops and that there was a relatively low use of VLEs at Oxford Brookes.) Their findings had led them to shift their course design to a much more learner centred view. – From VLE to PLE (personal learning environment) – they had a nice graphic of a dashboard for the student – although I suppose if you think about it the concept isn’t all that different from Blackboard’s “My sites” “My PDP” and “My timetable” They’re also using Ning. http://elesig.ning.com

The second breakout session was about the in class use of mobile technologies to support formative assessment and feedback

Tim Linsey from Kingston University described how they had used a variety of mobile technologies in the classroom, supporting members of staff with mentors to discuss how to use it. The staff and mentors met up every few weeks to exchange experience and both parties found this useful. Among the technologies used were electronic voting systems, inbound text messages, (to a mobile phone). Tablet PCs & wireless data projectors (This was effectively taking the interactive whiteboard a bit further as the lecturer could move around and students could interact with the presentation. They also tried to use interactive pads (whatever they might be!) But they found them too finicky and no-one used them in anger.

Students reported that they were able to focus their learning on areas of weakness, and diminish misunderstandings, that it was easier to give responses, and it was possible to discuss a wider range of interactions. They also felt a greater sense of involvement from because they could see group feedback in real time.

Staff, found the process useful for identifying misconceptions and challenges, adapting their teaching practices, enhanced and assessment and feedback, and delivering enhanced teaching.

The project also provided information about the most appropriate conditions in which can each technology be used, the impact they have on learning and on teaching practices. Curiously only 1 person used the text messaging which surprised the project team as this was the simplest to set up

Some of the other findings were that the time to set up this kind of technology can be an issue
Very positive about the role of mentors. But students also responded very positively. 84% would like other lecturers to use it. 89.3% wanted their current lecturer to continue to use it and the general feedback from students was that they wanted even more of this kind interaction.
Then we heard from Phil Gravestock – University of Worcestershire whose project was about digital storytelling. They don’t hang about with this, getting their students to start three days after arrival! The benefits were that it was low tech, easy to learn, accessible. Students don’t have to work at learning technology to get started. There was a nice quote from one of the students “Story without digital works, but digital without story doesn’t”. The point is that technology not important, narrative is

We were then shown a couple of the stories – they were very simple audio-visual presentations in which the students told us something about themselves. The images were often crude and the text hard to read, but the narratives were quite powerful.

But the point here is that students need help with the story not the technology. They can do the technology anyway.

Finally we heard from Richard Hall of De Montfort University
Their project was about making institutional sense of web 2.0. But they too were clear that this was not about technology, but about empowerment. They had developed podcasts that move face to face sessions forward, and provided synchronous classrooms related to social networking tools, and were using wikis for variouys activities. Their philosophy seemed to be based on the premise that we’re using 21st century technology in a 19th century pedagogical context. I think they may have a point, even if there’s some hyperbole in that remark

He also raised an interesting question “Do we assess the affective side of education, do we even engage with it?” He used sound clips too in his presentation and s one of the students said (sounding quite surprised) “I was doing more, getting more involved in it, and actually started enjoying it”

Students seemed to enjoy the flexibility. They want academic staff to make savvy decisions For example if you do use web 2.0 what’s your strategy if it stops being available.