Saturday, 12 July 2008

At last...something!

I've managed to collect some limited data and produce a final report. I'm a bit disappointed with the way the project unfolded as I had a few problems (which you can read about in the report).

My main issue now is that I've been working on the report in a Word document (as we currently have no internet access at home) and now I've come to upload/copy it into Google docs (on my mum-in-laws computer!!) all the formatting has messed up and the graphs have gone.

C'est la vie.

I'll email the report to Bronwyn but here is the link to the not very clear version.

Apologies for not being around much lately (life got very busy) but I hope I kept up with everyone's projects and made comments. Thanks for all the feedback from those who kept up with my progress.

Friday, 23 May 2008

Summary of Evaluation Plan

Here is my presentation summarising my evaluation plan. It's about 6 minutes long and took me hours! Had a couple of technology glitches which slowed me down and then spent ages re-doing the audio - not used to hearing myself present and spent a lot of time 'umming' and 'errring'. I then discovered the cut button in Audacity (the audio software I used) and spent a while cutting out the 'errs' and 'ums' - although got bored of this so there a still a few in there!! I suppose this is another online skill that will improve with practice.



Thursday, 22 May 2008

Slow progress with presentation...


I'm part way through creating a presentation to summarise my evaluation plan. Sorry it's not up and running but I've found it a bit tricky to find audio software that works on this machine! I'm now using Audacity which seems to be fine. Hoping to get it finished today/tomorrow.

I won't be collecting my data for a few weeks as the students are currently on exam (study) leave! They come back on 11 June so I'll hopefully be doing my observations and surveys shortly after that. I'll then rush to catch up with everyone :)

[image source: x-eyedblonde]

Monday, 12 May 2008

Completed Evaluation Plan

Here is a link to my completed plan.

Evaluation Plan Draft #2. (or if that doesn't work try copying this to your browser web address bar http://docs.google.com/Doc?id=dg6w7fth_5fhbbkdff)

Saturday, 10 May 2008

It's been a while...

I'm concious of the fact that I've not done much blogging or commenting lately so just a quick note to say I'm getting on with the evaluation plan - hoping to finish it over the weekend if possible. I've hit a slight snag in that half the students go on study leave from next Tuesday and the rest join them in another fortnight. Desperately hoping I've not left it too late to get access to some kids!

Monday, 28 April 2008

Evaluation Plan - Some Feedback


Here is Bronwyn's feedback on my first draft of the evaluation plan. May give you some pointers.

Thanks Bronwyn, your comments have clarified some issues for me.

[image: adewale_oshineye]

Friday, 25 April 2008

Ideas for evaluation plan

Some brief notes on my thoughts so far for my evaluation plan - using headings from the template provided (and not completed yet).

Background:

Moodle is a learning management system (LMS) or virtual learning environment (VLE) which was introduced to South Western College (invented name for the pupose of my project) in September 2005. It is available to all teaching staff, administrators and students. It is used as a repository for college administration (e.g. such as policy documents) and training documentation. It is also used as a resource repository with space available for every subject area. The areas are generated by course qualification. Some subject areas use the available area to provide all course documentation, resources that supplement the face-to-face classroom learning and relevant activities and links. The level of use of these areas varies greatly from course to course.

This section describes any information which is needed to provide the reader with an understanding of the background of the interactive multimedia that is being evaluated.

Purposes:

The purpose of the evaluation is to carry out a formative review of Moodle in terms of its impact on learning. In particular the following two e-learning guidelines will be investigated.
  1. ST7 Will the e-learning foster students’ curiosity and creativity?
  2. ST9 Do the technologies employed help students successfully meet the learning outcomes?


This section thoroughly describes the purposes of the evaluation. A single plan can address a variety of purposes, but all must be delineated clearly. Evaluation is always a political process and all parties must accept the purposes for the evaluation to be successful.

Audiences:

To focus in on the learner experience of using Moodle the evaluation will concentrate on gathering information from students who have experienced its use in their lessons.

The curriculum managers for the two departments will see the results of the evaluation.

This section specifies all the primary and secondary audiences or consumers of the evaluation. In general, it is recommended to open the evaluation up to as many people or agencies as the client will allow.

Decisions:

Formative evaluation - answers to questions on how well is it working so far? Does it help? Does it stimulate creativitiy and curiosity? Does it help learners achieve? If yes, how can good practice be disseminated to other staff. If no, how can these issues be addressed in the future?

This section is probably the most difficult, but it should be included if the evaluation is to have meaningful impact on decision-making. Trying to anticipate the decisions which can be influenced by an evaluation takes creativity and trust. Many developers do not wish to anticipate negative outcomes for their efforts, but these too must be considered.

Questions:

Are learners benefitting from the use of Moodle in their face-to-face lessons - learning outcomes?

Does using Moodle enable learners to develop skills/interest in their subjects - curiosity and creativity?

Does using Moodle enable students to achieve at a level equal to or above their achievments in a wholly face to face environment- learning outcomes?

A key element of a sound evaluation plan is careful specification of the questions to be addressed by the evaluation design and data collection methods. The clearer and more detailed these questions are, the more likely that you will be able to provide reliable and valid answers to them.

Methods:

Observation of lessons - to see how students interact with Moodle and whether it fosters creative activity and curiosity. Also to judge whether lesson outcomes can be achieved when using blended learning. I've put a list of statements together in an observation checklist to give me pointers to look for. I'd appreciate any feedback.

Student survey - to gather student opinions on the use of Moodle and their perceptions of the effects of the blended approach to their learning. I've started a survey but struggling a little with the questions to include - again any feedback welcome.

Informal interviews - as a result of survey responses and/or observations to help clarify any unexpected issues that arise to reflect 'responsive evaluation'.

This section describes the evaluation designs and procedures. There are scores of designs and hundreds of procedures which can be used. The keys to success are matching these options to the purposes and questions of your client and keeping within the budget and time line of the study.

Sample:

This depends on factors outisde my control - availability of teachers and students at a busy time in the academic year. It's likely to be between 20-40 students, aged 16-19 studying courses at level 1, 2 or 3.

This section specifies exactly which students, trainers, and other personnel will participate in the evaluation. If necessary, a rationale for sample sizes should also be included.

Instrumentation:

This section describes all the evaluation instruments and tools to be used in the evaluation. Actual instruments should be included in appendices for review and approval.

Limitations:

This section spells out any limitations to the interpretation and generalizability of the evaluation. It should also describe potential threats to the reliability and validity of the evaluation design and instrumentation.

Logistics:

This section spells out who will be responsible for the various implementation, analysis, and reporting aspects of the evaluation.

Time Line:

This section presents the schedule for implementation, analysis, and reporting of the evaluation.

Budget:

This section "costs out" the finances for the evaluation. Personnel time usually is the major cost factor. Evaluators often charge from two hundred to several thousand dollars per day depending on their expertise and reputation. Other significant cost factors are travel, data preparation (e.g.transcribing taped interviews), and document duplication.

Sunday, 13 April 2008

Article summary

Learner Experiences of e-Learning: JISC

My article isn't really an article but an evaluation report description. JISC (Joint Information Systems Committee) is a UK organisation with a mission to "provide world-class leadership in the innovative use of ICT to support education and research". (Supporting Eduaction and Research, para 1). E-learning is one of their eight strategic themes.

"JISC funds a range of programmes, services and activities that promote and support the use of e-learning. The majority of this work is funded under the e-Learning programme, which aims to identify how e-learning can benefit learners, practitioners and educational institutions, and advise on its implementation." (E-Learning, para 2)
The evaluation project (or research study) was completed in 2006 and looked at collecting "learner stories on their experiences with e-learning" (Learner Experiences of e-Learning, para 1, under Aims and Objectives heading) in the belief that this kind of perspective was generally ignored in research into e-learning pedagogy. (As stated by Sharpe et al., 2005).

Research questions centred on learner perceptions and strategies related to e-learning. This links to my small project as I try to discover students views on their blended learning classrooms.

Tools used by JISC in this study were as follows:
  • learners keeping 'audit logs' of their experiences
  • observations of learners using elearning in subject specific contexts
  • surveys
  • in-depth case studies of specific students and their activities (sampled from subject areas) to create learner diaries
  • audio logs and thinking-aloud observations
  • learner interviews (focussing on outcomes of other methods used)
  • focus groups (as a "triangulation method to transcend from the data gathered from individual learners and validate and reflect on these results in a larger group" (Learner Experiences of e-Learning, para 6, under Project Methodology heading))
The mix of various methods for data collection and use of triangulation suggest a eclectic-mixed methods-pragmatic paradigm. The project feels like a responsive evaluation as each method of data collection is used to inform the next. The nature of the project suggests a formative evaluation method - gathering data about what student's are thinking of their e-learning so far.

My project, on a much smaller scale (!) is going to take a similar approach, concentrating on finding out what the learners actually feel about their use of ICT in their lessons and whether they are of benefit.


References
  1. E-Learning. JISC. Retrieved on 10 April 2008 from http://www.jisc.ac.uk/home/whatwedo/themes/elearning.aspx
  2. Learner Experiences of eLearning. JISC. Retrieved on 10 April 2008 from http://www.jisc.ac.uk/whatwedo/programmes/elearning_pedagogy/elp_learnerxp.aspx
  3. Supporting Education and Research. JISC. Retrieved on 10 April 2008 from http://www.jisc.ac.uk/home/whatwedo/themes/elearning.aspx

Friday, 4 April 2008

Project Thoughts

Some information about the type of evaluation for my project.

Paradigm
Model
Method
Eclectic-Mixed Methods-Pragmatic

From a variety of methods pull together an appreciation of current practice to inform future practice.
Stake's Responsive Evaluation method

Start with some observations of current practice and then develop queries to explore further.
Formative Evaluation

To provide an insight into a developing project to make the most of strengths and minimse and flaws.
  • Project: The use of Moodle to deliver e-learning in a blended learning environment. Specifically:
  1. ST7 Will the e-learning foster students’ curiosity and creativity?
  2. ST9 Do the technologies employed help students successfully meet the learning outcomes?
  • I'm interested in getting a better understanding of how Moodle, as a container for e-learning, impacts on learning.
  • The expected outcome would be practical suggestions for future blended learning strategies.
  • I would like to gather views from multiple perspectives - student, teacher, Moodle adminstrator.
  • I would like to observe some lessons where blended learning is planned. This should give me an indication of student response to e-learning resources and perhaps provide evidence of curiosity, creativity or achievement of learning outcomes. From these observations I would expect to develop further questions to use in surveys or interviews (i.e. responsive evaluation)
  • I would like to survey/interview students to gather views on the use of Moodle in their lessons. Do they enjoy this type of learning? Is it helpful? Is it efficient? Is it fun?
  • I would like to survey/interview teachers who have used or are using Moodle. What do they really think about blended learning? Is it effective?
  • I would like to interview the Moodle administrator. This person's impressions could be interesting as she communicates with staff about the facilities of Moodle and helps teachers set up resources for their classes.
  • I would like to review any available usage data for Moodle. Although the data is limited (due to technical issues!) it is possible to view student access of resources - dates/times. This information may provide only a superficial reflection but may help support or refute other evidence.

NB Given the time of year, in the lead up to the summer exam period, I may have to rethink some of the techniques listed, as time will be precious for students and staff.

Tuesday, 1 April 2008

Evaluation Paradigms and Models...and Methods

Paradigms
The evaluation paradigms set out in our reading were quite tricky to follow and I organised myself a mind-map diagram to try to summarise the main points. However the program I used (freeware called FreeMind) wasn't great for turning it into a useful link or image. I'll include it here but you may find it difficult to read! (I've printed it and it's fine on paper).

However, other course participants have written some clear summaries that helped me get my head around the topic - I like Gordon's post and table .

I think the paradigm that best suits my approach to these things is the Eclectic-Mixed Methods-Pragmatic paradigm. I think this is a practical approach, which is suited to secondary school or further education college situations because time for evaluation is limited (and often not attepmted very thoroughly) and conclusions need to be seen as having relevance to the future development of teaching and learning.

Models
When it comes to evaluation models there are two that strike me as appropriate.

  1. Mutiple methods evaluation
  2. Stake's responsive evaluation
I like the idea of responsive evaluation in so much as the evaluator adapts their methods as the evaluation unfolds, using a range of tools much like the multiple methods evaluation technique. I think this is appealing to me because the reflection is led by the students, teachers or administrators and not by my own enthusiasm for e-learning. Responsive evaluation may lead to a more realistic outcome for me - avoiding a bias I have towards computers!!

I found this site helpful for a break down of evaluation techniques: Evaluation: A practical guide to methods.


Methods
Finally, I read the information relating to methods of evaluation. In the context of my evaluation project the method of formative evaluation appears most relevant. My project will involve looking at learners who are using resources delivered through Moodle during their face to face lessons. Moodle has been available to teaching staff for two years but there are varying degrees of uptake. By looking at the impact on learning across two curriculum areas, some conclusions may be reached on whether these elearning resources are beneficial or not. This would provide feedback during what could be considered a developmental stage in the use of Moodle. In an ideal world I can see that all four methods would be planned for - from analysis through to implementation.

Article
This article from JISC about Learner Experiences of e-Learning is interesting for me as it has some questions that I may consider in my project.

  • How do learners engage with and experience e-learning?
  • What is their perception of e-learning?
  • What do e-learners do when they are learning with technology?
  • What strategies do e-learners use and what is effective?

Thursday, 13 March 2008

eLearning Guidelines for Quality

I have permission to do an evaluation project at a local college. The curriculum covers a wide range of courses from level 1 foundation studies through to level 3 advanced studies (all post-16 age group). In 2005 Moodle was introduced as a college wide virtual learning environment, VLE, (or learning management system) replacing an in-house intranet. All courses in the college have a presence on the VLE although the depth of that presence differs from course team to course team.

I thought I could concentrate on looking at two curriculum areas - probably small enough for me to manage but enough to provide a comparison of usage.

  • Modern Foreign Languages - they seem to take a web enhanced approach whereby students are expected to use a 'virtual language lab' that is provided through Moodle (with additional software that has been linked to Moodle). This is a department that has a leader who is enthusiastic about the benefits of technology and has supported staff through the introduction of new methods of teaching/learning.
  • Business/ICT - they seem to take a web supported approach whereby students are provided with some online learning material through Moodle to support their face-to-face classroom learning. The development/use of material is left to the discretion of each classroom teacher.
Two elearning issues that I would be interested to evaluate are :-
  1. Do students enjoy the 'blended learning' approach? The students are working towards graded examinations that can help them progress to further or higher education and often there is pressure to gain high grades. This pressure can sometimes lead to resentment of any 'new' or 'different' teaching approaches and I'm curious as to how teachers deliver the online part of their lessons and whether the students enjoy those types of learning experiences and feel they are beneficial/helpful. A variety of learning styles can be catered for more easily through eleaning resources?
  2. Do students learn more effectively as a result of 'blended learning'? Is there anyway to tell? For years I have heard collegues discuss how students today bore easily when presentated with traditional teaching approaches and they therefore don't put in the required effort to achieve to their potential - they're de-motivated by face-to-face classroom learning. Has the introduction of elearning resources improved this situation?
The following guidelines all have relevance to the issues identified.
  • SD3 Do students gain knowledge relevant to employment and/or current thinking in their field?
  • SD4 Do students acquire transferable skills such as communication, information and technology skills?
  • SD5 Do students acquire the learning skills for successfully completing the course?
  • ST4 Does the course require students to engage in analysis, synthesis and evaluation as part of their course and program requirements?
  • ST5 Have activities been identified that allow individuals and groups to learn through experience, including opportunities to demonstrate, reinforce knowledge, develop understanding and practice skills?
  • ST6 Are problem solving skills addressed through project and inquiry-based teaching & learning?
  • ST7 Will the e-learning foster students’ curiosity and creativity?
  • ST8 Rather than just recalling facts, does the course help the student digest, reflect on and review new learning?
  • ST9 Do the technologies employed help students successfully meet the learning outcomes?
But I think the two guidelines I could focus on are:-
  1. ST7 Will the e-learning foster students’ curiosity and creativity?
  2. ST9 Do the technologies employed help students successfully meet the learning outcomes?
I like this guideline too:-
  1. ST5 Have activities been identified that allow individuals and groups to learn through experience, including opportunities to demonstrate, reinforce knowledge, develop understanding and practice skills?

How might the guidelines help address the issues?
The guidelines provide questions that help focus on the issues. This should help with the planning of evidence collection and the data/information eventually gathered should lead to conclusions for the two issues raised.

[Image by Richard]

Sunday, 2 March 2008

Evaluation is.......

...a thoughtful consideration of events which can inform future events for the better.

In my experience evaluation is...... a department meeting and a rushed report based on empirical observations by teachers and a review of course results and student responses to a tick box questionnaire! Better than no relection at all, but not always satisfactory.

Evaluation is important in all learning - to enable all students to learn effectively the teacher must be aware of what works and what doesn't. eLearning also has the additional issues that resources are often costly to produce and teachers need to be sufficiently trained to deliver them.

There are so many variables in education that it's improtant to reflect. It's important to know that you're on the right track. What might work with one group or individual may not work with others. What might seem a good idea may not in hindsight be appropriate. Also, it's important to find out what works well so that best practice can be used/shared for future learning.

In teaching I have experienced the following types of evaluation activity:

  • Lesson observations - by peers, by line managers and by external inspectors
  • Student surveys at the mid-point and end of courses
  • Team meetings between course leaders and teachers
  • During external inspections - focus groups with students
  • Moderation of learning and assessment materials
After watching Bronwyn's presentation, and with Flight of the Concords playing in the background (very funny btw), I had an idea for an evaluation project. The college I worked at for 8 years, and where I now do some supply teaching, introduced Moodle a couple of years ago and it would be interesting to see what the impact on learning has been. Some departments use the facilities of Moodle more than others, so it may be a useful exercise to evaluate the effectiveness of its use - useful to me for this course but also useful to the college in judging future development. Next task - find out if they'll let me run a small scale evaluation project!

Tuesday, 26 February 2008

Quick Reflection


I've spent some time this morning updating Google Reader with the sites relevant for this new paper - blogs, wikis etc. I also checked out some of the new blog entries made on sites relevant to the previous paper - facilitating online communities - and found this post "Towards Reflective BlogTalk" by Konrad Glogowski. It's an interesting read for anyone thinking of setting up student blogs in their classroom, as Konrad journals his continuing efforts to improve the learning of his students. Is this ongoing evaluation in practice? I suppose I'll come back to this once I read more about evaluation.

I found some of the comments to Konrad's post typical of many teacher's responses to using new teaching/learning methods in that they focussed on the practicalities of the technology and the classroom environment as a barrier to learning. Konrad's discussion of his successes/failures provides reflection that others can benefit from. This is one of the aspects of blogging that I find really helpful. Just yesterday I read a blog post, by Debbie, reflecting on a learning situation in Second Life and it instantly became a 'starred' item for future reference.

Not sure how this is going to relate to 'formal' evaluation but I enjoy this kind of reflection.

[Photo by r0b1]

Monday, 25 February 2008

Evaluation of eLearning for Best Practice

First blog post of 2008. Happy New Year!

It's the first week of a new paper on the Graduate Certificate in Applied eLearning and the participants are going to be blogging about their learning. So this is where I'll do mine :)

Nothing to report so far - I'm just finding my way around the course wiki and blog. More soon!