Here is my presentation summarising my evaluation plan. It's about 6 minutes long and took me hours! Had a couple of technology glitches which slowed me down and then spent ages re-doing the audio - not used to hearing myself present and spent a lot of time 'umming' and 'errring'. I then discovered the cut button in Audacity (the audio software I used) and spent a while cutting out the 'errs' and 'ums' - although got bored of this so there a still a few in there!! I suppose this is another online skill that will improve with practice.
Friday, 23 May 2008
Summary of Evaluation Plan
Subscribe to:
Post Comments (Atom)
12 comments:
Yvonne, this is Lei from myplick. We fixed the problem for you. Thanks for using myplick.
Marvellous isn't it how the community helps. Thanks Lei you solved the prob so quickly.
Yes it does work now.
I like the presentation Yvonne, it was a bit of fun to produce it in the end, don't you think?
I gathered from your plan that you are focussing on the student's perspective. One of the criticisms that has been levelled at Moodle in times past was that because it is so intuitive to use, the students usually seem have positive things to say about it. However the facilitators have not always been so kind. I am not sure what the most recent incarnation looks like but such things as sharing content, document management and enabling groups have all received bad press from them.
I am wondering why you have chosen not to look at the course grades as an indicator for whether the students are meeting the learning outcomes. Maybe time? Isn't that the whole point of assessments, to test whether the learning has been done? There are all sorts of discussions that you could have around the issue of assessment, but it could provide corroborating evidence perhaps. What would you conclude if the students thought that they had achieved the learning outcomes but they had all done poorly in the test? Interesting minefield perhaps.
Gordon
Yvonne your presentation brings your evaluation plan alive. It is just the right length and is very interesting.
As Gordon says it would be good to look at student's grades to match against their perceptions of whether they met the learning objectives.
However this could be done by the teachers once you have shown them the report of your evaluation. For a small project like yours with the time constraints imposed by the project, it is not feasible to do everything.
A question: If students do not believe that Moodle/online learning helps them to meet the learning objectives, is there any point in looking at their grades?
so your project is the first phase in evaluating the worth of Moodle.
Hi Yvonne
I enjoyed your presentation, it was very interesting. I think perhaps I should have used audacity as well - it appears quite user friendly. Following on from the comments made by Gordon and Bronwyn, I must admit that I picked up on the "evidence of improved achievement" comment. But what it made me think about was just how far we go with say, formative evaluation. How much information can be gleaned with one set of survey results and meet the needs of stakeholders along the way? Then there is the adjustments as a result of one evaluation leading onto another study. Gordon has also mentioned this in his evaluation plan. A never ending task if you let it I suppose.
Hi All
I've not turned the computer on since Friday and look what happens! Someone fixes a tech issue (thanks Lei) and I get lots of questions to ponder.
Gordon, an interesting point about Moodle and facilitators. On one of my recent days in college I offered to set up a quiz for a History teacher (in Moodle). It did take quite a while to set up as there were a huge number of options for each multiple choice question but managed it. I put a link to it from the History page and thought all was good but then when I logged in as a student it couldn't be seen. I worked out I had to select every History student as 'enrolled' in the 'quiz' before they could see it! Irritating, time consuming and possibly beyond the 'average' IT ability of most. But anyway - chose to focus on students as I think teachers will be more likley to give Moodle a go if the student response is a positive one.
As for course grades, I just don't think a project of this scope would make looking at grades worthwhile. I'm not sure you could make any safe correlation. I think if I was studying a class over time with some significant portion of learning undertaken with Moodle resources then links to grades could be asserted. Taking such a brief snapshot leaves room for lots of variables that could impact on grades. Am I way off here? Did I pick the wrong evaluation question?
I think, as Hilary suggests, that like Gordon, further studies could lead from any results from this minor one.
Thanks for your comments and making me think!!
Cheers
Yvonne
Bronwyn poses the question: "If students do not believe that Moodle/online learning helps them to meet the learning objectives, is there any point in looking at their grades?"
What would you conclude if they did not believe that the on-line learning helped them achieve the learning objectives but their assessment results were better than with f2f tuition? Could personal bias be a factor?
Hi Gordon
It's an interesting point. What would I conclude? I'm not sure I would be happy to make a conclusion without a lot more information.
* Are exam result comparisons valid between purely f2f and blended learning groups?
* Are there other factors that could affect results - class size, group dynamics, initial student scores, teacher confidence/skill with technology?
If students don't feel they achieve effectively then that could lead to many questions about how the technology is being blended with f2f learning. As a resource I think online learning can support effective learning but not for all topics/subjects and so perhaps further investigation in how the tools are used is needed. I know when I first was interested in elearning there was no information about pedagogy/theory but lots about what the tools/software could do. If students don't enjoy the use of those tools, perhaps they're not being used effectively to support learning.
Or perhaps other f2f methods are what the students prefer. That's always an option - not all youngsters have to like computers.
Yvonne
Yvonne I have to agree the project is too small to get meaningful data regarding grades. I don't think you have picked the wrong question.
In response to Gordon's question "what would you conclude if they did not believe that the on-line learning helped them achieve the learning objectives but their assessment results were better than with f2f tuition? Could personal bias be a factor?"
Probably the other variables Yvonne mentioned I would think. And better grades unfortunately does not always indicate a deep level of learning - it depends on how the assessments are designed. So I am dubious about grades as a key factor in measuring effectiveness. You have made a very good point all the same Gordon and generated some interesting discussion.
An interesting question Yvonne - "Are exam result comparisons valid between purely f2f and blended learning groups?" They can be if you account for the variables but it is pretty subjective and depends on the research design.
Personally I feel that looking at grades is just one of the tools in an evaluation. In the early days of elearning researchers focussed on grade comparisons between f2f and online learning.
They used scientific experimental models of research design with control groups. Little evidence was shown that one mode was better than the other - but this is not the issue is it?
What is more important as you are doing is finding out if students are using the tools, how they are using them and if they are learning through using them. Enjoyment is a big factor in learning.
As you say the "if students don't enjoy the use of those tools, perhaps they're not being used effectively to support learning."
HI Yvonne,
thanks for the nice presentation. It was very clear, and never mention the uhm anymore, because it's like a pink elephant.
Looking at grades and assessments results is at least an indication of the effectiviness of learning (whether or not you can compare it with other learning formats).
What we normally do in f2f training is let the participants fill out question how well they think they can perform on certain tasks and after the course they do it again. This is based on their interpretation but gives an idea of the before and after. I tried to do the same for the evaluation of the online course, only they had to fill out both questions at once. If you combine this outcome with questions like appreciation, you go a long way. But I do agree that small samples might give you a bias ed picture.
Have fun with the data collection.
Helga
Hi Helga
Thanks for your comments (you might have to explain 'pink elephant' as I've not heard that one before!).
You're right that looking at grades gives an indication of "effectiviness of learning" but I would still maintain that in a classroom situation there are many variables that lead to learning - not just the 'technology' we are evaluating. I think the evaluation tools would need to be far more sophisticated to take account of the different variables to get close to a valid interpretation of how effective elearning has been.
I appear to be developing a skepticism for evaluation!! Not really, just a healthy concern that I don't jump in with both feet and make unjustified conclusions?!
Thanks again,
Yvonne
Hi Yvonne - apologies for my slowness in responding to your presentation. First up - as i have commented on my blog - I'm hugely impressed with your and other presentations. Am blown away that you managed to cut out all your ummmms and ahhhhs! I'm curious whether you have ended up interviewing students and how this has gone? How many etc... Am just debating whether to include this in my evaulation also? Interesing comments re-grades and i agree it would be difficult to isolate the factor that attributes a students grade in a study of this size! Rika :]
Post a Comment