ICT in Education

Thoughts on: Evaluating online learning: Models and methods (Gunawardena et al, 2000)

October 17, 2007

Gunawardena, C., Lowe, C. & Carabajal, K. (2000). Evaluating online learning: Models and methods. In C. Crawford et al. (Eds.), Proceedings of Society for Information Technology and Teacher Education International Conference 2000 (pp. 1677-1684). Chesapeake, VA: AACE

This one is about methods for evaluating the success (or otherwise) of online learning activities and forms the basis of our final assignment in Network Based Learning (EDGI915) – where we are meant to examine two of the NBL activities we have undertaken over the last 12 weeks and see if it’s up to scratch.

I suspect that this is going to be trickier than it looks as none of the activities on first glance seem to have been overly successful if you look at them based on how much interaction there was between the students – hopefully this paper will offer some useful suggestions for breaking the activity down (as well as the interactions) and seeing why this might have been.

“Online learning designs are often based on constructivist learner-centred principles which provide more learner control, facilitate the sharing of multiple perspectives and places emphasis on individual learners creating their own meaning.”

“Behavioral objectives with a stated outcome for all learners, is not the goal of many online learning projects.

Evaluation questions:

1. How can we describe online participation, interaction patterns and group dynamics?

2. Were students satisfied with the experience of participating in the conference? Did they feel that it had been a worthwhile use of their time and one they would be willing to repeat?

3. Did participants learn?

4. Was knowledge constructed?

An additional issue to consider is that of: How is knowledge constructed in online learning networks through the process of social negotiation?

1. How can we describe online participation, interaction patterns and group dynamics?

“Participation analysis techniques examine the capacity of a conference to engage members and reveal comparative patterns of participation among learners from varying backgrounds. An evalutation tool we have employed to address these questions is the model developed by Levin, Kim and Riel (1990) for analysing instructional interactions on electronic message networks. This model has four dimensions of analysis: 1) Participant structures analysis, 2)Intermessage reference analysis, 3)message act analysis and 4) message flow analysis”

These methods didn’t suit their particular context, that being more of a community of practice type scenario. Message act analysis doesn’t consider the content of the messages, which didn’t help either. What did help was “we found ther data such as unsolicited participant reactions online, solicited participant reactions, both instructor and student perspectives on interaction and the analysis of computer transcripts to be more useful in forming a picture of events that occurred in the online community (Gunawardena, 1993)”

2 Were learners satisfied with their online learning experience?

“Hiltz (1990) discussed an approach to determining learner satisfaction by examining the social psychological (characteristics of the users); human relations (characteristics of the groups and organisations within which systems are implemented); and technological determinist (characteristics of the system); factors that impact student satisfaction with and subsequent use of computer conferencing. “

Generally speaking, Gunawardena (et al) found that “social presence alone is a strong predictor of satisfaction in a text-based computer conference”

3. Did participants learn?

Gunawardena et al assessed this by looking at the transcripts of text chat/discussion boards and by directly asking the participants their opinions.

“In order to understand the myriad forms of learning that occurs in a computer conference, we have often asked students to keep weekly journals documenting all aspects of learning. These journals have given us a unique perspective of each individuals learning process. Other techniques we have used are to ask students to critique their online learning experiences and to apply and transfer what they have learned from the computer conference to developing a computer conferencing design”

4. Was knowledge constructed?

“Perhaps the most challenging and the most exciting question one can ask in evaluating online learning is: Was knowledge constructed within the group by means of the exchanges among participants?”

“We developed an outline of the process of negotiation which appears to occur in the co-construction of knowledge. The outline led to the development of the interaction analysis model which has five phases, reflecting the complete process of negotiation  which must occur when there are substantial areas of inconsistency or disagreement to be resolved. The phases of learning outlined in this model occur at both the individual and social level and can be described as:

Phase 1: Sharing/Comparing
Phase 2: Dissonance
Phase 3: Negotiation/Co-construction
Phase 4: Testing tentative constructions
Phase 5: Statement/application of newly-constructed knowledge”

Conclusion

“The complex nature of online learning calls for the use of multiple methods and multiple sources of data to understand group as well as individual learning”

“When used in conjunction with quantitative data, qualitative data can overcome some of the shortcomings of using quantitative data alone”

So ultimately, this paper doesn’t say a lot – great.
(At least some of the questions it raises are a little interesting – if not overly helpful) 

Entry Filed under: 915, analysis, computer mediated communication, evaluation. .