As we’ve grown we’ve noticed the increasing importance of working with criteria. Academia excels at establishing and debating criteria, frameworks and models – continually revisiting and revising criteria to fit evolving needs.

However, as is often the case, many of these criteria are not fully exploited in commercial projects where time, efficiency and results are the driving variables, not to mention the mild fear that arises when anyone mentions the word “criteria” or “framework.”

Why do people so often shy away from using criteria?  Perhaps in part it is because we don’t understand how to interpret them or employ them. Maybe we fear that incorporating criteria in R&D will be prescriptive and thus restrictive to our creativity. Well, it really depends on how we understand the role of criteria and how we use them.

Recently we have raised our awareness of the value of using criteria in the continual design and development of the Idapted online language learning system (OLLS) and developed a better understanding of how we integrate criteria into how we do things, and how we make criteria work for us.

However, caution must be exercised.  Criteria can be taken too seriously, as a carved-in-granite stone tablet – a rule that everything must fit into -like when a company employs a zealous new HR manager who insists on implementing the most complex, systemic, up-to-date performance evaluations criteria upon reality, rather than letting reality help shape it in a two-way process.

This usually happens when someone enthusiastically wants to ‘academically’ back something, discovers a host of criteria, and so excitedly tries to get everyone on board to crow-bar reality into those criteria!

What we have clearly found is that criteria should be seen as guides. They are most valuable as measuring sticks that can be used as aids in the design and evaluation of a learning system (or an aspect of a learning system) and as means to ‘test’ its effectiveness as part of the feedback loop.

Whilst there are not essentially any specific criteria for an online language learning system with integrated live teacher, there are a number of criteria that we’ve found exceptionally useful in designing the Idapted OLLS and the lessons delivered through the system. Whilst originally devised for CALL software evaluation, their application arguably still has immense value for any online language learning system developments. Furthermore, there is no reason why these criteria cannot be used prior and during the design process.

Hubbard (2006) discussed three types of evaluation criteria for CALL: checklists, methodological frameworks, and Second Language Acquisition (SLA) based Approaches. We’ve found each of these three types can be used ‘together’ as a way of viewing the development of learning materials online, from 3 slightly different angles.

i) Checklists

In many ways checklists are what the name suggests: lists of questions that you can scan through quickly and answer ‘yes’ or ‘no’.  In this way you can look through in order to check that things are being done / have been done / will be done without missing key points.

Although we’ve found that checklists are often more useful after something has been developed, the checklist can also be used during the design process as a constant reminder so that nothing is neglected.

On occasions we’ve used the below checklist by Kevin Kruse as a general guide:

• Does the program immediately capture attention when it is run?
• Does the training program explain its relevance to learners? Does it answer the student’s question, “What’s in it for me?”
• Are learning objectives presented near the beginning of each lesson?
• Are learning objectives specific and observable?
• Is the presentation of content made interesting with a variety of media or through an engaging treatment?
• Does the program provide a variety of interactive exercises beyond simple multiple-choice questions?
• Are learners given the opportunity for frequent practice?
• Is feedback on practice immediate and specific?
• Does the program include a post-test or other final assessment?

(Kruse, 1997)

We found that putting the checklist in table format with a comments box to the right makes the checklist much more useful. The checklist-plus-comments from various people involved in the design process can be passed on, and it can form the basis of a general specification sheet.

Whilst a checklist is to some extent limiting as the only means of evaluation, combined with other frameworks, and modified accordingly, we’ve found that a checklist can be an excellent way of quickly ensuring that everyone in the design process is aligned with the learning objectives we seek to achieve.

ii) Methodological Frameworks

A framework is rather different from a checklist in that it is not judgmental but descriptive. We’ve found that for this reason it ‘gets us thinking’ more. Hubbard (1988) came up with the following CALL software evaluation framework: Operational description, teacher fit, learner fit.

Here’s a very simplified explanation:

Operational description: How the user can use the system, the flow of pages, lessons, screens, activities and their ‘usability’.

Teacher fit: How the system is designed to deliver the lessons (both in the Self Study and in the Interaction) that fit the desired pedagogic approach or teaching methodology.

Learner fit: How the system fits the ‘target learner’ in terms of meeting the objectives of the syllabus at the same time as meeting the learner’s social, cultural and learning objectives, as well as learning styles and preferences.

This framework was designed for a teacher to evaluate CALL software prior to using software for teaching. This is why operational description precedes teacher fit and learner fit. We’ve recently moved on to use a later framework, (Hubbard, 1992) in which development, evaluation and implementation are combined. I’d like to discuss this in more detail in a subsequent post.

iv) SLA –based approaches

SLA-based approaches focus more specifically on the language acquisition angle of looking at course evaluation.

Chapelle’s framework, at first glance, is very similar to Hubbard’s framework above, but is more explicit in its focus on the SLA considerations.

Language learning potential: The degree of opportunity present for beneficial focus on form.

Learner fit: The amount of opportunity for engagement with language under appropriate conditions given learner characteristics.

Meaning focus: The extent to which learner attention is directed to the meaning of the language.

Authenticity: The degree of correspondence between the CALL activity and target language activities of interest to learners out of the classroom.

Positive impact: The positive effects of the CALL activity on those who participate in it.

Practicality: The adequacy of resources to support the CALL activity.

(Chapelle, 2001)

We’ve found it useful to employ Chapelle’s framework as a guide in lesson development because it encourages us to look closely and specifically at the aspects of language that the learners need to learn and how effectively the CALL software, system or activity, meets those needs. We use Chapelle’s framework to examine and build lessons and tasks within a course, rather than evaluate the system as a whole.

There’s a lot to be said about how criteria can be understood, adapted, studied and employed in R&D of instructional products and systems as technology and pedagogy evolve.

The aim of this post has been to provide a general introduction to how Idapted started to look at criteria in CALL evaluation, and to give an overview and some examples of the 3 main types of evaluative criteria that can be used as instructive guidelines in development and evaluation of the Idapted OLLS and lessons delivered through it.

In further posts I’d like to delve a little deeper into criteria and how we are currently beginning to pay more and more attention to the various types of criteria used in design, development, evaluation and implementation of instructional systems and materials.

References

Chapelle C. (2001). Computer Applications in Second Language Acquisition. Cambridge: Cambridge University Press.

Hubbard P. Evaluating CALL Software Chapter 13 from Ducate and N. Arnold (eds) 2006. Calling on CALL: From Theory and Research to New Directions in Foreign Language Teaching.

Kruse, K. e-learning guru (accessed 20 August 2010) available here.