Evaluation Techniques for Interactive System

What is Evaluation?

Evaluation is the process we use to assess our system to ensure it behaves as expected in the user requirement. This process not only happens after development, we should use this throughout the development process. Ideally, evaluation should occur throughout the design life cycle, with the results of the evaluation feeding back into modifications to the design.

Goals of Evaluation

The evaluation has three main goals,

1. Assess the extent and accessibility of the system's functionality
Design of the system should enable users to perform their intended tasks more easily, this is not only about appropriate functionality available in the system but making it clearly reachable by the users.

2. Assess user's experience of the interaction
This includes assessing how easy the system is to learn, its usability, and the user's satisfaction.

3. Identify any specific problems with the system
These may be aspects of the design which, when used in their intended context, cause unexpected results, or confusion amongst users. It is specifically concerned with identifying trouble-spots which can then be rectified.

Evaluation through expert analysis

We can collaborate with users to evaluate our design at every stage. But it is more expensive and time-consuming. And also it is less effective when evaluating half-developed designs with users.

So we can involve experts (designers, human factor experts,..) to assess the design upholds accepted usability principles.

We can achieve this expert analysis evaluation in three different approaches.

1. Cognitive walkthroughs

This is a usability evaluation method in which one or more evaluators work through a series of tasks and ask a set of questions from the perspective of the user.

This process will need,

• A specification or prototype of the system
• A description of the task the user is to perform
• A complete, written list of the actions needed to complete
• An indication of who the users are

Given this information, the evaluators step through the action sequence to critique the system and tell a believable story about its usability.

1. Is the effect of the action the same as the user's goal at that point?
2. Will users see that the action is available?
3. Once users have found the correct action, will they know it is the one they need?
4. After the action is taken, will users understand the feedback they get?

Evaluators will keep records during the cognitive walkthrough of that is food and what needs improvement in the design.

2. Heuristic evaluation

Heuristic evaluation is that several evaluators independently critique a system to come up with potential usability problems.

We can use the following Nielsen's ten heuristics to assess the design of the system.

• Visibility of system status
• Match between system and the real world
• User control and freedom
• Consistency and standards
• Error prevention
• Recognition rather than recall
• Flexibility and efficiency of use
• Aesthetic and minimalist design
• Help users recognize, diagnose and recover from errors
• Help and documentation

3. Model-based evaluation

Model-based evaluation is combining cognitive and design models to the evaluation process.

Examples,

• GOMS model
• Keystroke-level model
• Design rationale
• Dialog models

Evaluation through user participation

Evaluation through expert analysis is useful for filtering and refining the design, but it is not a replacement for actual usability testing with the people for whom the system is intended.

User participation in evaluation tends to occur in the later stages of development when there is at least a working prototype of the system in place. This may range from a simulation of the system’s interactive capabilities, without its underlying functionality.

We can use empirical or experimental methods, observational methods, query techniques, and physiological monitoring methods to conduct evaluations with user participation.

Styles of evaluation

Before considering the evaluation techniques, we will distinguish two distinct evaluation styles.

1. Evaluation performed under laboratory conditions
2. Evaluation performed in the work environment

Empirical methods: experimental evaluation

One of the most powerful methods of evaluating a design or an aspect of a design is to use a controlled experiment. This provides empirical evidence to support a particular claim or hypothesis. It can be used to study a wide range of different issues at different levels of detail.

Any experiment has the same basic form. The evaluator chooses a hypothesis to test, which can be determined by measuring some attribute of participant behavior. A number of experimental conditions are considered which differ only in the values of certain controlled variables. Any changes in the behavioral measures are attributed to the different conditions. Within this basic form, there are a number of factors that are important to the overall reliability of the experiment, which must be considered carefully in experimental design. These include the participants chosen, the variables tested and manipulated, and the hypothesis tested.

Observational techniques

A popular way to gather information about the actual use of a system is to observe users interacting with it. Usually, they are asked to complete a set of predetermined tasks, although, if the observation is being carried out in their place of work, they may be observed going about their normal duties. The evaluator watches and records the users’ actions.

Query techniques

Another set of evaluation techniques relies on asking the user about the interface directly. Query techniques can be useful in eliciting detail of the user’s view of a system. They embody the philosophy that states that the best way to find out how a system meets user requirements is to ‘ask the user’. They can be used in the evaluation and more widely to collect information about user requirements and tasks. The advantage of such methods is that they get the user’s viewpoint directly and may reveal issues that have not been considered by the designer. In addition, they are relatively simple and cheap to administer.