An Integrated Approach to Interaction Design and Log Analysis

Logging The User Interaction: An Introduction

A good understanding of people – what they are like, why they use a certain piece of software, and how they might interact with it – is essential for successful design of interactive systems, which help people achieve their goals. While each user is unique, and may have a particular background, context, interest and motivation to use a system, it is necessary to learn what is generally true about the users of a system and what behavioral patterns are common. Specifically, the designer should learn (1) the users’ goals in using a system; (2) the specific tasks undertaken in order to achieve some goals; (3) the language or terminology used by users to describe what they are doing; and (4) the users’ experience and skills at using a certain kind of system (Tidwell, 2006). Some common methods and techniques used before and during system design in order to understand the users’ needs and to establish system requirements, as well as during the implementation and testing in order to evaluate the usability and effectiveness of a system, are direct observation, interviews, surveys, personas, focus groups.

While these methods are excellent tools for evaluating the quality of the interaction between human and system, the quality of the system in supporting the users to achieve their goals and the user satisfaction, they have a number of drawbacks. First, people are often incapable of accurately assessing their own behaviors, especially when removed from the context of their activities (Pinker, 1999) and therefore interviews and surveys may not provide true answers. Second, direct observation may be obtrusive – the users may be distracted, or they may not behave naturally. Third, they are expensive to run, and therefore provide information from a rather limited sample of users, so the results are often informative, but may lack statistical significance, may miss unusual cases, and may not capture behavioral patterns or trends.

Logging the user interaction with the system provides a complementary tool for analyzing the interaction and evaluating a system. It provides the means to acquiring large quantities of data about patterns of interface usage, speed of user performance, rate of errors, or frequency of requests for online assistance (Shneiderman & Plaisant, 2005). An important ethical issue, which indirectly affects user behavior and therefore the validity of the results, is whether users are told and know that their activity is logged. However, when logging is done in order to evaluate a system rather than user preferences or private activities, and when no personal information is captured, this problem is minimal.

An interesting set of constraints on what data can practically be logged, and on designing a logging system, is dictated by the software architecture of the system being investigated. The simpler situation is that of a standalone system, when the entire user activity runs on the same machine, and where all the data resides. In such situations, if the logging module is designed and built as part of the system, then all user actions, all user events and all data being generated or manipulated can potentially be logged. Logging the interaction with third-party software is more challenging: while operating system-level actions such as keystroke or mouse events, or opening/closing a file, or starting/stopping a certain application can be captured and logged, semantic events specific to a certain application are usually impossible to capture. For example, while it is possible to capture the text typed by a user, it is not easy or even possible to determine if the text was typed as a query for a search engine, or for filling in a form. This problem can be addressed by video-recording the interaction or by using screen-capturing software (e.g., Morae: http://www.noldus.com, so that the researchers can subsequently examine the interaction, interpret what is happening, insert annotations or mark significant events. While these tools can be helpful in analyzing the captured data, they rely on the manual-intellectual annotation done by the researcher, and are therefore very labor intensive and error-prone. Moreover, the format used for the logs is usually proprietary, which forces the researchers to buy proprietary analysis software that is not customizable. So, in order to fully benefit from the power of user activity logging, it is preferable that the designer of the logging module has access to the source code of the system being evaluated.