Masters Thesis - Issues of Saliency and Recognition in the Search for Web Page Bookmarks
4. The Experiment
4.1) Participants
30 mainly post-graduate students took part in the experiment, with an average age of 30-35, and an age range of 15 - 65. 12 were female and 18 were male.
All participants had normal or corrected-to-normal vision and were regular users of World Wide Web, with an average of 7 years experience. All but one reported that Internet Explorer was their main web browser. Some had taken part in other eye tracking experiments but none were aware of the research hypotheses in the present study. All were paid 3 for the 30-minute duration of the experiment.
Prior to the 30 participants who completed the experiment, six could not be calibrated with the eye tracker, data from two participants were excluded as the participants did not follow the instructions and a further four were eliminated to ensure that the remaining sample of 30 contained only native English speakers.
Most of the participants reported that they had never seen the websites used in the test, although six participants stated that they were familiar with one or two of the websites, but didn't use them regularly.
4.2) Materials and design
A 2x3 within subjects design was used, the first factor being the bookmark structure (either top-down or bottom-up), the second factor being the number of information components, or 'cues' available in the bookmark (One, two or three cues) (Table 4).
| Bookmark Structure | Number of cues | ||
|---|---|---|---|
| 1 | 2 | 3 | |
| Top-down | a) Site name | c) Site name - Article title |
e) Site name - Section name - Article title |
| Bottom-up | b) Article title | d) Article title - Site name |
f) Article title - Section name - Site name |
The format of the experiment was straightforward - participants were asked to view a series of websites then find its corresponding bookmark in the menu that followed each time.
A set of 24 web pages containing articles on international news and current affairs were collected from news websites and saved as screenshots (Figure 10) (See Appendix A & Appendix B for a full listing). The chosen web pages all had a clear site name, article title and section name, ensuring equal opportunity of encoding for later recognition.
As they were static screenshots, the body text of the article was often not fully in view, and participants were not able to 'scroll down' to read the rest of the news story. The original title bar text was deleted from each website screenshot to allow the full manipulation of the bookmark text (this is necessary as the title bar and bookmark text is the same, as mentioned in section 1.2.
Figure 10
One of the news websites used in the test: Note that the title tag text has been removed from the browser's top bar to prevent it from clashing with the manipulated bookmark text in the search task.
For each website, a corresponding set of screenshots were created of Internet Explorer 6 with the 'favorites' menu displayed, The bookmark corresponding to the web page was located somewhere on the menu (Figure 11).
Lastly, a questionnaire was prepared to collect demographic data (Appendix D).
The experimental conditions were distributed so that websites 1-4 were followed by bookmarks of condition type 'a' (site name alone), websites 5-8 were followed by bookmarks of condition type 'b' (article title alone), websites 9-12 were followed by bookmarks of condition type 'c' (site name and article title), and so on (see first row, Table 5).
Figure 11
A bookmark menu screenshot used in the test.
The presentation order of the websites was counterbalanced so that all conditions occurred equally, and further randomised to eliminate fatigue and practice effects. Lastly, participants were randomly assigned to one of six groups (Table 5), and completed 24 trials, one for each website/bookmark menu combination.
| Group | Website 1 - 4 | Website 5 - 8 |
Website 9 - 12 | Website 13 - 16 | Website 17 - 20 |
Website 21 - 24 |
|---|---|---|---|---|---|---|
| 1 | a | b | c | d | e | f |
| 2 | b | a | d | c | f | e |
| 3 | c | d | e | f | a | b |
| 4 | d | c | f | e | b | a |
| 5 | e | f | a | b | c | d |
| 6 | f | e | b | a | d | c |
The bookmark conditions for each trial were manipulated by taking six screenshots (one for each condition) of the bookmark menu in Internet Explorer. Across all six menu screenshots, the distractor bookmarks were identical, the target bookmark was in the same location but the format was different, depending on the experimental condition (Table 4).
The order of the distractor bookmarks in each menu were changed haphazardly for each trial as well as one or two bookmarks being substituted for new ones each time (See Appendix C for typical set of distractor bookmarks). The target bookmarks appeared once in each position in the menu from number 4 to 27, with 29 bookmarks in the menu each time.
4.3) Apparatus
The website screen shots were presented on a 15 inch flat screen monitor, with a screen resolution of 1024 x 768 pixels.
Eye movements were recorded with an LC Technologies Eyegaze development system. The Eyegaze eye tracker consists of a standard desktop computer running Windows NT/2000, an infrared camera mounted beneath the monitor (Figure 12) and software to process the eye movement data ('Eyegaze development system').
An additional smaller monitor was used to ensure that the eye was in the centre of the camera's field of view. The Eyegaze system determines the eye's gaze direction by the pupil-center/corneal-reflection method. A small LED at the center of the camera lens directs infrared light into the eye, causing a reflection in the cornea and increasing the brightness of the pupil to make it more easily identifiable (Figure 13).
Figure 12
The eye tracker used in the present study: a desktop computer with an infrared camera mounted beneath the monitor.
Figure 13
The 'bright pupil' effect shown in the eye shown in the infrared camera monitor.
Image processing software is then used to identify and locate the centres of the pupil and corneal reflections. Finally, the gaze point (the coordinates of where the person is looking on the monitor) is found by computing the angle between the corneal reflection and the centre of the pupil.
The eye tracker is accurate to within 0.45 degrees of visual angle, which at 51cm from the screen covers approximately 3.8mm. This corresponds to 12.8 pixels on the monitor used in the test, which had a dot pitch of 0.297mm. Eye movements were sampled 60 times a second, with tracking errors not exceeding 6.3mm.
Although the eye tracker can tolerate head motion of around 3cm in all directions, participants were asked to use a chin rest (Figure 14) to minimise loss of eye movement data. A small wad of tissue is placed in the chin rest to improve comfort, certainly necessary for sessions lasting longer than a few minutes.
Figure 14
A chinrest is essential in keeping head movements to a minimum in order to maintain tracking of eye movements.
Fixations were detected at 100ms or above, an appropriate cut-off point for tracking the movement of the eyes in reading tasks (Hyönä, Niemi & Underwood, 1989; Inhoff & Radach, 1998).
Finally, a monitoring console similar to those used in lab-based usability evaluations was used to observe the participants during the main part of the experiment (Figure 15).
Figure 15
Monitoring console equipped with three CCTV cameras, used to observe participants.
4.4) Procedure
On arrival, participants were shown the monitoring console and told that it would be used by the experimenter to monitor the progress of the test while keeping a distance from the participant, so as not to distract them or make them feel self-conscious during the test. It was further explained that none of the video feedback would be recorded (none of the participants objected to this arrangement).
Next, the participants were shown the eye tracker and given a brief explanation of how it worked and why it was necessary to use the chin rest. Participants were then helped to get comfortable for the duration of the test by making appropriate adjustments to the chinrest and the monitor to accommodate individual variations in seated head position. At all times, approximately the same viewing angle between the face and the screen was maintained. Participants were seated on average 51cm from the screen.
Once the participants were comfortable in the chin rest, the camera was adjusted vertically and the participant was asked to move slightly to the left or right so that one of their eyes was in the centre of the camera's field of vision. Lastly, once the camera's focus and aperture was set, the participant was calibrated with the eye-tracker.
The calibration procedure lasts 15 seconds and consists of the participant following a series of 9 dots around the screen with their eyes, starting in various locations. Through this, the system can accurately plot the person's gaze point. Once this profile of the person's eye has been captured, there is no need for them to be calibrated again, even across different test sessions.
In the present study, six participants could not be calibrated due to low contrast between the eye and pupil, large pupils being partially obscured by the upper eyelids, eye reflections being distorted by super-compressed lenses, and partially obscured pupils caused by 'lazy eye'.
Next, custom software was launched which presented participants with on-screen instructions and the experiment itself. After reading the instructions, (Figure 16) the participants completed 4 practice trials while the experimenter sat beside them to answer any queries.
Figure 16
The main instruction screen.
Extra care was taken to check that the participants understood what they had to do before proceeding to the main session. The experimenter went to the far side of the room behind the monitoring console and left the participant to complete the main session without distraction.
Each website appeared on screen for 18 seconds, with each bookmark screen appearing for a maximum of 30 seconds. Participants pressed the space bar on the keyboard to indicate that they had found the target bookmark. If they could not find the target within 30 seconds, the trial ended and the next trial began. Once the participant had reached the end of the main session (lasting around 15 minutes) they were given the questionnaire to complete (Appendix D).
Many significant experimental design issues were solved through a thorough multi-iteration piloting phase.