Masters Thesis - Issues of Saliency and Recognition in the Search for Web Page Bookmarks
7. Discussion
7.1) Was there a difference in saliency?
Response times
Faster response times were taken as indicating superior recognition when participants were searching for a target bookmark within a set of distractor bookmarks. There was no significant difference in the time it took to find either type of bookmark (top down or bottom-up), meaning that in a broad sense, they were both equally salient.
However, the number of cues on display was a significant factor in recognition. Two cues were found to be the optimal length for a bookmark, while one cue was clearly inadequate. Adding a third cue did not bring any significant benefit in terms of recognition. This is most likely to due to the 65 character limit on the bookmark menu - only some or none of the third cue may actually have been visible, negating its usefulness.
This said, there are subtle hints in the response times that top-down bookmarks were much more sensitive to the existence of extra cues than were bottom-up bookmarks. The bottom-up bookmark with one cue had the slowest response time out of all the conditions, but decreased sharply to the fastest response time overall when a second cue was added. This does indicate that the site name may be relatively less salient than the article title - The site name appears to 'need' extra information to spark the same level of recognition that the title of the article can attract on its own. Fortunately, the eye movement data permit us to explore this subtle effect in more detail.
Number of fixations
The assumption in the present study was that higher fixation frequency in a visual search task indicates uncertainty in recognising targets (Jacob & Karn, 2003). This certainly appears to be case for bookmarks with bottom-up structures, which were fixated more frequently than top-down bookmarks, regardless of the number of cues. This clearly shows that bottom-up is more salient than top-down as a bookmark structure.
As was hinted at by the recognition times, the site name was far more sensitive to the existence of extra cues than was the article title, and as before, these extra cues had a diminishing marginal benefit.
Fixation duration
Longer fixation times on particular cues indicate that they are less meaningful (Goldberg & Kotval, 1999). The pattern of fixation durations on the leading cues strongly suggest, as with fixation frequency, that the name of the site is less salient than the name of the article, and by extension, that top-down is less salient than bottom-up as a bookmark structure.
When viewed in isolation, the name of the site was less salient in absolute terms, as it was fixated for far longer than was the article title.
Fixation durations on the title of the article were unaffected by extra information, indicating that it was 'salient enough' with or without extra information. The addition of a second cue to site name reduced the fixation time by quite a large margin, indicating that the site name was much more meaningful when processed in the context of the extra information.
Encoding
When we consider the encoding phase of the test, when participants read through the website in order to identify its bookmark, we saw that site name actually received slightly greater attention than the article title. It was fixated more frequently and for longer on average, serving as further evidence that site name is less salient. Despite being subject to more potentially more encoding, the name of the site was still not as salient as the article title.
7.2) What factors lead to greater saliency and recognition?
Semantic value
Information that can be characterised in terms of existing knowledge structures, or 'schemata' are easier to remember (Alba & Hasher, 1983; Bartlett, 1932). Article titles tell stories of international events, which should have greater potential for being remembered in terms of what people already know about the world (eg, a terrorist attack may always be linked to certain countries in a schema), rather more than an abstract site name. By extension, this should be true of most well-formed page titles that refer to rich content.
Meaning is essential if we are to remember something effectively (Rumelhart & Norman, 1985). The article title tells a story, a scenario - it has a strong intrinsic meaning. Site names however, at least for news websites, probably have a lower capacity for rich meaning as they are simple, yet abstract names unconnected to the news stories they provide,.
Imagery evoking potential
Imaginable and concrete items can be easier to remember as they are represented more richly in memory (Paivio, Yuille & Madigan, 1968). Article titles tend to have more imaginable, concrete words that the site name, which can often be rather abstract, so an advantage in recognition value may arise from this difference. Highly imaginable words are encoded in both verbal and visual channels, and can be recognised more effectively due to the inferential connections that are established between them. Words that are less imaginable may only be encoded verbally, so unable to take advantage of this inferential power in recognition (Clark & Paivio, 1987).
At a broader level, the whole article title tells a story, so for this simple reason it may evoke more imagery than the name of a mundane news website, committing it to memory more effectively as in a mnemonic process. (Clark & Paivio, 1987).
7.3) A critique of the experimental method
Errors
The 'absolute drift' and 'relative warp' that existed in the eye movement data was a great concern and does raise the question of whether the results are valid. However, exhaustive measures were taken to ensure the accuracy of the data. The error factors were well known and every piece of data was checked by hand, potentially ensuring a greater level of accuracy than would have been achievable with a fully automatic analysis.
One solution to reduce the error rate would have been to recalibrate the participants at several times during the test, as some researchers advise (Stampe, 1993). This is indeed an excellent way to ensure data accuracy but it is likely to introduce more artificiality into the test situation. Participants would have been made even more aware of their eye movements, potentially exaggerating trends in the data.
Word-level effects
Readers fixate longer on low frequency words than on high frequency words (Hyönä & Olson, 1995; Inhoff, 1984; Rayner & Pollatsek, 1989), and this may have accounted for some of the longer fixation duration on the site name than the article title.
As the websites in the test were all authentic examples, it is of course true that the bookmark elements were not balanced for word frequency (nor for syntactic difficulty or word length). An informal analysis of the wording used in the target bookmarks (Appendix A) showed no particular extremes in vocabulary.
Furthermore, since the average length of the article title was approximately 7 words and the site name approximately 3 words, there was by definition a greater probability of lower frequency words appearing in the article title.
Two site names longer than one word were presented in the bookmarks as domain names ("startribune.com" and "bostonherald.com"). Since these forms lack spaces between the words, they may have been harder to read, thus affecting fixation frequency and duration (Rayner, Fischer & Pollatsek, 1998). However, this effect was only reported in the context of at least sentence-length pieces of text, not simple two-word combinations. Furthermore, people are very much used to reading domain names without spaces on the web, so we would expect very little interference in word identification.
Lastly, one site name ("THE PRESS") was presented in upper case on its bookmark, while the distractor bookmarks were presented in a mixture of lower case and title case. This may have artificially decreased recognition time, as this difference in appearance is sufficient for parallel processing of the target bookmark, while the distractor bookmarks are still processed serially (Nakayama & Silverman 1986). This is the same principle that allows for superior recognition of custom icons on the bookmark menu.
Miscellaneous critiques
The fact that a participant could not find a bookmark does not necessarily mean that they couldn't recognise it - people sometimes do 'skip' menu items (Byrne, Anderson, Douglas & Matessa, 1999). An informal review of the results during the manual error correction found that this did sometimes happen on random search and even with systematic search, but as this happened on only 11 out of a total of 1376 trials, occurrences were too low to affect the overall means.
A fixation on the far right of an element in the bookmark may include parafoveal processing time for the next element (Inhoff & Radach, 1998), thus wrongly attributing some fixation time. However there is some evidence that this may not be the case (Henderson & Ferreira, 1993).
Websites were not balanced for the size of the site name logo - most were smallish and in the top left corner but some were very large and in the middle of screen (see Appendix B), thus potentially being much more salient. However, correlation results confirm that logo size was not a factor in recognition.
Participants were mot requested to reset their gaze point before each search task, as some researchers advise (Zelinsky & Sheinberg, 1995). This meant that participants were looking somewhere other than the top of the bookmark menu as they would if they had activated the menu in a real task. Participants' gaze often fell directly on the menu, potentially leading to more chance bookmark finds.
The encoding task in the experiment was slightly different compared to a real life web browsing situation. Here they had to pick up elements that would identify the website, but when browsing the web, you are likely to look at things that interest you and you may not consciously try to remember elements to identify later on. The encoding of the website is not motivated by a desire to return to the site, but only by the demands of the experiment.
This is where a longer-term ethnographic study may be more appropriate, as it would reduce much of the artificiality inherent in lab-based studies.
Similarly, encoding and recognition must be affected by interest and engagement - many participants reported that they were interested in some stories but not others, but this effect should be eliminated by the randomisation that was applied to the trial set.
Some of the difference in saliency between the site name and the article title could be attributed to shift in context from coloured logos to plain text. There may have been interference in the recognition of site name partially to fact that the encoding took in visual elements of colour and shape (as they were logos) as well as verbal elements, but recognition was solely on verbal elements. (Clark & Paivio, 1987; Zelinsky, 2000). However, there is empirical evidence that claims that colour is not a strong contextual cue in recognition (Pointer & Bond, 1998).
It may have been beneficial to interview participants to find out what they consciously found more salient. When some participants were asked this in informal chats, all reported that they were looking for the article title while searching the bookmark menu. These are interesting responses, but people can be notoriously inaccurate when attempting to report on their inner cognitive processes (Nisbett & Wilson, 1977).
7.4) Limits and generalisability of the results
The study could be criticised on the grounds that any real life situation would include location memory for menu entries (Hornof & Kieras, 1999). Apart from memory for location, the other recognition cues in this case is approximate memory for the length of the bookmark text. In this way people can find the entries quicker by looking for low spatial frequency features that can be exploited in a parallel search (Lansdale & Ormerod, 1994).
The main counter-argument is that bookmark lists can be left for a long time or re-arranged, and menu positions can be forgotten. Another justification is that pages saved on the hard drive are often left in archives a long time and can be re-ordered in many ways thus destroying any memories for entry location. In these cases the users would have to rely on the text content of the bookmark, hence the purpose of this study.
Lastly, these results should generalise most appropriately to sites that share a similar information hierarchy.
7.5) Future work
There may be more factors affecting the salience of bookmarks than could be revealed in the present study, which tests for short-term recognition. For example, further studies could explore longer cut-off times on the visual search task, or introduce longer delays between the viewing of the website and the appearance of the bookmark menu. The effects of familiarity could also be investigated: Are people more likely to recognise pages that come from websites that they use regularly? Lastly, it would certainly be worthwhile to perform a longer-term ethnographic study to explore the full depth of recognition and salience issues.
Eye movement data on the websites themselves could be analyzed to make recommendations for the bookmark text based on the pattern of eye movements while encoding. For example, the URL was looked at quite often in the present study, which indicates that it may be a significant navigational cue. Similarly, the previous research in graphical bookmarks (Cockburn et al., 2003) could be replicated using eye tracking measures for a more detailed analysis of recognition value.
Finally, further studies could be performed to refine the test protocol so that companies can use the technique to find out how to organise information structures in large scale information retrieval tasks. Application areas include Information Architecture, Knowledge Management, database engineering and Web design.