I am doing a talk about going inside the black box of the search index for the Enterprise Search Summit in September in San Jose (more on that later).While I have a lot to say about indexes, I used the opportunity to check around and look for current research on the topic, and pretty much struck gold. Although this paper is from 2006, it is exhaustive and detailed, with both practical and theoretical information, including finding that inverted indexes are both significantly faster to search and easier to maintain than relational database management systems, signature files and suffix arrays. It also has a thorough annotated bibliography. Best of all, Zobel and Moffat agree with me on lowercasing all words in the index and including stopwords, which they say “have an important role in phrase queries”. Inverted files for text search enginesby Justin Zobel and Alistair MoffatACM Computing Surveys. 2006;38(2) (56 pages).Available from: http://doi.acm.org/10.1145/1132956.1132959Unfortunately, this article is firmly behind the ACM firewall, so if you or your institution don’t have a subscription, you have to go through a few hoops to get it. Click the PDF link, you will be denied access and have to go through their free registration form. After that, there’s a little form and you can buy the article for $10 by credit card. I think it’s worth it.

Search usability research findings
Whitney Quesenbery and her colleagues convey the findings of a long study about how search is used at the UK’s Open University, She gave a talk at the Enterprise Search Summit, and presented more formally at the Usability Professionals Association conference, in June 2008The study included search log analysis, heuristic reviews, remote and local usability testing on the search user experience, over the course of several years, and they are linked from Whitney’s valuable Search Usability page. Designing for Search: Making Information Easy(PDF) covers both search and content. It recommends focusing improvements first on the most frequent terms, the short head of search popularity. The results of tests with eyetracking “heat map” visualizations show that both students and those outside the system will scan the whole search results page, and confirm the user tests stressing the strong value of meaningful titles. Search is now normal behavior. what do we do about that?(PDF) has a different perspective. In addition to the classic long tail frequency of search terms, it showed that the most popular search terms (the short head) remained much the same over the course of three years, though there is also some seasonal variation. Gratifyingly, because I’ve been saying this for a while, the need for search as a supplement to navigation went down, when the site navigation changed. The study finds that topical metadata and improving titles makes search results significantly more useful. A comparison of four search engines found significant differences in results, in particular, in the variety of top results for common terms. Only one of the four search engines hid duplicate entries, consistently had all links on the first page be appropriate, and displayed links to several different locations, rather than a single subsite or directory.It’s great to see more research done over time and with a large amount of data. I’m keeping a listing of what I’ve found at CiteULike with the tag search-interface, and planning to update my Search Usability page

This entry was posted on Sunday, August 31st, 2008 at 3:30 am and is filed under Tooling. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.