SEARCH-LOGSPsychic analysis of AOL users
and their search logs
|Here is search logs of 650,000 AOL users. It's very interesting to view search history of particular person and analyze his personality. Let's do it together! Read more about AOL search database scandal or view research papers on web searching.|
Bernard J. Jansen a,*, Amanda Spink b,1
a School of Information Sciences and Technology, The Pennsylvania State University,
2P Thomas Building, University Park, PA 16802, USA
b School of Information Sciences, University of Pittsburgh, 510 IS Building,
135 N. Belleﬁeld Avenue, Pittsburgh, PA 15260, USA
Received 15 April 2003; accepted 15 July 2003
Available online 30 August 2003
The Web has become a worldwide source of information and a mainstream business tool. It is changing the way people conduct the daily business of their lives. As these changes are occurring, we need to un- derstand what Web searching trends are emerging within the various global regions. What are the regional di erences and trends in Web searching, if any? What is the e ectiveness of Web search engines as providers of information? As part of a body of research studying these questions, we have analyzed two data sets collected from queries by mainly European users submitted to AlltheWeb.com on 6 February 2001 and 28 May 2002. AlltheWeb.com is a major and highly rated European search engine. Each data set contains approximately a million queries submitted by over 200,000 users and spans a 24-h period. This longitudinal benchmark study shows that European Web searching is evolving in certain directions. There was some decline in query length, with extremely simple queries. European search topics are broadening, with a notable percentage decline in sexual and pornographic searching. The majority of Web searchers view fewer than ﬁve Web documents, spending only seconds on a Web document. Approximately 50% of the Web documents viewed by these European users were topically relevant. We discuss the implications for Web information systems and information content providers.
2003 Elsevier Ltd. All rights reserved.
Keywords: Web searching; Session duration; Query language; Search engine evaluation
The Web is changing the way many people locate information. As the Web is becoming a worldwide phenomenon, we need to understand what searching trends are emerging. These trends include how searchers utilize Web search engines in the search process and the viewing of Web documents. There is a growing body of Web research concerning how users interact with Web search engines (Spink, Jansen, Wolfram, & Saracevic, 2002). However, the majority of research in this area has focused on users of United States Web search engines. There is a need to understand what searching trends are emerging within di erent global regions. To our knowledge, there has been limited large-scale research examining the interactions of users with European Web search engines. Examining the Web searching behavior of di erent users from di erent world regions is an important area of research with potential to impact our understanding of global Web search and the design of Web search engines.
In this paper, we examine the interactions of the users of a major and predominantly European search engine. We report general searching characteristics and trends, including session duration, query length, languages, and result pages viewed. We also examine the number of Web documents viewed, and analyze the relationship between sessions, queries, and pages viewed. Finally, we evaluate the success of these searches by analyzing the topical relevance of documents retrieved and viewed.
We begin with a review of the literature, followed by the research design utilized to obtain and analyze this Web search engine data. We use these Web queries to isolate trends in searching and page viewing, also known as click through or page view data (i.e., the Web page/s a user visits when following a hyperlink from a search engine results page). This analysis includes the temporal aspects of Web page viewing. We discuss the implications of these results for Web search engine users and designers, and Web sites targeting the European market. We conclude with directions for future research.
There is a growing body of research examining the search patterns of users of predominantly US search engines (Jansen & Pooch, 2001; Jansen, Spink, & Saracevic, 2000; Silverstein, Henz- inger, Marais, & Moricz, 1999; Spink et al., 2002). Jansen and Pooch (2001) present an extensive review of the Web searching literature, reporting that Web searchers exhibit di erent search techniques than do searchers on other information systems. Jansen et al. (2000) conducted an in- depth analysis of the user interactions with the Excite search engine. Silverstein et al. (1999) conducted a large study with a sample of more than a billion queries from the Alta Vista search engine. Spink et al. (2002) analyzed trends in Web searching, reporting that Web searching has remained relatively stable over time, although they noted a shift from entertainment to com- mercial searching.
Overall, we see that Web searching sessions are very short as measured by number of queries. There has been less analysis of session temporal length, but it is assumed to be short. Users view a very limited number of results pages. 2 From the studies cited, the majority of Web searchers, approximately 80%, view no more than 10–20 Web documents. The page viewing characteristics of Web searchers have not been analyzed at any ﬁner level of granularity. We do not know how many Web documents Web searchers actually view. There has also been a focus on primarily US search engines, with much less study of European and other Web search systems. From previous research, indications are that over 80% of users of US based search engines are from the US (Spink, Bat- eman, & Jansen, 1999). This stream of research provides useful information and a methodology for considering the Web search process when evaluating search engine usage in other regions.
Limited research has focused on users of European Web search engines. Three studies have examined this area of Web searching (Cacheda & Vina,~ 2001a; Holscher€ & Strube, 2000; Spink, Ozmutlu, et al., 2002). Holscher€ and Strube (2000) examined European searchers on the Fireball 3 search engine, a predominantly German search engine, and reported on the use of Boolean and other query modiﬁers. The researchers note that experts exhibit di erent searching patterns than novice users. Cacheda and Vina~ (2001a, 2001b) reported statistics from a Spanish Web directory service, BIWE. 4 Table 1 provides the key results for the Fireball and BIWE studies.
The researchers report on number of page results viewed, queries, operator usage, and terms. Spink, Ozmutlu, Ozmutlu, and Jansen (2002) compared Excite (American) and AlltheWeb.com (European) search engine users. They found that AlltheWeb.com users tended to create longer sessions and search more for information on people and places, rather than the Excite user focus on e-commerce. A summary of their results is displayed in Table 2.
In general however, there has been limited research on European based Web searchers. Eu- ropean users may interact di erently with Web search systems relative to their US counterparts. Their searching topics may di er. They may have di erent preferences in viewing results. In this study, we seek to address these issues by examining the searching patterns of actual Web searchers using a predominately European Web search engine.
The research questions driving this study are:
(1)What are the trends in Web searching characteristics by European users of the Allthe- Web.com search engines?
(2)How many Web documents do AlltheWeb.com European Web search engine users view, and how long do they spend viewing these documents?
(3)How topically relevant are the Web documents they are viewing?
2When a Web search engine user submits a query, the search engine returns the results in ‘‘chucks’’, of usually about
10results. These ‘‘chucks’’ are referred to as results pages and are presented to the user sequentially from the top most ranked results page to the maximum number of results the search engine presents.
|364||B.J. Jansen, A. Spink / Information Processing and Management 41 (2005) 361–381|
|Comparison of Fireball and BIWE study results|
|Fireball studya||BIWE studyb|
|Mean terms per||1.66||Not reported|
|Terms per query|
|1 term||8,873,001||55%||Not reported|
|2 terms||5,005,653||31%||Not reported|
|3+ terms||2,374,248||14%||Not reported|
|Session size||Not reported||Not reported|
|Results pages viewed|
|Terms not repeated||Not reported||9356||8%|
|in data set|
|a Holscher€||and Strube (2000).|
|b Cacheda and Vina~||(2001a).|
|c Data reported using 71,810 initial queries.|
|d Statistics reported for ﬁrst page and all other pages.|
These issues are important for the examination of European Web searching, as the Web be- comes a more global tool for information searching.
We obtained, and quantitatively analyzed, actual queries submitted to AlltheWeb.com, 5 a major European Web search engine at the time of the study owned by FAST. Since the study, an outside company has purchased the FAST corporation (Kane, 2003). According to Allthe- Web.com personnel, most European users of AlltheWeb.com are from Norway and Germany. All queries were submitted to the European Web site for the AlltheWeb.com search engine. The queries examined for this study were submitted to AlltheWeb.com on 6 February 2001 and 28
|B.J. Jansen, A. Spink / Information Processing and Management 41 (2005) 361–381||365|
|Comparative results from Excite and AlltheWeb.com studya|
|Mean terms per query||2.6||2.4|
|Terms per query|
|Mean queries per session||2.3||2.9|
|Mean pages viewed per query||1.7||2.2|
|% of use of 100 most frequently||22%||14%|
|occurring query terms|
|a Spink, Ozmutlu, et al. (2002).|
May 2002, each spanning a 24-h period. The queries were recorded in transaction logs and rep- resent a portion of the searches executed on the Web search engine on these particular dates. The transaction logs hold a large and varied set of queries (over one million records).
Each record within the transaction log contains three ﬁelds:
(1)Time of day: measured in hours, minutes, and seconds from midnight of each day as logged by the Web server;
(2)User identiﬁcation: an anonymous user code assigned by the AlltheWeb.com server. The All- theWeb.com server software derives this code using the Internet Protocol (IP) address of the searcher s machine. The code is unique and persistent.
(3)Query terms: terms exactly as entered by the given user. Additionally, the 2001 transaction log contained:
(4)Language: the user preferred:
The 2002 transaction log contained:
(5)Page viewed: the uniform resource locator (URL) that the searcher visited after entering the query.
The transaction log contained searches from individuals, common user terminals, automated processes, and agents. We were interested in only those queries submitted by individuals for this study. From the transaction log, we therefore culled a subset of queries that we deemed were likely submitted by an individual. To do this, we separated all sessions with less than 101 queries into a separate transaction log, which we used for this research. We chose 101 queries because it is almost 50 times greater than the reported mean search session (Jansen et al., 2000) for human Web searchers.
Given that there is no way to accurately identify individual from non-individual searchers, most researchers relying on transaction logs for data collection must either ignore it (Cacheda & Vina,~ 2001a) or assume some temporal or interaction cut-o (Montgomery & Faloutsos, 2001; Silver- stein et al., 1999) to the session. Using a cut-o of 100 queries, we were satisﬁed that we had retrieved a subset of the transaction log that contained queries submitted primarily by human searchers in a non-common user terminal, but also broad enough not to introduce bias by too low of a cut-o threshold.
Using the time stamp ﬁeld and user identiﬁcation code, we located the initial query and recreated the chronological series of actions in a session. A session is the entire series of queries submitted by a user during one interaction with the Web search engine. A query is the entire string of terms submitted by a searcher in a given instance. A term is any series of characters separated by white space. A results page is the chuck of results presented by the search engine, usually Web sites or Web pages. The language is that selected by the user, with the default on the search engine being ANY. The Web page viewed is the Web document located at the URL recorded and pre- sented by the Web search engine in the results page.
When a searcher submits a query, views a document from the results listing, and returns to the search engine, the AlltheWeb.com server logs this second visit with the identical user identiﬁcation and query, but with a new time (i.e., the time of the second visit). This is beneﬁcial information in determining how many of the retrieved results the searcher visited from the search engine, but unfortunately it also skews the results in analyzing how the user searched on system.
To address the ﬁrst research question, we collapsed the data set by combining all identical queries submitted by the same user to give us the unique queries in order to analysis sessions, queries, languages and terms, and pages of results viewed. We could then located a user s initial query and recreated the chronological series of actions by each user in a session. As outlined in Jansen and Pooch (2001), an initial query is the ﬁrst query submitted by a particular user in a session.
For the second research question, we utilized the complete un-collapsed sessions in order to obtain an accurate measure of the temporal length of sessions and the number of pages visited. The page viewed ﬁeld permitted us to addresses what Web document the user visited.
For the third research question, we randomly selected 530 records from the transaction log. Each record contained the query submitted by the Web search engine user and the Web page viewed after the user submitted that query. Three independent raters reviewed the Web document visited from each of these 530 queries for topical relevance, assigning a binary relevance judgment of 1 (for relevant) or 0 (for not relevant) based on the rater s interpretation of the query.
Topical relevance is a standard measure utilized in information retrieval to evaluate the ef- fectiveness of a query based on the documents retrieved (Saracevic, 1975). The reviewers received training regarding the topical relevance judgment process and were given instructions for deter- mining topical relevance. We calculated agreement across the three raters using rwg, and we found it to be quite high (rwg ¼ 0:95). From these topical relevance rankings, we were able to calculate relative precision (i.e., the ratio of the number of relevant documents retrieved to the number of documents retrieved at a certain point in the results listing).
From this analysis, we could determine the trends in Web searching over an approximately one- year period. In order to facility comparison with other studies, we generally use the procedure and terminology outlined in Jansen and Pooch (2001).
In the following sections, we report the results of our analysis.
We present the aggregate results for the analysis reported in Table 3.
Overall, we see a trend toward greater simplicity and increased variability of terms. The percentage of 1-term queries increased from 25% in 2001 to 33% in 2002. The number of users modifying queries decreased approximately 6%, from 47% of all users in 2001 to 41% in 2002. Concerning overall session length, the percentage of shorter sessions trended higher, with 53% of users with a one-query session in 2001 to 59% in 2002. The percentage of queries containing Boolean operators remained at 1%, which is low, even by Web standards. There was an in- crease in the percentage of users viewing more than the ﬁrst results page, although this may be a result of the more na€ive searching rather than an increased persistence in locating relevant results.
Based on the term analysis, there was a broadening of topics. The number of unique terms increased (from 12% to 15%) and the percentage of terms not repeated in the data set also in- creased, from 7% in 2001 to 10% in 2002. There was a corresponding decrease in the percentage of usage represented by the top 100 most frequently occurring terms, from 15% to 14%. Taken together, these may indicate a broadening of search interests of AlltheWeb.com users.
In the following sections, we examine the results of our analysis in more detail at three levels of granularity, the session, query, and term levels of analysis.
We report session length analysis in Table 4.
An increase in single query sessions resulted in shorter sessions for all except a small (about 4%) group of very persistent users. This trend parallels what is reported from analyses of US search engines, namely a move toward great simplicity in searching (Spink et al., 2002), but di ers from the longer sessions lengths found by Spink, Ozmutlu, et al. (2002).
Table 5 presents the session duration, as measured from the time the ﬁrst query is submitted until the user departs the search engine for the last time (i.e., does not return) for the 2001 data set. Unfortunately, the login time were not properly recorded for the 2002 data set.
|368||B.J. Jansen, A. Spink / Information Processing and Management 41 (2005) 361–381|
|Comparative results statistics for AlltheWeb.com data sets|
|AlltheWeb.com 2001||AlltheWeb.com 2002|
|Mean terms per||2.4||2.3|
|Terms per query|
|Mean queries per||3.0||2.8|
|Results pages viewed|
|Terms not repeated||100,649||7%||212,040||10%|
|in data set|
|Use of 100 most||196,390||15%||303,176||14%|
With this deﬁnition of search duration, we can measure the total user time on the search engine and the time spent viewing the ﬁrst and all subsequent Web documents, except the ﬁnal document. This ﬁnal viewing time is not available since the Web search engine search records the time stamp. Naturally, the time between visits from the Web document to the search engine may have not been entirely spent viewing the Web document.
However, this may not be a signiﬁcant issue as shown from the data in Table 5. The mean session duration was 2 h, 21 min and 55 s, with a standard deviation of 4 h, 45 min, and 36 s. However, we see that the longer session durations skewed our result for the mean, masking signiﬁcant details. Fully 52% of the sessions were less than 15 min. This is inline with earlier reported research on Web session length (He, Goker,€ & Harper, 2002). Perhaps even more sur- prisingly, over 25% of the sessions were less than 5 min.
|B.J. Jansen, A. Spink / Information Processing and Management 41 (2005) 361–381||369|
|Occurrences and percentages of session length for AlltheWeb.com 2001 and 2002|
|Occurrences and percentage of AlltheWeb.com session duration|
|Mean||2 h and 22 min|
|Standard deviation||4 h and 37 min|
5.3. Language preferences
We analyzed the 2002 data to determined language preferences by user, with results reported in Table 6.
These Web users appear to not be concerned with specifying the language of the Web docu- ments they retrieve, with the vast majority of searchers accepting the AlltheWeb.com default of ANY. The use of the particular query terms themselves itself may provide the needed selectivity for most Web users (i.e., the use of a query containing terms in Russian, for example, will retrieve primarily Web documents written in Russian). It has been reported that the majority of Allthe- Web.com customers are German (Spink, Ozmutlu, et al., 2002); however, the top non-English
|370||B.J. Jansen, A. Spink / Information Processing and Management 41 (2005) 361–381|
|AlltheWeb.com terms, queries and sessions analysis by language|
|Language||Queries||Mean terms||Sessions||Mean queries||Total terms|
|per query||per session|
|Entire data set||957,303||2.3||345,093||2.8||2,225,141|
3 The queries were blank.
a The default language selection is ANY.
b Non-language options such as DOMAIN and ALL.
language preference was French, followed by Spanish, with German at a distance third. Italian and Russian also had fairly high rates of occurrences.
We report query length analysis in Table 7.
Query lengths of 1–3 terms inclusive account for 83% of all queries in 2001 and 84% of all queries in 2002. The percentage of queries with 1 term has increased by 8%. After 3 terms, there is a sharp decline in the frequency of occurrences, dropping to almost minimal occurrences after 4 terms per query. Similar to trends in sessions, this trend with European Web searchers parallels that reported from analysis of US search engines (Spink et al., 2002).
|B.J. Jansen, A. Spink / Information Processing and Management 41 (2005) 361–381||371|
|AlltheWeb.com query lengths|
Table 8 presents a more in-depth analysis of the number of pages viewed per query submitted. There was a sharp decrease in the number of viewings between the ﬁrst and second, and the second and third pages of results, with very few users viewing more than four or ﬁve pages of results. The percentage of European searchers viewing only one page of results is signiﬁ- cantly higher (5–25%) than reported in previous research (Jansen et al., 2000; Silverstein et al., 1999). European users appear to have a low tolerance for wading through large numbers of
We present a term analysis in Table 9.
AlltheWeb.com results pages viewed
|372||B.J. Jansen, A. Spink / Information Processing and Management 41 (2005) 361–381|
|AlltheWeb.com top occurring terms and frequencies|
From both transaction logs, we extracted the top terms, removing the terms without content (and, or, de, la, le, etc.). We then took the top 25 terms from each year. For better identiﬁcation of trends, if a term appeared in one list and not the other we added terms and the frequency of occurrence to each list for those terms. The combined list is what is presented in Table 9.
Three trends present themselves from the term level of analysis. First, all of the top terms are English language terms, despite AlltheWeb.com being primarily a European search engine. Sec- ond, technology terms dominate the top term usage list, with terms such as Internet, Linux, software, Web, Windows, and XP. These types of terms certainly seem to stand out as indicators of a major topic for AlltheWeb.com searchers. Third, another topical area for AlltheWeb.com searchers is entertainment, with terms such as CD, games, MP3, lyrics, music, and video. These topical areas have held fairly constant over the analysis period.
Although a term analysis is useful, it is sometimes di cult to determine the speciﬁc usage of a term intended by a searcher within the framework of a particular query. In these cases, a term co- occurrence is more helpful (Leydesdor , 1989; Wolfram, 1999). We present in Table 10 a term co- occurrence for the 2001 data set in a correlation matrix fashion.
From the term co-occurrence analysis, the predominance of technical searching is even clearer. Nearly half of the top occurring term pairs are technology related (e.g., Windows mac, Windows os, and bug ﬁxes). Business related pairs are a distance second.
In Table 11, we present the term co-occurrence for 2002 in a correlation matrix fashion.
In Table 11, we do not see the clustering that was displayed in Table 10 for the 2001 data set. This diversity reinforces our ﬁndings with the initial term analysis that these Web users are searching for an increasingly variety of topics and domains. Business and entertainment pairs have replaced technology as the predominant grouping.
We classiﬁed a random sample of approximately 2500 English language queries each from the 2001 and 2002 data sets, into 11 non-mutually exclusive, general topic categories previously de- rived by Spink et al. (2002). Table 12 presents the results of this classiﬁcation.
People, places or things category remained the top ranked category with a large percentage increase from 2001 to 2002. Percentage drops occurred in several other categories, most noticeably computers or Internet and sex or pornography. The category rankings changed somewhat. The sex or pornography category, for example, dropped from 4th to 6th place. This decrease in sexual searching as a percentage of overall Web searching parallels that reported in studies of US searching (Spink et al., 2002). This analysis conﬁrms that reported by Spink, Ozmutlu, et al. (2002) who found little European commercial searching compared to the large shift to e-com- merce searching in the US. It also parallels the increase in commercial content on the Web (Lawrence & Giles, 1999).
Although most searchers viewed only the ﬁrst one or two pages of results, this does not tell us the number of Web documents they actually visited. They may have viewed all documents pre- sented or they may have viewed none. To address this issue, Table 13 shows the Web documents viewed per session. Initial results for were presented in Jansen and Spink (2003).
The mean Web documents viewed was 8.2, with a standard deviation of 26.9. Previous studies report that most Web searchers rarely few more than the ﬁrst result page, which is usually 10 results (Spink et al., 2002). While 10 documents is in line with the average, our analysis shows that over 66% of searchers examine fewer than ﬁve documents in a typical session and almost 30% view only one document in a given session.
Frequency of term co-occurrence for top 25 AlltheWeb.com terms for 2001
|Agree-||Alterna-||Bug||Cell(s)||Com-||Fix||Fuel(s)||Hat||Joint||Linux||Mac||Me||Micro-||Ms||Nt||Oper-||Os||Power||Red Sys- Tech-|
Frequency of term co-occurrence for top 25 AlltheWeb.com terms for 2002
|2000||Basic Cup Down- Engine||Estate||For||Free||His-||Job||Map||Pics||Pic-||Real||Re-||Sale||Search||Sex||Skills Soft- Uni-||Visual||Win-||World|
|376||B.J. Jansen, A. Spink / Information Processing and Management 41 (2005) 361–381|
|Comparison of AlltheWeb.com general topic categories|
|Rank||2001 (2503 English queries)||2002 (2525 English queries)|
|1||22.5% People, places or things||41.5% People, places or things|
|2||21.8% Computers or Internet||16.3% Computers or Internet|
|3||12.3% Commerce, travel, employment, or||12.7% Commerce, travel, employment, or|
|4||10.8% Sex or pornography||9.5%||Entertainment or recreation|
|5||9.1%||Entertainment or recreation||4.9%||Health or sciences|
|6||7.8%||Health or sciences||4.5%||Sex or pornography|
|7||4.8%||Society, culture, ethnicity or religion||2.6%||Government|
|8||4.7%||Performing or ﬁne arts||2.5%||Unknown|
|9||2.9%||Education or humanities||2.3%||Education or humanities|
|10||2.7%||Government||2.1%||Society, culture, ethnicity or religion|
|11||0.6%||Unknown or other||1.1%||Performing or ﬁne arts|
|Pages viewed per AlltheWeb.com session|
The low number of documents viewed also holds when we move from the session level of analysis to the query level. Table 14 presents the Web documents viewed per query.
The mean documents viewed per query were 2.5, with a standard deviation of 3.9. Allthe- Web.com users viewed ﬁve or less Web documents per query over 90% of time. The largest percentage of users by far, just fewer than 55%, viewed only one document per query.
Prior research on Web searching has not reported the duration of viewing of Web documents by Web search engine users. We present this information for this data sample in Table 15.
The mean time spent viewing a particular Web document was 16 min and 2 s, with a standard deviation of 43 min and 1 s. However, some lengthy document views skewed our mean. Over 75% of the users viewed the retrieved Web documents for less than 15 min. More surprisingly, perhaps, nearly 40% of the users viewed the retrieved Web document for less than 3 min. Fewer than 14% of the users viewed the Web document for less than 30 s. These results for Web document viewing time are substantially less than has been previously reported using survey data (Cyber Atlas, 2002). These results suggest the need for Web site designers to place more emphasis on the pre- sentation of Web documents given the short assessment time by searchers. Again, the time be-
|B.J. Jansen, A. Spink / Information Processing and Management 41 (2005) 361–381||377|
|Results viewed per AlltheWeb.com query|
|Duration of page views by AlltheWeb.com users|
|Page view duration||Occurrences||%|
tween visits from the Web document to the search engine may have not been entirely spent viewing the Web document.
This portion of the study used a random subset of records from the 2001 transaction log that included the Web site the searcher actually visited. Three independent raters visited the sites and evaluated the Web document to determine topical relevance. Topical relevance is a relevance based on a direct topic matching between the search terms used and the terms in the retrieved document, not necessarily related to the user s information seeking stage or information need (Greisdorf & Spink, 2001). Our analysis explores the question of whether search sessions are short because the searchers are potentially ﬁnding topically relevant information. The results are re- ported in Table 16.
We had the three independent raters view 530 URLs and evaluate these pages for topical relevance based on their interpretation of the query submitted. Each rater assigned a topical
|378||B.J. Jansen, A. Spink / Information Processing and Management 41 (2005) 361–381|
|Topical relevance results for pages viewed by AlltheWeb.com users|
|Topical relevance score||Number of documents||%|
|Total Web documents reviewed||530|
relevance Web document a rating of 1. A non-relevant page received a rating of 0. So, the maximum topical relevance score a Web page could receive was 3, meaning that all three reviewers rated the document as topically relevant.
Approximately 52% of the time, two or more raters evaluated a page to be topically relevant. Approximately 48% of the time, two or more raters evaluated a page to be not topically relevant. These percentages, taking in total, represent precision for this set of results retrieved by this search engine. This conﬁrms earlier survey data that users are ﬁnding topically relevant information on Web search engines (Spink et al., 1999) despite the simplistic searching methods.
Our study identiﬁed some interesting searching patterns by AlltheWeb.com users. Web searching by these European users trended toward greater simplicity from 2001 to 2002. Queries decreased in length and sessions were shorter. Sessions were temporally short, about 15 min on average. About 25% of the sessions were less than 5 min. Boolean usage was almost non-existent. The range of topics searched for increased, and the users employed a greater variety of terms.
These searchers are generally unconcerned with specifying the preferred language of the re- trieved Web documents, although within some linguistically groups this did occur (e.g., French and Spanish). This may be due to the relatively high rate of transference of terms from these to other languages. Therefore, linguistically terms from these languages will appear in Web docu- ments that are written primarily in other languages. Searching for pornography decreased slightly as a percentage of overall Web searching and an increase in commercial searching was not ap- parent. Spink, Ozmutlu, et al. (2002) also found a similar result.
Web searchers do not appear to make lengthy judgments on the relevance of information re- trieved. Approximately 75% of the users spent less than 15 min viewing the retrieved Web doc- uments. Twenty percent of the Web users viewed a Web document for less than a minute. These results seem to indicate that Web searchers are typically not spending a great deal of time combing the documents to ﬁnd relevant information.
From our analysis, it appears that generally the precision Web users can expect is about 50%, meaning that one out of every two of the Web documents viewed will be topically relevant to their query. However, note that this analysis is for Web documents viewed, not documents retrieved.
The results of this study have several implications. For search engine designers, there is still work to be done. Although, search engines are currently helping people ﬁnd information, with a precision of about 50% for documents viewed, there is certainly room for improvement. For infor- mation content providers, the abstract that appears in the results listing seems to have signiﬁcant impact on attracting or dissuading searchers from visiting the site. Also, Web documents must be well-designed, easy to load, and relevant information ease to ﬁnd, given the short amount of time users spend on a particular Web site. For search engine users, it appears that about one in two documents viewed will be relevant, indicating a need for persistence in looking needed information.
This study contributes to the Web searching literature in several important ways. First, the data comes from real users submitting real queries and viewing actual Web pages. Accordingly, it provides a realistic glimpse into European public Web searching, without the self-selection issues or altered behavior that can occur with lab studies or survey data. Second, our sample is quite large, with between 150,000 and 350,000 users per data set. Third, we obtained data from a very popular European based search engine. Finally, it provides a detailed examination of the Web document viewing patterns and viewing duration of Web users.
As with any research, there are limitations that should be recognized. First, the query data comes from only one major European Web search engine, introducing the possibility that the queries do not represent the queries submitted by the broader European or global Web searching population. However, Jansen and Pooch (2001) suggest that characteristics of Web sessions, queries, and terms are very consistent across search engines.
Second, we do not have information about the browsing patterns of the users once they leave the search engine to visit a Web document. It is possible that they are browsing using the hy- permedia structure of the Web. However, given that the duration between departing and returning to the search engine, this is unlikely in most situations. Similarly, we do not have information about the demographic characteristics of the users who submitted queries, and there is no knowledge of the underlying cognitive motivation concerning the searcher s information.
Finally, there are limitations related to the use of transaction log analysis as a research tool. The identiﬁcation of a user is dependent on the logging software of the search engine and the IP address of the searcher s computer. We used a numerical limit to deﬁne the upper boundary of a session (i.e., 100 queries), while other researchers have ignored the issue (Cacheda & Vina,~ 2001a) or utilized a temporal cut-o (Montgomery & Faloutsos, 2001; Silverstein et al., 1999). We believe that existing research supports a numerical rather than a temporal boundary.
In previously published research, there is a high degree of consistency at the session and query levels of analysis across multiple Web studies (Abdulla, Liu, & Fox, 1998; Cacheda & Vina,~ 2001a; Croft, Cook, & Wilder, 1995; Holscher€ & Strube, 2000; Jansen et al., 2000; Montgomery & Faloutsos, 2001; Selberg & Etzioni, 1997; Silverstein et al., 1999; Spink et al., 2002; Wolfram, Spink, Jansen, & Saracevic, 2001). The similarities exist even with researchers studying various search engines and utilizing a variety of analytical methods, deﬁnitions, and metrics.
However, there is only one previously published research study that we could locate examining temporal lengths of Web sessions (He et al., 2002). He et al. (2002) report a session length of about 12 min. Based on survey data, Cyber Atlas (2002) reports a session length of over 32 min. In their studies, Silverstein et al. and fellow researchers (1999) use a session length of 5 min, and Mont- gomery and Faloutsos (2001) uses a session length of 2 h. Our research supports a session length of about 15 min. We believe more research in this area is needed to clearly deﬁne the temporal length of Web sessions.
Our results provide important insights into the current state of European Web searching and Web usage. The short sessions lengths combined with short queries of many Web searchers are puzzling issues for designers of Web information systems. This does not seem to be a successful strategy to maximize recall or precision, the standard metrics for information retrieval system performance. However, it appears that Web search engine users are ﬁnding topically relevant information with this searching strategy. Our research also highlights the need for further research comparing users of US and European Web search engines.
We thank AlltheWeb.com and especially Per Gunan Auran for providing the Web query data sets without which this research could not have been conducted.
Abdulla, G., Liu, B., & Fox, E. (1998). Searching the World-Wide Web: implications from studying di erent user behavior. In Proceedings of the world conference of the World Wide Web, Internet, and Intranet, Orlando, FL (pp. 1– 8).
Cacheda, F., & Vina, A. (2001a). Experiences retrieving information in the World Wide Web. In Proceedings of the 6th IEEE symposium on computers and communications, July, Hammamet, Tunisia (pp. 72–79).
Cacheda, F., & Vina,~ A. (2001b). Understanding how people use search engines: a statistical analysis for e-business. In
Proceedings of the e-business and e-work conference and exhibition 2001, October, Venice, Italy (pp. 319–325). Croft, W. B., Cook, R., & Wilder, D. (1995). Providing government information on the Internet: experiences with
THOMAS. In Proceedings of the digital libraries conference, Austin, TX (pp. 19–24).
Cyber Atlas (2002). November 2002 Internet usage stats [Web site]. Nielsen//NetRatings Inc. Retrieved 1 January, 2003, from the World Wide Web: http://cyberatlas.internet.com/big_picture/tra c_patterns/article/0,5931_1560881,00.html.
Greisdorf, H., & Spink, A. (2001). Median measure: an approach to IT systems evaluation. Information Processing and Management, 37(6), 843–857.
He, D., Goker,€ A., & Harper, D. J. (2002). Combining evidence for automatic Web session identiﬁcation. Information Processing and Management, 38(5), 727–742.
Holscher,€ C., & Strube, G. (2000). Web search behavior of Internet experts and newbies. International Journal of Computer and Telecommunications Networking, 33(1–6), 337–346.
Jansen, B. J., & Pooch, U. (2001). Web user studies: a review and framework for future work. Journal of the American Society of Information Science and Technology, 52(3), 235–246.
Jansen, B. J., & Spink, A. (2003). An analysis of Web information seeking and use: documents retrieved versus documents viewed. In Proceedings of the 4th international conference on Internet computing, 23–26 June, Las Vegas, NV (pp. 65–69).
B.J. Jansen, A. Spink / Information Processing and Management 41 (2005) 361–381 381
Jansen, B. J., Spink, A., & Saracevic, T. (2000). Real life, real users, and real needs: a study and analysis of user queries on the Web. Information Processing and Management, 36(2), 207–227.
Kane, M. (2003). Overture to buy search services [Electronic Journal]. CNET News.com. Retrieved 1 March, 2003, from the World Wide Web: http://rss.com.com/2100-1023-985850.html?type ¼ pt&part ¼ rss&tag ¼ feed&subj ¼ news.
Lawrence, S., & Giles, C. L. (1999). Accessibility of information on the Web. Nature, 400, 107–109.
Leydesdor , L. (1989). Words and co-words as indicators of intellectual organization. Research Policy, 18, 209–223. Montgomery, A., & Faloutsos, C. (2001). Identifying Web browsing trends and patterns. IEEE Computer, 34(7), 94–95. Saracevic, T. (1975). Relevance: a review of and a framework for the thinking on the notion in information science.
Journal of the American Society of Information Science, 26(6), 321–343.
Selberg, E., & Etzioni, O. (1997). The MetaCrawler architecture for resource aggregation on the Web. IEEE Expert, 12(1), 11–14.
Silverstein, C., Henzinger, M., Marais, H., & Moricz, M. (1999). Analysis of a very large Web search engine query log.
SIGIR Forum, 33(1), 6–12.
Spink, A., Bateman, J., & Jansen, B. J. (1999). Searching the Web: a survey of Excite users. Journal of Internet Research: Electronic Networking Applications and Policy, 9(2), 117–128.
Spink, A., Jansen, B. J., Wolfram, D., & Saracevic, T. (2002). From e-sex to e-commerce: Web search changes. IEEE Computer, 35(3), 107–111.
Spink, A., Ozmutlu, S., Ozmutlu, H. C., & Jansen, B. J. (2002). U.S. versus European Web searching trends. SIGIR Forum, 32(1), 30–37.
Wolfram, D. (1999). Term co-occurrence in Internet search engine queries: an analysis of the Excite data set. Canadian Journal of Information and Library Science, 24(2/3), 12–33.
Wolfram, D., Spink, A., Jansen, B. J., & Saracevic, T. (2001). Vox Populi: the public searching of the Web. Journal of the American Society of Information Science and Technology, 52(12), 1073–1074.