Topic: The State of Search Engine Safety

[Samker]

Abstract

One year after releasing The Safety of Search Engines in May 2006, we reassess the state of search engine safety and evaluate changes in search engine safety levels over time. This report also follows our second study, The Safety of Search Engines - Revisited, released in December 2006.

In this study, we compare the safety of leading search engines, using McAfee SiteAdvisor’s automated Web site ratings. We find that AOL returns the safest search results, while Yahoo! returns the greatest percentage of risky results. Since May 2006, search engine results have become safer, primarily due to improved safety of sponsored results on Google, AOL, and Ask. Despite this improvement, dangerous sites are found in search results of all of the top five search engines, and sponsored results continue to be significantly less safe than search engines’ organic results.

Key Findings

Overall, 4.0% of search results link to risky Web sites, which marks an improvement from 5.0% in May 2006. Dangerous sites are found in search results of all 5 of the top US search engines (representing 93% of all search engine use).

The improvement in search engine safety is primarily due to safer sponsored results. The percentage of risky sites dropped from 8.5% in May 2006 to 6.9% in May 2007. However, sponsored results still contain 2.4 times as many risky sites as organic results.

AOL returns the safest results: 2.9 % of results rated red1 or yellow2 by McAfee SiteAdvisor. At 5.4%, Yahoo! returns the most results rated red or yellow.

Google, AOL, and Ask have become safer since May 2006, with Ask exhibiting the greatest improvement. The safety of search results on Yahoo! and MSN has declined.

1. “Red” rated sites failed McAfee SiteAdvisor’s safety tests. Examples are sites that distribute adware, send a high volume of spam, or make unauthorized changes to a user’s computer.

2. “Yellow” rated sites engage in practices that warrant important advisory information based on McAfee SiteAdvisor’s safety tests. Examples are sites which send a high volume of “non-spammy” e-mail, display many pop-up ads, or prompt a user to change browser settings.



Introduction

With 90% of U.S. consumers using search engines and 80% of Web site visits originating from search queries, search engines have a meaningful impact on the Web sites which Internet users visit. However, our May 2006 Search Engine Safety study revealed that search engines expose users to dangerous sites posing security risks including spyware, spam, and scams. While the level of risk depends on the degree of dangerous online activity, it also depends on the extent to which search engines keep risky sites out of search results. As the leading search engines compete for market share by tweaking their search algorithms and updating their search advertising platforms, the safety of search results is likely to change over time. Our current analysis finds that, on average, search engine results have become safer within the last year. In particular, the improvement in safety is primarily due to decreased prevalence of risky sites in sponsored results at Google, AOL, and Ask. Still, these sponsored results remain riskier on average than non-sponsored results, indicating that there remains room for further improvement.


Methodology

We compare the safety of search results from five search engines: Google, Yahoo!, MSN, AOL, and Ask. First, we compiled a list of approximately 2,300 popular keywords derived from lists of common searches from Google Zeitgeist, Yahoo!, AOL, Ask, Lycos, MSN, Wordtracker, Hitwise and other industry sources. We assess the first five pages of search results for each keyword on each of the five search engines. We analyze site safety based on result position and result type (organic versus sponsored).

Our site safety assessments come from McAfee SiteAdvisor’s Web safety database of 8.2 million of the most trafficked Web sites, covering in excess of 90% of all Web traffic. Since our earlier search engine analysis in May 2006, McAfee SiteAdvisor has increased its coverage of Web sites, site downloads and browser exploits -- growing from 3 million sites to 8.2 million.

We analyze Web site safety using overall McAfee SiteAdvisor ratings as well as component ratings of specific behaviors: browser exploits, e-mail, downloads, scams, annoyances (such as pop-ups), and links to other such sites.



Analysis
Search Engine Comparison

While all search engines return some unsafe results, listings at Yahoo! are particularly risky. In our analysis, 5.1% of Yahoo! results are rated red or yellow. AOL returns the safest results with 3.6% rated red or yellow. Overall, 4.4% of search results are rated red (2.6%) or yellow (1.7%).



Since May 2006 the safety of search results from Google, AOL, and Ask has improved, while the safety of search results from Yahoo! and MSN has declined. AOL returns the safest results, with 2.9% of results rated red or yellow by McAfee SiteAdvisor. Within the past year, Ask’s search results have improved the most, dropping from 6.1% red or yellow results in May 2006 to 3.5% in May 2007. Yahoo! returns the highest percentage of risky sites: 5.4% of results are rated red or yellow by McAfee SiteAdvisor, up from 4.3% in May 2006.

Change since May and December 2006:

May 2006   Dec 2006 (vs. May 06)   May 2007 (vs. Dec 06)
Google    5.3%   4.2% ()   3.4% ()
Yahoo!   4.3%   5.1% ()   5.4% ()
MSN   3.9%   4.6% ()   4.2% ()
AOL   5.3%   3.6% ()   2.9% ()
Ask   6.1%   4.2% ()   3.5% ()
Overall   5.0%   4.4% ()   4.0% ()
            
Organic   3.1%   3.0% ()   2.9% ()
Sponsored    8.5%   8.0% ()   6.9% ()


safer since last study  less safe since last study

Results by Safety Facet: Exploits, E-mail, Downloads, Scams, Links

Our December 2006 report added an analysis of safety by type of security risk. Since December 2006, search results have improved in all areas tested by McAfee SiteAdvisor, including risky downloads, high volume or spammy e-mail, exploits, scams, and risky links.

December 2006   May 2007
Risky Downloads   1.1%   1.0% ()
High Volume/Spammy E-mail   1.8%   1.3% ()
Exploits   0.13%   0.03% ()
Scams   1.2%   0.9% ()
Risky Links   1.4%   1.0% ()

 
 
% Search Results Rated Red/Yellow By Security Risk
Risky Downloads   High Volume/ Spammy E-mail   Exploits   Scams   Risky Links
Google   1.0% ()   1.3% ()   0.03% ()   0.3% ()   1.0% ()
Yahoo!   1.0% (-)   1.4% ()   0.04% ()   2.1% (-)   1.5% ()
MSN   1.1% ()   1.3% ()   0.02% ()   0.8% ()   0.9% ()
AOL   0.9% ()   1.0% ()   0.04% ()   0.2% ()   0.6% ()
Ask   0.9% ()   1.2% ()   0.03% ()   0.5% ()   0.9% ()
Overall   1.0% ()   1.3% ()   0.03% ()   0.9% ()   1.0% ()

 
 
% Red/Yellow Search Results By Security Risk
Risky Downloads   High Volume/ Spammy E-mail   Exploits   Scams   Risky Links
Google   29.4% ()   38.2% ()   0.9% ()   8.8% ()   29.4% ()
Yahoo!   18.5% ()   25.9% ()   0.7% ()   38.9% ()   27.8% ()
MSN   26.2% ()   31.0% ()   0.5% ()   19.0% ()   21.4% ()
AOL   31.0% ()   34.5% ()   1.4% ()   6.9% ()   20.7% ()
Ask   25.7% ()   34.3% ()   0.9% ()   14.3% ()   25.7% ()
Overall   24.5% ()   32.5% ()   0.8% ()   22.5% ()   24.5% ()



Organic vs. Sponsored Results

The safety of sponsored (paid advertising) results has improved since our May 2006 study. The percentage of sponsored results that link to risky sites has dropped from 8.5% in May 2006 to 6.9% in May 2007. That said, sponsored results are still riskier, with 2.4 times as many red and yellow sites as in organic results.

Improvements in paid search safety were found on Google, AOL, and Ask, all of which include Google’s sponsored listings (Ask includes ads from its own advertising platform as well.) These improvements seem to arise in part from Google’s July 2006 evaluation of landing page quality when setting each ad’s minimum bid requirement. Within the past year, MSN and Yahoo! also made changes to their advertising platforms in order to improve ad quality and relevance. Yet the safety of their sponsored results has deteriorated.

Organic results have also become moderately safer: The proportion of risky organic listings dropped from 3.1% in May 2006 to 2.9% in May 2007. Since May 2006, the percentage of risky sites found in organic results decreased on Google, AOL, and Ask, but increased on Yahoo! and MSN.



Percentage of red and yellow sites in sponsored results
Sponsored Results   May 2006   December 2006   May 2007
Google   8.5%   7.3% ()   5.4% ()
Yahoo!   6.5%   8.0% ()   9.0% ()
MSN   6.3%   10.7% ()   7.2% ()
AOL   10.2%   8.1% ()   4.4% ()
Ask   11.1%   6.5% ()   4.1% ()

 
Percentage of red and yellow sites in organic results
Organic Results   May 2006   December 2006   May 2007
Google   3.6%   3.1%  ()   2.8% ()
Yahoo!   2.4%   2.8% ()   2.7% ()
MSN   3.0%   2.6%  ()   3.2% ()
AOL   3.2%   3.1% ()   2.9% ()
Ask   3.4%   3.2% ()   3.3%  ()



Sponsored Results Promoting Scam Sites

Scam sites (such as sites selling free software, ringtone sites with misleading billing practices, and deceptive work-at-home ploys) are primarily found in sponsored results, far more than in organic results. Scam sites represent 3.2% of all sponsored results, but only 0.07% of organic results. If search engines work to keep scam sites out of sponsored results, they could significantly reduce consumer interaction with these sites. Our tests indicate that the percentage of scam sites within sponsored results has decreased to 3.2% from 4.1% in May 2006.



Safety by Position

The safety of Google’s sponsored links varies according to the position where those links appear. In particular, sponsored results displayed at the top of Google search results are generally safer than sponsored results displayed on the right side. Google only puts ads at page top if the ads meet certain criteria – not officially disclosed, but generally understood to include click-through rate, relevance, and bid. We’re reassured to see that the ads Google regards as higher quality are also ads SiteAdvisor evaluates more favorably. Nonetheless, even Google’s most prominent ads are still, on average, less safe than organic results.

Google’s Results by Rank
Organic       Sponsored: Top of page       Sponsored: Right side of page
Position   % Red/Yellow       Position   % Red/Yellow       Position   % Red/Yellow
1   2.5%       1   3.9%       1   3.8%
2   2.5%       2   4.4%       2   5.8%
3   2.6%       3   3.2%       3   6.4%
4   2.7%                   4   6.2%
5   2.8%                   5   6.0%
6   3.1%                   6   8.1%
7   3.3%                   7   5.5%
8   2.7%                   8   6.7%
9   2.7%                       
10   2.9%                       



Category Analysis

Keywords related to music and technology continue to be among the most dangerous search terms. Comparing the safety of Google Zeitgeist categories of search terms, the category “Digital Music” (which includes search terms “free music downloads”, “bittorrent”, and “itunes”) returns the highest percentage of risky sites at 19.1%. The category “To Do Online” (which includes online activity keywords such as “chat” and “wallpaper”) returns results which are 17.5% risky.

Top Ten Most Dangerous Google Zeitgeist Categories:
Google Zeitgeist Category   % Red/Yellow Results
1. Digital Music   19.1%
2. Tech Toys   18.1%
3. To Do Online   17.5%
4. Technology Queries   13.5%
5. Christmas Craze   11.8%
6. Singers   11.6%
7. Popular Male Singers   11.5%
8. Top Music Queries   10.6%
9. Popular Software   10.1%
10. Popular Brunettes   9.7%




At the individual keyword level, file sharing programs, including “bearshare” (45.9% risky search results), “limewire” (37.1%), “kazaa” (34.9%), and “winmx” (32.0%) are among the most dangerous search terms. File sharing downloads are often bundled with adware and unwanted toolbars. Desktop downloads such as “screensavers” (42.0%) and “wallpapers” (31.1%) also topped the list of most dangerous keywords.

Top Ten Most Dangerous Keywords 3:
Keyword   % Red/Yellow Results
1. bearshare   45.9%
2. screensavers   42.0%
3. limewire   37.1%
4. kazaa   34.9%
5. lime wire   33.3%
6. winmx   32.0%
7. wallpapers   31.1%
8. hentai   29.7%
9. halloween wallpaper   28.6%
10. ringtones   28.4%



3 This table excludes searches for sites’ domain names.



Searching for Adult Search Terms

Adult search terms are more than twice as dangerous as non-adult search terms, and searching for adult terms has become a riskier activity since our analysis in December 2006. The percentage of risky search sites found in adult keyword search results increased from 8.0% in December 2006 to 9.4% now. Higher percentages of risky search results for adult terms are found on Google, Yahoo!, MSN, and Ask, while AOL now returns a lower percentage of risky results for adult terms. The increased danger of adult search terms is primarily due to an increased percentage of risky sites within sponsored results for adult terms, which rose from 13.3% in December 2006 to 22.9% now.

In our tests, which maintain the default content filtering settings on each search engine, the riskiness of sponsored search results for adult terms varies greatly depending on the search engine. Searching for adult search terms, users are eight times more likely to encounter a risky sponsored site searching for adult terms on Yahoo! than on Ask. On Yahoo! sponsored results for adult terms are 29.3% risky compared to 3.7% on Ask.




Discussion

Overall, search engine results have become safer since May 2006. Both organic and sponsored results on Google, AOL, and Ask now contain a lower percentage of sites rated red or yellow by McAfee SiteAdvisor than they did last year. However, Yahoo! and MSN have become less safe over the past year. MSN and Yahoo! were the two safest search engines in our May 2006 tests, but now their search results contain the highest percentages of risky sites. In December 2006 AOL replaced MSN as the safest search engine and AOL remains the safest overall in our current analysis.

Search engines vary most in the safety of their sponsored results. Ask has 4.1% risky sites, compared to 9.0% on Yahoo! – so users clicking on an ad are more than twice as likely to encounter a risky site on Yahoo! than on Ask. These differences indicate search engines’ control over the sites they promote. Search engines can look the other way while dubious ads run rampant, but they can also set and enforce tough editorial policies to keep bad ads out. Unfortunately, current guidelines primarily focus on ensuring appropriate ad copy as opposed to Web site safety. If landing page quality is correlated with Web site safety, then Google’s implementation of landing page evaluations may be helping filter out dangerous advertisers. Still, we are struck by search engines’ failure to block even the most notorious and widespread of scam ads – a decision we suspect arises out of search engines’ business objectives. Excluding bad ads would reduce search engines’ revenues by leaving fewer bidders buying ad space – so search engines have strong incentives to retain even unsavory advertisers.

Law enforcement intervention – and the threat of such intervention –- may provide a further check on bad ads. For example, legislators have recently cracked down on deceptive affiliate marketing of free ringtone offers, where scammers advertise ringtones as free when there are often hefty fees attached. The current lack of online advertising regulation enforcement has enabled bad actors to clutter search results with misleading claims. With greater intervention, fear of legal consequences may help purge search results of false advertising.

While we are encouraged by the recent overall improvement in search engine safety, search engine users still face serious security risks when using a search engine. The average search engine user performs 43.1 searches per month and clicks on 2.3 results per search – yielding one dangerous site every 8 days, on average. An active search engine user that performs 11 searches per day is likely to visit a dangerous site at least once a day.

As we look to the future of search engine safety, we wonder if the trend towards safer search results will continue. Will the distinction between safer and riskier search engines become even more pronounced over time? If so, safer search engines will benefit from enhanced consumer trust and consumers will benefit from an improved user experience. But is it realistic to expect search engines to keep all scammers and bad actors out of search results?

Fortunately, consumers do not have to wait for search engines to eliminate risky sites from their results in order to search safely. McAfee SiteAdvisor’s safety ratings can help steer users away from dangerous sites and towards safer options.


Results by Search Engine



Sponsored vs. Organic Results



Sponsored Results by Search Engine



Organic Results by Search Engine