Google's strength is not in searching for head or torso terms like "facebook" or "2013 calendar". It is in searching out tail terms; those obscure terms which might include a typo or two, and the phrasing may not be correct.
Unfortunately (and I don't have much time to explain this) one simply _can not_ judge a search engine's comparative performance using random users. And the Bing guys are falling into that trap.
EDIT: To expand more on what I wanted to write.
// DISCLAIMER: This is my personal view, and has nothing to do with my employer.
There are 3 large (fuzzy) classes of queries on search engines: head, torso and tail. The head queries are small in number but large in volume: queries like "Facebook", "Google", "Yahoo", etc. (yes, people do 'search' for these on search engines. Just a handful of such queries may make up 10% of the entire search engine traffic, which would be 100M queries per day (or so) on Google. It is depressing to see the numbers). In general, any search engine worth its salt will get the head queries right.
The torso is basically deciles 2-6 or so (there's no hard and fast definition). Here you'd have most musicians, actors, popular restaurants, etc. Bing and Google both would do a mostly adequate job here, for the first result. But: if you want to explore the concept a little, Bing goes astray and Google does not. For example, search for "idle hand tattoo" on http://bingiton.com/ . The first result is the same, to my local tattoo parlor. But the second result on Bing is from Mesa, AZ; while Google sticks with the right result and offers up other, related sites (Yelp, etc.). Google knows that "Hand" in the title is not the same as "Hands", and gives me not only the store, but also their Yelp, FB and Twitter pages; in other words: Google _understands_ your query better.
And finally, the tail. Here we have obscure error codes, weird technical terms, etc.; most of which are seen maybe once or twice a day. These constitute the bottom ~30% of the query stream. This is where Google really shines. It is very rare indeed to look for an obscure term and find that Bing does a better job than Google (I have yet to find an example).
So, as a user, I have a choice of 2 search engines; and one of them I know will do a better job on obscure terms. Guess which one will I pick?
When Bing claims that "most people prefer Bing in our user studies" (or something similar), their conclusion is flawed because the study itself is flawed. You can't sit a person down in front of a computer and ask him to evaluate your performance on tail queries! Where are those tail queries coming from? If they are provided by Bing ("here's an obscure term, search for it and tell us what you think"), then the user has no way to evaluate how relevant the results are, because, by definition, it's an obscure term! On the other hand, if you ask the user to come up with an obscure term, it will most likely be a term they are familiar with and they'll already know the answer, so they won't really hunt for information. So in my "idle hand tattoo" example, they'll see the top result and claim satisfaction; when, in real life, I would like to see the Yelp reviews, maps, etc. for the place.
If they (Bing) really want to compare how well they're doing, here's a suggestion: setup a search engine (like bingiton, but with only 1 pane of results, and neutral branding), and make it the default for a large pool of users (with their permission). Randomly, switch the backend to Bing or Google. Then, monitor the heck out of what the users do (all with their permission): how often do they click on the first result, how often do they re-formulate queries, the time to the first click, how often do they quickly come back to the results page from a bad click, etc. etc. etc.
And this is why I use Google. I rarely need to search for "facebook"; I do have a brain. When I'm searching, its more often than not for something I can describe but don't know the name of, and Google blows Bing out of the water on these. Try describing a movie to see what I'm talking about.
Absolutely agreed. I was also 5-0 Google. Examples: "mdc date" "python math" Google is the clear winner here, as well as many other scenarios where more specific searches yield incredibly higher quality results.
It doesn't surprise me at all that Google tends to handle the long tail questions better. --They're much larger and have had many more years to collect data, build synonym dictionaries, etc.
When I tried to come up with obscure queries, Google does better at the technical queries, and Bing is doing better at the non-technical queries. But the technical queries usually weren't that far off.
Yes, they're much larger, bot not much older. Microsoft was in search business years before Bing, and I also think they now have Yahoo!'s dataset too.
They're just better in search. I generally dislike Google (because I don't like their business model, which is advertising [I don't care for privacy much, just hate ads]), but nothing is as good as them in search, nor could be (for the foreseeable future).
Google's strength is not in searching for head or torso terms like "facebook" or "2013 calendar". It is in searching out tail terms; those obscure terms which might include a typo or two, and the phrasing may not be correct.
Unfortunately (and I don't have much time to explain this) one simply _can not_ judge a search engine's comparative performance using random users. And the Bing guys are falling into that trap.
EDIT: To expand more on what I wanted to write.
// DISCLAIMER: This is my personal view, and has nothing to do with my employer.
There are 3 large (fuzzy) classes of queries on search engines: head, torso and tail. The head queries are small in number but large in volume: queries like "Facebook", "Google", "Yahoo", etc. (yes, people do 'search' for these on search engines. Just a handful of such queries may make up 10% of the entire search engine traffic, which would be 100M queries per day (or so) on Google. It is depressing to see the numbers). In general, any search engine worth its salt will get the head queries right.
The torso is basically deciles 2-6 or so (there's no hard and fast definition). Here you'd have most musicians, actors, popular restaurants, etc. Bing and Google both would do a mostly adequate job here, for the first result. But: if you want to explore the concept a little, Bing goes astray and Google does not. For example, search for "idle hand tattoo" on http://bingiton.com/ . The first result is the same, to my local tattoo parlor. But the second result on Bing is from Mesa, AZ; while Google sticks with the right result and offers up other, related sites (Yelp, etc.). Google knows that "Hand" in the title is not the same as "Hands", and gives me not only the store, but also their Yelp, FB and Twitter pages; in other words: Google _understands_ your query better.
And finally, the tail. Here we have obscure error codes, weird technical terms, etc.; most of which are seen maybe once or twice a day. These constitute the bottom ~30% of the query stream. This is where Google really shines. It is very rare indeed to look for an obscure term and find that Bing does a better job than Google (I have yet to find an example).
So, as a user, I have a choice of 2 search engines; and one of them I know will do a better job on obscure terms. Guess which one will I pick?
When Bing claims that "most people prefer Bing in our user studies" (or something similar), their conclusion is flawed because the study itself is flawed. You can't sit a person down in front of a computer and ask him to evaluate your performance on tail queries! Where are those tail queries coming from? If they are provided by Bing ("here's an obscure term, search for it and tell us what you think"), then the user has no way to evaluate how relevant the results are, because, by definition, it's an obscure term! On the other hand, if you ask the user to come up with an obscure term, it will most likely be a term they are familiar with and they'll already know the answer, so they won't really hunt for information. So in my "idle hand tattoo" example, they'll see the top result and claim satisfaction; when, in real life, I would like to see the Yelp reviews, maps, etc. for the place.
If they (Bing) really want to compare how well they're doing, here's a suggestion: setup a search engine (like bingiton, but with only 1 pane of results, and neutral branding), and make it the default for a large pool of users (with their permission). Randomly, switch the backend to Bing or Google. Then, monitor the heck out of what the users do (all with their permission): how often do they click on the first result, how often do they re-formulate queries, the time to the first click, how often do they quickly come back to the results page from a bad click, etc. etc. etc.