What is Bing It On?
Bing It On is a valiant marketing effort by Microsoft search engine Bing. The campaign claims that People Chose Bing Web Search Results Over Google Nearly 2 to 1 in Blind Comparison Tests and is fueled by BingItOn.com, a website where people could see for themselves in a blind comparison test (similar to the old Pepsi Challenge comparing Coca-Cola and Pepsi) which search results they prefer, Google’s or Bing’s. The website strips formatting of the SERPs (although it doesn’t do a great job of this as things like author pictures still appear for Google) so that users are unable to distinguish between which search engine is on the left or right. The user can search using a custom search term or use the search suggestions provided by Bing. Search engines switch sides randomly, and the user indicates which side’s results they prefer. They can also answer that it was a “draw” if they cannot choose a winner for that round.
Bing Wins 2:1?
I was a bit perplexed by this campaign as I was certain about the superiority of Google, having compared the two of them before with my own side-to-side searches. However, I was open to testing objectively which search engine I actually preferred when using the Bing It On website. I was also interested in which engine my peers would choose.
My Personal Results: 3 Google, 1 Bing, 1 Draw
Bing’s study conducted by an independent research company tested “nearly 1000 participants [and] 57.4% chose Bing more often, 30.2% chose Google more often; 12.4 % resulted in a draw”. These statistics were intriguing, but I still didn’t quite believe the numbers seeing as the results of myself and a few friends all had results in favor of Google. Then it dawned on me that there was quite a bit of public data out there on Twitter. Bing was having people tweet with the hashtag #BingItOn. Finding the answer to which search engine people actually preferred would be as simple as extracting the outcomes from Twitter’s search function for the hashtag. This also confused me as I would assume that if the results did NOT favor Bing, then they wouldn’t want people tweeting this information since it would be contrary to their claims. I couldn’t understand why Microsoft would spend so much money promoting the Bing It On campaign on television and all over the internet if there was data that said otherwise.
What the Twitter Data Says About Which Search Engine Wins?
I was able to extract data from Twitter (ranging from 9/07/2012 to 9/11/2012) using an advanced search term for sample size of n=286. This data included foreign language tweets as well, and the results were determined using Google Translator. The data from Twitter showed quite a different picture than the data that Bing is claiming:
The Actual Results of Bing It On
|
Google Wins |
Bing Wins |
Draws |
|
| Number of Tweets |
208 |
69 |
9 |
| Percent |
72.73% |
24.13% |
3.14% |
While Bing is saying that people prefer Bing’s search results almost 2 to 1 compared to Google, my data shows that people actually prefer Google’s search results close to 3 to 1.
#BingItOn Results From Twitter
You may be asking yourself why Bing would promote this campaign so heavily if there is public data available that says differently? The answer is that the campaign carefully masked this data through an intelligent social media campaign.
Obfuscation of Twitter Results With @BingRewards Contest
I had expected that it would be as easy to collect the data as searching for the Bing It On campaign hashtag. However when I actually tried this, I found that I could hardly find any results. As it turns out Bing brilliantly handled the social media aspect of this campaign. Bing ran another (might have been more than one) campaign in tandem with Bing It On, all of which used the BingItOn hashtag. One of which was a contest which overshadowed results for the Bing It On campaign. In order to compete, it was necessary to tweet a reason why you started using Bing. With the myriad of tweets using the BingItOn hashtag, it was hard to clearly see the results to the Google v. Bing challenge.
See Tweet below:
Why did you start using Bing? Visit binged.it/QzI0rO & answer for a chance to win 100k @bingrewards credits! #BingItOn
— Bing Rewards (@BingRewards) September 10, 2012
It was necessary to use the following advanced search query to filter out unrelated tweets (after utilizing the advanced search terms, results required sifting with Excel):
#BingItOn -@BingRewards -"win Xbox" -"2 win Surface" -"People chose Bing nearly 2 to 1 in a blind test" -"some great prizes" -"bing rewards" -"The most popular searched question on Bing" -#spon
Another interesting thing to note is when you receive the results from BingItOn.com, you are presented with social sharing options. Sharing via this method however, does not share your results; it only shares the link for the campaign. It was necessary for the user to take manual action to share their results. Many people were sharing what they thought were results, but instead was only an invitation for their friends to try Bing It On.
Not important, but this video had 173 Likes and 678 Dislikes. The response has been mostly negative.
Which search engine’s results did you prefer: Google’s or Bing’s? I’d be interested to see what your results were in the comments below.
UPDATE: Barry Schwartz conducted a poll that yielded similar results to my study. He found that Google won 55% of the time, Bing won 38% of the time, and 7% experience a tie.
UPDATE: A post published on The Bing search blog pointed to a study conducted by SurveyMonkey (mentioned on Search Engine Land) where “more people preferred Bing results labeled as Google than Google results labeled as Google” as a result of a Confirmation Bias (Google is the best search engine).
UPDATE (10/3/2013): Freakonomics posted about a study conducted by a Yale professor, finding evidence contrary to the claims of Bing. See comments on Search Engine Land and (Bing’s Response).
I just did the “taste test” (4 Google, 1 draw), and also noticed that you couldn’t share your results from the results page. I think that says everything you need to know about their expectations. If they really thought users would choose Bing 2:1, you betcha they would have made sharing that as frictionless as possible. As it was, I had to Print Screen, open GIMP, crop image, upload image to FB, and write a snarky comment. So much work, but I have that kind of commitment.
The parallel promotion using the same hashtag was a brilliant tactic, though. Sounds like they had a reputation management firm involved.
@cjvannette Thanks for the comment! On an interesting note, Bing has hit an all time high for their market share (http://searchengineland.com/august-search-share-bing-hits-all-time-high-133021). I bet the Bing It On campaign had something to do with this.
@pshapiro @cjvannette
I’m not sure I agree. They’ve been on a steady pace of incremental growth for the last couple years, and this teeny increase is consistent with that and could easily be attributed to IE9 adoption. If we were seeing a jump of whole percentage points, we might be able to credit the campaign.
@cjvannette Actually, you are probably right about that. I think I got ahead of myself 🙂
I was thinking about the bias issue. It might be onto something.
The first time I went through the “taste test,” my queries were mostly medium-tail and informational, stuff like “iran contra crack cocaine” and “seo blog strategy.” I picked Google every time. The second time, I did shorter KWs and e-commerce or general-interest queries, stuff like “women’s shoes” and “Kanye West.” I picked Bing every time.
If Bing did relatively random sampling, I think they would end up with more participants who use search engines for shopping and researching basic subjects, because that’s what average people do most often. I don’t think average folks use search engines for complex informational requests like figuring out if Oliver North can be blamed for the decline of the American inner city. Bing seems to excel at presenting basic searches in an attractive way, while it seems to lag in parsing more complex queries.
So the question is, are these tweets biased toward more informational, longer-tail queries? It’s possible. Plenty of Twitter users love shopping and celebrities, but are they aware of BingItOn yet? Would they tweet about it? Or is it more likely to attract the attention of egghead Twitter users like me?
@cjvannette I love the thought process here. I have heard people make the claim that long tail searches will do better on Google than on Bing. It is important to note that things like geolocation, personalized searches, and search suggestions were stripped as well and could impact the search experience.
Fits with my experience. I typically only find Google results to be more useful if I’m looking for something obscure, which is why Bing is my default. That actually points to a weakness of the Bing it On! campaign among the tech crowd, which is more likely to try corner-case searches.
Mine was 5-0 in favor of Google, though to be fair one of them was close enough it could have been considered a draw.
I haven’t seen even 1 person in my G+ circles who posted a result favoring Bing. I suspect their survey was as objective and unbiased as their bought research showing MS products were cheaper than open source equivalents.
Bing made another blogpost about the Bing It On Challenge: http://www.bing.com/community/site_blogs/b/search/archive/2012/09/17/the-bing-it-on-challenge-takes-on-la-with-maria-menounos-and-derek-hough.aspx
They took on Los Angelas this time…and mentioned that you can take the challenge in Microsoft Stores nationwide.
I noticed today that @Bing is also using Promoted Tweets for this campaign.
I favored Bing 3-0-2. I personally like the overall look and the different sublinks that are under the actual site links in searches (like downloads and search bars from the website displayed directly on the search). I also find Google to be much more bias, showing many more results from the companies they own and support on the first two pages, as opposed to Bing, which was perfectly fine with showing youtube results right along-side their bing video results.
@JuniorCole Thanks for the feedback! Google often shows these sublinks as well, although I am unsure about how the frequency differs between the search engines. I would love to see search examples. -Paul
I actually discovered the youtube results (for the few of my searches that gave them) only appeared for Google. Same with the sublinks and search bars. The reason I know for a fact those were Google? I often preferred those qualities as well, and ended up favoring Google 4-0-1 .
Thank you. I’m so glad someone is finally talking about this. I was about to write a blog post calling their bluff, but you did it for me. 🙂 Also, I think it is important to note that both Google and Bing (in the Bing It On Test) were stripped of all extra content. For example: Google has advanced search results like; flights, hotel search, shopping results and “knowledge graph” results.” Bing has an annoying (IMO) UI that is crowded with their Facebook feed, and their ads. One of the reasons I use Google is because the experience is better all around.
Great post.
@Bryant Jaquez Thanks for the comment buddy 🙂
Go ahead and write your post if you want.
Excellent point about extra data being stripped. THIS IS A BIG DEAL. Since Google is putting so many of their eggs in the “knowledge graph” and “semantic web” basket, this heavily effects results.
I also personally think they should have done a better job at stripping information from the searches also (since they this was their goal). It wasn’t really a blind experiment design for me (and I’m sure others), so it possible that biases are still a factor.
The problem with your methodology is that you ignore a major component of error in your presentation of results. Let’s think about who takes the Bing it on challenge. I think it’s fair to say that most participants favored Google before the trial rather than Bing. If most people favored Google, and some percentage of them got Google as their preferred search engine, it goes to reason that they would “brag” about their “victory” online.
If a Google fanyboy tries it and receives Bing as their better engine, they will be less likely to flaunt their “failure” online. As a result, you will have a disproportionate number of Google fans posting their results.
@Demonstros You are absolutely correct. If this were to be a more academic study, that would be considered one of the sources of bias. I would like to direct you to Barry Schwartz’s poll which yielded similar results:http://www.seroundtable.com/bing-challenge-google-15758.htmlAlso I noticed this recently…Bing update their Facebook cover photo to reflect the Bing It On campaign:
https://www.facebook.com/photo.php?fbid=10151498539053438&set=a.10150843234883438.512162.70424008437&type=1&theater
The comments also reflect a great deal of pro-Google results. This is intriguing since you wouldn’t expect “Google fanboys” to like the Bing Facebook page. What do you think? I personally liked the page to keep a tab on Bing.
My 3 tries come up 2-1 Google. Same results 3 times ,2 Google, 1 Bing ,2 undecided.
Google won all five points.
Google 4 to 1 in most instances
I did it multiple times, and not once did bing win.
I prefer DuckDuckGo to both Google and Bing. But I will say this, I admittedly am bias to be anti Microsoft period, they have done shady “results” with other products of theirs such as DirectX if anyone’s forgotten. Granted, a few search results were nice to have bings pictures, but with others, were simply annoying and in the way. DuckDuckGo, If you haven’t tried it give it a chance, its a GREAT search engine.
Thanks for dropping by. DuckDuckGo is great (but I end up just using the !bangs and don’t use the actual engine). Another great alternative is Blekko.
lol i do that when im looking for wallpapers/images, which honestly i do use bing for. my brother in law likes bing because it gives him rewards, which admittedly is enticing, bust still… microsoft… no thanks!
GOOGLE WON 5 OUT OF 5 !!!!!! YEAH
GOOGLE 5 OUT OF 5 YAH GOOGLE !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
I just did this at bingiton.com, 8 wins for google, 0 wins for bing, 0 ties. The best part about this is that when i tried to get Bing as a winner, I chose the least helpful results, and Bing won that. 😛 When google wins, the website asks for a rematch, but when bing wins, the website just says “try bing, get rewarded”. Hahaha
I tried it,got google 5 bing 0.
tried again google 5 bing 0
tried again google 3 bing 2
tried again google 5 bing 0
tried again google 4 bing 1
lol i did the test twice and got google every time on both tries :p
I love my google!
I got Google 3 out of 5 the one time I tried it. I had a feeling that ad was BS. Every time I’ve even attempted to use Bing I have not liked the results. Some one else said that they get more YouTube videos from Bing, but I don’t think that’s true. Every time I Google something if there is a YouTube video it’s near the top of the findings and sometimes that’s all I get. Google is by far BETTER!
I just did it and got Google for all the answers. Bing is a total piece of **** and will never be better than Google.
There were very subtle differences. I asked “Who is Spongebob?” And only one of the searches provided info on Tom Kenney. I searched “ps4” and only one of them had a 17-min-old article with new information. It was obvious I had to choose those pages, but I can’t believe Google one-upped bing every single time. 5-0. See for yourself, here’s a picture:
fbcdn-sphotos-e-a.akamaihd.net/hphotos-ak-ash3/625428_10200982165104807_858433866_n.jpg
Bing obviously has a long way to go to compete with Google.
I just took that test, I selected Google 4 times and a draw on the last selection.
The results were Google,Google Google, Bing it on, Draw lol and stated that Bing won lol.
Obviously Microsoft inserted itself in the fourth pick lol. What a joke. 🙂
I did the Bing It On and I never got Bing. I don’t like the layout of it all
Did the Bing thing… Never again… The big G rules!
i preferred google 4-0-1 the only one i chose bing over google is when searching for porn. i find bings porn results better than googles this is not a troll post, im serious, i even did a google search on bing being a great porn search engine and found a lot of results of people who feel the same way!
Sorry Bing you’re either very wrong, delusional or just flat out lying. I chose 5 random search enteries and Google provided the most and best information hands down, all 5 times.