Tuesday, February 9, 2010

Microsoft 90 days challenge to beat Google

Microsoft engineer Chad Carson wasn't thrilled about surrendering his solo window seat on the Alaska Airlines flight from San Jose to Seattle so he could talk shop with his boss Sean Suchter and colleague Eric Scheel.

But that innocent decision last July 22 would spark a 91-day sprint to a previously unreached Internet milestone.

By the time Flight 321 was over Oregon, the group in Row 6 had evolved from a technology klatch to a cabal of plotters who scrawled a schematic tangle of boxes on a sheet of paper to map out something no big Internet search engine had yet achieved. The three members of Microsoft's new Silicon Valley search team would try to make their company's Bing a window into America's stream of consciousness, serving up the chatter on Twitter and blog posts, with the latest updates on everything from celebrity gossip to breaking news.

Real-time search, as it was called, was a daunting technical task. It required a system that could ingest the fire hose of Twitter data, pluck popular topics from that digital torrent using entirely new computer algorithms, and — within seconds — allow people to search them.

"We looked at each other," Suchter said of the dawning excitement in economy class that day, "and we said, you know, if we got those three things together, you know, we could make this work!"

Even in the summer of 2009, the big search engines could produce only a static snapshot of the Web at any


--------------------------------------------------------------------------------

Advertisement

--------------------------------------------------------------------------------
one time. Building the first mainstream real-time search engine would be a PR and industry coup for Microsoft. Bing had been well received following its launch in June, but it remained a distant third to what has become America's default search engine — Google.

For Suchter and his new team, the next three months would bring the exhilaration of plotting a coup over a higher-profile rival, and the bone-weary fatigue of trudging into their Mountain View office for 31 straight days, facing yet another caffeine-fueled 12-hour session.

And there was anxiety. Less than two days before Microsoft execs were to appear at a leading Internet conference in San Francisco, with rumors rife about a big announcement, Suchter's team still didn't have their cool invention actually working.

A valley team

Bing's center of gravity remains in and around the company's Redmond, Wash., headquarters; perhaps 5 percent of the company's search resources reside at its Mountain View campus.

But a year ago, Microsoft recruited Suchter from Yahoo to build a new team in Silicon Valley, one that would draw on the bountiful local talent and experience in Internet search to quickly build new, high-profile products on the Bing platform.

"Suchter" literally means "searched" in German, so his career may have been predetermined. A stocky, unassuming 34-year-old who moved to Silicon Valley in 1998 after graduating from the California Institute of Technology, Suchter was a valley search veteran who had been Yahoo's vice president for search technology since 2003.

A self-described "serial" home remodeler with his wife — "I just like to build things" — Suchter drives an ultrafast Yamaha sport bike. "It's all about the feeling of moving fast," he said. An internal Yahoo e-mail that became public after Suchter quit in late 2008 called him "a Gibraltar rock at Yahoo."

Early last year, the newly installed general manager of Microsoft's Search Technology Center Silicon Valley set about assembling his team, bringing in a roster of talent that included veteran engineers like Carson who had toiled for generations of valley search companies like AltaVista, Verity and Yahoo.

There were valley heavyweights, such as former IBM database expert Ashok K. Chandra, a professorial presence who sounds like a poet when he compares creating computer algorithms to the view from the summit of Mount Whitney. But Suchter also wanted young blood, recruiting newly minted Ph.D.s like Stanford product Shubha Nabar, with just 18 months in search.

"We all look at each other and say this a lot — we're basically a little startup within Microsoft," Suchter said. "This is the team I would have gotten, were I starting a startup."

Now the little startup in the belly of the whale had to deliver.

Getting the go-ahead

The afternoon of the Seattle flight, Suchter stood before his boss in Redmond, Harry Shum, and pulled the dog-eared sheet of paper from his back pocket. This, Suchter told Shum, handing him the marked-up page, is what the team wants to do.

"I know I've got to get worried when you're giving me your plans drawn on a piece of paper and not in PowerPoint," Shum said. But he approved the effort.

Real-time search was a hot topic. As Microsoft launched Bing in June, the explosive growth of social and microblogging sites sparked a growing critique of Web search — that it was static, because it couldn't capture the growing legions who peppered the Web with their chatter on Facebook and Twitter.

What Suchter, Carson and Scheel had seen was that several components of a real-time search engine already existed within Microsoft, including technology to ingest the Twitter stream. But it was considered a garage-style hack, not a potential consumer product. Suchter's team worked with the Microsoft researchers developing the tool to make sure it could do what Bing needed.

The biggest technical challenge for the Microsoft team wasn't digesting the stream of Twitter data. It was writing their algorithms to surface tweets and blog postings that had what search engineers call "social authority."

Just like e-mail, people use Twitter to spam.

"We found one guy we called 'the movie spammer,' " Suchter said. Chatter about movies is a popular topic on Twitter, and people like the movie spammer set up multiple Twitter accounts to generate what appears to be a community chatting about "Avatar," when links in those tweets might actually go to a Viagra ad.

The team had to give Bing the ability to instantly detect patterns of use that would block people with "negative social authority" like the movie spammer, and elevate the tweets of real people commenting on the movie.

That meant they had to develop new algorithms at the forefront of search technology that would determine significance, weighing "the newness and the hotness and the information content,'' Suchter said.

By early September, at an all-hands meeting at Microsoft's Silicon Valley campus, the team was ready to show a demo. Microsoft set up a screen in the parking lot — concealed from the adjacent Highway 101, knowing Google was right around the corner.

The team got the system working 40 minutes before the demo. "I was pretty stressed," Suchter said.

The demo showed how real-time search would work conceptually. Now the team had to make their algorithms run on Bing's systems, and the pressure was just beginning. Hoping to make a splash at the influential Web 2.0 Summit in late October in San Francisco, the company decided to move up the launch, scheduling Qi Lu, yet another prominent former Yahoo executive, to speak.

Google co-founder Sergey Brin would be there. Many in Microsoft felt the valley's perception was that younger, more nimble innovators like Google had more technical agility than lumbering Microsoft. They itched to change that perception.

Launch preparations

Between Sept. 21 and the scheduled launch Oct. 21, the team came to the office every single day.

The team was constantly tweaking code within its new algorithms in an effort to make them run faster and be more accurate. But just four days before Web 2.0, they realized they needed to make a major change to the software architecture. In addition, it had become clear that bing.com/twitter would require double the number of servers within Microsoft's data centers.

"That kind of thing — you're making decisions right before you're going to launch — that's what you do at a startup," Suchter said. "That's exactly the feeling we wanted to have. This is what the team is like. This is what being in Silicon Valley is."

Nobody saw the product completely put together until the day before the Web 2.0 Summit, when the engineers switched it on and watched Bing spit out a constantly updating stream of banter about world events, technology and celebrities.

"I feel like I've got my finger of the pulse of the nation," Nabar blurted, catching the mood of the giddy engineers.

Microsoft's PR team didn't get the green light until the night before the launch, and reports of the new product dutifully appeared in tech blogs the next morning, drawing hundreds of people to a conference ballroom. As Microsoft executive Yusuf Mehdi took the stage to unveil the product, an engineer in the audience picked up his cell phone and texted Redmond.

They were live. Applause erupted in the ballroom. A few hours later, Google hustled out an announcement that it, too, would soon offer real-time search. Its version launched Dec. 7.

How did it feel to beat Google?

"That was fun — retroactively," Suchter said. "We didn't know we were going to catch them. We kind of thought we would, but who knew?"

No comments:

Post a Comment

 
eXTReMe Tracker