Test Case: Google, Yahoo, MSN: Finding new content

Joel Bresler emailed me this morning and since his email is self-explanatory, I present it here (slightly modified):
———-
I have recently launched a web site dedicated to the American folk song, "Follow the Drinking Gourd." The site presents an interesting test case, for the following reasons: First, there has been virtually no new research on Follow the Drinking Gourd in twelve years. Second, this site represents the most comprehensive research published on the song. Third, some web pages that recently ranked highly on a "follow the drinking gourd" search just a few weeks ago were removed in February. I analyze search engine performance on several different parameters: Read more

Matt and Big Daddy

While I was gone, everyone was buzzing about Matt Cutt’s blog post, titled Indexing Timeline. I wanted to consolidate pieces of his post as well as his later comments in one place, so that those affected by the Big Daddy crawl/index issues wouldn’t have to wade through hundreds of comments to see everything. I have intentionally left some bits out that weren’t germaine to the issue, or were just repeats of things said elsewhere, or were obviously part of the original post, so they were easy to find. Feel free to go to the original post to see everything. So, here, in no rational order is my summary/compilation of Matt’s remarks.

Read more

Googleguy discusses crawling and supplementals

Over at WMW, Googleguy discusses one aspect of the problems people are seeing regarding crawling, indexing, and supplementals. Unfortunately, he only focused in on one small part of the whole thing, so there are still tons of questions to be answered. Nevertheless, those of you who are seeing strange things should head over there and read up about it. One of the things he mentioned:

Read more

PR = Deep Crawl

During the Meet The Crawlers session at SES, someone asked why all of their pages weren’t indexed. Google’s answer: Every search engine crawls in a different way. Mentions vs indexed. There are instances where we know about the url, but we did not crawl it. Your site may not have enough PageRank for us to do a deep crawl.

Read more

Indexing Speed of The Big Three

I love comparing Google, Yahoo! and MSN to each other. Today, it’s all about speed of indexing a new site. I launched a site 3 weeks ago, and put up a small smattering of links to it about a week later. So how do the big three stack up in terms of indexing the pages? Read more