Do you ever wonder what the process Google and Yahoo use to index your webpages is?  The following analysis is based 100% on my personal experience.  Hope you enjoy!

Google can find you almost instantly and then indexes your site in small chunks while revisiting lots of times. If you update your content regularly, or add new content regularly, you go in the “every-day-visit” from the Google-bot category. Google has more filters too, and its index is volatile… so your duplicate pages can easily fall from the index (They dont exactly fall - they go to supplemental results, which is practically the same).

Yahoo finds your page a bit slower than Google does, not significantly slower though. It indexes your main page and then checks your backlinks. At some point, something flips, and slurp decides to spend 4-5 days (depending on your site’s size) rummaging through your pages and bandwidth. Then almost 99% of your pages appear in its index, and yahoo starts sending constant visitors. If your pages are not in the index after the Inktomi raid, then your pages have some sort of problem.

As for MSN… I don’t really know. I think it finds your index page a few months late, and then decides to index one more page every 3 month.