Research & Development for 3D & the Web

Bots, Spiders, Crawlers

The job of collecting information about web pages is performed by automation using an agent called a ‘crawler, spider or robot (bot) which is different depending on the search engine. They look at every URL on the internet and collect keywords, links, relevant data and phrases on each page. Before it can do that, the network administrator must allow these agents to look at the content the website provides. Crawlers/bots etc. look at the coding behind the scenes too.

Google have made it pretty easy to see when ‘Googlebot’ has visited your webpage. You can install the ‘Google Toolbar’ to view this which can be downloaded for free from http://toolbar.google.com. Or else in Google Chrome and I’m sure in Firefox, you can get addons for you browser which should let you view page statistics. There were 7 or 8 listed when I wrote this for Google Chrome anyway.

Page rank is a score from 0-10. Ten is highest and zero is lowest. If you look at your own site and your competitors you can see who will appear first in the list of search results based on this too. Without professional SEO help, most pages will get a rank of around 2-3.

Because the googlebot really only reads in text, an amazing site all done in Flash can sometimes be ranked 0 when it could be higher with the inclusion of some text, as heartbreaking as that might be..

Another important thing is ‘Backwards Links’ which are links which link directly to your site from elsewhere. The more you have from relevant sites that Google knows about, the higher your Page Rank will be.

Only about 20-25% of your ranking for the major search engines comes from what the users see. The rest comes from the coding and the other things behind the scenes. Bots look for certain things when crawling through your site, I’ll try to cover each of them in the next series of posts as I learn about them.