You always seem to hear about the “spiders” that search engines send crawling the web and unless you are one of the “techies” you probably really do not understand what these little creatures are. These web spiders are also known as “web crawlers, web robots or web scutter are simply programs or automated scripts that browse the web in a methodical automated manner. These things have also been called ants, automated indexers, bots and worms..

This automated search process is called web crawling or spidering and many sites use them to scrape up dated information from their search. These crawls now are mainly used to create copies of all visited pages in order to later index them creating faster search results. However a spider is also capable of being programmed for automatic maintenance on a particular site such as verifying HTML code or checking the links provided. And for the real creepy life of a spider it can be programmed to crawl the Internet harvesting email addresses to be used in a nefarious manner. This manner is usually SPAM and is where the black hat boys and the white hat boys of internet marketing seem to split pathways.

The crawl generally starts with a list of URLs to visit which are called seeds. As these URLs are visited the relevant hyperlinks are identified on the visited pages and the spider adds them to the list of URLs for visits. Then these are later visited again seeking certain search parameters which will aid in the indexing of the visited page.

Now confused just yet? Then lets visit a bit on these parameters. These are set for certain preferences of the program and if we think of the spider searching for “food” then whatever content we place in our websites better be as tasty to these little crawlers as we can make it. It would just seem natural that is we give the crawler what it wants then there will be return visits and the ensuing indexing should be balanced in our favor. Huh??

Probably better for all if we leave the email harvesters to their dark deeds and take our lessons to what will benefit us monetarily…..SEARCH ENGINES!! Away we go!

SPIDERS LIKE:

–keep your code neat on your sites. The fewer lines of code you have in relation to the amount of
text the better off you are.
–Since this is always about keywords you need to pay attention here___AGAIN!. Keep your keyword
density within a normal range of 3%-7%. A lower percentage will cause your content to not be
recognized and a high percentage is just stupid!
–any of your site pages that link back to your home page should just be crawling (Pun
intended) with backlinks. Many of the guides you read state that top quality sites have averages
of over 300 backlinks.
–Unique and Original will always be the descriptive words
regarding your content. If you duplicate what is all over the web why bother? Just be a bit
inventive and put yourself into your work. Brand it as your own and build your own reputation!
–SITE MAPS>>>this is the big one that everybody is pushing now. Create a site map of your site
so that the crawlers will have a road map to what they are looking for!

Now for a bit of what the spiders do not like:

–Not to be repetitive but spiders DO NOT LIKE to come upon pages with more code than text. Keep
it simple and write real words not just a bunch of code!
–nested tables. It appears that when you do this you present more “markup” code than actual
informative content. When you attempt to present content through tables it comes with the
price tag of rather large HTML files. While not a big problem with broadband these present quite
a bit of difficulty with dial up connection and now that many are accessing the web through their
cellphones and other hand held devices a large file is just not attractive. And if it is not attractive
and useful to the visitor then the search engine spider wants very little to do with it.
–do not be a keyword stuffer. Keep your keyword load to the average 3-7% and you will be quite
a bit more attractive with each crawl.
–use very few doorway pages as these relate to high density keyword content.

Well, just a bit of spiders, crawling and webs so not it is bed time. Time to learn what the power of suggestion will be to my dreams with these images in my mind.