Making Your Site Web Crawler Friendly
Posted on | April 1, 2005 | No Comments
If you’ve been reading up on search engine optimization, you know how important it is for your site to be “web crawler friendly”, which is another way of saying “search engine friendly”.
But what does it really mean and how can you achieve it? This article will look at ways to improve your site in the eyes of the engines and their little envoys, the web crawlers.
What Web Crawlers Do
Web crawlers, or search engine spiders, visit sites all over the web, following their internal links and “reading” each page. Everything they find goes back to their search engine’s index, a big catalog that holds a copy of every webpage that’s been web crawled. Whenever a website is changed – say some new information is added – the web crawlers will revisit (once every few weeks or months) and send an update to the index. This “indexing” is the first part of the process.
Next, the search engine goes to work sorting through all these pages to establish their relevance for a given search query. This process determines the site’s ranking, in other words how highly it appears in search engine results pages (SERPs).
So you can see there is a direct relationship between the information that gets web crawled and your site’s eventual ranking. Having a web crawler friendly site can make the difference between a top 10 ranking and being buried on page 28 of the SERPs.
Is Your Site Web Crawler “Readable”?
There are many aspects of your webpages you can change in order to make them more friendly to the web crawlers.
Keywords
Selecting the right keyword phrases – the ones your target market is most likely to use when searching for the products you sell – is an artform unto itself. You can learn more about keyword research here.
For the purposes of this article, let’s just say that the keywords you select and put in your meta tags, your on-page content, and in your incoming links will play a big role in how the engines index and rank your site… so choose wisely!
Meta Tags
If you want the web crawlers to come crawling to your door, put your top keyword phrases in your title tag, description tag, and (optionally) in your keyword tag. These days, few web crawlers bother reading the keyword tag so its importance has diminished, but the title tag is highly revered by all search engines and should contain your top one or two keyword terms.
The description tag is shown in the search results returned by some engines, so it needs to be written like a mini-advertisement for human readers as well as optimized for web crawler action.
Content
The content of a website has become so important to the search engines that very often the first job an SEO tackles, right after selecting keywords, is ensuring the text of each page is optimized… in other words, the copy has been skillfully peppered with the selected keyword terms.
Search engines count keywords in content, giving them a convenient tool (one among many) to help them match a webpage with a search query. (Learn more about content optimization.)
Behind the Scenes
Factors hidden in your site’s HTML code can block web crawlers, so any SEO professional worth his/her salt will check your code to make sure it’s accessible. They will look for impediments such as flash-based programs, dynamic databases, frames or text embedded in graphics, all of which can render a site less web crawler friendly.
Responsible SEOs will also remove any spammy tricks they find in your code and discourage you from using dubious tactics such as invisible text loaded with keywords. These short-term fixes can have long term negative consequences when the search engines catch up with you.
These days, many SEOs also encourage the use of site maps to help the web crawlers navigate from your home page to your inside pages – especially important if your menus don’t use text links.
Off-page Factors
This includes things like link popularity and click-throughs. Search engines analyze how pages link to each other and gage the importance of a linking page as a factor in determining your rankings.
And lately the engines have become increasingly adept at filtering out linking strategies they deem suspicious or artificial. Your best bet is one-way, unpaid links from high-ranking sites related to yours… in other words, relevant and important sites.
In fact, the engines are so obsessed with “importance” they even monitor click-throughs, dropping the ranking of pages that don’t get a lot of visitors and promoting pages that do… just one more way in which your site is rewarded or punished depending on its search engine traffic.
Summing Up
Since the search engines change their criteria on a regular basis, what constitutes web crawler friendliness today may be somewhat different tomorrow.
But there are some time-honored optimization strategies you can depend on: having clean HTML code with no tricks; using good keywords in the right places; and building link popularity from related sites with high rankings… these will always work in your favor.
More Web Crawler Resources
- Googlebot: Google’s Web Crawler
- Yahoo! Slurp: Yahoo!’s web crawler
- Web crawler definition by Wikipedia
Comments
Leave a Reply