Article printed from SiteProNews: http://www.sitepronews.com
  HTML version available at: http://www.sitepronews.com/archives.html

Are You Playing Hide and Seek with Search Engines?
by Mike Banks Valentine ©Copyright 2004

You could be HIDING your site from the Search Engines!

HIDING? Yes, you heard me right, I said hiding from search
engines! Let's take a look at a few of the ways you might do
that without meaning to do so. Hide and seek! Peekaboo!

Secure Server

Search engines do have the ability to spider secure server
hosted pages, but often these pages require either that a
visitor fill out a form or log-in with a password and user
name before being allowed past a certain point. If any page
requires filling out of forms or passwords to reach, search
engine robots will simply leave. They can't log in because
they can't fill out forms, leave email addresses or enter
passwords.

I was contacted by a webmaster for a 4500 page ecommerce
web site. He wondered why search engines were ignoring such
a large site. I asked the URL of the site and visited the
home page. I noted that upon loading, there was an immediate
passing of the URL http://anybusiness.com site to a secure
httpS://anybusiness.com page. This has two immediate faults
that may be a problem - the forwarding method and different
server. If the instant forward is by javascript, bad news.

First, search engines often either penalize or downgrade sites
that use immediate URL forwarding, especially from a home page.
URL forwarding suggests doorway pages (a search engine no-no)
or affiliate URL's forwarding to an affiliate program site, or
the worst of all scenarios, cloaking software on your server.
You may not be doing any of these things, but the robots don't
know, don't care, and don't index your site, plain and simple.

Secondly, secure servers are very often a separate web site,
meaning that the secure server is actually a different machine
and is an entirely different site from the non-secure server
site unless your site is hosted on a dedicated server on it's
own IP address, security certificate at the same domain. This
can happen when secure shopping carts are hosted by a third
party host so that a small ecommerce site needn't purchase a
security certificate or set up complex shopping carts.

For example, if your shopping cart is hosted by Yahoo stores
or other application service providers (ASP's), pages hosted
in the shopping cart don't reside on your domain and can't be
recognized as pages on YOUR site unless you also host your
domain with the same company. Unfortunately, many shopping
cart ASP's use dynamic IP addresses (IP address is different
each time you visit) and use database generated dynamic 
pages.

The process of serving dynamic pages is not the problem. The
problem is simply that the URL of those pages contains several
characters that either stop or severely curtail search engine
spiders. Question marks (?) are the biggest culprit, followed
by ampersands (&), equal signs (=) percent symbols (%) and
plus signs (+) in the URL's of dynamic pages.

These symbols serve as alarm bells to the spiders and either
turn them away entirely or dramatically slow the indexing of
your pages. This is stated simply in the Google "Information
for Webmasters" page 
http://www.google.com/webmasters/2.html

  "Reasons your site may not be included.

  "Your pages are dynamically generated. We are able to index
  dynamically generated pages. However, because our web crawler
  can easily overwhelm and crash sites serving dynamic content,
  we limit the amount of dynamic pages we index."

Just because your site is dynamically generated, creating long
URL's full of question marks, equal signs and ampersands like
www.domain.com/category.asp?ct=this+%28that+other%29&l=thing
doesn't mean you are in search engine limbo. There are simple
solutions available for your webmaster. Here are a couple of
articles explaining an elegant solution called "mod_rewrite".

You can read about that technique if technically inclined:

http://alistapart.com/articles/urls/
http://alistapart.com/articles/succeed/

This technique is simply creating a set of instructions for
your web server to present URL's in a different form that
replaces those "bad" question marks and ampersands with slash
marks (/) instead. The method will require that your webmaster
is a bit more technically savvy than most home business CEO's
who created their own web site. Some hosts will help here by
simply turning on the "mod_rewrite" for shared hosting clients.

Don't play hide and seek with the search engines! Tell them
EXACTLY where to find every page on your site and if there's
any question that they will find every page on your site,
give them a map.

A Site Map

Hard code those dynamic URL's for most categories within the
categories of different sections of your web site into your
comprehensive site map. As long as those dynamic links (even
those that include ?=+%& symbols) are hard coded into a site
map, the spiders will follow them. Clearly those 4500 pages
mentioned earlier would be too much for a site map listing.
But the main category pages could be provided for the engines.

I visited the site map page of the webmaster mentioned above
and saw fourteen pages listed on the site map. That explains
why they have fourteen pages, not 4500, indexed by Google.

How to find out how many pages of your site are indexed?
Go to Google search and type "allinurl:www.domain.com" with-
out the quotes. Replace "domain" in the above example with
your own domain name. This query operator will return a list
of every page of your site. Look in the blue bar across the
top of the Google results page and you'll see the number of
pages indexed at your site!

That should do it. Get indexed and stop playing hide-and-seek!

================================================================
Mike Banks Valentine is a Search Engine Optimization Specialist 
practicing ethical SEO for Online businesses http://SEOptimism.com
Learn SEO http://SearchEngineOptimism.com/SEO_Tutorial/
Then Take a Search Engine Quiz to test your Skill Level
http://SearchEngineOptimism.com/search_engine_quiz.html
================================================================

Copyright © 2004 Jayde Online, Inc.  All Rights Reserved.

SiteProNews is a registered service mark of Jayde Online, Inc.