Article Categories

• Advertising
• Affiliate Marketing
• Article Marketing
• Blogs & Podcasts
• Branding
• Business
• Cloud Technology
• Ecommerce
• Email Marketing
• Facebook
• Google
• Keywords
• LinkedIn
• Linking Strategies
• List Building
• Local Search
• Marketing
• Miscellaneous
• Mobile Applications
• Page Rank
• Pay Per Click
• RSS
• Sales Copy
• SE Optimization
• SE Positioning
• SE Submission
• SE Tactics
• Search Engine Marketing (SEM)
• Security
• Social Media Marketing
• Social Networking
• SPAM
• SPN Featured Articles
• Technology
• Video Marketing
• Virtual Office/Telecommuting
• Web 2.0
• Web Design
• Web Development
• Webmasters
• Website Promotion
• Website Traffic
• WordPress
• Writing

SiteProNews Blogs

• Editor's Corner
• Darren Dunner
• Jerry Bader
• John Sylvester
• Kalena Jordan
• Mel Strocen
• Titus Hoskins

Search, the Final Frontier

By Robert Cerff in SE Submission

“Second star to the right and straight on till morning.” – James T. Kirk

I am still amazed on a daily basis just how many people don’t know, don’t realize or simply don’t care just how powerful the average internet browser can be. It almost seems as though the average user either has their browser homepage set to their own website homepage (usually by the technician that set the network up) or to a search engine or directory of preference.

I have seen some people enter email addresses into the web address bar or even a full URL’s into the search bar. Some would even argue that entering the URL into the search bar and hitting the infamous “I’m Feeling Lucky” button is quicker than having to move the mouse to the address bar and typing it in. While this may be true, I’m still not buying it. But after looking into our larger client’s websites it would seem that the company name is more often one of the top 10 keywords. I would imagine that many fail to bookmark or add the website to favourites.

While it is very easy to gain a top ranking for your company name, especially if your domain name matches, how many know how much potential traffic (good qualified traffic) is lost by not ranking first for your own company’s name. Some marketers even bid on their own name in paid ads simply so that nobody else can rank higher for searches. Sometimes your domain name won’t match your company name; perhaps you’ve opted for a slogan or keyword phrase as a domain name. Will your website still be returned at number 1 for your company name?

Consider the following: You have spent thousands (insert any currency here) of your hard earned pennies into offline marketing. Slowly over the years you have created a good well established brand. A neighbourhood brand if you wish. Then, as time passed, you decide that you would move with the times and seek the additional benefits of online marketing. After having a website created, you just sit back and wait for the traffic. After all having a website guarantees you success online, doesn’t it? Surely you will be successful online, especially if you already have an established market?

Sometimes with a bit of luck you websites can indeed rank well for good converting keywords. This usually requires a lot more work. But surely your website will rank top for your company’s name? This may come as a shock, but, “NO!” Quite often you will find an article or a page of information that may actually rank higher for your company’s name than your very own website. This won’t come as a surprise to any seasoned SEO consultant.

“Okay, so what does this Matter?” Some may ask. Well while the search may lead to another website that endorses yours, many times it does not. Can you imagine the damage that could occur if someone searched your company name, only to find that some blogger had taken your entire customer support department to task. Even worse than that would be if you didn’t even know. Imagine losing all those years of building a good solid, respected brand name only to have it ruined in moment because of a careless search.

Going back to my original point you can see how valuable it is that you have a top ranking for your company name. This is just one incident where the variations of how people search, and more importantly what they can find, that you should consider when marketing your website.

Some web browsers have a built in default search function that will return a search results page should a URL not be found. Even if the searcher was to put in a correct URL it may occur at times that either the site or the network are simply too busy to return the correct results. If these search results are returned, will your website have the number one ranking?

As any marketer would suggest, study your market. Learn the search techniques of those that you expect to be searching for you online. By doing this you will pick up those points that are often overlooked. While the goal remains a number 1 ranking for you key search phrases sometimes you can harvest a lot more traffic by just getting the smaller details right.

Robert Cerff is a search engine analyst and marketing consultant for Prop Data Internet Solutions. He has ten years experience in e-commerce, online marketing and web development. http://www.propdata.co.za

Why Starting A Relationship With The Search Engines Can Be Hard

By Erin Ferree in SE Submission

In a singles bar, patrons typically have the same sorts of insecurities running through their heads:

“I’ve been hurt before.” “Should I trust her?” “Can I let my walls down and let him in?”

These insecurities really get in the way of finding a happy, lasting relationship.

Search Engines have a lot in common with singles. They’ve been hurt, and they aren’t so quick to trust any more.

What You Need To Know About The Search Engines’ Pasts

Search engines have had a difficult relationship history. They’ve tried to be open-hearted and trusting. But unethical website owners, spammers, and other cheaters have used their trusting ways against them and worked their way into the top of the rankings with lies.

And search engines get hurt by that. Their job, above all else, is to help the people using them to search for information. When spammers rather than valid websites get to the top, the search engines can’t do their jobs well. That frustrates searchers, and the search engines don’t help them find what they need.

So how can you get the search engines to trust your website enough to list it? The answer is to know what they are looking for when you start a relationship with them.

Don’t just give them another line (of code)

The search engines no longer use the META Keyword tag when ranking your site. This tag is hidden at the top of your website’s HTML. It used to be one of the major areas the search engines looked at when determining what your site was all about.

The intent of this tag was for site owners to list words and phrases explaining what their sites were about. But it was rarely used in ethical ways. People would list terms they thought would help their sites come up in popular searches “free,” “money,” and “sex” were all popular back in the early days of the Internet.

After a while, this practice was so widespread that the search engines decided to do something about it. They stopped looking at this tag at all.

These days, the most common use of the META Keyword tag is your competitors researching your optimization strategy. If you already use a META Keyword tag, consider having it removed. If you’re coding or optimizing your website, leave it off and concentrate on putting your keywords in the body text of your website instead where they belong.

They want you to put all your cards on the table.

A while ago, some website owners decided they wanted to have very little copy on their sites. They still wanted to come up strong on the search engines. The search engines were no longer really looking at the META tags. The problem was how to have a site that looked as though it had very little text but still had enough information for the search engines to give it a good ranking.

Some of these website owners came up with what they thought was a brilliant solution. They’d add more text to the site at the bottom of the page below the copyright. And they’d make this text the same color as the background which basically meant that it was invisible to human website visitors. But the search engines would still see it in the code, read it, and count it toward their ranking.

The search engines caught on to this little trick pretty quickly. If they catch you at it, they’ll blacklist your site, which means that they’ll kick it out of their results listing. And it will be really difficult to get back in! So make sure our site isn’t using any invisible text and that all the text on the site is visible to humans and the search engines.

They hate doors and mirrors.

Have you ever searched for something online and clicked on what you thought was the perfect match, only to come to what was little more than a full-page advertisement or a bunch of links? Disappointing, isn’t it? And then there’s a link at the bottom of the page pointing to the site you originally thought you were originally going to? By this point, you already have a sour taste in your mouth.

Or you’re looking for a piece of information and you click a couple of links that look like exactly the same site with slightly different words. You get a sense of deja vu and start to feel a bit like you’re losing your mind, right?

This tactic is called doorway and mirror sites. The search engines feel exactly the same way that you do about them because, again, the search engines want searchers to be happy with the results they get from their queries, not disgruntled or confused.

When you start your relationship with the search engines, make sure that your site optimization strategy focuses on ethical ways of getting their attention like being content-rich, having links with other high-quality sites (not link farms!), and updating your site on a regular basis. Those tactics will ensure that you start a love affair with the search engines instead of just heading for a bad breakup.

Erin Ferree is a brand identity designer who creates big visibility for small businesses. As the owner of elf design, Erin is passionate about helping her clients stand out in front of their competition and attract more clients. One of the best ways to do that is with Search Engine Optimization, which you can learn about in her eLearning product, Raise Your Ranking, which is available at http://www.howtoraiseyourranking.com.

The Ultimate Guide to Directory Submissions

By David Eaves in SE Submission

Submitting to directories is a great way to build links and increase your search engine rankings. In semi-competitive industries it can produce great results. If you add article creation and social media marketing into the link building equation then you can achieve great results for any industry.

Free or paid directories?

When choosing directories going for the paid ones can be better, mainly because the links seem to last a lot longer. Many of the free directories seem to disappear or delete links after a year or so. However, there are a few free directories out there that should always be used – directories that have stood the test of time.

Niche or general directories?

Submitting to a combination of both niche and general directories is usually a good idea.

For most industries there are a variety of niche directories out there – the best way to find them is through a search engine. Do a search on Google, MSN or Yahoo for directories in your niche area – those that come up in the first few pages of results are usually the best ones to use.

With general directories it’s better to go for those that are more established. The older a directory is, the better.

PageRank – does it still matter?

Because directories are generally quite large, they need a certain amount of PageRank to get all of their pages indexed properly. However a high PageRank isn’t the be-all and end-all. Google’s recent crackdown on directories1 has made visible PageRank even more irrelevant when it comes to choosing directories. There are directories out there that have no PageRank whatsoever that can offer value.

If a directory is ranking well in the search engines then you can rest assured that it has enough PageRank. If you’re unsure then check how many links it has via Yahoo! Site Explorer 2 or another link popularity checker.

Anchor text

It’s always best to get the main search phrases that you are targeting in the links to your site. However this isn’t always possible and unfortunately many of the best directories won’t let you do this. Sometimes you can get around this problem by slipping in a search phrase at the end of your company name.

Before submitting have a good look around a directory and get a feel for what you can and can’t get away with. Some directory owners will let you use a search phrase on its own, whereas others are very strict and will only use your business/website name. In between you have those where you might just be able to slip a search phrase in.

Doing your homework comes in handy – if you try to use a search phrase on its own and they change it to your business/website name then it’s very unlikely you’ll be able to get them to change it to your website name with a search phrase at the end. If you’d submitted it like that in the first place you may have got away with it.

Always try to get one of your search phrases in and vary the anchor text as much as possible – this will appear more natural to the search engines.

Varying your description

Many directories will provide you with your own page about your business. If you have the same description on every page across different directory websites then many of these will be seen as duplicate content by the search engines and your links will get devalued.

To avoid this, write a unique and substantial description for every single directory (200+ characters works best). Make sure the descriptions accurately mirror your products and services and that they read well.

Deep linking

Many directories allow you to add extra links directly to internal pages of your website. You should take turns in linking to different pages of your website using different variations of the phrases you’re targeting on each page. Using the same anchor text to link to the same page over and over again will appear unnatural to the search engines and this could work against you.

How many directories should you submit to?

There’s no fixed number of directories that you should get listed in. Work out a 12 month directory submission budget for each site and then do so many each week or month for the full duration.

When you’re building links to your site via directories or any other method, you should do it over time. Submitting to 100 directories in a week and then forgetting about it won’t be as effective as spreading the 100 directory submissions over a 3 month period.

The top 10 directories

Finally, here’s a list of the top free and paid general directories to get you started.
5 of the best free directories include:

• Open Directory Project
• World Site Index
• Domaining.in
• Web World
• Search Sight

5 of the best paid directories include:

• Yahoo! Directory
• Best of the Web
• Business.com
• Aviva
• Ezilon

Author:  David Eaves has been working in the search engine optimisation13 industry for the last 5 years. More of his articles can be found at his SEO blog.

All Search Engines Love Spiders: How Meta Commands Can Help You Love Them Too

By Scott Buresh in SE Submission

Nearly all search engines utilize spiders (which are also known by their original name, robots) to go out and scour the web looking for web pages. These search engine spiders then bring the data back to be indexed by the engine.

Since roughly 1996, individual meta commands have existed that can be used on individual web pages to modify how these search engine spiders behave. The most useful of these commands are fairly universal and respected by almost all search engines. What follows is a list of some of the more popular spider commands and instances in which you might want to use them.

<meta name=”robots” content=”index”>

This meta command is one of the most common ones used – and it is also the least necessary. It tells search engine spiders to come on in and put the page in their index. However, all search engines do this by default anyway. Basically, if you want to put it in there for fun, be my guest, but this command is not giving you any special treatment. All search engines are going to index your page, unless you specifically tell them otherwise.

<meta name=”robots” content=”follow”>

The follow command is different from the index command. It basically requests that the search engine spiders follow the links that are on a particular page. Again, however, this piece of code is completely unnecessary because all search engines are going to follow the links on a page, unless otherwise directed.

<meta name=”robots” content=”noindex”>

The noindex command, the opposite of the index command, tells search engine spiders not to index the content of a page. It’s important to note however that search engine spiders will still follow the links on a page that uses only this command.

When not used for legitimate purposes, this tag can be dangerous because it can put you at risk for penalization by most, if not all search engines. This is because you can use a noindex tag to hide pages with multiple links that you don’t want visitors to see but that you do want all search engines to index.

There are however some legitimate uses for the noindex command. For example, if you have a dynamic site and you’ve created static pages to replace some of your dynamic pages, which can make them easier for search engine spiders to access, you could put a noindex tag on the dynamic version.

As Google mentions in its Webmaster Help Center:

“Consider creating static copies of dynamic pages. Although the Google index includes dynamic pages, they comprise a small portion of our index. If you suspect that your dynamically generated pages (such as URLs containing question marks) are causing problems for our crawler, you might create static copies of these pages.”

In cases like these, it is acceptable to use the “no index” command on the dynamic version of the page, so that your content will not be treated as duplicate. You are not tricking all search engines, you’re just redirecting them.

<meta name=”robots” content=”nofollow”>

This tag tells search engine spiders that it’s OK to go ahead and index a page and list it but that they shouldn’t follow any of the links that are on the page. This can be useful if, for example, you had some partners that requested a link on your site that you felt obligated to give, but you wanted to hold onto as much Page Rank as possible. Now this is of course between you and your own personal god, but you would be able to in effect have a partners page, add the nofollow attribute to the meta tags, and basically not pass on any of your Page Rank to any of the sites to which you are linking. The nofollow command in effect tells all search engines that this is the end of the line.

<meta name=”robots” content=”noindex,nofollow”>

Obviously, noindex and nofollow are powerful tags – and in combination, they can make a page and the subsequent pages to which it links invisible to nearly all search engines.  This combination command tells search engine spiders, “Do not read this page; do not follow any of the links on this page; do not include this page in your index.”

This command has its beneficial uses. For example, it can be placed on pages on a site that have duplicate content for legitimate reasons. A website might have both a page for the United States and a page for England that cover the same product with exactly the same content. However, nearly all search engines would see this as duplicate content and could devalue both pages. So placing this command on one of them means that search engine spiders will walk on by and you won’t be penalized.

<meta name=”robots” content=”noarchive”>

Finally, almost all search engines today, including Google and Yahoo, offer a cached version of a site alongside its listing that provides a snapshot of what the page used to look like. The noarchive tag, therefore, is available to be used in circumstances where there is content on your website that is of a timely nature and therefore that you might not necessarily want search engine spiders to cache for people to have access to moving forward.

For example, a business might run a one-time special that has a ridiculously low price to drum up some business while things are slow. The business will want to be able to shut that sale down as soon as sales are back up to a solid level. However, it is conceivable that someone could click on the cached version of the business’s site, see the old deal that was out there, and insist on getting it for themselves. By using the noarchive tag, you are telling search engine spiders, in effect, “This page is subject to frequent changes, and I don’t want my visitors to have access to some of this content at a later time.”

Conclusion

The commands discussed above are just a few of the ones in existence, and new ones are being added frequently. While nearly all search engines support these commands, there are still some that don’t.  The ones in this article, however, are fairly universally understood by search engine spiders, no matter from where they originate. As more universal commands are introduced, I will write about them in future articles.

Author:  Scott Buresh is the CEO of Medium Blue, which was recently named the number one search engine optimization company in the world by PromotionWorld.  Scott has contributed content to many publications including Building Your Business with Google For Dummies (Wiley, 2004), MarketingProfs, ZDNet, WebProNews, DarwinMag, SiteProNews, ISEDB.com, and Search Engine Guide.  Medium Blue serves local and national clients, including Boston Scientific, DS Waters, and Wake Forest University Baptist Medical Center. Visit MediumBlue.com to request a custom SEO guarantee based on your goals and your data.

Avoiding Becoming An SB Site Spammer

By Sean S. Sampson in SE Submission

MAKING SURE YOU DON’T SPAM

Social bookmarking offers you a lot of freedom, and can help you drive traffic to your website overnight. Still, abusing this freedom can lower the credibility of your business, your brand, and your company name. Moving beyond the potential for spamming will help you to build a strong and regular community, instead of being booted off a social bookmarking network.

It’s important to review your submissions to make sure you’re not overusing keywords. Keyword overuse is a turnoff for readers anyway, and you should aim for a keyword density between 6-8% at the maximum level. Most readers will easily pick up on articles that simply do not make sense, and this will limit your ability to promote quality work. Instead of focusing exclusively on SEO, create something unique with an attractive headline instead.

Google and Yahoo! are also becoming vigilant about catching spam-type articles and you will lower your chances of indexing as a result. The most important factors for indexing by Google include:

• Publishing content that is relevant to the heading or topic
• Quality formatting
• Appropriate links and references
• Appropriate amount of keywords

Reading all terms of use and guidelines will ensure that you can make the most of your participation, and grow with your community so you are developing a strong presence.

Avoiding Becoming A Spammer If You’re Using Duplicate Content.

Duplicate content makes it easier for you to write about your topic of choice, giving you a pre-researched format and guideline. Still, it’s important to make many changes to the article, and obtain permission rights from the author when necessary. A single sentence or paragraph that is copied word for word can be detrimental to your search engine placement, and may even lead to copyright infringement.

THE SOFTWARE QUESTION: TO BUY, OR NOT TO BUY

You’ll find a variety of submission directory software packages and applications available on the web today, but it’s important to review the risks involved with using an automated service. Since many of these can be blocked by spam filters, the efforts may not be worth your time. These software packages are designed to submit your content and information on multiple sites in one simple step.

Some social bookmarking submission software packages are designed to be compatible with multiple websites, but you may find that some of your content simply becomes pushed onto spamming sites and bogs. This is why submitting to quality directories and sites is even more important today. Automation does not necessarily guarantee quality, and only you can make the decision on whether this is appropriate for your business, your blog, or your website.

Still, you may find some value in free sites such as Socializer 2.0. Socializer is free web service from eKstreme.com that allows you submit one link to several social bookmarking websites by adding some code to your site. The company has also developed customizable WordPress and Movable Type plugins and you can find instructions on downloading this information directly from the website.

You can book mark your articles to one folder in your IE Favorites, export it to it’s own folder and import all book marks in that folder to Del.icou.us and them tag/label them. Many of the other Social Bookmarking Sites have features that will allow you to import those favorites directly into your account from Del.icio.us.

IMPORTANT: Remember to mark all your bookmarks in all Social Bookmarking Sites as public. When you import bookmarks, most sites will automatically or by default mark them as private. When they are marked as private, no one including search engines can find of view your bookmarks. If you leave them marked as private, it of course then defeats the purpose of Social Bookmarking from a marketing standpoint.

Author:  Sean S. Sampson provides valuable resources, and information to help the home business owner, and online business owner prosper with their business. For more information about Social Bookmarketing, please go to http://www.makinglotsofmoneyonline.com

Go Ga Ga For Google Or Yell About Yahoo?

By Daniel Jupp in SE Submission

Search engine marketing is about words. Pure and simple. It’s about putting the right words in front of the right people. The twist in the tail is how you find those right people. Combining SEO with an efficient pay per click campaign is one way to go. But how do you navigate the search marketing waters and choose the correct platform for your pay per click budget?
Deciding between Google and Yahoo as your search host of choice is a critical business decision. As Google snap up business after business in a bid to expand their services portfolio, it can seem like a no brainer. But Yahoo has much to offer, not least of which are strong conversion rates and a more competitive cost per click…

Partner Networks
While Google bristles with enough features, advances and new additions to make a NASA spaceship feel retro, Yahoo has concentrated on expanding its partner network. Google rightly claims that it is the global search engine of choice but Yahoo’s investment in its new Panama platform and extensive affiliate structure is slaying many with its charm. Profits are up, conversions are up and advertiser numbers are increasing. Yahoo is using this networking to generate around 5.7 million targeted leads every month, leads which they say are more than likely to convert into cold, hold sales cash.
Advert Submission
Although the user interface can be a daunting prospect for the uninitiated, Google’s submission procedure does offer one considerable advantage; speed of use. Advert submission is instant, putting a new or revised pay per click advert online immediately providing simple guidelines are adhered to. Little if any forward planning is required on the part of the advertiser to introduce new campaigns and special offers. Similarly, pausing discontinued products or oversubscribed services is also the work of minutes, giving pay per click advertisers the highest level of control over their internet presence.
Unfortunately for Yahoo, Panama pay per click is susceptible to editorial backlogs, delaying keyword approval by anything up to a couple of days. Changing an existing advert is also subject to a time delay, again postponing the effectiveness of the campaign. However, users do enjoy greater leeway in terms of advert length – used wisely the increased character permit is a useful marketing tool.
Bidding
It’s all change on the bidding front at Yahoo with Panama following the Google example and doing away with transparent bidding. Where once you could see rival bids and buy your desired position accordingly, blind bidding has shifted the emphasis to quality campaign infrastructure.
The process is more complicated still on Google as their non-transparent system factors in external factors such as keyword and advert relevance, campaign history and the holy grail of Google AdWords, the account quality score.
Reports
AdWords counts a user-friendly reporting procedure amongst its plus points, giving account holders automated feedback on campaign performance. The perceived value of keywords, click through rates and impression counts are available for client perusal on a regular basis. Yahoo is yet to jump on the automated report bandwagon, although reports can be created manually to analyse account performance.
Payments
Yahoo operates a pre-pay system as standard as part of their payment terms. When you set a daily budget, a whole month’s worth of advertising is taken in advance from your bank account. In contrast, Google spreads the cost, using a credit limit to take payments over an interim period.

Author:  Daniel Jupp is the founder and managing director of pay per click consultancy, Top Position. For more information and to request a free 30 day management trial, visit www.topposition.co.uk

Use Robots.txt, Save the World

By Sante J. Achille in SE Submission

Robots.txt help the search engines learn all about your website

There is a growing interest in the little known file that every website should have in the root directory: robots.txt
It’s a very simple text file you can find all about at the robotstxt.org website
Why should you use it ? Here are some good reasons for you to consider.
Controlled access to your content

With a robots.txt file you can “ask” the search engines to “keep out” of certain areas of your website. A typical area you might like to exclude is your images folder: If you aren’t a photographer, painter and your images are for your website use only, there are good chances you don’t want them to be indexed and showing up on image search engines, for people to download, or hotlink.
Unfortunately grabbers and similar software (such as Email harvesting applications) will not read your robots.txt file disregarding any indication you may provide in this respect. But that’s life isn’t it, always someone being disrespectful to say the least …
You can keep search engines away from content you wish to keep out of sight, but remember your robots file is also subject to attention of hackers seeking sensitive objectives you might inadvertently list: keeping out the robots while inviting the hackers – keep this in mind.

(cont. from front…)

The growing importance of robots.txt

At SES New York a robots.txt summit was held where major search engines (Ask, Google, Microsoft, Yahoo!) participated, sharing interesting information on this file. Here are some numbers.
According to Keith Hogan from Ask:

i) Less than 35% of websites have a robots.txt file
ii) The majority of robots.txt files are copied from others found online
iii) On many occasions robots.txt files are provided by your web hosting service

It looks like the majority of webmasters aren’t familiar with this file. This is going to play a major role as the size of the web continues to grow: Spidering is a costly effort that search engines tend to optimize. Those web sites demonstrating optimal command (which in turn determines efficiency) will be rewarded.

During the summit, all search engines announced they will identify (or autodiscover) sitemaps via the robots.txt file. In essence search engines are now able to discover your sitemap via a link in the following format:

Sitemap: <sitemap_location>, where <sitemap_location> is the complete URL of your Sitemap Index File (or your sitemap file, if you don’t have an index file).
Being compliant to Google Terms of Service

Robots.txt can help prevent you getting banned or being penalized by Google. In a move to eliminate search results pages because “web search results don’t add value to users” Google has recently added the following sentence to their terms of service:

- Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.

How to implement a robots.txt file

If your website doesn’t support a sitemap and you do not have any areas to exclude, include an empty robots.txt file in your root directory. By doing so you are acknowledging full spidering of your entire site.
Carefully review the robots exclusion protocol available at robotstxt.org. If you must exclude numerous areas of your website, build your file in a step by step manner and monitor spider behaviour with a log analyser tool.
Test your robots.txt file with a few online tools and keep in mind that every spider has a different behaviour and spidering criteria.

Avoid useless spidering traffic

When your website grows to a significant size and achieves optimal visibility, spidering significantly increases to hundreds (if not thousands) of hits per day and will put your server and bandwidth to the test.
Recently I was called on to examine a blog burdened by a very unusual and extremely heavy spidering activity: the log file I examined reported an excess of 8 Gbyte of invisibile (spider) traffic over a 1 month period. Given the reduced amount of daily visitors (less than 200) and the reduced size of the blog (less than 100 posts), there was something wrong in the architecture.
It took just a few minutes to identify the problem: There was no robots.txt file.
At each request for a robots.txt there was a redirect to the home page of the blog triggering a complete download of the blog home page. Each download of the home page was approximately 250 K. There were thousands of unnecessary hits on the home page. This was causing a spidering frenzy that ceased when an empty robots.txt file was created and uploaded to the server. Traffic is now down from 8 Gbyte to 500 Mbyte.

Keep the spiders informed, help save the world

The web is growing by leaps and bounds, the use of a robots.txt file helps the search engines effectively allocate their resources and is a tangible sign of respect and courtesy. If you don’t have a robots.txt file on your website set one up now. Use it to inform the crawlers on how your site is organized, and how often it is changing. I think we should all do our part to avoid waste of resources, saving energy and helping to save the world.

Author:  Sante has an engineering degree and has worked the web since 1994. An accomplished speaker, Sante has appeared at many European SES Conferences, including the first Italiain SES held in Milan in April 2006. Appointed as an ICT consultant to the regional government in Abruzzo, Sante has also appeared at the Reykjavik Iceland Internet Marketing Conference and will be presenting at SES Milan in late May.