| |
| AUGUST 29, ISSUE #376 |
Search the Web |
|
|
|
Online Web Tools |
Fr-e-e Traffic Exchange
Fr-e-e Meta Tag Generator
Fr-e-e Site Popularity Checker
Fr-e-e Search Engine Submitter
|
|
Webmaster Tools |
Hundreds of Top webmaster apps arranged in 12 major categories. New programs added weekly.
Webmaster Tool Directory
|
|
Webmaster Resources |
A comprehensive directory of the Web's top webmaster resource sites. 10 major categories.
Resource Directory
|
|
WebSite Tools |
Get Top Website Tools. Over 25 FR-E-E Services to choose from.
Visit Bravenet.com
|
|
Tip of the Day |
|
Cross promoting is a powerful technique used by many Internet marketers to increase the exposure of their
products, services or publications. Contact non-competing fellow marketers targeting your market to discuss an
exchange.
Today's tip by Shelley Lowery, taken from her acclaimed
Web Design Mastery series.
|
|
SPN Site of the Day |
|
BigWebmaster.com is a directory of webmaster resources
with over 1,390 links to resources in 15 major categories.
Does your web site qualify as a SPN Site of the Day? Webmaster resource sites can apply via email:
sotd@sitepronews.com
|
|
SPN App of the Day |
Host Evaluator (383 KB) is fr'ee software that
analyses server connection speed over time, tests for server uptime and alerts you if the server is not responsive
by system alert and/or email. Features of web hosts can be added in a wizard type format for comparison. The
program can assist in determining the best host, based on attributes like storage space, bandwidth, cost, e-mail,
support, redundancy, features, connection speed and uptime. Outputs a table listing the hosts best suited to your
needs. For Windows 98SE/ ME/ 2000/ XP.
If you have a Webmaster App that you would like listed on the SPN site, send us an email with details to:
wapps@sitepronews.com
|
|
Fr-e-e Newsletters |
|
|
|
MyZips.com Downloads |
|
|
|
Must Read Ebooks |
|
SPN offers one of the best eBook libraries on the Web. Check out our current selection of
Commercial and 156 plus
Fr-e-e eBooks.
Authors of EBooks may submit their publications to SPN via email:
ebooks@sitepronews.com
|
Top 5 DeskTop Themes |
|
|
|
Link to SiteProNews |
|
Link your site to SiteProNews, the fr-e-e,
newsletter and resource site for Webmasters.
Or, Add SPN to your site with just 2 lines
of JavaScript code. Top content for your site without any of the work.
Check out the SPN Promotion Partners
page. Some great sites have opted to support the SPN newsletter
|
|
Custom I.E. ToolBar |
Put one-click search power at your fingertips. Add a
free Custom Toolbar to Internet Explorer.
Download
the Toolbar file, then run the file to install. Open I.E., go to the View menu, then Toolbars. Select "ExactSeek
I.E. Toolbar" and you're done. One click searching and much more at your fingertips.
|
|
SiteProNews Partners |
|
SubmitPlus - Promote your site to 110
search engines... FR-E-E!
Template Monster - World #1 website
templates are available for immediate download.
PreWired.com -
Providing ISPs & Publishers an Instant Web based revenue stream!
$9.50 Domain Names...
Register, transfer or renew your domain names and get a FR-E-E Service Suite.
FindMyHost.com...
Review detailed Report Cards of web hosts who made the grade.
Web-Source.net... Your Guide to Professional
Web Site Design & Development.
|
|
|
Stopping and Directing Spiders
By Michael Bloch
|
Not all agents, (otherwise known as crawlers, bots, robots and spiders), that visit your site will be of benefit. Even
the "good" spiders such as the ones Google sends out to index your site may visit places that you don't wish them
to.
Malicious spiders or web strippers can cause you a great deal of grief by taking up server resources and increasing
your bandwidth usage - this can result in excess bandwidth fees. People use web stripper applications (also known
as offline browsers) to download your entire site. Sometimes their goal is fairly innocent - to go through your site
while offline using a locally stored copy. In other circumstances, there may be a much more devious goal - plagiarism
or hacking.
How To Ban Spiders
If you notice entries such as Teleport Pro and WebStripper in your traffic reporting, there is something you can do
about it - either via a robots.txt file or through meta-tags.
Robots Meta-Tag
The robots exclusion tag is very simple to implement, but it's mainly of benefit in keeping search engine spiders
out of sensitive areas. Unfortunately, most web stripping applications ignore it.
The following META tags can be used and should be placed between
your <head> and </head> tags:
<META NAME="ROBOTS" CONTENT="NOINDEX">
This will prevent most search engine spiders and some web strippers from accessing the page.
FR'EE RoboScreenCapture:
The Easiest Screen Caputuring Tool Around
Software Developers - we want to hear from you. Just fill-out the quick survey below for your copy.
RoboScreenCapture is the most robust screen capturing tool available, including an advanced image editing
utility wrapped into one easy-to-use tool!
Click Here to Complete the Survey
for Your Fr'ee RoboScreenCapture!
|
Another method:
<META NAME="ROBOTS" CONTENT="NOFOLLOW">
The page will be still be indexed, but any hyperlinks in that page will not be followed by the spider.
The best method is to combine the two:
<META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">
The page will not be indexed and no links will be followed.
Robots.txt File
The robots.txt file is a more powerful strategy. It is a text file that contains instructions on what to allow/disallow
agents and spiders to view and spider on your site. These rules are called The Robots Exclusion Standard.
ThinkHost doesn't place a default robots.txt file in your web when you open an account, so you'll need to create one
in notepad and upload it via FTP to your docs directory. If you are using Microsoft Frontpage, save the file to the root
directory of your disk based web and then upload via FrontPage's standard HTTP:// publishing function.
Never use a blank robots.txt file as some search engines may see this as an indication that you don't want your
site spidered at all! Have at least one entry in the file and remember to skip a line between entries. Also ensure
that the spider/agent that you are banning doesn't turn out to be a legitimate software browser.
To prevent specific agents and spiders from having any access to your site, put these lines into the robots.txt file:
User-agent: NameOfAgent
Disallow: /
You must record the name of the agent exactly as it appeared in your traffic reports; for example WebZip/4.0.
User-agent: WebZip/4.0
Disallow: /
Skip a line between entries. You could do the same to exclude search engine spiders such as Googlebot.
The "/" means disallow access to any directory.
You can also prevent access to specific folders:
User-agent: *
Disallow: /cgi-bin/
In this example the * indicates "all" but please note that the wildcard (*) cannot be used on the Disallow line,
use "/" instead.
Example robots.txt file
If you would like some sort of guide and further examples of a robots.txt file, you can take a look at the one we use
on the ThinkHost site. View it here:
http://www.thinkhost.com/robots.txt
Our file is by no means complete, but it does contain a number of "idiot" bots that repeatedly attempt to
strip our main site. Please be aware that robots.txt will not stop all web stripping activity as many strippers can
fake agent names, but it will help you save on bandwidth.
Good Spiders
If you would like to be able to identify the "good" spiders that may visit your site, you can view a listing of the most
popular search engines' robots in our tutorial,
Understanding your web site traffic
About The Author
Aricle by Michael Bloch of Team ThinkHost. Thinking Hosting? ThinkHost!
Printer Friendly Version of this Article
|
Webmaster Resource Sites & Services
|
|
Metamend.com - Optimize
your website with a recognized world leader in search engine optimization. SEO services to meet any web site's needs.
DesignGalaxy.net - Professional graphics,
including logos, flash and website templates.
TheCgiSite.com - A directory of programming resources.
Add Me! - A leading resource for website and e-business promotion.
|
FreeWebSubmission -
Submit your website to the Top 50 search engines as ranked by Alexa. OR, auto-submit to 16 major
engines with one mouse click.
Submitsolution.com - Top 10 listings
Guaranteed in Yahoo, Altavista, MSN, and more.
FreeTechMail.org -
A search engine where you can preview and subscribe to thousands of IT newsletters.
BraveNet... Top Rated Web
Tools for Webmasters.
|
|
Recommended
Webmaster EBooks & Tools
|
The Insider's Guide To Dominating the Search Engines
Introduces you to revolutionary, "tried-and-tested" optimization strategies which will increase your website
traffic by 10, 100, or even 1000 times ... GUARANTEED. See results start to materialize in as little as 2
days....and see significant results appear in just 1 month.
Want a Simple Explanation of
How to Make Money Online?
Our friend Jim Daniels, who makes $200k a year from his basement,
will show you how. His simple, five page website explains the Actual Steps to Making YOUR Living Online. If
you're tired of hype, see this fr-e-e guide.
Download Your Free Copy Of "Website Optimization Made Simple"
This unique, brandable, e-book includes step-by-step site optimization instructions, a fr-e-e website submitter, a fr-e-e
refer-a-friend service, and viral marketing ideas that will explode your traffic. Get your fr-e-e copy now.
|
 |
