2.3.4 Make a robots.txt file
Sorry, you are not logged in. Please login to get full access to this page or register if you're not registered yet.
Like a traveler coming to a country, the Robot needs a guide — which is, the robot.txt file. It's a specific guide, though, as it'll only tell the Robot which cities he shouldn't see.
Like, if you're at the crossroads and one of the roads leads to a private place you shouldn't visit, there will be a NO TRESPASSING sign on the way.
No Tresspassing Sign
A robots.txt file will prevent the Robot from going to some pages with sensitive material, web pages that you don't want to be found through Google search (for instance, the "shopping cart"), pages that are not important or can be negative for your rankings. And, you can direct the Robot to other, keyword-rich pages, instead.
So, if there's something to hide, a robots.txt file is a must for your website. It helps you keep the Robot away from anything that's not good for your Search Engine rankings. Yep, just tell him not to go here or there — and he'll believe you.
You can make a robots.txt file yourself (and here's a web source that'll help you: http://www.robotstxt.org), though it's rather your webmaster's business. So ask your webmaster to write a robots.txt file for your site, or do it yourself.
Remember, if you rewrote the dynamic URLs we talked above, use robots.txt to forbid the old URLs like this:
http://www.myshop.com/showgood.php?category=56&good=54146
After you have the robots.txt file, run it through a validator to ensure it's written correctly.
Hundreds of robots.txt validators can be found on the web. You can apply Google's tool, or this one, for instance:
http://www.invision-graphics.com/robotstxt_validator.html
As soon as the robots.txt file's correct, you needn't worry, as it will only do you a lot of good, and no harm.
DO IT NOW! Make a robots.txt file and validate it. Add it to the root directory of your website.
2.3.5 Make different paths to reach a page
Think of our map and the Robot driver, again. If you can get to a city from two other places, chances are good that the Robot will get there. But if there are 6 roads from 6 different places, leading to one city, chances are much bigger.
Different Roads Leading to the Same Place
So let's make as many paths as possible.
DO IT NOW! Make different pages on your website link to each other.
2.3.6 Fix broken links
You know what's a broken link and how bad it can be? Well, I'll tell you.
About a month ago, I was driving to Walker River, NV. Just wanted to see the place, as my granddad came from there. I only had a 20-year-old map of the place, and it surely turned out to be too old: On the way I chose, the bridge was broken quite long ago. Well, I was a bit disappointed and had to take another road.
But what if it weren't me on the broken bridge? The Robot would try to find another way to go. But he's not that determined to visit all your pages. In the case of a broken link, the Robot may simply leave the page not crawled.
And what is actually a broken link?
Now what you have to do is check your pages for broken links — and fix them.
DO IT NOW! Find and fix broken links on your web pages. Upload the changed pages to your web server.
Here's a free online tool to help you:
http://validator.w3.org/checklink
Like any other tool you might find on the web, this one has instructions and tells you where those links are and what you have to do to fix them.
4 comments
Add comment Hide commentsShow commentsPlease log in or register to leave a comment.
