Forgot Password?
SEO Certificate


2.3 Make your site Search Engine-friendly: 11 things to consider

First, meet Mr. Robot. He'll help me to provide examples, to tell you more clearly how website and Search Engine logic works.

The Robot (also called Spider, or Crawler) is a great traveler. Driving his car from one city to another, then to a different country, all over the world.

Well, but in our terms countries are actually websites, cities are web pages, and all the country roads, tracks and highways are links between different pages.

Say, here's a map of such a country,

A Map for

Still, Mr. Robot is not that independent as it may seem. He inspects all the websites, evaluates them, and he's always on the phone to report to the Search Engine. The Search Engine updates its index with the data the Robot reports about a website. And, as soon as the Robot's made a world trip, he's ready to start another one, to check the old spots and explore new ones — to keep the Search Engine's index up–to–date.

And what's a Search Engine's index? It's a huge database, where record is kept of everything that robots (like our guy) find on the Web: web pages and any information on them. In terms of SEO, it's crucial for a web page that all that's good on it is recorded in the Search Engine's index (= is indexed). If it's not there, the page won't be found through search, or will not bring you the results you wish.

This all means that you want the Robot to visit all your important pages, and look at every detail that can be found there.

Now, one by one, let's see the things important for the Robot, and even more important for you.

2.3.1 Make a robots.txt file

Like a traveler coming to a country, the Robot needs a guide — which is the robots.txt file. It's a specific guide, though, as it'll only tell the Robot which "cities" he shouldn't see.

Like, if you're at the crossroads and one of the roads leads to a private place you shouldn't visit, there will be a NO TRESPASSING sign on the way.

No Trespassing Sign
No Trespassing Sign

A robots.txt file will prevent the Robot from going to some pages with sensitive material, private info or any other web pages that you don't want to be found through Google search (for instance, the "shopping cart"), pages that are not important or can be negative for your rankings. And, you can direct the Robot to other, keyword–rich pages, instead.

So, if there's something to hide, a robots.txt file is a must for your website. It helps you keep the Robot away from anything that's not good for your Search Engine rankings. Yep, just tell him not to go here or there — and he'll believe you.

You can make a robots.txt file yourself, though it's rather your webmaster's business. Here's an example of a creative one to inspire you —

What I want you to understand is the fact that the directives in this file are extremely important and can greatly influence the number of pages that will appear in search results after you make these or that tweaks. Obviously, the number of pages appearing in SERP will affect the number of pageviews and clicks you get.

That's why you should be extremely accurate to NOT disallow indexing of the entire website (which is, frankly speaking, quite common among business owners who try to make SEO tweaks themselves). So be cautious, because even a single mistake or a wrong symbol can make the whole website disappear from a Search Engine's index.

After you have the robots.txt file, run it through a validator to ensure it's written correctly. Hundreds of robots.txt validators can be found on the web. You can apply Google's tool, or this one:

As soon as the robots.txt file is correct, you needn't worry, as it'll only do you a lot of good, and no harm.

DO IT NOW! Make a robots.txt file and validate it. Add it to the root directory of your website.

2.3.2 Find fast and reliable hosting

No Trespassing Sign

Like most travelers, the Robot loves high speed and hates driving slowly. And, I'd rather not tell you what he thinks of traffic jams.

So let's make a good and smooth road for him. That is, let's get fast and reliable hosting. This will guarantee that your web server is never down when a Search Engine spider tries to index it, and it's always fast enough, both for the Robot and for users.

Yep, it's really worth it. Why? We all know that a site can be down sometimes. If this only happens on very rare occasions, it's not that bad. But if you've got problems with hosting, and your site doesn't respond quite often, the Robot may leave this kind of site not checked. Not to say that you simply might be losing sales, because users can't reach you.

So here's what I advise: host your site on reliable servers that are very seldom down and that are fast. By the way, your users will like speed as much as the Robot does

The faster your hosting, the sooner your site loads, the more visitors like it, and the faster they give you their money ;). But that's not just about user experience! Page loading time is one of more than 200 ranking factors, which means that the optimization of your loading speed may positively affect rankings.

According to the latest research, almost half of web users (including my son Andy, who hasn't seen the dial–up era) expect a site to load in 5 seconds or less, and they tend to abandon a site that isn't loaded within 8 seconds. And it's mighty nice of Google that they offer a special tool you can find at to help us optimize the website performance.

You needn't be an Internet guru to understand and remember:

fast and unfailing hosting ensures
Search Engine love and helps
quick and stable sale.

DO IT NOW! Try to get hosting that is reliable and fast.

2.3.3 Rewrite dynamic URLs

A common problem for online stores, forums, blogs or other database–driven sites is: pages often have unclear URLs like this:, and you cannot say which good or article it leads to. Though instead, they could have URLs like–linen.html, or, where you can easily see what's on the page.

So the problem with such URLs is: no one (neither users, nor even the Robot) can tell what product can be found under the URL before they land on this page.

For simplicity, let's assume that URLs like this,, having parameters (here it's item=) are called Dynamic URLs, while URLs like–linen.html are static. As you can see, static URLs are much more user–friendly. For users, URLs with too much of "?", "" and "=" are hard to understand and pretty inconvenient. It may surprise you, but some guys prefer to memorize their favorite URLs, instead of bookmarking them!

Secondly, static URLs are generally shorter and may contain keywords, which are positive signals for Search Engines. Thirdly, a Search Engine wants to only list pages in its index that are unique. And sometimes Search Engines decide to combat this issue by cutting off the URLs after a specific number of variable characters (e.g.: ? & =). To make sure all your pages with unique and useful content appear in a Search Engine's index, you'd rather make your URLs static and descriptive, if possible.

And now I'll tell you about a nice trick to make URLs look good to Search Engines.

There's a file called .htaccess. It's a plain–text file, and using it, you can make amazing tricks with your web server. Just one example is rewriting dynamic URLs. And then when a user (or a robot) is trying to reach a page, this file gets a command to show a page URL that is user– and crawler–friendly.

This is, basically, hiding dynamic URLs behind the SE–friendly URLs. I'll give you an example for an online store.

As a rule, a page URL for some product looks like this:

where there are two parameters:
category — the group of goods
good — the good itself

At the same website, you may be offering Dove soap in the category of beauty products, with the URL:

And a bra by Victoria's Secret, under:

To Search Engines, both pages appear like showgood.php. They just can't understand that these are two different pages offering two different products.

You can rewrite pages, so the Robot will see that both pages contain different information:–products/dove–soap.html instead of the first URL, for Dove soap and–secret–underwear/bra.html

for a bra by Victoria's Secret, as you may guess.

Thus you'll get "speaking URLs" that are understood by Robots and users and are easy to check.

Writing an .htaccess file is not a trivial task and requires special knowledge. Moreover, it's your webmaster's business. I personally never do this myself. So if you have a database–driven site, search the Web for a special SEO service that will write an .htaccess file for you.

Or, if you're using a fairly well–known 3rd–party engine, you can write the .htaccess file yourself, using some scripts that you can find in the Internet. To do the search, you can type in the_name_of_your_site's_engine "URL Rewrite" htaccess or something like that.

For instance, I used the following query: phpBB "URL Rewrite"

And got a number of results:

Google results for phpBB URL Rewrite
Google results for phpBB URL Rewrite

Now, the idea is: it's of great use to rewrite URLs. So find the URL rewrite tools if you need them — or just find your webmaster.

Then, one more thing, the old URLs that have parameters should be "hidden" from Search Engines. So, use robots.txt we talked above to forbid the old URLs like this:


Add comment Hide comments

Comments page:
#1186 2010-04-23 23:27:47 Adeel Akhter

AT the top it is mentioned "it is helpful for newcomers to know that the htaccess is only of apache servers and this is only on linux hosting, when you have windows hosting then you need a isapi rewriting solutions. This information is never given on the web hosting firms and if you choose windows hosting because you have MS access Database then you are going to make a huge mistake as all the support is for .htaccess, go for a linux hosting solution, it is far easier and better supported not to mention that on some accounts you also get a sitebuilder program"

can you please make it little more clear for me?? As far as i have understood you mean we should look for linux based hosting ??? is it so ?

#1196 2010-04-27 10:29:15 Dan Richmond


Linux servers are more preferable because e is more info on them, it is easier to find a specialist to help you setting them up and, moreover, most of the sites are located on Linux servers.

Theoretically it is possible to configure the .htaccess file on Windows server as well but you need to ask your hosting company to what extent they will allow and support that.

#1135 2010-04-04 06:31:22 Marc Wang

This is an awesome tool.

Just posting a note to "pat you on the back", Dan for putting this great content up. Unfortunately, such effort seldom gets noticed or worse, is under-appreciated.

So, how bout it guys, a round of applause to Mr. Dan Richmond, Cheers.. : )

Now back to more SEO ingestion..

#1142 2010-04-07 12:14:39 Dan Richmond

Thanks Marc

#1043 2010-03-11 01:35:58 Cameron Clarke

Hi, I am working through this great 'SEO In Practice' book in conjunction with SEO PowerSuite Pro. I got to this section on static URLs versus dynamic URLs and then started delving into it further to look at changing my dynamic pages to appear as static.

I have been going through websites on URL rewriting and started looking at making changes to my .htaccess file to do this.

I currently have 1069 pages indexed by Google, which is less than 50% and most of the non-indexed pages utilise the same dynamic variable (?pc=) which Google is setup to index for my site in their Webmaster Tools.

In the process however, I thought I would see what Google had to say about it and found this comprehensive blog ( and was wondering, based on the information in the blog, whether it would be actually detrimental to make the changes.

Appreciate your latest (up to date) thoughts on this issue.

#1051 2010-03-12 06:39:36 Dan Richmond


nice question :) The problem is not that Googles can't crawl dynamic URLs.
My point here is, a static-looking URL will be useful for your page's SEO if (instead of all strange-looking parameters like numers, letters and other characters) it uses your keywords.

#919 2010-01-05 04:35:53 Dan Richmond

@Paul Watchorn

It can also help you get your pages re-crawled fasted when they have been changed. Also, it's a good way to get new pages indexed.

#895 2009-12-30 04:58:47 Paul Watchorn

This is really interesting about the site map, so a quick qusetion: if I do a link command, and all the pages show up as indexed, is is still important to have a site map?

#833 2009-12-12 22:55:12 Ken Taylor

@Dan Richmond
Hey Dan, It's the second paragraph of your post:
"If you're just asking how you get to know the speed og your hosting, then the answer is 1) Bandwidth 2) How much the server is loaded (meaning, if you have a shared hosting, the server may need to run thousands of requests to/from other sites, which obviously influences your site's performance). Dedicated hosting won't cause this problem."

Thank you for clarifying and also enlightening me on a benefit of having 'dedicate hosting'. Something I'll look into. Thanks again Dan.

#821 2009-12-09 10:44:21 Dan Richmond

@Ken Taylor

<i>Where in the Host account features do you look for the feature that tells you about how fast the Hosting is? I've never seen anything. I'm guessing here, are you referring to the 'band-width' the hosting is offering?</i>

Are you asking where WebSite Auditor find this data? Than the answer is:

We make our own analysis of your pages, plus we use data from some third party services that measure the access speed of your website from different locations from all over the world. Usually this doesn't directly correlate to your hoster's bandwidth etc - we just try to open your website from various IPs spread all over the world and see how fast we can open your site.

If you're just asking how you get to know the speed og your hosting, then the answer is 1) Bandwidth 2) How much the server is loaded (meaning, if you have a shared hosting, the server may need to run thousands of requests to/from other sites, which obvoiusly influences your site's performance). Dedicated hosting won't cause this problem.

#806 2009-12-08 03:54:39 Ken Taylor

Finally, a clear easy-to-understand definition on the differences between "Dynamic" and "Static" webpages. Thank you.

One suggestion: Where in the Host account features do you look for the feature that tells you about how fast the Hosting is? I've never seen anything. I'm guessing here, are you referring to the 'band-width' the hosting is offering?

And finally, thank you for providing the links to help make submitting site maps easier. I'm using RapidWeaver for Mac and have a special 'SiteMap' plugin that apparently does both HTML and XML. So it's pretty easy setting up a sitemap and getting right the first time around.

#711 2009-10-23 02:58:06 Watch Naruto Online

About robots.txt:
As i know that this file has a function to tell robot/crawler to direct it if we want the robot craws any pages at our blog or not.

about loading:
sometimes when we put a lot of image this can make our blog loading slowly.but we still can put image for our blog but avoid to put animation image..if we still want to put animation image put it at the footer not at the top.

and the best format for image is also use a lot of image that has gif. format for can make the size of image smaller and really help our blog loading faster.

#412 2009-06-26 08:51:57 Dan Richmond

<b>@Marlene Wilkinson</b>

You're right that a page overloaded with images will load slower but depending on the speed of your hosting you can "afford" more or fewer images on your webpage to be loaded within reasonable time.

Comments page: