Due to the ever changing criteria of the search engines, some new methods have been discovered and some old ones are listed here for a reminder. Don’t forget the importance of a solid site architecture.
As a standard rule, any page below the second level of your site architecture may not be indexed. It is highly suggested to keep your file structure as flat as possible. Keep each directory categorized, if possible. The reason? I have seen search engines only indexing up to 45 files per folder. If you have one main folder with 200 files, you risk not having all of the files indexed and included in the search engine database.
Underscore or Hyphen?
When naming your directories or files, giving “keyword separation” is often done with a “space” – most notably an underscore or hyphen. In my testing, I have seen more reliable indexing and better overall results using a hyphen instead of an underscore.
Search engines have long said that they treat an underscore or hyphen as a space. However, my testing has revealed that it isn’t always the case. I strongly recommend that your naming conventions include using hyphens instead of underscores.
Don’t forget the importance of having a Site Map. This will allow for the search engines to find your most important pages and spider them on a regular basis. Despite what you may have been led to believe, it is not vital to list every page of your site on your Site Map, just the main sections. The search engines can take it from there.
For an example, the Site Map I created lists each main category or page. The category pages are “mini-site maps” to take the search engines deeper into the site. The Site Map is also a great PageRank feed, as all the main sections of the site are highlighted on one page. As you will notice in the example, the page is not just a list of links, but includes simple descriptions of each page. This is vital, as you have link text and body text together. The search engines eat pages like this alive.
If you haven’t been using the Robots.txt file out of laziness, it is time to use it again. I have seen Googlebot recently skim across sites that do not have the Robots.txt file, and deep crawl others that have the Robots.txt file. If you are unfamiliar with the file you can see an example of mine, and you can also read my in-depth article. It is a few years old, but still noteworthy.
I highly recommend using the Robots.txt file to allow for the spiders to index your site as deep as possible.
Google Dance Update:
The Google Dance is rumored to start next week. The Google staff is well known for their sense of humor, so look for something on April 1st. I am recommending that you post all of your new pages by midnight tonight, as Google will be spidering sites heavily this weekend in preparation for the upcoming Dance.
LookSmart Pricing Change
You may have seen this week that LookSmart has lowered their prices on adding new listings or changing your current listing. The new fee structure is $29.00 for a new listing and $19.00 to modify an existing listing. This is a reduction from the previous price of $49.00. It is hoped that with the new pricing structure, LookSmart can get an injection of new revenue.