To celebrate the launch of SEOmeter.com, we are accepting 100 websites in our monitoring system for free. Yes, you heard it right. Free admission with no string attached! To be eligible for free admission, you should add a comment to this blog post, and include your site URL in the comment.

Here are other requirements for free admission:

1. Only one site is allowed for each IP address. If there are multiple entries for a given IP, the one mentioned in the earliest comment will be accepted, and the rest will be automatically rejected.
2. Only top-level domains or sub-domains are allowed. No internal page or folder is allowed.
3. Site should be cached by Google within the last two weeks of your post. Please check with Google by using cache: command.

This post/offer will be closed once 100 eligible entries have been chosen.

There are some internal changes made to our database. Now all the websites monitored by us are categorized by their topics internally; the classification is made by our editor, not by submitter. In the top-20 page, we then list the top-20 sites in each category in terms of their average crawling cycles. The top-20 rankers in each category are automatically updated as their stats change over time. The top-20 list will give you an idea of what kinds of sites are favored by search engines in each category. You should be proud of yourself if your site make the list.

The web is an ever-growing and dynamically changing world. Given the gigantic scale and dynamic nature of the web, it’s a nearly insurmountable task to maintain an up-to-date search engine index for the entire web. Therefore, one of the most important tasks of a search engine is to determine what part of the web is important and thus worth crawling more often, and reflect any updated content in their index in a timely manner. From a webmaster’s point of view, our mission is then to become a search engine’s favorite, and to make search engine robots visit our site more often than others. Here are a (non-exhaustive) list of factors that can affect the search engine’s crawl rate.

1. Relevant and Authoritative Backlinks
It’s a well known fact that backlinks help major search engines’ crawlers find your site and can give your site greater visibility in their search results. Especially links from relevant content and authoritative sources are considered a more powerful vote by search engines, and therefore are more likely to bring search engine robots to your website. Submitting your site to reputable and well-categorized web directories or major social networking sites helps your site get more exposed to crawlers.

2. Content Update and Pinging

Regular and frequent content update is another important factor that attract search engine robots. For example, the purpose of Google’s fresh crawl is to detect content update, and reflect the change in the search engine results immediately.

If your site is a blog, you can try existing pinging services such as pingomatic.com or Google’s Blog Search pinging service to proactively inform search engine robots of new posts and content changes.

3. Internal Link Structure
Another factor that affects search engine’s crawling rate is how the current page of a website is linked from other pages within the same website domain. Search engines determine the relative importance of the current page on a website based on the site’s overall internal link structure. Pages that are heavily linked to internally (e.g., site-wide pages) are considered important by search engines, and therefore receive more frequent visits from spiders.

4. Sitemap and Robots.txt

Creating a search engine sitemap for your site helps your site indexed more deeply as well as more frequently. With Google, you can create XML/TXT-formatted sitemap and submit it to your Google webmaster tools account. A typical sitemap contains a list of URLs for crawler to retrieve. If the sitemap is formatted in XML, you can specify extra information for crawlers, such as frequency of content change, last modification date, or relative importance of a page.

While sitemap informs crawlers which pages to retrieve, robots.txt does the opposite. That is, robots.txt prevents spiders from retrieving all or part of your website, which otherwise is publicly accessible by human. As webmasters become more SEO-savvy, they start to make use of robots.txt more actively (e.g., to eliminate duplicate content). But at the same time, it increases a chance for them to fumble robots.txt, and unwittingly block search engine spiders. In order to prevent any costly mistake, always arm yourself with the up-to-date syntax of robots.txt recommended by major search engines such as Google and Yahoo, and look out for Google’s crawl error reports.

5. Server Speed

Not to interfere with search engine’s crawling, the web server where your site is hosted should respond to a request in a reasonable time. Fast response time offers visitors good surfing experience. The same logic applies to search engine robots as well. Given that the search engine’s primary role is to provide users good searching experience, having your website hosted on a fast web server helps your site indexed faster and updated more frequently by search engine.

6. Set Crawl Rate Feature in Google Webmaster Tools

In your Google webmaster account, you can choose three different types of crawl speed for your website: Faster, normal, slower. The set crawl rate option is available only for top-level domain or sub-domains, but not for any internal pages or folders. An once requested crawl rate need to be renewed every 90 days. However, it’s reported that this feature does not guarantee an immediate effect on Google’s crawl rate.

Two updates from SEOmeter.com:

Website submission page has been added. We decided to charge an annual (but modest) fee for each submitted site for monitoring. The reason behind introducing the annual fee is to support the cost of our server and database resources which will increase over time as more sites are monitored by us.

The other update is, an AJAX comparison menu was added in the site information page (see the screenshot below). You can enter two more urls in the comparison form to compare the current website against two other sites in terms of crawling history.

Currently there is one known issue with the AJAX comparison form. For whatever reason, AJAX refuses to retrieve results on any mod-rewrited page. So for example, the comparison form works fine on http://www.seometer.com/index.php?search=google.com, but not on page http://www.seometer.com/cache/google.com . We are working on fixing this issue. The issue was fixed.

SEOmeter.com is open to the public. Our site is still in beta, and we are still working to make it fully functional, including website submission procedure. In the mean time, you still can get a glimpse of our site. We will get back to our blog in a few days, and describe in more detail what our site is about, and what motivated us to develop SEOmeter.com.
Please tune in!