I spent some time this morning responding to an email on this topic, so I thought I’d post it here too as a reference.

There are three studies that I know of that have attempted to answer or predict the CTR of organic results. Each has a different methodology and sample data, but the results are similar.

First up, this one is based on the AOL data that was leaked a couple years ago. Someone plugged all the click data into a formula and got these results. Note, that this is AOL data, and it’s widely believed that those searchers behave differently than more savvy Google or Yahoo users: AOL Clickthrough Study Data

Second, this study was done by Cornell University using eye and click tracking equipment. It’s probably the most experimentally sound, but of course the people also knew they were in an experiment and were given specific search terms like “shopping” that might not be relevant to their state of mind. Cornell Organic Click-through Study

Finally, there is a third study done by me in my former life as a corporate SEO, which pre-dates the first two studies I mentioned. This one used keyword volume estimates and # of visits from a given keyword in a given ranking to a specific website. Because the dataset was smaller, I extrapolated the results to a logarithmic function which I have since forgotten. But the spreadsheet still contains the ranks.

The beauty of these results is twofold: 1. I think it better represents the B2B searcher and, 2. because it’s a function you can extrapolate the CTR on any given ranking (although since I don’t remember the formula, this sheet only goes back to result #200, but if you’re ranking past 20 you shouldn’t care about CTR anyway). Here’s a sample of my results, and you can download the full spreadsheet here:

Rank CTR
1 42.30%
2 11.92%
3 8.44%
4 6.03%
5 4.86%
6 3.99%
7 3.37%
8 2.98%
9 2.83%
10 2.97%

The arguments against bulk submission of your website to hundreds of general directories can be summarized in the following way:

1. Search engines will frown upon you getting 1,000 new links in one day.
2. The value of 1,000 directory links from general-topic sites is next to nothing.
3. Topical directories that are hand-edited will provide much more link-value, so you should spend resources on that instead.

I’d like to use an opportunity (or “crisi-tunity” if you’re a Simpsons fan) that’s recently come up for me to refute some of these claims. But first, let me refute #1 right away…

Rate of Change
When search engine spiders visit your webpage, there is a certain process that occurs. The spider reads the HTML of the page, probably makes a few statistical notes (# of words, # of links (internal and external), URL parameters used, dates on the page, etc.) and then puts it in the queue for analysis by a heavier-duty piece of software.

Your site gets put in the queue to be spidered in a few different ways: someone with a search-engine toolbar visits your site, someone links to your site, someone does a search for your specific URL. All of which give the site a different priority, depending on whether or not it’s in the index. Statistically speaking, this means that your site and my site are almost never being spidered simultaneously.

When you add in the fact that your page can’t be indexed until it goes through the indexing algorithSimpsons fanm (the heavy-duty software I referenced above) which can take a couple days, there is just no way for your new pages and mine to be added to the index at the same time.

Because links are an element of the page, they are “counted” in the same manner. When you spread this example across 1,000 sites, all with different indexing periods and rates there is just no way for a search engine to “see” you receiving hundred of links at the same time. They will appear to trickle in over the course of several weeks or months, especially when you consider the fact that not all the approvals go through at once either. I run several directories, and I only approve submissions once or twice a week.

There, #1 is officially dead, no?

The meat and potatoes
Last year a friend of mine was designing a brand new website for a company that sold promotional items. They were starting off with a brand new domain and wanted to get search engine traffic right away. Although my buddy asked me if I would help out, at the time I was confined by an agreement to only perform SEO for one company. So instead I hooked him up with a few tips on on-page optimization and a vendor who does directory submission.

Long story short the company ended up bailing on my friend after he delivered the site. They never paid him, so all that happened was the new site was built and live with on-page SEO basics, and it was submitted to 1,000 general-topic directories. One year later, the company has basically dissolved, but their site still stands (along with some analytics code I told my buddy to put on the site). Here’s what has materialized:

• The site now has a pagerank of 4 (from N/A at the start).
• The site shows 553 backlinks in Yahoo.
• The site is receiving about 20 visitors a day from Google.

So it seems that those links are neither worthless nor ignored. They have produced tangible results in 2 major search engines. Granted, this won’t make you a million dollars, but for a $50 “fire-and-forget” submission package, why wouldn’t you?

What I Advocate and Response to #3
With regard to topical web directories and finding the really strong ones to submit to, I fully encourage you to take advantage. There will never, ever be anything wrong with achieving a high-quality link, from a site that is topically-related to yours.

However, if you have a brand new site where you’re just trying to get some “air under the wings” (or you’re running an Advanced Domaining Strategy) why not spend a little cash to get things moving? You can (and should) always supplement this with topical link-building, but that kind of strategy means you need to know your business, which is something you (conceivably) can’t outsource.

So spend your time building quality links, and outsource the foundational stuff.

The Article Distribution Service iSnare.com has been billed as one of the best tools around to increase a website’s presence. And I’ve been a big proponent of it since I first came across the service.

The idea is simple enough: submit an article to this service, it is reviewed by humans for quality and then gets auto-distributed to 1000s of article-aggregation websites, many on general topics, and a few on whatever topic you choose for your article.

After using it a few times, I began to notice that pages I promoted with the service would tend to rise in Google’s SERPs for my targeted terms, and then slowly fall back down. They would usually settle at higher positions than where they started, but I wondered why the Rome effect was so strong (that was a subtle reference to a rise/fall timeline).

So, I decided to study the Google results on fresh articles, and their mentions in search engines. I used the old trick of searching a unique phrase. On August 4th I used a unique phrase from each article on Google’s engine: 0 results. I then submitted both articles to iSnare for distribution. On August 8th I got an email that both articles had been approved and syndicated; a second Google search revealed 0 results for both.

0 results again on Aug. 9th. Then on Aug. 10th I saw the first signs of life: 7 results for Article 1 and 8 results for Article 2. By Aug. 15th, Article 1 had 437 results, and Article 2 had 458 results. There are two points of note here:

Point 1: I submitted both articles under the same category. They were approximately the same length (around 450 words). I submitted them on the same within minutes of each other, and yet Article 1 lagged behind Article 2 for some reason.

Point 2: At this point (Aug. 15th) there were no supplemental results for either article. All 400+ results were fully viewable in the main index.

On the 16th of August the dupe filter must have kicked in on Article 1, because supplementals appeared and total results dropped to 361. Article 2 continued to thrive with 556 results on the 16th, with still no supplementals showing.

Eventually the dupe filter must’ve kicked in on Article 2 as well, and by August 30th, both result counts were below 50 (39 and 34 for 1 & 2, respectively).

As of today, Big G shows 11 results, of a total of 16 for Article 1 (so, approx. 4 supplementals). Article 2 fared better in the end, today displaying 16 results of 22 total (so, approx. 6 supplementals).

The [recently exported] PageRank for the top 10 results on each article range from 0-2, with the majority being 0 (and 2 N/As!).

So now some theories:

1. Article 1’s target phrase was more competitive than Article 2’s. My theory is that the more competitive an area, the greater number of filters (or in some cases, reviews) a page must pass to become part of the index. This is explained best in the theory of long-tail keywords, where phrases that don’t mean much in a marketing sense have a lot of impact on John Q. Searcher.

2. To compete with social bookmarking, Google needs to be buzz-aware. When a site creates a certain amount of buzz (linking, textual-references, etc.) Google needs to get in there and evaluate it for ranking. It will weight these sites with additional trustrank to get on top of the coming wave. A second (and potentially third) filter will later decide if the page is worth keeping in the index. Possibly by analyzing search volume for a phrase vs. the amount of “buzz”.

What might a takeaway be from this experiment? In my case, the combination of the “buzz” created with the article distro, plus the already-established authority (or Trustrank) of the site was enough to put the [brand new] pages I was targeting into the top 10 for their intended keyphrase.

As with most SEO activities, it is recommended to use this tool appropriately, and in combination with other tools.
Any thoughts?

Update: Looks like Aaron Wall and I may have been thinking along some similar lines.  He just posted about  new domains getting ranked in Google over old sites, and mentioned the following:

“Also think of the search business model as though you are a search engine. To them, being the first person to do something is a sign of quality because to be the first person in a market requires some market timing / knowledge / investment / luck.”

“Catch a wave” theory, explained in financial terminology.

I was reading a thread about meta tags on the DigitalPoint forums today.

.NET magazine wrote an article on SEO recently and stated that meta tags no longer matter for search engine optimization. Were they correct to do so?

The SEO industry is in state where most of the mechanical aspects of optimization can be handled by a competent web designer who stays on top of the basics (.htaccess, robots.txt, static URLs), which is why we see so many design firms now touting that they can perform SEO while designing a site.

So that takes care of the ’search engine’ part of SEO. But what about optimization?

Search engines (specifically Google) still use meta tags, but not to rank your site. At least not directly.

Let’s start with three statements:

Fact 1: Google has access to your meta- title and description tags.

Fact 2: Google will display your title and description tags in search results, unless the engine feels that writing its own description from your text, or using your DMOZ entry is more relevant (algorithmically determined, happens less than 20% of the time in my experience).

Fact 3: Google will rearrange its search results by click through rate and other (measurable) factors to provide he most relevant results to users.

You may wish to debate fact #3, but for now let’s assume it’s true in this world.

The acceptance of these three facts means that you need to write a title and description that will impress visitors and increase your click through rate. A better visitor experience will push your site higher in the rankings.

So how do you write a good title and description? Well, that means it’s time to take a page from the PPC-world…

So it turns out the article I submitted to the MarketingPilgrim SEO Scholarship Contest was chosen as a finalist. This means my article will also be judged by a panel of professionals for the grand prize.

I had hesitated to enter the contest, but I’m glad that I did. I wanted to go a different direction than the previous entries, and talk less about why SEO is important and general, and tackle a specific issue that I had noticed causing problems for people. As I say in the article, I talked about the same thing at an AdWords conference and I had the whole table listening intently.

Anyway, if you’re arriving here for the first time, please be aware that I’ve recently taken down much of the content on this site and I’m gradually replacing it with better, more informational articles. I decided that since the guys over at MP were good enough to give me a big fat link to this site, I’d better make it worthwhile to visit. Bookmark my site and stop by later.

Or you might see me again in future articles!

Update: So I tied for second in the final round. One of the judges had this to say of my work:

I’ve got to give it up for the direct, hands-on learning and image integration shown by Ryan Bell in Instant PPC Success.

Thanks for a fun contest guys!

You’ve just completed your site. You have a unique offering to the public, but how are you going to get the word out?

If you’re going to get the maximum benefit from your future SEO activity you need to lay the proper foundation. Step one in the process is getting the proper URL structure set up.

The first decision you need to make is to decide whether you want your site to appear as www.mysite.com or just mysite.com in search results. There isn’t necessarily a difference, and it’s a matter of personal choice.

One factor that might help you make your decision is that less computer-savvy folks might be more inclined to add the www. before your address, whereas “techie” types will be more likely to use just mysite.com.

If your website is hosted on an Apache server, you can add this code to your .htaccess file:

RewriteEngine On
RewriteCond %{HTTP_HOST}  !^www.example.net$       [NC]
RewriteRule ^(.*)         http://www.example.net/$1  [L,R]

For an IIS server (with isapi_rewrite), you can use the following code:

Add this to your httpd.ini file:

RewriteCond Host: ^mysite.com
RewriteRule (.*) http://www.mysite.com$1 [I,RP]

If you want to rewrite your URL to the non-www version your Apache code would like:

RewriteEngine On

RewriteCond %{HTTP_HOST} ^www.mysite.com [NC]
RewriteRule ^(.*)$ http://mysite.com/$1 [L,R=301]

Next up, creating the proper URL structure for your navigation.