Blog

There's an ongoing obsession with tracking search engine result pages (SERPs). Both seasoned search marketing specialists and know-enough-to-be-dangerous webmasters can't get enough of them. So what's so special about these stats and why do people track them?

Why Track SERPs?

There are generally three reasons why people track SERPs: Research, Trends and Performance.

Research tracking allows an SEO specialist or webmaster to know where a website ranks based on a set of keywords. It's one thing to know if a website is in a search engine's index, but it's another thing to know whether or not it shows up in the SERPs. Most people want to know whether or not it shows up in the top ten or twenty results, or if it resides deep in the SERPs where nobody will ever find it?

Trend tracking looks at the affect of an ongoing SEO campaign. That campaign can consist of simply updating the HTML code of a website or involve a sophisticated online link building campaign. Regardless, it's important to know if the campaign is affecting a website positively or negatively in the SERPs for the target keywords.

Performance tracking relates to the actual traffic that the site receives from search engines. Instead of focusing on where a website resides in the SERPs for targeted keywords, performance tracking only focuses on keywords and traffic that actually send visitors to the website.

The Problem With SERP Tracking

There are major hurdles to SERP tracking. First off, search engines don't like it. They see SERP Tracking as an attempt to game their system and they also don't like the burden it puts on their systems (usage that in their mind should only be used for what they consider to be legitimate requests). There have been attempts in the past to provide APIs to allow these type of queries, but search engines like Google quickly determined that their API wasn't being used for the purposes they wanted it to be used for, and subsequently discontinued it. There is of course a large interest in having access to a commercial API for their search engine, but Google has continually shown absolutely no interest in providing that service.

The next problem is accuracy versus overloading/triggering the beast. The most accurate results are those that would appear for most regular users — ten results per page. The problem is that if you want to get the top ten results for the first one hundred results, you have to hit the search engine ten times! The easiest way around that is to get the top one hundred results from one page (a setting that an easily be made in the search preferences or in the query request of the URL). That way you only have to hit the search engine once — there's less impact on the search engine's resources and you have all of the data you need. Unfortunately, those results aren't entirely accurate.

Several years ago, search engines began to implement indented results — results to the same website that were bundled together, but the second result was indented. This mainly occurs when a search engine's algorithm decides that there are two pages on a website that deserve to appear for the same keyword search. At first, this was rare, but as time passed it became more common. So common in fact, that if you do a top one hundred search for a popular term, almost every result will have a secondary indented result. If you didn't pick on what I just said, let me be more clear; The top one hundred search results only have 50 websites!

Indented results occur in greater numbers for result pages that have more than ten results on them. For example, if you do a search that displays the first twenty results, and if a website shows up for the sixth result and also the nineteenth (normally page one and two), the nineteenth result will bubble up and indent itself underneath the sixth result — thus making the nineteenth result the seventh result and pushing everything else down. Now imagine that scenario on a one hundred result page where all of the result's websites are repeated. The bubbling up of secondary results throws everything off compared to its ten results per page counterpart.

Finally, the elephant in the room is universal, subscription, local and custom search results (I guess that makes four elephants). All of these results can significantly change the search results — especially on the first page. Not only that, they can work in conjunction with each other and also occur randomly.

Future of SERP Tracking Is Passive

It's only a matter of time when even the typical one through ten search result page will become a moving target. Once that occurs, it will be practically impossible to report any results that have any degree of accuracy. Coupled with search engines' distaste for people tracking their SERPs, it's also a matter of time until they implement measures that will make it extremely difficult to track SERPs en masse, or at the very least, affordably. Of course, this could be years away, but I do believe it's coming.

Ultimately, I believe that the future of SERP tracking will be done passively. Passive SERP tracking involves using the referral data from search engine traffic to extrapolate not only the search engine and keyword, but also what page and/or position the result was on. We recently created a free Pepper (stats add-on) for Mint that captures passive SERPs in order to showcase this method SERP Tracking. There are also companies, like Enquisite, that are currently using passive SERP tracking to create and enhance their own analytics.

The beauty of passive SERP tracking is that it doesn't require using any search engine resources, meaning, it's search engine friendly. I recently had an email discussion with Matt Cutts about enhancing passive SERP tracking. He said:

We've talked about doing this, although we don't have any plans right now; how would you propose that we pass along the rank information? I can see a lot of pros and cons to any particular approach.

We (Sitening) racked our brains on how to do this — it's not as easy as one might think — and the best we could come up with was to either incorporate it into Google Analytics (which we believe Google could easily do) or to use Webmaster Tools by appending the links in search results. For example, if you could go to Webmaster Tools and turn on the ability to track the "rank" of a link by using an appended variable, that might work. So, instead of having the SERPs return http://domain.com/, they would return http://domain.com/?rank=4 (where ?rank=4 states the rank as 4). It would be an opt-in ability and if turned on, would affect all links that appear in the SERPs for that domain. The user would also have the ability to specify the characteristics of the appended variable in order to make sure it worked correctly with whatever technology they were using. Of course, it's never that simple. If you already have a link in your index that fits that variable (?x=#), then Google may need to add an ampersand when it appends it — or they could just say too bad, don't use crappy URLs if you're going to opt in for the ranking service.

SERP Tracking Should Really Be About Performance

The only SERPs that really matter are the ones that bring traffic. Although it's nice to know where a website resides in the SERPs, if you aren't getting any traffic from it, it's meaningless. Ideally, if you can connect and relate passive SERP data with ongoing campaign data, the analytics can become quite useful. For example, you can better correlate campaign efforts with increased traffic and conversions. You can also use that data to determine the effectiveness of different search marketing techniques.

For Raven, we're going to continue to use traditional SERP tracking for monitoring trends, but not necessarily for accuracy. Although we would love to capture ten results per page instead of one hundred results, it's cost prohibitive and it requires the use of too many resources from the search engines. Coupled with the problems discussed in this article — including the four elephants — we believe it's only a matter of time until traditional SERP tracking results will become impossible to maintain any degree of accuracy.

Although traditional SERP tracking may always play a role in the Raven suite of SEO tools, we're going to start focusing on how we can integrate SERP trends and performance with campaign data. We believe that's where the most valuable information will ultimately be derived.

Comment

"The problem with social media is... there are more people writing it than reading it."

Mixx

After being granted a credibility boost from the New York Times recently, it seems like Mixx is here to stay. The underdog of social networking, Mixx has a smaller following than other user generated content (UGC) sites, but was seen as worthy enough to be invested in by the Los Angeles Times.

Positive Impact

You can use Mixx's smaller user base to your advantage. Any item that you submit to Mixx will be given a better chance of exposure, because there's not as much competition as there is with a site like Digg. If you select your niche carefully and link to quality content, you can easily hit the front page of Mixx.

Negative Impact

Mixx can be a bit skewed. Before I wrote this article, visiting three different categories on Mixx presented the same story on "brooklyn decker. brooklyn decker topless." Those three categories were "Random and Crazy," "Animals" and "Odd News." That title (and subsequent adult link) had no business being under any of those three categories and should have been buried.

Furthermore, under the "Football" category was a submission with a link to a business which had nothing to do with football. Mixx users can vote down these entries; but if they're submitted to a niche category that isn't too popular, those entries are going to remain visible for quite a while. Until Mixx users or entries become moderated in some kind of fashion, Mixx will struggle to compete with its more successful counterparts.

Sphinn

Sphinn is a social site for Search and Internet Marketers. It's designed to allow you to share and discover news stories, read and take part in discussions, discover events of interest and network with others.

Positive Impact

There are some really smart people at Sphinn, and if you take the time to sift through the submissions, you're going to learn something. The leaders in the search industry are either at Sphinn or being talked about on Sphinn. Take the time to read their stories — you won't be sorry.

Negative Impact

I often find some of the submissions at Sphinn as not "sphinnworthy", meaning they shouldn't have made it onto the front page. Submissions such as endlessly promoting your own work, submitting an entry about how a person has managed to reach so many number of Sphinns, and people submitting something that a search celebrity has written, knowing it will make the front page — this gets a little nauseating at times. Fortunately, I do see this as improving in the future, as a recent entry by Rob Kerry suggests.

Twitter

Twitter is a service for friends, family, and co-workers to communicate and stay connected through the exchange of quick, frequent answers to one simple question: What are you doing?

Positive Impact

Sometimes it's not what you say but how you say it. Twitter's limit of 140 characters should not put people off. Think of your followers as a captive audience - they want to know what you're doing, how you're doing it and which web page you're viewing. Remember, some of the most enticing advertising slogans and tag lines are short and to the point. Ignoring Twitter because of its limited character capability is a missed marketing opportunity.

Negative Impact

It's easy to miss something if you're not looking for it. To first put out your message on Twitter, you need to establish followers. This can be a time-consuming process. Beware of users who arbitrarily just add you to their list of followers and expect to be followed in return.

During times of heavy usage, Twitter's servers are known to take a turn for the worst. This can be a pain if you're at a Web conference and want to Tweet a link to a live blog. Still, there's always mobile Twitter.

Digg

Contrary to popular opinion, Digg is not the be-all, end-all of social media sites. Digg power users (top 100 diggers) have way too much authority on what goes hot and what doesn't. As recently as January 2008, Digg changed their story promotion algorithm. This is how Kevin Rose described the latest algorithm update:

"As we've talked about in the past, Digg's promotional algorithm ensures that the most popular content dugg by a diverse, unique group of diggers reaches the home page. Our goal is to give each person a fair chance of getting their submission promoted to the home page."

Positive Impact

If your submission gets on the front page of Digg, expect a significant traffic increase for the website you submitted. Make no mistake, being popular on Digg can bring huge exposure.

Negative Impact

Getting on the front page is a challenge in itself. The rewards are great if you succeed, but it's certainly not a sure thing. The recent algorithm change will hopefully see some parity and common sense in the popular news stories, but probably not.

'The Digg Effect'. A popular story on Digg can create so much traffic that it becomes a headache - servers crash and websites go down under the weight of being dugg. If you submit on Digg, be prepared for a possible onslaught.

Unfortunately, a lot of the traffic resulting in being dugg does not get you much ROI. You're not going to find sustainable and renewable traffic from the Digg demographic. The vast majority of diggers are after a one-time hit.

del.icio.us

Social bookmarking on del.icio.us is essential if you're an SEO specialist or just an avid Web 2.0 user. The ability to have a place to store all your online favorites is not only convenient, but brilliant in its simplicity.

Positive Impact

When you submit and tag a bookmark, it provides exposure to the bookmarked website and also ensures that it will be indexed in major search engines. Taking the time to create quality tags can make the difference between getting bookmarked a few times or hundreds of times. Since del.icio.us relies on the social aspect of bookmarking, the traffic-to-user relevancy is generally pretty high.

You can also subscribe to particular tags of users. If there are only parts of a user's bookmarks that you like in particular — say, something tagged with "SEO" — you can subscribe directly to that tag's feed.

Negative Impact

Many users of del.icio.us are now using the social bookmarking service as their own private bookmarks collection. This completely takes away the social aspect of del.icio.us and the point of the website altogether.

StumbleUpon

Channel surf the internet with the StumbleUpon toolbar to find great websites, videos, photos and more based on your interests. StumbleUpon learns what you like and makes better recommendations.

Positive Impact

Dependable traffic. If someone stumbles upon your site, the chances are very good that the content is something they were searching for and will bookmark. You're far more likely to get a renewable source of traffic from Stumbleupon than you are from being dugg.

Negative Impact

Hateful or abusive reviews. Having a difference of opinion is just something some people can't stand. Be prepared to take some flack and be careful about which category you're submitting to. Be certain it's relative to the website and make sure you hit that fine balance between newsworthy content and relative information.

Reddit

A source for what's new and popular online. vote on links that you like or dislike and help decide what's popular, or submit your own! Reddit has been in the news recently, being blamed for a new Middle Eastern conflict.

Positive Impact

There's something for everyone. Whether you're promoting your own website or a client's site, you should get decent leverage out of submitted content. Reddit has ample categories to choose from.

Negative Impact

Such is Reddit's popularity that social agendas can be pursued through the site. See 'reddit's leaked algorithm'. That may be tongue-in-cheek, but the intent behind it is still true. Reddit is too repetitive in regards to the news stories that make the front page. At any given time, agenda-specific news items are almost a definite for going hot. Submitting to Reddit? Try hitting up Google Trends first; you want a sure thing, right?

Article References:

• Top 3 Problems of Social Media
• Social Bookmarking For Journalists: 101

Comment [15]

Why is the Raven blog a "Do Follow" blog? The short and simple answer has to do with rewarding our blog community. There's too many high quality comments that don't deserve to have their links "no followed" in the comments.

Running a "Do Follow" blog does take more effort. You have to be more vigilant about the quality of comments left on your blog and you have to give greater scrutiny to external links. However, there's excellent "Do Follow" and anti-spam tools that are available to bloggers. Andy Beard has a comprehensive list of "do follow" plugins. The list includes plugins and guides for WordPress, Blogger/Blogspot, Typepad and Moveable Type blogging platforms.

Five Reasons for Running a "Do Follow" Blog

Although I've already touched on some of the reasons why it's best to run a "Do Follow" blog, here's my full list (feel free to expand the list in the comments).

1. It encourages quality people to leave quality comments
2. It rewards quality commenters for their participation
3. It improves search engine performance (if you follow the comment policy below)
4. It helps you grow your RSS subscriptions and builds a better blog community
5. It facilitates the purpose of linking, instead of crippling it with "No Follow"

"Do Follow" Comment Rules / Policies

Running a "Do Follow" blog does take extra effort, which is why you should have a few comment policies. These are the policies that we use on Raven.

1. If your external link isn't related to the genre and focus of SEO, SEM, search marketing, etc..., we will remove your external link from your comment.
2. If your external link goes to a website that is questionable in any way — no PageRank, spammy, etc... — we will remove your external link from your comment.
3. If your comment is unrelated to the blog entry, abusive or outright spammy, we will remove your comment.

After you change your blog to a "Do Follow" blog, make sure you submit it to the "Do Follow" Blog Directory. That will make it easier for other "Do Follow" bloggers to find your website and is also an excellent resource for yourself. As long as you use spam blocking services, like Akismet, you should be able to effectively manage the amount of spam that will pass through your blog.

Thanks to Randa Clay for the "U Comment / I Follow" badge.

Comment [7]

Three thousand dollars. Think about how much money that is. It's a lot for anyone. Now think about what you would normally do with that money. Pay off debt, use it for a down payment on a car, make a couple of mortgage payments perhaps, possibly put it in your kid's college fund or even a help pay for a family vacation. All of those would be worthwhile uses of your money.

Now how about one year's worth of exclusive access to SEO tools and access to seasoned search marketing gurus? I don't know about you, but I can find an awful lot of information about virtually any topic I desire on the Internet for free — especially information on SEO techniques. So how could that possibly be worth $3,000 per year?

Jim Boykin is a great guy and very well respected amongst his peers. He uses innovative SEO techniques and his knowledge of SEO and the Search Marketing business is second-to-none. But is the Internet Marketing Ninjas program a venture one step too far? Lets take a further look at what we're getting for our three-thousand dollar layoff.

The Internet Marketing Ninjas Program offers exclusive interviews with top SEOs, PPC experts, social media gurus, affiliate marketing experts, and more. Buy the videos now, and get free access to our Private Ninja SEO Toolbox with your year's subscription! These are the tools that the ninjas at We Build Pages use every day to propel clients to the top of the rankings.

Exclusive Interviews

Rand at SEOmoz recently commented that 90% of all SEO knowledge is "out there". He also blogged about how information spreads in the search world. The exclusive video's that the Internet Marketing Ninja's are offering as part of the subscription had better contain all of that ten percentile of hidden knowledge. Otherwise, I'm paying for something I could find relatively easy with a few clicks of a mouse (granted I have to know where to look, but that shouldn't prove too difficult).

Private Ninja Toolbox

By taking a look at the tools they're offering, the majority of them are tools that were in the public domain, but have now been removed. So, what makes the tools now so exclusive that they are part of a three-thousand dollar per year program? There are some never before released tools that will be part of the program, but they are going to have to be really special for the price tag.

Perhaps this is the way SEO and Search Marketing is heading - SEO at an exclusive price. But I can't help but think that Jim is targeting large corporations that keep their SEO in-house. Either that, or it's an incredible piece of linkbait...

For now, I think I'll stick to something more affordable and practical...cough...Raven SEO Tools. And I still think that most of the knowledge you need can be retrieved through free online communities and blogs. The rest really comes down to real life experience and simply practicing the art of SEO.

Comment

I've never liked the idea of renting links. Text Link Ads, one of the pioneers in link selling, continues to use misleading terminology in their marketing through their use of the term "Buy Links." The problem is that you're not buying links, you're renting them. Not only that, you're renting them at the detriment to your website.

The original reason for renting links was to artificially boost your Google PageRank and to improve your search engine results (SERPs). It seemed simple enough when it first began. Publishers could specify an area of their website where advertisers could display small text ads and everyone would win (especially Text Link Ads with their obscene cut of the transaction). However, similar to pay-per-click (PPC), all of the fun is over once the funds run out.

Then one day, Matt Cutts rolled into town to halt all of the fun — albeit temporary fun. As most SEO specialists know, Matt has preached ad nauseum about the evils of paid links. Although in the grand scheme of things, the move by Google to penalize websites for renting and hosting paid links is bit sketchy, the line has been drawn very clearly by Google's Search Engine Spam Team — they do not approve.

There is no doubt that Google can now easily detect paid link blocks on websites — especially the dynamically generated ones from link brokers. They also provide a simple way for people — particularly competitors — to report paid links. From my point of view, there's not much advantage to renting links from brokers. You get temporary links and run the risk of being penalized by Google.

As a seasoned SEO specialist, my healthy paranoia believes that Google takes it all a step further. My paranoia is that Google also tracks and determines rented links based on the "time they existed" and through "pattern recognition." For example, if a paid link campaign ends, so do the links. If Google can — and I'm sure they can — track websites that frequently have a block of links that change every few months, and they can associate the anchor text, URLs and HTML block areas with those websites, then they will have just caught both the sellers and the buyers.

Ultimately, the only paid links that are safe(r) are permanent links that appear naturally inside the copy of original content. Also, the only safe way to purchase links is to either work with private, niche link brokers or to contact website owners directly. I'll be discussing this more in a future blog entry.

Comment