Have you ever held a magnifying glass to your computer monitor and counted the pixels one by one? That’s the level of detail revealed about your website by search engine crawlers: every word, title, image, meta tag and load time for every page. They know more about your site than you do.
What crawlers learn feeds search engine algorithms that determine how your site ranks in search results, affecting clicks, conversions, shopping cart size and brand loyalty. That’s a lot of power.
The good news is that you can capture Web crawlers’ power and put it to work for you. For example, you can determine, in minutes, the 20 percent of website performance issues that — if solved — would give you 80 percent performance gains.
They Know Your Weaknesses
Search engine crawlers flag site latency as a weakness, sometimes even skipping slow-loading pages. Or, due to poor site configuration, they may crawl inefficiently, focusing intently on some pages while bypassing others altogether. This presents a particular challenge for large online retail sites with extensive dynamic content, meaning complex Web pages — often with personalized content — that necessitate large object libraries.
The crawlers relay their findings to the search engines which, in generating website rankings, penalize slow and poorly indexed websites with suboptimal placements. Optimizing your website structure and speed helps ensure your site isn’t buried on the third, fourth, or fifth page of search results. Since most Internet shoppers don’t click beyond page one, it pays — literally — to know how well your website is indexed and what’s slowing it down.
In short, to succeed with SEO, you need to know exactly what the crawlers see.
Approaches That Fall Short
Separate search engines equals separate analytics. Google doesn’t tell you about Bing, nor Yahoo about Google. Scrutinizing only one set of crawler data means working from incomplete information. You could scrutinize all of them one by one, but that wastes time and risks overlooking valuable cross-engine correlations.
A content delivery network (CDN) used for site acceleration can collect information about multiple crawlers — but beware when taking this approach. A traditional CDN may aggregate content to such a degree that you end up with a “black box,” losing visibility of the details critical for honing SEO performance.
There is a way to overcome all of these limitations at once.
Become Omniscient
Aggregate the analytics using your CDN as a super-observer and conduit. A CDN well designed for today’s e-commerce challenges is poised to collect and analyze Web crawler activity because it monitors all HTTP requests for site information. If you have a global monitoring channel already in place, then it’s easy to aggregate, analyze and report on data for all crawler requests from Google, Bing, Yahoo and others.
This straightforward approach of integrating SEO analytics with your CDN gives you immediate, detailed insights — like which objects have the slowest delivery times — for quickly fixing website performance issues to boost search engine rankings.
Once You Know the Crawler’s Secrets…
With the new insights you gain, you can optimize your e-commerce site for search engines ad infinitum, especially with the comprehensive, granular data available through CDN-integrated SEO analytics. Plus, search engine algorithms evolve over time, so the moving target of SEO could be all-consuming. Fortunately, in this case, the 80-20 rule applies.
Twenty percent of the problem pages or objects cause 80 percent of site performance problems, and that’s where to focus your SEO efforts. In particular, watch these key pieces of crawler data:
- repeated requests for the same page/object
- overall number of pages/objects viewed by a crawler
- the number of pages/objects returning error codes
- objects with excessive delivery times
By establishing performance thresholds for these key elements, you can pinpoint problems quickly. With detailed, actionable insights in hand, you can then optimize your website for better SEO ranking with relatively little effort and budget.
Drive More Business With SEO Analytics
Internet retailers that tap into the power of Web crawler activity can improve site performance, achieve greater returns on SEO investments, and gain advantage over competitors that remain in the dark. The greatest beneficiaries may be smaller retailers that can’t afford big pay-per-click campaigns. For them, an informed SEO strategy with the right tools and analytics can level the playing field with deep-pocketed rivals.
The SEO analytics from your CDN reconnaissance will help you achieve better SEO for higher placement in organic search results, driving more clicks and conversions. Plus, faster site speed will improve user experience, increasing brand loyalty and monetization.
Through the secrets they tell, Web crawlers possess considerable power. Use your CDN to listen and learn, and put their power to work for you.
Ronni Zehavi is CEO of Cotendo.
Loading ...
