science behind Search Engines optimization (SEO)

The science behind Search Engines optimization (SEO)

I'm often asked how search engines do what they do. Search engine technology is pretty complex so let's start by breaking it down into its individual parts. Then I'll attempt to simplify the parts that make up the whole.

Let's start by breaking up the search engine process into three prime areas: The spider, the index, and the search leg.

The spider crawls the servers on the internet to find HTML documents, then dumps them into the index.
The index filters the HTML pages and stands by as a repository of data which answers to the search leg.
The search leg is the connection between a browser, the index, and a search result.

I drew a diagram right below here to help visualize the process.

In the diagram, you can see the three entities that need to communicate with each other for indexing and ranking to take place: the web site, the search engine, and the browser.

Let's start by looking at some of the processes:

The filtering process between the spider and the index.
This is where the search engine applies its unique algorithms for organizing the data in its index. In its most basic form, genuine SEO occurs as a continuous loop of research, analysis, submission, monitoring, and reporting. This method allows for nonstop improvement of the webpages as they travel through the search engine filter process month after month.

This process helps webpages and their content to be in prime form and to demonstrate its compliance with what a search engine spider wants to index. A search engine is tasked with finding expert advice, knowledge, information,and opinion. While, at the same time, also ranking what it finds most pertinent for a particular query.

The science of relevance is a responsibility Yahoo, Google, AOL, MSN, and other search engines take very seriously. I have found that by demonstrating respect for this science and understanding how a search engine works is the quickest way to acquire good organic search results.

A closer look at the individual components in the search engine process:

The website

The website can't be found without a properly configured server. The server hosts your website and is at the very foundation of the entire process. Its capability to communicate well with the spider is of significant importance. The server DNS configuration, its organization of associated files, and its means of directing the search engine to certain documents or URLs are also unquestionably important.

Think of the server as a gatekeeper that lets the search engine spider in; if there are any problems between the two communicating, the search engine spider will give up and leave without indexing your web pages. The spider has a very limited time to respond to any type of issues while crawling the web and it requires everything in proper working order. So it would appear most likely that websites with top organic/natural rankings are the websites that have their servers configured and organized properly.

The HTML code which makes up your website simply waits on the server for the search engines' spiders and web surfers. Spiders like to consume simple HTML code; they can't process JavaScript or complex code as quickly or easily. Therefore, it opens up the possibility of having your web pages skipped over and not being indexed.

The search engine

Google and other top search engines are always searching for original and outstanding quality content to rank at the top of their organic listings on their search engine results pages. They're trying to provide a high quality user experience for those users searching their databases by producing relevant, unbiased, and useful information.

In order to fulfill this task while relying on a computerized procedure of interpreting HTML code, the search engine must store and filter that data in a tamper proof location. And that location is identified as the index. Therefore, after gathering data from servers on the web, the spider filters it into the index.
The index is the heart of search engine credibility; if the index is any way corrupted, the search results will be also. For this reason, search engines always take so much care in filtering and monitoring their index.

Simply getting your website into a search engine index is good but not good enough. Getting a high ranking on the search results page is the goal. There's a huge difference between the two.

Metaphorically, being in the index vs. top 10 search results is the equivalent of flying a kite vs. flying a fighter jet. There is a huge difference between distinguishing your website content as that of an expert in the eyes of a top search engine vs. getting your website into the search engines' index. This difference depends on the training and skills of your CEO professional or in-house team.

Keep in mind, in 1996 it was fairly easy because the search leg had fewer than a hundred million documents to sift through. In 2004, it was estimated that Google was managing anywhere from 45,000 to 80,000 servers. Nowadays, I don't know if Google even knows how many servers they're managing while indexing its eight billion+ web documents. At this time, the search leg must navigate itself through billions of documents and return millions of relevant results within 16-hundredths of a second.

Okay, all you mathematicians out there figure this one out. If you Google for your keyword and it serves 50,000 search results. In order to get your site into one of the top 10 positions it would have to rank within the top 0.02 percent of all search results. Now let's assume your keyword competes with 1.5 million pages. To find your site in the top 10 search results, it must rank within the top 0.0007 percent, [(10/1,500,000)x100] of all the search results!

The Browser

The browser is the place where a search request originates from on a desktop. It connects to and communicates with the search leg of a index to return a search engine result page (SERP).

When you hit the search button, your browser is calling to the search leg, and accessing the index and then receiving the search engine's best estimate at answering your search query with the most relevant results.

The CTR

Now let's turn our attention to the CTR (clickthrough-rate). The CTR for organic results on Google is about 28 percent for the No. 1 position and about 3-12 percent for the No. 2 through No. 10 positions.

When you take the time to consider how many searches occur daily for the most popular keywords and phrases, then you'll begin to understand the amount of traffic that you can get. For example, there were over 5.5 billion searches in January 2006 alone. In fact, research and case studies have documented significant increases in the number conversions when a website enters the top 10 rankings in Google.

SEO best practices

It's never been more important for your team to understand how search engines work and to comply with their guidelines. There's a big payoff for those organizations that take the time to follow the guidelines; because if they do follow them they will find it much easier to achieve a far more superior result. Complying with search engine guidelines is a very important part of SEO best practices.

We as an industry need to adopt standards. The sooner we develop uniform industry standards, the sooner we can eliminate the devious search engine optimization techniques that still tarnish our industry. This will bring more credibility and prosperity to our trade.

SEO and the bottom line

As reported in the recent Outsell annual Ad Spending Study, companies are allocating 33 percent of their online budgets to website updates. The report further states that better natural search results are driving this trend. Organic SEO gives you a better functioning website, which in turn, delivers a better bottom line-- a win-win for C-level executives.

science behind Search Engines optimization (SEO)

I developed this page to help clients more fully understand the process search engines go through to determine what web site will appear on the top results of a given key word search.