SEO > Programming Project

How Search Engines Look at Information

Mon, 23 Jun 2008 12:12:02 MST

This is the third in a series of articles attempting to look at how programs are built and how one would begin to think about building their own search engine. This article is about how information on the web is laid out and how the major search engines must look at the context the creator provides.

Part I: Creating a New Program

Part II: Beginning The Search Project – Understanding Short Cuts

At its core a search engine needs to find information locked up in documents, databases or web pages. After finding the information in those documents a search engine needs to deliver the information to someone based on a question. Hopefully the question itself can provide some context to help in delivering the proper result.

You could look at it as if all this data in the world were flat, it starts off equal and depending on the way someone chooses to organize these bits, the more context they have. If I place something in a pdf document, or an html document, or an image, I have a different set of organizational context surrounding it.

A web page has meta tags that help to make sense of it, hypertext links to and fro that help to define it, and html and css that help to provide emphasis to certain parts, and the actual text information. This is at least the way a human could interpret it. So far search engine bots are not likely to interpret the document in the same way a human would. And that is understandable since two different humans would likely interpret the layout and context of the document differently also.

Pdf's, word documents, and text style documents, may or may not have any formating to provide emphasis, and these tend not to be linked to in the same way as an html document. Although both html documents and pdf's do have url's, so the domain, or site itself may provide some context.

But information is not really flat, since those individuals that create a document inherently provide context. If we consider a web site, there are two different elements, the site as a whole and the individual document. The site as a whole may be made up of thousands of urls and may or may not be related to a single concept. You would expect the homepage, or root directory files to contain the main concept of the whole domain. This does not include subdomains, which can be related to different computers.

The basics of any search engine are how to interpret what someone types in. If someone types in “car”, then what should come up? What if a human were asked that same question, what would they come up with? But it is likely the question would be longer and more specific. To a certain extent people have been trained to speak (type) the way search engines want them to, since not many people in the real world would just say “car”. But on the web people type in “car” and other relatively nonsensical phrases. And since this web language has developed, many search engines and site owners understand this and have tailored their titles and results to reflect this development.

Beginning The Search Project – Understanding Short Cuts

Mon, 19 May 2008 12:11:55 MST

This is the second in a series of articles attempting to look at how programs are built and how one would begin to think about building their own search engine. This article is about understanding the nature of taking shortcuts and how the major search engines need these to perform up to current expectations.

Part I: Creating a New Program

Part III: How Search Engines Look at Information

The first thing you want to do is determine what you want to accomplish. If I were starting my own search engine then I first want to think about what exists and why. What don't I like about Google or Yahoo! search or MSN search.

In regards to Google and the other major search engines, everyone dislikes bad results. But you have to determine what is a bad result. If you are searching for contemporary information then for the most part the results are easy to find. This is especially true with Google's search engine, since they frequently source news collections sites, such as Digg. And it is a point like this that needs to be highlighted. This is one of many shortcuts that Google takes to have better results. If there were no public relation sites, such as openpr.com or news aggregators, such as digg.com then Google would have a tougher time getting those results to the public. They would have to work much harder at understanding what is todays news and what is yesterday's news. And yesterday's news is just history, locked in documents called web pages.

So news sites, public relation sites and so on are shortcuts that Google has capitalized on to make sure their results are better, then if they were simply relying on their own algorithms. Another of these shortcuts are separate programs to interpret certain types of search queries. Let's use the example:

“grams in a pound”

The first thing you will notice is that they get the right answer. The next thing you should realize is that they had to write a separate program to deal with that type of language. They provide their traditional results, but also the answer to your question.

But if you try a little different search, say “1 pound is x grams”, they do not translate this into their program. You just receive the traditional results - and this is a matter of differentiating language, something which at this point needs to be very hard-coded.

So lets say that:

1.We want to produce good results
2.We don't want to write separate programs specific to certain topics

But we must understand the nature of shortcuts, and since the web changes (i.e., all sorts of people out there are adding all sorts of new structures all the time), the search engine's role must adapt. I suppose the ideal situation is some all-encompassing artificial intelligence that would adapt on its own, understanding that the web changes and that people are changing it. But for now lets further understand the use of shortcuts and how to make them work better.

Creating a New Program

Mon, 19 May 2008 11:41:16 MST

This is the first in a series of articles attempting to look at how programs are built and how one would begin to think about building their own search engine.

If you have the opportunity to build a program, you will notice that there are few resources to help you think about the overall structure. There are plenty of web resources for specific issues, but not too many when you want to think about the overall idea of how things should fit.

For web program ideas there are many open-source programs you can open up and look at – oscommerce, pligg, wordpress...etc but in some cases you are starting from scratch and these open-source programs are rather all over the place. A lot of companies use these open-source programs and end up hiring professionals to do any modifications, even just color changes. Anyway, open-source always has the downside of being a little too understandable to just about anyone with a little knowledge, and that may not be a good thing for some applications. So you could modify any of these open-source programs for your own purpose, but that may just drive you in the wrong direction.

Some projects require SEO, and some programs do not. SEO may also steer you away from some open-source applications, since you might want to be innovative. Some programs are needed because they are for a job which may have strict requirements. But there are some projects that you just do for yourself.

Perhaps, if you are like me you want to make something that you need for yourself, or maybe you are just interested in creating something new and interesting. If you are the type that likes to get into new projects, which may or may not result in a business, then you are generally starting from scratch.

There are many programs that need creating, or improving, whether they are just something small or if they are something large. Many people have been thinking about, or are currently in the midst of creating their own search engine, simply because they do not like the way the current crop of companies deal with information. But when you think about creating your own program you may come with all sorts of opinions, but soon you will face a reality.

When you are thinking about the overall structure of a project you should realize that the thinking part will consume a great deal of time. Why are you doing it? What exactly do you want to accomplish? So the philosophy is a major part, and that can take months, then a little programming, then some more thinking.

Part II: Beginning The Project - Understaning Short Cuts

Part III: How Search Engines Look at Information