Search Engines Use Software That Combs The Web

Search engines are interesting. They are automated pieces of software designed to search the web looking for keywords. If you’re reading this, then you’ve most likely used search engines at least once in your life.

Have you ever asked yourself how do search engines find new content to index? I assume that you did. The name search engines is created by the word “searching”, which means finding something with a lot of information either on the internet or in the real world. And one of the ways to do this is by crawling.

It’s important to remember that search engines themselves are software. Their main task is to crawl the web, looking for new and updated content. The first of their two main objectives? To build the best, most relevant search index possible.

 Search Engines Use Software That Combs The Web

What Is a Search Engine? Definition Plus 10 Examples

Public domain image via Pixabay

What Is a Search Engine?

Also known as a web search engine and an internet search engine, a search engine is a (usually web-based) computer program that collects and organizes content from all over the internet.

The user enters a query composed of keywords or phrases, and the search engine responds by providing a list of results that best match the user’s query. The results can take the form of links to websites, images, videos, or other online data.

How Do Search Engines Work?

The work of a search engine can be broken down into three stages. Firstly, there is the process of discovering the information. Secondly, there is the organization of the information so that it can be effectively accessed and presented when users search for something. Thirdly, the information must be assessed to present search engine users with relevant answers to their queries.

These three stages are usually called crawling, indexing, and ranking.


Search engines use pieces of software called web crawlers to locate publicly available information from the internet, which is why this process is known as crawling. Web crawlers can also sometimes be referred to as search engine spiders. The process is complicated, but essentially the crawlers/spiders find the webservers (also known as just servers for short) which host the websites and then proceed to investigate them.

A list of all the servers is created, and it is established how many websites are hosted on each server. The number of pages each website has, as well as the nature of the content, for example, text, images, audio, video, is also ascertained. The crawlers also follow any links that the website has, whether internal ones that point to pages within the site, or external ones that point to other websites and use them to discover more pages.


Information found by the crawlers is organized, sorted, and stored so that it can later be processed by the algorithms for presentation to the search engine user. This is known as indexing. Not all the page information is stored by the search engine, instead, it’s just the essential information needed by the algorithms to assess the relevance of the page for ranking purposes.


When a query is entered into a search engine, the index is scoured for relevant information and then sorted into a hierarchical order by an algorithm. This ordering of the search engine results pages (SERPS) is known as ranking.

Different search engines use different algorithms, and so give different results. Over the years, algorithms have become more and more complex as they attempt to present more relevant and accurate answers in response to the queries of search engine users.

10 Examples of Search Engines

1. Google

Google is the biggest search engine in the world by far. It handles over 5 billion searches each day and has a market share of over 90% at the time of writing (August 2019). Developed originally by Larry Page and Sergey Brin in 1997, Google has become so successful that it has become synonymous with search engine services, even entering the dictionary as a verb, with people using expressions such as: “I googled it” when they’ve searched for something online.

2. Bing

The origins of Microsoft’s Bing can be found in the technology company’s earlier search engines, MSN Search, Windows Live Search, and Live Search. Bing was launched in 2009 with high hopes that it could usurp its rival Google, but despite attracting many fans, things haven’t quite worked out that way. Even so, Bing is the third largest search engine worldwide after Google and Baidu. It is available in 40 different languages.

3. Yahoo!

Yahoo! Search is another big player in the search engine world. However, for much of its history it has supplied the user interface, but relied on others to power the searchable index and web crawling. From 2001 to 2004, it was powered by Inktomi and then Google. From 2004, Yahoo! Search was independent until a deal was struck with Microsoft in 2009 whereby Bing would power the index and crawling.


Originally known as Ask Jeeves, is a little different from Google and Bing, as it uses a question and answer format. For a number of years, was focused on becoming a direct rival to the big search engines, but nowadays, answers are supplied from its vast archive and users contributions, along with the help of an unnamed and outsourced third-party search provider.

5. Baidu

Founded in the year 2000 by Robin Li and Eric Xu, Baidu is the most popular search engine in China, and the fourth most visited website in the world, according to Alexa rankings. Baidu has its origins in RankDex, a search engine previously developed by Robin Li in 1996. As well as its Chinese search engine, Baidu also offers a mapping service called Baidu Maps and more than 55 other internet-related services.


AOL, now styled as Aol. and originally known as America Online, was a big player in the early days of the internet revolution, providing a dial-up service for millions of Americans in the late 1990’s. Despite AOL’s decline as broadband gradually replaced dial-up, the AOL search engine is still used by a significant minority of searchers. On June 23, 2015, AOL was acquired by Verizon Communications.

7. DuckDuckGo

DuckDuckGo (DDG) has a number of features that distinguish it from its main competitors. It has a strong focus on protecting searchers’ privacy, so rather than profiling users and presenting them with personalized results, it provides the same search results for any given search term. There’s also an emphasis on providing quality rather than quantity when it comes to search results. DDG’s interface is very clean and not overladen with adverts.

8. WolframAlpha

WolframAlpha markets itself as a computational knowledge engine. Instead of answering the queries of searchers with a list of links, it responds with mathematical and scientific answers for their questions, using externally sourced “curated data”. WolframAlpha was launched in 2009 and has become a valuable tool for academics and researchers.

9. Yandex

Launched in 1997, Yandex is Russia’s largest search engine, and the country’s fourth most popular website. Outside of Russia, the search engine also has a major presence in Ukraine, Belarus, Kazakhstan, and other countries of the Commonwealth of Independent States. As well as search, Yandex offers many other internet-related products and services, including maps and navigation, music, eCommerce, mobile applications, and online advertising.

10. Internet Archive

The Internet Archive provides free public access to a wide range of digital materials. A nonprofit digital library based in San Francisco, it’s a great tool for tracing the history domains and seeing how they have evolved over the years. Besides websites, you can also find software applications and games, movies/videos, music, moving images, and a huge collection of public-domain books. The Internet Archive also campaigns for a free and open internet.