If you’ve ever tried to research a difficult topic on Google, DuckDuckGo, Bing, or any of the other major search engines, you know what a battle it is. You don’t just get irrelevant results, you get the feeling the search engine is working against you. Instead of matching your keywords, it returns matches for vaguely similar spellings. Instead of matching all your search terms, it gives you popular pages that match just one. You may start to think the search engines are conspiring against you, and in a sense you’re right.

Why the search engines fight you

Here’s the secret: Search engines think you’re stupid. They think you can’t construct a proper search and they have to “help” you by guessing your real intent. Statistically, this isn’t so unreasonable. Most people have no idea how to construct a search string. They can’t spell. Search engines have dumbed themselves down to the level of these people. This is great if you can’t remember the spelling of a name and you’re looking for popular articles, but it’s murder when you’re trying to get an answer to a difficult query.

The first problem is that the engines heavily favor popular sites. There are reasons they do. Before Google, search sites treated all pages as equal and considered only how well they matched. This meant anyone could get into search results just by sticking in a few keywords. People regularly got lists of porn sites when searching on innocent topics. Google’s breakthrough was to consider the quality of the match.

The trouble is that it’s gone too far. A popular page which is a partial match outranks an obscure one which is a full match. Even words which aren’t matches outrank matching words. Recently I tried to search on DAA (distributed access architecture) and was overwhelmed with results that (I think) showed up because the pages contained the word “data.” Sometimes the engines will warn you, asking if you really meant that word, but at least as often they won’t.

Just searching on my own last name is often a pain. I just did such a search on Startpage, which told me, “Including results for mcgrath. Search only for ‘mcgath’?” If I wanted results for McGrath, I would have searched for McGrath!!!

Another problem is that recent items rank much higher than older ones. Suppose you’re trying to get background on an ongoing situation where a big news story just broke. You might want to learn about a long-running war, a school system, or a business. Doing the obvious search will get you a ton of versions of the news story, mostly copied from each other. Getting past them to find general information is a pain.

How to outsmart the search engine

As a serious researcher, you have to maneuver search engines into giving you the results you want. To do it, you have to understand how they work and what all your options are.

Typing in a question is a bad strategy. Years ago there was a search site called Ask Jeeves that claimed to handle questions, but it never worked well. Ask.com still exists, but it no longer encourages people to phrase their searches as questions. Whatever search site you use, just enter the relevant terms.

Putting words and phrases in quotes helps. Quotes tell the engine to treat a phrase (e.g, “Homeland Security”) as a phrase rather than two unrelated words. They’re also useful with single words. The quotes say, “and I mean it.” There’s a case for putting every word in a search in quotes. You’ll still get irrelevant results, but you’ll get fewer of them. For all their claims of using algorithms, the engines are never entirely predictable.

Adding terms to make your search more specific may help, but it doesn’t always. Sometimes the engines use the additional terms as an excuse to drag in results that contain any one of the terms. When you have more than three terms, putting each of them in quotes helps. A different combination of search terms, rather than more terms, may be more helpful.

You’d think that long-tail terms would be the best ones, but that’s where the engines try hardest to frustrate you. They wag their finger at you, telling you there aren’t a lot of results containing that term, and you really ought to search for a different term. Again, putting those terms in quotes may help.

If one search engine doesn’t work, try another. You might get better results. Please stop telling people to “google.” Google is a big, rich company and doesn’t need your free advertising, even if it loves it.

Search restrictions

Most engines support qualifiers to restrict the search. They can be a huge help. The “site:” qualifier, which many of them support, gives you results from just one domain. It can be a top-level domain, such as .edu, .de, or .gov. If you have an idea of where the best results will be, this trick can get you directly to the results you need.

Date restrictions are usually available, though not all search sites respect them consistently. You can typically ask in “Advanced Search” for results from the past 24 hours, month, or year. Unfortunately, it’s harder to ask for just older results, which are sometimes what you want when doing research.

Escape the search engine

Sometimes the best approach isn’t to use a search engine at all. There are many searchable sites with information on specialized topics. Stack Exchange is one of the best-known examples. The responses to questions are user-contributed and user-rated, and most of the time they’re pretty good. A useful trick is to type in a question that interests you, even if you don’t feel brave enough to submit it. Doing that will show you questions similar to yours, often with helpful answers.

Wikipedia is a useful alternative, if you know how to avoid its pitfalls. Its information on most topics is more stable than search results. It can be dead wrong, so be careful. Some topics are edited by someone with a personal agenda and not checked much by others. The information can be years out of date. The best part of the articles is often the links — at least the half of them which aren’t broken. Look at an article critically before trusting it. If in doubt, look at its editing history. If edits are few, or if there are signs of an editing war, be especially skeptical.

The winning search strategy

When you’re researching on the Web, flexibility is important. Don’t just settle on the first page of results you get. If it’s not what you want, look for different ways to find better information. With practice, you’ll accumulate a range of techniques and become an expert researcher.