Right now, in the West, web search is highly centralized. About 90 percent is in the hands of one company with one index of the web: Google. The other 10 percent is split between Bing with it’s own index and a few others. But wait it gets worse, most of the “few others” use Bing’s index and some use Google’s index. Only a couple of other, very small, search engines actually crawl the web and have their own search index.
Both major search engines are really set up to find brand name, corporate, commercial web pages, or at least, they favor those. The non-commercial blogs, personal sites, the fun websites the Indieweb movement wants to encourage, don’t get the same sort of traction.
So we have too few search engines which are not focused on independent, non-commercial websites. What do we do?
IMHO one way to fix this is Decentralizing Search:
- Have 8 major search engines with their own indexes. We may be moving toward that but progress is slow and the Web so vast that it is very expensive to set up and run a major web crawling search engine. This won’t happen overnight.
- Have many small directory type search indexes. There are still enough good directory scripts around that one can set up a directory, with a lot of features, at a very low cost.
- Blogrolls, link pages and maybe webrings. We used to surf the web, we can learn to do it again.
It’s #2 above I want to discuss. If we create many hundreds of directories and search engines, large and small, all dedicated to the listing of interesting or fun independent web sites and pages, then we create our own discovery network. Each index is a unique collection, presented differently from one another it helps break the dependency on Google and Bing in the near to mid term, at least until more major search engines get established.
And by “directory” I mean: A. a directory intended to help navigate the web NOT to sell links for SEO; B. traditional directories like Indieseek.xyz, also newer types of directories running hand coded scripts, hybrid directories that somehow incorporate a crawling spider, directories that incorporate webrings, local business directories, niche directories, even automated directories of some sort. Whatever kind of index human ingenuity can invent. By this means we start taking back the web and remaking it into something we can better enjoy.
This is not going to replace big crawling search engines. And one directory, even 100 times the size of Indieseek, is not going to make a difference, but large numbers will. If everywhere we throw a stick we hit some directory people will start exploring, just like they explore bookstores, libraries or snoop through the bookshelves at a friends house.
20 years ago, when all the search engines sucked, I and others searched for websites with a battery of maybe 10 favorite search engines and directories in our browser bookmarks. It’s not hard to do that again IF the tools are available. That is my vision of decentralized search. We don’t have to wait passively for others, big companies and venture capital, to solve the problem for us, we can do it now. Ourselves. That is the beauty of the Web.
Indieseek.xyz is my stab at a proof of concept. I know Kicks Condor is working on his own directory vision and that other concepts like Wiby.me are out there. Coders take a stab at it. If we can do this, you can do this.
This was also posted to
Liked this post? Follow this blog to get more.