Right now, in the West, web search is highly centralized. About 90 percent is in the hands of one company with one index of the web: Google. The other 10 percent is split between Bing with it’s own index and a few others. But wait it gets worse, most of the “few others” use Bing’s index and some use Google’s index. Only a couple of other, very small, search engines actually crawl the web and have their own search index.
Both major search engines are really set up to find brand name, corporate, commercial web pages, or at least, they favor those. The non-commercial blogs, personal sites, the fun websites the Indieweb movement wants to encourage, don’t get the same sort of traction.
So we have too few search engines which are not focused on independent, non-commercial websites. What do we do?
IMHO one way to fix this is Decentralizing Search:
- Have 8 major search engines with their own indexes. We may be moving toward that but progress is slow and the Web so vast that it is very expensive to set up and run a major web crawling search engine. This won’t happen overnight.
- Have many small directory type search indexes. There are still enough good directory scripts around that one can set up a directory, with a lot of features, at a very low cost.
- Blogrolls, link pages and maybe webrings. We used to surf the web, we can learn to do it again.
Number 2.
It’s #2 above I want to discuss. If we create many hundreds of directories and search engines, large and small, all dedicated to the listing of interesting or fun independent web sites and pages, then we create our own discovery network. Each index is a unique collection, presented differently from one another it helps break the dependency on Google and Bing in the near to mid term, at least until more major search engines get established.
And by “directory” I mean: A. a directory intended to help navigate the web NOT to sell links for SEO; B. traditional directories like Indieseek.xyz, also newer types of directories running hand coded scripts, hybrid directories that somehow incorporate a crawling spider, directories that incorporate webrings, local business directories, niche directories, even automated directories of some sort. Whatever kind of index human ingenuity can invent. By this means we start taking back the web and remaking it into something we can better enjoy.
This is not going to replace big crawling search engines. And one directory, even 100 times the size of Indieseek, is not going to make a difference, but large numbers will. If everywhere we throw a stick we hit some directory people will start exploring, just like they explore bookstores, libraries or snoop through the bookshelves at a friends house.
20 years ago, when all the search engines sucked, I and others searched for websites with a battery of maybe 10 favorite search engines and directories in our browser bookmarks. It’s not hard to do that again IF the tools are available. That is my vision of decentralized search. We don’t have to wait passively for others, big companies and venture capital, to solve the problem for us, we can do it now. Ourselves. That is the beauty of the Web.
Indieseek.xyz is my stab at a proof of concept. I know Kicks Condor is working on his own directory vision and that other concepts like Wiby.me are out there. Coders take a stab at it. If we can do this, you can do this.
This was also posted to
/en/linking.
Liked this post? Follow this blog to get more. 
I am very tied up trying to finish mine, but you’re doing a lot of good writing
and I wish I was done so I we could be synced up on this.
I think the other question we need to ask is: how do we make a directory that’s
not a directory? Like: is there a new kind of directory that is an evolution of
the tried format? And I think the main point of pain is having to enter this
giant catalog through a straw.
Just like Google is ‘entered into’ through a few search words—terms that rarely hit
the mark and need to be gamed—could the directory widen the straw somehow?
This is what is done by providing a hierarchy—but I also wonder if there are
other novel forms. Like: say the directory changed day-by-day to suggest common
categories or to show where I’ve been editing or to suggest a few categories.
I also wonder if a ‘pinned post’ might be useful: here are a few suggested
categories, here’s one I added recently that’s kind of sweet, here are a few
links that I’m considering throwing out.
Also on:
You could do this with RSS. Example: Include the, say, 5 newest posts on a listed site via RSS. You would have to come up with a way to index the content of each RSS post maybe daily or every 12 hours and some way to weed out shorty posts like checkins. Discard old upon update. That would give you fresh changing content on the cheap. Still pretty server intensive.
Flag categories that are new or updated with an “Updated” flag for X number of days. Generate top lists via javascript: 5 newest categories, 5 most popular categories, etc. Likewise have a “New” flag for X number of days for each new listing.
Sort order: you could have New listings appear at the top of their categories and first in any searches for the first X number of days.
I like the idea of the Admin. pinning a category to the top saying “I’ve been doing a lot of work here check it out.”
One thing I’m going to try to have done is have a “Random” link coded up. Click on “Random” and you get sent off to a random site from the whole index. Adventure!
Idea: Play with your search and find 5 queries that return really cool results. Maybe queries most would not think of. Turn those into links on a sidebar. Change these up every few months.
Tracking: this is important knowing where traffic is coming from, but also what pages people visit what they click on. Ditto logging search terms from your own search boxes.
Tag clouds: WSNLinks uses keywords meta to generate the tag cloud box. The clouds refresh with each pageview so they are randomish. I don’t think those keyword tags are included for regular search box search, but click on a tag in the tag cloud and it initiates a search for that term. Kinda different. Might be some ideas there.
General rules I’ve learned: 1. Don’t rely on user input if it takes any effort. 2. Don’t over complicate the UI. 3. Users like New so a page listing the newest sites gets traffic.
I can’t wait to see your directory when you roll it out!