I am highlighting two search engines that are indexing the only Indie Web and/or Indieweb pages (please see below for my definitions of these terms.)  Each search engine is different from each other so I’m going to highlight both their similarities and differences.

Similarities

Both engines allow submissions of URL’s for inclusion.

Both engines are human reviewed at submission.  This is a good thing, because a human editor can fiter out spam and websites that just don’t belong better than any algorithm.

Both will recrawl at unknown intervals for changes and to detect dead URL’s.

Both crawl for onpage text and maybe meta tags.  I’m guessing here but this seems likely.  Therefore, they may not do so well indexing pages that are mainly, photos, artwork or images with very little text.

Both discover new pages/websites only by submissions and by URL’s added to the index that the human editor might add manually.  So neither has a general crawler that just keeps following hyperlinks like a general search engine crawler.

Both have a nice “Random” result link which is a handy and fun starting point to surf the web.

Differences

Wiby.me – Wiby is designed to index Indie Web (aka Independent Web) pages, like the old non-commercial HTML static pages of the 1990’s and early 2000’s.  It does not spider a whole website.  It only crawls the unique URL submitted.  So only one page.  Because of that it is not really designed to crawl non-commercial blogs.  Wiby’s suggestion for blog webmasters is to submit one or two really good posts if you want to be in the index.  Because it actually crawls onpage text this makes Wiby more sophisticated than a web directory like Indieseek.xyz and gives it a better search function.

Searchmysite.net – Searchmysite is designed to crawl deeper (maybe 50 pages or permalinks) within a single domain. (I don’t know how it treats subdomains.)  This makes it better at indexing Indieweb blogs and indeed, as I write this, most of Searchmysite’s index appears to be Indieweb blogs.  I cannot speak for the editor but I don’t see why Searchmysite would not also accept and crawl static HTML websites (ie. Retro or vintage HTML sites) so long as the site has some value and content that it can index, but they might not.  Again, this is going to be mainly text.  If you are looking for non-commercial blogs Searchmysite is the better option to start with.

Future of Indie Search

I’m highlighting these two niche search engines because in their own ways, they represent the future of Indie Web search.  I’d like to see many more of them.  Directories, like Indieseek.xyz, have their place, but being able to collect and search by,  actual on-page content goes far beyond what a traditional directory is capable of doing.  These guys could put me out of business and I’d be just fine with that!

 

Liked this post? Follow this blog to get more. Follow

Bookmarked

Bookmark: Microsoft proposes method to automatically submit URLs from WordPress to search engines | Search Engine Land

Microsoft wants to build into the core of WordPress the ability to automatically push new and updated URLs to Bing and other search engines.

Since so much of the web is built on WordPress, this could save current and future smaller engines a lot of crawling time.  I think this could be good for the Indieweb and Independent Web of non-commercial bloggers.

I like that this isn’t just for Bing but other search engines could use it too.  It ight help even the crawler playing field against Google.

However, beware, I can see this quickly being abused by SEO’s so I guess each search engine would have to develop some way to filter out the dreck from the real content from this firehose.

Liked this post? Follow this blog to get more. Follow

None of this is new news, it’s just new to me and in case it slipped under your radar I’m sharing it.

It starts with this interview in HackerNoon: Startpage is Still a Privacy-First Search Engine, says System1 CEO

System1 is an ad tech company that is getting into the Internet privacy business.  They are best known for buying the privacy meta-search engine Startpage.

But wait there is more.

Last year we acquired the Waterfox browser, which is known for, among other things, being privacy friendly. We are also working on a private mapping solution in our MapQuest business. We believe a combined offering, which could include VPN and other privacy-related services, would be a very interesting privacy bundle for our users. Stay tuned!

So they own Startpage, Waterfox browser, and good old MapQuest, a pioneer in online mapping.  Yeah they have the beginning building blocks for a suite there if they can put them all together.

DuckDuckGo is more than just a search engine, they are a privacy company.  Brave browser is now expanding into privacy search. Now System1 has shown that they have bigger plans than just a search engine. Interesting.

Other References:

Waterfox web browser sold to System1 | ghacks

Verizon sells mapping service MapQuest to ad-tech firm System1 | Josh Loe

Liked this post? Follow this blog to get more. Follow

I previously reviewed Mojeek.com in September, 2018.  (Note: I’m now posting all search related stuff here instead of on my personal blog.)  Since then there has been a lot of movement in the search industry, with an explosion of Bing Clones with only a few of those Bing Clones worth using.  Plus a smattering of new engines with their own indexes.

What makes Mojeek different is:

  1. Privacy – Mojeek was a privacy respecting search engine way before privacy became cool.
  2. Building their own index – this is really important because there are only three other large indexes of the Web in English: Google, Bing and Yandex.  Mojeek comes in at #4.

The combination of privacy respecting and having it’s own index are what makes Mojeek a potential, long term, big league player in the search engine business.

What has Changed Since 2018?

 

At first glance not much has changed since my original review above.  Mojeek still has the same ultra-clean look it had before.  The big changes come in the search results because Mojeek has been busy adding a couple of hundred servers: increasing their index size by crawling, and refreshing old pages in the index more often.

  • I’m seeing a fresh crawled date on at least one listing for every search I do.
  • I’m seeing that many other listings have been recrawled in the last few weeks so the index is fresher than ever.
  • There are definitely more pages in the index.

It’s Not Perfect

Mojeek has it’s spells of weirdness.  Note these 2 searches.

Example:

Search for “marlin firearms” brings up both the main website for Marlin in the organic results plus the Wikipedia entry in the sidebar.

Now look at a search for “savage firearms” and you see the Savage website in the organic results but this time no Wikipedia entry!  The problem lies with Wikipedia which, for some reason, lists Savage as “savage fire arms” (“fire arms” as two words) and Mojeek can’t bridge that gap.

The thing is when I first ran these searches I was looking for the Wikipedia entry, not the organic results, to get some history on these companies.  That’s why I noticed it.  Not a big deal just a little weird. However, do note that the organic results are pretty good, for both searches, and not exactly like the organic results Google or Bing would give you.  That’s a very good thing.

There will also be times when Mojeek just didn’t have the website I was looking for but it did return a cluster of websites that mention the name of the website I wanted.  I am not an expert but I take that as a sign that Mojeek’s crawler just has not indexed that site yet.  I used to see this a lot back in the early search engine era (when there were many search engines of varying sizes and even on Google when it was very young)  but younger web citizens who grew up with only all seeing Google might not have encountered this.

How I Use Mojeek

  • Mojeek is my default on my laptop, where 98% of my writing gets done.
  • I use it for most searches first.  If it does not give me what I need then I resort to another privacy engine as a backup.  Yes I often have to use the backup but Mojeek comes up with some good pages I would never get on G or B.
  • I use it go find Wikipedia articles.
  • Spell Checking: Okay this is embarrassing, I’m a terrible speller.  Mojeek has a great spell checker and I have one social network place I post at where my browser spell checker does not work, so I often use Mojeek to check the spelling of a word.  I have no idea if Mojeek’s spell checker is any better than any other but it works well and maybe the uncluttered results page makes it my preferred – crutch.
  • Navigational Searches.

Conclusion

I think I could get by with just Mojeek if I could only use one engine for some reason.  But that really is not the point.  Google and Bing are giving you their opinion of what the best pages are for a given search.  It’s just an opinion.  We should all be using several search engines for our research to get a broader view – more opinions.  Out of all the trillions of pages on the web it is ludicrous to think that the first 10, 20 or 30 results on Google are everything that needs to be said on a topic.  Every single search engine has a bias built into the algorithm in some way. Every. Single. One.  Which is exactly why you should use several search engines with different indexes.

Mojeek is growing into it’s role as a general web search engine.  Attempting to index the ever growing Web is a noble endeavor.  Their index has gotten much larger, fresher and their algo gives good results. As they grow, they will continue to get even better.

Back when I started on the Web in the 1990’s there were many major search engines, each showing just a part of the Web.  I commonly used a suite of my favorite search engines and directories just to find a few websites or pages.  Today my suite of privacy search engines consists of Mojeek, DuckDuckGo and once in a great while Startpage.  I think you should add Mojeek to your searching suite.

Liked this post? Follow this blog to get more. Follow

Mojeek.com, my favorite upstart privacy search engine, turns 15 years old today. Congratulations, Mojeek!

There is a discussion on Hacker News, you might want to check out.  And an interesting reflection and history on how and why Mojeek got started and why they guard your privacy here.

Factoid: Mojeek is one of four search engines world wide with a large English language index of it’s own.

Liked this post? Follow this blog to get more. Follow

Bookmarked

Bookmark: How to Surf the Web (nowadays)

This is a great retro 1990’s page chock full of resources and tips on how to surf the web in 2021.  There are links to new tools and starting points for web surfing.  This looks like a lot of fun and I intend to use this as my starting point.

Back in the 1990’s we surfed the web, because search engines pretty much sucked.  Yes it took time to surf the web but it was endlessly entertaining back in those days before everything became a commercial for some product.  A lot of the tools we used to use to surf are long gone, but this guide lists replacements.  Good stuff.

In addition to that page, the Webmaster at Sadgrl.online has built a remarkable website and I encourage you to explore the whole site.  It’s not just nostalgia, she has created webrings, search engines, discussion groups and listed extensive guides and resources.  Well worth browsing.

Liked this post? Follow this blog to get more. Follow

If you are not using Google, chances are the search engine you are using is probably powered by Microsoft’s Bing search engine.  This is because setting up a search engine that actively spiders the web to create it’s own index is expensive and a lot of work.  But Microsoft is willing to license out Bing search for a fee to just about anybody large and small.  This means that a lot of search engines from Yahoo to AOL use Bing as their primary web search source.

Don’t get me wrong, Bing is a good search engine but when you get to 15 or more Bing clones you start getting cross eyed trying to figure out what makes one different from both the others and Bing.

So this is my list of just a couple of the best search engines that use Bing in their primary index and what makes them special.

  1. DuckDuckGo – Bing and Yandex. Privacy, DDG does not track you nor even keep a record of your visit.  DDG is a meta-search engine that mainly uses Bing but also brings in answers from hundreds of other sources depending on the query.  For instance they use Apple maps for local searches and Wikipedia where relevant.  For most people, DDG is a good default search engine that will work great as a daily driver without spying on you.  DDG is one of my two default engines.
  2. MetaGer – Mainly Bing in English.  Privacy.  MetaGer is a metasearch engine owned by a German non-profit and they don’t track you or spy on you.  What is cool about MetaGer is they take privacy one step further, under each listing in the search results you will see a link to “OPEN ANONYMOUSLY” if you click on that MetaGer will try to open the site using an anonymous proxy!  This really adds a lot of value to the search results.  Journalists, those looking for sensitive information, people in authoritarian countries should make a note of this.  They also integrate Wikipedia into the search results in a minimalist way.
  3. SwissCows – Bing in English.  Privacy, based in Switzerland and protected by strict Swiss privacy laws, SC does not track you.  SwissCows bills itself as family friendly and therefore Safe Search cannot be set lower than the Moderate setting.  The combination of both privacy and family friendly search results makes SwissCows the perfect choice for a family or a kid’s computer or tablet or anything a child might use.
  4. Ecosia – Bing. Charity donation, in this instance planting trees all over the world every time you search.  Because of climate change fears, Ecosia has become a popular search engine.  It gives you the whole feature rich Bing search experience but links it to a good cause.  Privacy on Ecosia is sort of mid-range: better than if you were using Google, but Bing is still getting some of your information.
  5. Givero – Bing.  Charity donations.  Like Ecosia (above) searches done on Givero spur donations to charitable organizations, only you get to choose amongst seven different charities of an ecological, tech, child rescue or animal rescue missions.  Again, privacy is mid-range like Ecosia (above) but the charitable causes are good.

The list is for everyday users.  The best thing to do is figure out what is important to you (privacy or good causes) and give one of the likely search engines a try.  Not being tracked or spied upon is a good feeling.  Likewise, having your search activity contribute to good causes is a good feeling too.

Liked this post? Follow this blog to get more. Follow

Internet gatekeeper Google has threatened to cut the country/continent of Australia off from access to Google search engine.  This is because Australia is close to passing a law that would require Google and also Facebook, to pay newspapers for linking to them and using their content.

For background see: Google Threatens to Ban Aussies to Avoid Paying for News.

I’m not going to get into whether the proposed Australian law is wise or the right solution to the problem of Aussie newspapers dying, I want to focus on Google and it’s, foolish move to threaten a sovereign country.  I also want to underline how dangerous it is to only have two major English language search engines.

Threatening Australia: Google has more to lose.

  • Google made the threat in such a way, in front of the Australian Senate, that makes it very hard for either side to compromise.  Maybe Google wanted a direct confrontation, but frankly, Australia has a history of not shying away from a good fight when it feels threatened and they are tough as nails.
  • Google made the threat in such a way that it is almost patriotic for the Aussie public to boycott Google.
  • Google is afraid of the precedent it would create if forced to pay for news.  Google should be more afraid of the reaction by other countries if Google pulls the plug: all of a sudden those anti-trust investigations against Google will take on new urgency and Google’s treatment of the Aussies will be held up as the textbook example of Tech Giant monopoly power abuse.
  • What happens if Australia finds it can live without Google search? Sure, it might be inconvenient for a few weeks, but Aussies will eventually find Bing, Duckduckgo, Startpage, Yandex and Yahoo.  If this happens, word will spread fast that there is life without Google.
  • Microsoft Bing search engine, will gain an opportunity for market share.  And there is no reason Bing can’t improve both it’s Aussie web search and it’s maps to help keep those new users.
  • Apple uses Google as the default search on it’s iPhone product.  Apple will not thank Google for making Apple look bad in Australia.

Google Search Monopoly

The elephant in the room here is Google’s search monopoly at 95% in Australia. A pattern that holds in much of the rest of the world too.  Obviously Google thinks that 95% gives them leverage in this battle, but it also sort of proves the point that Google has a monopoly.

It also highlights how fragile the web has become having only two major English crawling search indexes: Google and Bing.  The world really needs to step up and start some new, serious crawling search engines.  (Yes there is Yandex, which is perhaps medium sized in English.  I’ve always thought their results would be better as part of a meta-search than stand alone.  There is also the up and coming Mojeek.com search which is still growing and deserves our support.)

Liked this post? Follow this blog to get more. Follow

Liked

Like:  Detailed tests of search engines: Google, Startpage, Bing, DuckDuckGo, metaGer, Ecosia, Swisscows, Searx, Qwant, Yandex, and Mojeek | LibreTechTips

Since my last article, many other alternatives have cropped up, bringing some very interesting features and concepts, but it still remains to be seen if they offer acceptable results in the fundamentally important area of relevant search results. This comparison sets out to analyze and compare the current batch of alternatives in 2020.

 

This is a nice detailed review of the search engines named above.  Each undergoes the same tests and the results are analysed.

What this excellent article shows, is we need more search engines that have their own crawlers and indexes.  Most of the search engines tested rely on Bing for their core organic results which usually means: when Bing fails they all fail.  A near global duopoly of just Google and Bing is just ridiculous considering the size of the Internet audience in English alone.

 

I pointed this out in: How Many Bing Using Search Engines Can We Handle?  We have enough Bing retreads, it’s time for some more crawlers and indexes in English.

 

Liked this post? Follow this blog to get more. Follow

There is a whole raft of new search engine startups plus many established players that all share one thing: their primary search source is Bing.   Which leads me to the question of: how many Bing powered search engines can the market absorb?

Now don’t get me wrong, I’m not criticizing search engine projects for using Bing nor am I criticizing Bing for selling access to it’s search feed.  Many of these search engines have been able to differentiate themselves from the others and from Bing in innovative ways and what appeals to one person might repel another so having a large choice of Bing powered engines is not bad.  But in the end they are all retreads of Bing all dressed in different clothes.

Bing Powered Search Engines

 

MetaSearch Engines that Use Bing

The above lists are partially sourced from: https://www.searchenginemap.com/  along with other sources.

So What Does this Mean?

The good thing, is that all of the Bing powered search engines together, help to erode Google’s near monopoly of search.  I like to think of it as Bing’s, guerrilla war on Google, fought with surrogates.  “Death by a thousand cuts.”  And, all these Bing powered engines provide users with a variety of UI’s and features that they may prefer over just being stuck using Bing itself.  Good so far as it goes.

The bad thing is, we the users, are still stuck with a duopoly.  At the end of the day the entire global English language searchverse is still stuck with only two major search engines: Google and Bing.  That’s it, you only get two opinions for finding things on the web – no third, forth or fifth opinions.  No matter how you dress it up it’s still a duoculture.  There is nobody else to turn to.

The second list of meta-search engines are stuck in the same boat, they are stuck in the duopoly too, being forced to rely on Google and Bing (plus Bing powered engines.)  Some are able to bring in smaller, independent engines that have their own crawler and indexes like Mojeek, and Yandex but those are not deep enough.

Do They Realize It’s Bing?

Based on the comments I read across the web and on social networks, I’m pretty sure the average Joe or Jane user is unaware that the search results on their alternative search engine of choice are derived from Bing.  In fact, I’ve seen many users swear that, say DuckDuckGo results are so much better, deeper more relevant than Bing.  The problem is, that sooner or later the public is going to catch on.  And we can’t just keep adding more Bing retreads to the mix.

The DuckDuckGo Question

Nobody has woven together a variety of different search sources around Bing with more skill than DuckDuckGo.  Yet the backbone of DDG’s results remains Bing, to the extent that they are totally dependent on Bing to function as a search engine.  As more Bing clones launch their task of making themselves different becomes harder.  Yet, I do not see any evidence of DDG trying to crawl and create their own index.  To me, that would be a long term strategy.  DDG has a crawler but they seem to use it for their Answers feature.  Other search engines like SwissCows and Qwant are building their own indexes in other languages so we know it can be done.  But only Mojeek seems to be trying in English.  Again we are up against the barrier of the duopoly which suppresses all other attempts at competition.

How long can we keep cloning Bing for variety in search?  Ultimately the market, and regulation from the EU and/or US will decide.  What the Bing clones do show is that demand is out there for something different – something not Google.  But how long will we be happy with just cloning Bing?

Related:

EU, Android and Search Engines

9 Best Private Search Engines for 2020

Liked this post? Follow this blog to get more. Follow