It’s always neat on a new domain and website to see which crawlers (aka Spiders, Robots) find you and what they do.

Indieseek.xyz is lousy with crawler bots.

Google found us first.  Within 24 hours of my first testing posts (either a webmention to another site or when I added a link back on my Twitter profile, Googlebot was all over the site.  Googlebot is very competent and well behaved, but voracious.  This is to Google’s credit, that is what a search engine spider is supposed to be.  Google is always keen to find new sites.  Every day since then one, two sometimes 3 Googlebots have been in the directory, getting into everything.  They almost camp out there.

Bing or a bot pretending to be Bingbot found us second but a few days later.  I’m not sure is it is the real Bingbot because it mearly checks the index page and leaves a query string of “amazon”.  It came beck several times but I never detected it going deeper.  Has not been back in awhile.  Bing always seems to hold back on indexing.  You can check one week and they only have your index page.  You check again weeks later and they have 6 pages, a few weeks more and they have 6 more.

A few days after I started cross posting to Twitter the spambots showed up.  Some I can only detect because they try to leave spam comments.  So far the defenses are holding on those.  The others are better behaved: one from a popular SEO tools site has been, almost as voracious as Google in checking out the entire site, another which may be somebody’s experiment keeps coming back, and the third, unnamed, is from Asia and keeps coming back and just hanging on my help page.  Weird.  The later three are not of any value to me, but I’m okay on bandwidth so I let them carry on.

So far, no traffic from any search engine.  That does not surprise me as Indieseek has almost no inbound links.

The plan of action, is to keep writing posts in the Indieseek blog and keep adding listings in the directory and what will be will be.

 

Liked this post? Follow this blog to get more. Follow

Q.: Does your directory have a crawler/spider?

 

Why yes we do.  Although he’s getting kinda old and is a grumpy old git and he hobbles more than crawls.  When he’s not mopping the floors and washing the bottles he can be found adding URL’s to our directory.  Generally he can add about 20 URL’s a day, sometimes 25 if we juice him up with caffeine.  After about 20 site reviews he starts going all cross-eyed.  We tried naming him Bradbot but it never really took.  🙂

Seriously:

Not really.  Indieseek does have a meta tag fetcher.  When I put a URL in the field the meta tag fetcher tries to grab that page’s Title, Description and Keyword meta tags and fill in those fields.  It does the same thing when the public submits a URL and it saves some time.

Other than that, everything is manual.

Liked this post? Follow this blog to get more. Follow