Our bot

If you run a website and saw a visitor called ObscureInternetBot in your logs, this page is for you. It belongs to Obscure Internet, a museum of the lost web. It isn’t a search engine, a scraper or a spammer.

What it does

The museum recommends a small, hand-picked list of living websites through its Surprise Me button. The bot visits each one on the list about once a day to check it’s still there and still the same site, so we don’t send visitors somewhere that has closed, been sold or changed beyond recognition.

• It identifies itself as ObscureInternetBot/1.0 (+https://obscureinternet.com/bot/).
• It makes one request for the front page of a site on the list (plus any redirects, at most five), reads the first 64 KB, and gives up after 15 seconds.
• It doesn’t run scripts, log in, fill in forms or follow links around your site.
• It stores the page title, a fingerprint of the text and the names of outside services the page loads. It doesn’t keep a copy of your page.

Why your site

The bot only visits sites a person at the museum chose to recommend. It doesn’t crawl the web.

How to ask it to stop

Contact us with the address of your site and we’ll take it off the list, which stops the visits and the recommendation. You don’t need to give a reason.