Web crawler information
AntiCloneBot
AntiCloneBot finds copies of online shops (fake shops that copy a real store's products, texts and images) for the owners of the copied shops.
- Honors robots.txt
- Signed requests (Web Bot Auth)
- No logins, forms or orders
- Data stored in the EU
Opt out
Add these two lines to your robots.txt. AntiCloneBot will no longer visit your website automatically.
User-agent: AntiCloneBot Disallow: /
The block applies to the bot. Public pages stay visible to people: when a merchant reports a page as a possible copy of their store, we look at it in person in a normal browser, like any other visitor.
What it reads
- It usually looks at a website only once and reads few public pages: robots.txt, the homepage and, for online shops, the public product list or sitemap, with a few product images for a visual comparison.
- Pages that probably copy a shop are checked more closely and looked at again later to see whether they are still online.
What it never does
- No login, no forms, no orders
- No payment data
- No vulnerability tests
It honors robots.txt including Crawl-delay and slows down when a website answers 429 or 430.
How to recognise it
- User-Agent
AntiCloneBot- Signature
- Requests are signed with Web Bot Auth.
- Public keys
- https://anticlone.de/.well-known/http-message-signatures-directory
Data
- Kept
- Domain, shop system, page title, public shop IDs, image file names and fingerprints of texts and product images
- Product titles and shop contacts
- Only as keyed hashes
- Visitors
- Nothing about visitors
- Where and how long
- In the European Union, with no fixed deletion date; we delete the entries of your website on request