JobikoCatalogBot is the web crawler of Jobiko, a job search assistant. Every request it makes carries this user agent:
JobikoCatalogBot/1.0 (+https://jobiko.org/bot)
1. What it reads #
JobikoCatalogBot builds the job catalogue that Jobiko users search. It reads:
- the public job feeds that applicant tracking systems publish for their customers’ careers sites;
- open job data from public employment services;
- the page of a posting already in the catalogue, to check that the job is still open;
- the page of a posting a Jobiko user chooses to work on, and an employer’s site icon, shown next to its postings.
It reads public pages only. It never signs in to an account, and it never fills in or submits a form. It does not collect postings from job boards or aggregators.
2. robots.txt #
JobikoCatalogBot follows the Robots Exclusion Protocol (RFC 9309). Before it collects postings from a site, it reads that site’s robots.txt and skips every path disallowed for JobikoCatalogBot or for all crawlers (*).
To keep it out of your whole site, add:
User-agent: JobikoCatalogBot
Disallow: /
3. Pace #
It sends a few requests at a time to any one site. When a site answers 429 or 503, it backs off and waits at least as long as the Retry- header asks.
4. Keeping your site out of Jobiko #
Use the robots.txt rule above, or email support@jobiko.org with the subject “JobikoCatalogBot opt-out: yourdomain.com”. Name the domains or careers pages concerned. We exclude them from the catalogue and confirm by email.
5. Contact #
Questions about JobikoCatalogBot: support@jobiko.org.