What it is
anex.sh runs AI models for its customers. A customer can put the URL of an image, an audio clip, a video or an input file in a request, and the media fetcher downloads that file so the model can read it.
If you see it in your logs, one of our customers sent us a request that names a file on your site.
What it fetches
Only URLs a customer placed in a request. Every download is started by
a customer's request, so the fetcher does not crawl: it follows no
links, discovers no pages and builds no index. For the same reason it
does not read robots.txt.
It sends GET and HEAD requests. Before a
download it may ask for the file's size or its first bytes (a
Range request). A URL that many requests name can be
fetched many times.
User-Agent
Every request it sends carries this header:
User-Agent: anex-media-fetcher/1.0 (+https://anex.sh/docs/media-fetcher)
The version number can change. The name
anex-media-fetcher stays.
IP addresses
Requests come from changing IP addresses. There is no fixed range today, so identify the fetcher by its User-Agent.
How to block it
Refuse requests whose User-Agent contains
anex-media-fetcher, for example with a 403. The fetcher
does not retry under another name. The customer's request then fails
with an error that says the file could not be fetched.
Webhook deliveries
A customer can also register an address for us to notify when a job
finishes. Those POST requests download nothing and carry
their own name, anex-webhook/1.0. They are described in
the API reference.
Contact
Questions, or requests you did not expect: email support@anex.sh with the URL and the time of the request.