NudlisBot

NudlisBot is the name of the web crawler of the Nudevista.com search engine.

We request both desktop and mobile versions, images and other files from a single user agent.

We fully support the Robots Exclusion Protocol with minor additions.

Best Usage Practices:

File: https://your.tube/robots.txt

User-agent: NudlisBot
Disallow: /temp/

Crawl-delay: 1
Prefer-time: 12:00GMT

Videos-all: https://your.tube/videos.json
Videos-new: https://your.tube/videos_new.json
Videos-del: https://your.tube/videos_del.json

Description of instructions:

  1. User-agent

    2. If we don't find instructions for user-agent NudlisBot, we look for instructions for Googlebot. If we don't find them either, we assign default values to your tube.

  2. Disallow

    3. Paths or specific pages that the crawler is not allowed to access. These can be dynamic services (search, personalisation), temporary files or any other data that cannot be indexed.

  3. Crawl-delay

    4. Instructing the crawler to pause between requests. Values from 1 to 5 are accepted. Otherwise considered = 1.

  4. Prefer-time

    5. Our crawler traverses the sources twice a day. You can suggest the time of one of these rounds (the main one). The time is given in 24-hour GMT format.

  5. Videos-all

    6. JSON (or text/plain or CVS) file with all videos on the site. Format:

    1. id: int or string, required
    2. URL: string, required
    3. title: string, required
    4. description: string, optional
    5. duration: (in seconds) integer, required
    6. categories: string, optional
    7. tags: string, optional
    8. models: string, optional
    9. thumb_main: url, 600px width min, required
    10. thumb_secondary: string, optional
    11. thumb_video: url, 320px width min, optional

  6. Videos-new

    7. JSON (or text/plain or CVS) file including videos published in the last 7 days.

  7. Videos-del

    8. JSON (or text/plain or CVS) file including videos deleted (all time or last 7 days). Format:

    1. id: int or string, required
    2. URL: string, optional

From which IP and with which user-agent our web crawler goes:

Our official IPs:

Our official user-agent:

NudlisBot/3.17; +http://nudevista.com/bot.html

The full string varies and contains the most currently compatible installs possible. Currently it is:

Mozilla/5.0 (Linux; Android 14; Pixel 7 Pro) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.6668.89 Mobile Safari/537.36 (compatible; NudlisBot/3.17; +http://nudevista.com/bot.html)

We reserve the right to crawl from any number of other addresses and without specifying user-agent in order to verify or clarify the information received.

It is easy to check if an IP address belongs to our bot:

host 74.117.179.122
122.179.117.74.in-addr.arpa domain name pointer crawler122.nudevista.com.
And vice versa: 
host crawler122.nudevista.com
crawler122.nudevista.com has address 74.117.179.122

See you online!
Nudlis