Checks for disallowed entries in '/robots.txt' on a web server. The higher the verbosity or debug level, the more disallowed entries are shown. SYNTAX: http.pipeline: If set, it represents the number of HTTP requests that'll be pipelined (ie, sent in a single request). This can be set low to make debugging easier, or it can be set high to test how a server reacts (its chosen max is ignored). TODO Implement cache system for http pipelines http.useragent: The value of the User-Agent header field sent with requests. By default it is ''Mozilla/5.0 (compatible Nmap Scripting Engine http://nmap.org/book/nse.html)''. A value of the empty string disables sending the User-Agent header field. http-max-cache-size: The maximum memory size (in bytes) of the cache.