No Proxy

The darc.proxy.null module contains the auxiliary functions around managing and processing normal websites with no proxy.

darc.proxy.null.fetch_sitemap(link)

Fetch sitemap.

The function will first fetch the robots.txt, then fetch the sitemaps accordingly.

Parameters

link (darc.link.Link) – Link object to fetch for its sitemaps.

darc.proxy.null.save_invalid(link)

Save link with invalid scheme.

The function will save link with invalid scheme to the file as defined in PATH.

Parameters

link (darc.link.Link) – Link object representing the link with invalid scheme.

darc.proxy.null.PATH = '{PATH_MISC}/invalid.txt'

Path to the data storage of links with invalid scheme.

darc.proxy.null.LOCK: multiprocessing.Lock

I/O lock for saving links with invalid scheme PATH.