Sites Customisation

As websites may have authentication requirements, etc., over its content, the darc.sites module provides sites customisation hooks to both requests and selenium crawling processes.

To customise behaviours over requests, you sites customisation module should have a crawler() function, e.g. crawler().

The function takes the requests.Session object with proxy settings and a Link object representing the link to be crawled, then returns a requests.Response object containing the final data of the crawling process.

To customise behaviours over selenium, you sites customisation module should have a loader() function, e.g. loader().

The function takes the Chrome object with proxy settings and a Link object representing the link to be loaded, then returns the Chrome object containing the final data of the loading process.