Sites CustomisationΒΆ

As websites may have authentication requirements, etc., over its content, the darc.sites module provides sites customisation hooks to both |requests|_ and |selenium|_ crawling processes.

To customise behaviours over |requests|_, you sites customisation module should have a crawler() function, e.g. crawler().

The function takes the |Session|_ object with proxy settings and a Link object representing the link to be crawled, then returns a |Response|_ object containing the final data of the crawling process.

To customise behaviours over |selenium|_, you sites customisation module should have a loader() function, e.g. loader().

The function takes the |Chrome|_ object with proxy settings and a Link object representing the link to be loaded, then returns the |Chrome|_ object containing the final data of the loading process.