Sites CustomisationΒΆ
As websites may have authentication requirements, etc., over
its content, the darc.sites
module provides sites
customisation hooks to both requests
and selenium
crawling processes.
To customise behaviours over requests
, you sites customisation
module should have a crawler()
function, e.g.
crawler()
.
The function takes the requests.Session
object with proxy settings and
a Link
object representing the link to be
crawled, then returns a requests.Response
object containing the final
data of the crawling process.
To customise behaviours over selenium
, you sites customisation
module should have a loader()
function, e.g.
loader()
.
The function takes the Chrome
object with proxy settings and
a Link
object representing the link to be
loaded, then returns the Chrome
object containing the final
data of the loading process.