Sites CustomisationΒΆ
As websites may have authentication requirements, etc., over
its content, the darc.sites
module provides sites
customisation hooks to both |requests|_ and |selenium|_
crawling processes.
To customise behaviours over |requests|_, you sites customisation
module should have a crawler()
function, e.g.
crawler()
.
The function takes the |Session|_ object with proxy settings and
a Link
object representing the link to be
crawled, then returns a |Response|_ object containing the final
data of the crawling process.
To customise behaviours over |selenium|_, you sites customisation
module should have a loader()
function, e.g.
loader()
.
The function takes the |Chrome|_ object with proxy settings and
a Link
object representing the link to be
loaded, then returns the |Chrome|_ object containing the final
data of the loading process.