Default Hooks¶
The darc.sites.default
module is the fallback for sites
customisation.
-
darc.sites.default.
crawler
(session, link)¶ Default crawler hook.
- Parameters
session (
requests.Session
) – Session object with proxy settings.link (darc.link.Link) – Link object to be crawled.
- Returns
The final response object with crawled data.
- Return type
See also
-
darc.sites.default.
loader
(driver, link)¶ Default loader hook.
When loading, if
SE_WAIT
is a valid time lapse, the function will sleep for such time to wait for the page to finish loading contents.- Parameters
driver (
selenium.webdriver.Chrome
) – Web driver object with proxy settings.link (darc.link.Link) – Link object to be loaded.
- Returns
The web driver object with loaded data.
- Return type
Note
Internally,
selenium
will wait for the browser to finish loading the pages before return (i.e. the web API eventDOMContentLoaded
). However, some extra scripts may take more time running after the event.See also