Module Constants¶

Auxiliary Function¶

General Configurations¶

darc.const.REBOOT: bool¶

If exit the program after first round, i.e. crawled all links from the requests link database and loaded all links from the selenium link database.

Default: False
Environ: DARC_REBOOT

darc.const.DEBUG: bool¶

If run the program in debugging mode.

Default: False
Environ: DARC_DEBUG

darc.const.VERBOSE: bool¶

If run the program in verbose mode. If DEBUG is True, then the verbose mode will be always enabled.

Default: False
Environ: DARC_VERBOSE

darc.const.FORCE: bool¶

If ignore robots.txt rules when crawling (c.f. crawler()).

Default: False
Environ: DARC_FORCE

darc.const.CHECK: bool¶

If check proxy and hostname before crawling (when calling extract_links(), read_sitemap() and read_hosts()).

If CHECK_NG is True, then this environment variable will be always set as True.

Default: False
Environ: DARC_CHECK

darc.const.CHECK_NG: bool¶

If check content type through HEAD requests before crawling (when calling extract_links(), read_sitemap() and read_hosts()).

Default: False
Environ: DARC_CHECK_CONTENT_TYPE

darc.const.ROOT: str¶: The root folder of the project.

darc.const.CWD = '.'¶: The current working direcory.

darc.const.DARC_CPU: int¶

Number of concurrent processes. If not provided, then the number of system CPUs will be used.

Default: None
Environ: DARC_CPU

darc.const.FLAG_MP: bool¶

If enable multiprocessing support.

Default: True
Environ: DARC_MULTIPROCESSING

darc.const.FLAG_TH: bool¶

If enable multithreading support.

Default: False
Environ: DARC_MULTITHREADING

Note

FLAG_MP and FLAG_TH can NOT be toggled at the same time.

darc.const.DARC_USER: str¶

Non-root user for proxies.

Default: current login user (c.f. getpass.getuser())
Environ: DARC_USER

Data Storage¶

darc.const.PATH_DB: str¶

Path to data storage.

Default: data
Environ: PATH_DATA

See also

See darc.save for more information about source saving.

darc.const.PATH_MISC = '{PATH_DB}/misc/'¶

Path to miscellaneous data storage, i.e. misc folder under the root of data storage.

See also

darc.const.PATH_DB

darc.const.PATH_LN = '{PATH_DB}/link.csv'¶

Path to the link CSV file, link.csv.

See also

darc.const.PATH_DB
darc.save.save_link

darc.const.PATH_QR = '{PATH_DB}/_queue_requests.txt'¶

Path to the requests database, _queue_requests.txt.

See also

darc.const.PATH_DB
darc.db.load_requests()
darc.db.save_requests()

darc.const.PATH_QS = '{PATH_DB}/_queue_selenium.txt'¶

Path to the selenium database, _queue_selenium.txt.

See also

darc.const.PATH_DB
darc.db.load_selenium()
darc.db.save_selenium()

darc.const.PATH_ID = '{PATH_DB}/darc.pid'¶

Path to the process ID file, darc.pid.

See also

darc.const.PATH_DB
darc.const.getpid()

Web Crawlers¶

darc.const.TIME_CACHE: float¶

Time delta for caches in seconds.

The darc project supports caching for fetched files. TIME_CACHE will specify for how log the fetched files will be cached and NOT fetched again.

Note

If TIME_CACHE is None then caching will be marked as forever.

Default: 60
Environ: TIME_CACHE

darc.const.SE_WAIT: float¶

Time to wait for selenium to finish loading pages.

Note

Internally, selenium will wait for the browser to finish loading the pages before return (i.e. the web API event DOMContentLoaded). However, some extra scripts may take more time running after the event.

Default: 60
Environ: SE_WAIT

darc.const.SE_EMPTY = '<html><head></head><body></body></html>'¶

The empty page from selenium.

See also

darc.crawl.loader()

White / Black Lists¶

darc.const.LINK_WHITE_LIST: List[str]¶

White list of hostnames should be crawled.

Default: []
Environ: LINK_WHITE_LIST

Note