Task Queues¶
The darc.model.tasks
module defines the data models
required for the task queue of darc
.
See also
Please refer to darc.db
module for more information
about the task queues.
Hostname Queue¶
Important
The hostname queue is a set named queue_hostname
in
a Redis based task queue.
The darc.model.tasks.hostname
model contains the data model
defined for the hostname queue.
- class darc.model.tasks.hostname.HostnameQueueModel(*args, **kwargs)[source]¶
Bases:
BaseModel
Hostname task queue.
- DoesNotExist¶
alias of
HostnameQueueModelDoesNotExist
- id = <AutoField: HostnameQueueModel.id>¶
- timestamp: datetime = <DateTimeField: HostnameQueueModel.timestamp>¶
Timestamp of last update.
Crawler Queue¶
The darc.model.tasks.requests
model contains the data model
defined for the crawler
queue.
- class darc.model.tasks.requests.RequestsQueueModel(*args, **kwargs)[source]¶
Bases:
BaseModel
Task queue for
crawler()
.- DoesNotExist¶
alias of
RequestsQueueModelDoesNotExist
- id = <AutoField: RequestsQueueModel.id>¶
- timestamp: datetime = <DateTimeField: RequestsQueueModel.timestamp>¶
Timestamp of last update.
Loader Queue¶
The darc.model.tasks.selenium
model contains the data model
defined for the loader
queue.
- class darc.model.tasks.selenium.SeleniumQueueModel(*args, **kwargs)[source]¶
Bases:
BaseModel
Task queue for
loader()
.- DoesNotExist¶
alias of
SeleniumQueueModelDoesNotExist
- id = <AutoField: SeleniumQueueModel.id>¶
- timestamp: datetime = <DateTimeField: SeleniumQueueModel.timestamp>¶
Timestamp of last update.