Task Queues¶
The darc.model.tasks
module defines the data models
required for the task queue of darc
.
See also
Please refer to darc.db
module for more information
about the task queues.
Hostname Queue¶
Important
The hostname queue is a set named queue_hostname
in
a Redis based task queue.
The darc.model.tasks.hostname
model contains the data model
defined for the hostname queue.
-
class
darc.model.tasks.hostname.
HostnameQueueModel
(*args, **kwargs)[source]¶ Bases:
darc.model.abc.BaseModel
Hostname task queue.
-
DoesNotExist
¶ alias of
darc.model.tasks.hostname.HostnameQueueModelDoesNotExist
-
id
= <AutoField: HostnameQueueModel.id>¶
-
timestamp
: datetime.datetime = <DateTimeField: HostnameQueueModel.timestamp>¶ Timestamp of last update.
-
Crawler Queue¶
Important
The crawler
queue is a sorted set
named queue_requests
in a Redis based task queue.
The darc.model.tasks.requests
model contains the data model
defined for the crawler
queue.
-
class
darc.model.tasks.requests.
RequestsQueueModel
(*args, **kwargs)[source]¶ Bases:
darc.model.abc.BaseModel
Task queue for
crawler()
.-
DoesNotExist
¶ alias of
darc.model.tasks.requests.RequestsQueueModelDoesNotExist
-
id
= <AutoField: RequestsQueueModel.id>¶
-
link
: darc.link.Link = <PickleField: RequestsQueueModel.link>¶ Pickled target
Link
instance.
-
timestamp
: datetime.datetime = <DateTimeField: RequestsQueueModel.timestamp>¶ Timestamp of last update.
-
Loader Queue¶
Important
The loader
queue is a sorted set
named queue_selenium
in a Redis based task queue.
The darc.model.tasks.selenium
model contains the data model
defined for the loader
queue.
-
class
darc.model.tasks.selenium.
SeleniumQueueModel
(*args, **kwargs)[source]¶ Bases:
darc.model.abc.BaseModel
Task queue for
loader()
.-
DoesNotExist
¶ alias of
darc.model.tasks.selenium.SeleniumQueueModelDoesNotExist
-
id
= <AutoField: SeleniumQueueModel.id>¶
-
link
: darc.link.Link = <PickleField: SeleniumQueueModel.link>¶ Pickled target
Link
instance.
-
timestamp
: datetime.datetime = <DateTimeField: SeleniumQueueModel.timestamp>¶ Timestamp of last update.
-