Data Models¶
The darc.model
module contains all data models defined for the
darc
project, including RDS-based task queue and data submission.
- Task Queues
- Submission Data Models
- Hostname Records
HostnameModel
- URL Records
URLModel
URLThroughModel
robots.txt
RecordsRobotsModel
sitemap.xml
RecordsSitemapModel
hosts.txt
RecordsHostsModel
- Crawler Records
RequestsHistoryModel
RequestsHistoryModel.DoesNotExist
RequestsHistoryModel.cookies
RequestsHistoryModel.document
RequestsHistoryModel.id
RequestsHistoryModel.index
RequestsHistoryModel.method
RequestsHistoryModel.model
RequestsHistoryModel.model_id
RequestsHistoryModel.reason
RequestsHistoryModel.request
RequestsHistoryModel.response
RequestsHistoryModel.status_code
RequestsHistoryModel.timestamp
RequestsHistoryModel.url
RequestsModel
RequestsModel.DoesNotExist
RequestsModel.cookies
RequestsModel.document
RequestsModel.history
RequestsModel.id
RequestsModel.is_html
RequestsModel.method
RequestsModel.mime_type
RequestsModel.reason
RequestsModel.request
RequestsModel.response
RequestsModel.session
RequestsModel.status_code
RequestsModel.timestamp
RequestsModel.url
RequestsModel.url_id
- Loader Records
SeleniumModel
Base Model¶
The darc.model.abc
module contains abstract base class
of all data models for the darc
project.
- class darc.model.abc.BaseMeta[source]¶
Bases:
object
Basic metadata for data models.
- table_function()¶
Generate table name dynamically (c.f.
table_function()
).- Return type:
- Parameters:
model_class (Model) –
- class darc.model.abc.BaseMetaWeb[source]¶
Bases:
BaseMeta
Basic metadata for data models of data submission.
- table_function()¶
Generate table name dynamically (c.f.
table_function()
).- Return type:
- Parameters:
model_class (Model) –
- class darc.model.abc.BaseModel(*args, **kwargs)[source]¶
Bases:
Model
Base model with standard patterns.
Notes
The model will implicitly have a
AutoField
attribute named asid
.- DoesNotExist¶
alias of
BaseModelDoesNotExist
- Meta¶
Basic metadata for data models.
- id = <AutoField: BaseModel.id>¶
- class darc.model.abc.BaseModelWeb(*args, **kwargs)[source]¶
Bases:
BaseModel
Base model with standard patterns for data submission.
Notes
The model will implicitly have a
AutoField
attribute named asid
.- DoesNotExist¶
alias of
BaseModelWebDoesNotExist
- Meta¶
Basic metadata for data models.
- id = <AutoField: BaseModelWeb.id>¶
Miscellaneous Utilities¶
The darc.model.utils
module contains several miscellaneous
utility functions and data fields.
- class darc.model.utils.IPField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)[source]¶
Bases:
IPField
IP data field.
- db_value(val)[source]¶
Dump the value for database storage.
- Parameters:
value – Source IP address instance.
val (str | IPv4Address | IPv6Address | None) –
- Return type:
- Returns:
Integral representation of the IP address.
- class darc.model.utils.IntEnumField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)[source]¶
Bases:
IntegerField
enum.IntEnum
data field.- Parameters:
choices (IntEnum) –
-
choices:
IntEnum
¶ The original
enum.IntEnum
class.
- class darc.model.utils.JSONField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)[source]¶
Bases:
JSONField
JSON data field.
- class darc.model.utils.PickleField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)[source]¶
Bases:
BlobField
Pickled data field.
- class darc.model.utils.Proxy(value)[source]¶
Bases:
IntEnum
Proxy types supported by
darc
.- FREENET = 5¶
Freenet proxy.
- I2P = 3¶
I2P proxy.
- NULL = 1¶
No proxy.
- TOR = 2¶
Tor proxy.
- ZERONET = 4¶
ZeroNet proxy.