Data Models

The darc.model module contains all data models defined for the darc project, including RDS-based task queue and data submission.

• Task Queues
  • Hostname Queue
  • HostnameQueueModel
    • HostnameQueueModel.DoesNotExist
    • HostnameQueueModel.hostname
    • HostnameQueueModel.id
    • HostnameQueueModel.timestamp
  • Crawler Queue
  • RequestsQueueModel
    • RequestsQueueModel.DoesNotExist
    • RequestsQueueModel.hash
    • RequestsQueueModel.id
    • RequestsQueueModel.link
    • RequestsQueueModel.text
    • RequestsQueueModel.timestamp
  • Loader Queue
  • SeleniumQueueModel
    • SeleniumQueueModel.DoesNotExist
    • SeleniumQueueModel.hash
    • SeleniumQueueModel.id
    • SeleniumQueueModel.link
    • SeleniumQueueModel.text
    • SeleniumQueueModel.timestamp
• Submission Data Models
  • Hostname Records
  • HostnameModel
    • HostnameModel.DoesNotExist
    • HostnameModel.alive
    • HostnameModel.discovery
    • HostnameModel.hostname
    • HostnameModel.hosts
    • HostnameModel.id
    • HostnameModel.last_seen
    • HostnameModel.proxy
    • HostnameModel.robots
    • HostnameModel.since
    • HostnameModel.sitemaps
    • HostnameModel.urls
  • URL Records
  • URLModel
    • URLModel.DoesNotExist
    • URLModel.get_by_url()
    • URLModel.alive
    • URLModel.children
    • URLModel.childrent
    • URLModel.discovery
    • URLModel.hash
    • URLModel.hostname
    • URLModel.hostname_id
    • URLModel.id
    • URLModel.last_seen
    • URLModel.parents
    • URLModel.proxy
    • URLModel.requests
    • URLModel.selenium
    • URLModel.since
    • URLModel.url
  • URLThroughModel
    • URLThroughModel.DoesNotExist
    • URLThroughModel.child
    • URLThroughModel.child_id
    • URLThroughModel.id
    • URLThroughModel.parent
    • URLThroughModel.parent_id
  • robots.txt Records
  • RobotsModel
    • RobotsModel.DoesNotExist
    • RobotsModel.document
    • RobotsModel.host
    • RobotsModel.host_id
    • RobotsModel.id
    • RobotsModel.timestamp
  • sitemap.xml Records
  • SitemapModel
    • SitemapModel.DoesNotExist
    • SitemapModel.document
    • SitemapModel.host
    • SitemapModel.host_id
    • SitemapModel.id
    • SitemapModel.timestamp
  • hosts.txt Records
  • HostsModel
    • HostsModel.DoesNotExist
    • HostsModel.document
    • HostsModel.host
    • HostsModel.host_id
    • HostsModel.id
    • HostsModel.timestamp
  • Crawler Records
  • RequestsHistoryModel
    • RequestsHistoryModel.DoesNotExist
    • RequestsHistoryModel.cookies
    • RequestsHistoryModel.document
    • RequestsHistoryModel.id
    • RequestsHistoryModel.index
    • RequestsHistoryModel.method
    • RequestsHistoryModel.model
    • RequestsHistoryModel.model_id
    • RequestsHistoryModel.reason
    • RequestsHistoryModel.request
    • RequestsHistoryModel.response
    • RequestsHistoryModel.status_code
    • RequestsHistoryModel.timestamp
    • RequestsHistoryModel.url
  • RequestsModel
    • RequestsModel.DoesNotExist
    • RequestsModel.cookies
    • RequestsModel.document
    • RequestsModel.history
    • RequestsModel.id
    • RequestsModel.is_html
    • RequestsModel.method
    • RequestsModel.mime_type
    • RequestsModel.reason
    • RequestsModel.request
    • RequestsModel.response
    • RequestsModel.session
    • RequestsModel.status_code
    • RequestsModel.timestamp
    • RequestsModel.url
    • RequestsModel.url_id
  • Loader Records
  • SeleniumModel
    • SeleniumModel.DoesNotExist
    • SeleniumModel.document
    • SeleniumModel.id
    • SeleniumModel.screenshot
    • SeleniumModel.timestamp
    • SeleniumModel.url
    • SeleniumModel.url_id

Base Model

The darc.model.abc module contains abstract base class of all data models for the darc project.

class darc.model.abc.BaseMeta

Bases: object

Basic metadata for data models.

table_function()

Generate table name dynamically (c.f. table_function()).

Return type:

str

Parameters:

model_class (Model) –

database = <peewee.SqliteDatabase object>

Reference database storage (c.f. DB).

class darc.model.abc.BaseMetaWeb

Bases: BaseMeta

Basic metadata for data models of data submission.

table_function()

Generate table name dynamically (c.f. table_function()).

Return type:

str

Parameters:

model_class (Model) –

database = <peewee.SqliteDatabase object>

Reference database storage (c.f. DB).

class darc.model.abc.BaseModel(*args, **kwargs)

Bases: Model

Base model with standard patterns.

Notes

The model will implicitly have a AutoField attribute named as id.

DoesNotExist

alias of BaseModelDoesNotExist

to_dict(keep_id=False)

Convert record to dict.

Parameters:

keep_id (bool) – If keep the ID auto field.

Return type:

None

Returns:

The data converted through playhouse.shortcuts.model_to_dict().

Meta

Basic metadata for data models.

id = <AutoField: BaseModel.id>

class darc.model.abc.BaseModelWeb(*args, **kwargs)

Bases: BaseModel

Base model with standard patterns for data submission.

Notes

The model will implicitly have a AutoField attribute named as id.

DoesNotExist

alias of BaseModelWebDoesNotExist

Meta

Basic metadata for data models.

id = <AutoField: BaseModelWeb.id>

Miscellaneous Utilities

The darc.model.utils module contains several miscellaneous utility functions and data fields.

class darc.model.utils.IPField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)

Bases: IPField

IP data field.

db_value(val)

Dump the value for database storage.

Parameters:

• value – Source IP address instance.

• val (str | IPv4Address | IPv6Address | None) –

Return type:

Optional[int]

Returns:

Integral representation of the IP address.

python_value(val)

Load the value from database storage.

Parameters:

• value – Integral representation of the IP address.

• val (int | None) –

Return type:

Union[IPv4Address, IPv6Address, None]

Returns:

Original IP address instance.

class darc.model.utils.IntEnumField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)

Bases: IntegerField

enum.IntEnum data field.

Parameters:

choices (IntEnum) –

python_value(value)

Load the value from database storage.

Parameters:

value (Optional[int]) – Integral representation of the enumeration.

Return type:

Optional[IntEnum]

Returns:

Original enumeration object.

choices: IntEnum

The original enum.IntEnum class.

class darc.model.utils.JSONField(json_dumps=None, json_loads=None, **kwargs)

Bases: JSONField

JSON data field.

db_value(value)

Dump the value for database storage.

Parameters:

value (Any) – Source JSON value.

Return type:

Optional[str]

Returns:

JSON serialised string data.

python_value(value)

Load the value from database storage.

Parameters:

value (Optional[str]) – Serialised JSON string.

Return type:

Any

Returns:

Original JSON data.

class darc.model.utils.PickleField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)

Bases: BlobField

Pickled data field.

db_value(value)

Dump the value for database storage.

Parameters:

value (Any) – Source value.

Return type:

Optional[bytes]

Returns:

Picked bytestring data.

python_value(value)

Load the value from database storage.

Parameters:

value (Optional[bytes]) – SPicked bytestring data.

Return type:

Any

Returns:

Original data.

class darc.model.utils.Proxy(value)

Bases: IntEnum

Proxy types supported by darc.

FREENET = 5

Freenet proxy.

I2P = 3

I2P proxy.

NULL = 1

No proxy.

TOR = 2

Tor proxy.

TOR2WEB = 6

Proxied Tor (tor2web, no proxy).

ZERONET = 4

ZeroNet proxy.

darc.model.utils.table_function(model_class)

Generate table name dynamically.

The function strips Model from the class name and calls peewee.make_snake_case() to generate a proper table name.

Parameters:

model_class (Model) – Data model class.

Return type:

str

Returns:

Generated table name.