Data Models

The darc.model module contains all data models defined for the darc project, including RDS-based task queue and data submission.

• Task Queues
  • Hostname Queue
  • Crawler Queue
  • Loader Queue
• Submission Data Models
  • Hostname Records
  • URL Records
  • robots.txt Records
  • sitemap.xml Records
  • hosts.txt Records
  • Crawler Records
  • Loader Records

Base Model

The darc.model.abc module contains abstract base class of all data models for the darc project.

class darc.model.abc.BaseMeta

Bases: object

Basic metadata for data models.

table_function()

Generate table name dynamically (c.f. table_function()).

Return type

str

Parameters

model_class (Model) –

database = <peewee.SqliteDatabase object>

Reference database storage (c.f. DB).

class darc.model.abc.BaseMetaWeb

Bases: BaseMeta

Basic metadata for data models of data submission.

table_function()

Generate table name dynamically (c.f. table_function()).

Return type

str

Parameters

model_class (Model) –

database = <peewee.SqliteDatabase object>

Reference database storage (c.f. DB).

class darc.model.abc.BaseModel(*args, **kwargs)

Bases: Model

Base model with standard patterns.

Notes

The model will implicitly have a AutoField attribute named as id.

DoesNotExist

alias of BaseModelDoesNotExist

to_dict(keep_id=False)

Convert record to dict.

Parameters

keep_id (bool) – If keep the ID auto field.

Return type

None

Returns

The data converted through playhouse.shortcuts.model_to_dict().

Meta

Basic metadata for data models.

id = <AutoField: BaseModel.id>

class darc.model.abc.BaseModelWeb(*args, **kwargs)

Bases: BaseModel

Base model with standard patterns for data submission.

Notes

The model will implicitly have a AutoField attribute named as id.

DoesNotExist

alias of BaseModelWebDoesNotExist

Meta

Basic metadata for data models.

id = <AutoField: BaseModelWeb.id>

Miscellaneous Utilities

The darc.model.utils module contains several miscellaneous utility functions and data fields.

class darc.model.utils.IPField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)

Bases: IPField

IP data field.

db_value(val)

Dump the value for database storage.

Parameters

• value – Source IP address instance.

• val (Optional[Union[str, IPv4Address, IPv6Address]]) –

Return type

Optional[int]

Returns

Integral representation of the IP address.

python_value(val)

Load the value from database storage.

Parameters

• value – Integral representation of the IP address.

• val (Optional[int]) –

Return type

Union[IPv4Address, IPv6Address, None]

Returns

Original IP address instance.

class darc.model.utils.IntEnumField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)

Bases: IntegerField

enum.IntEnum data field.

Parameters

choices (IntEnum) –

python_value(value)

Load the value from database storage.

Parameters

value (Optional[int]) – Integral representation of the enumeration.

Return type

Optional[IntEnum]

Returns

Original enumeration object.

choices: IntEnum

The original enum.IntEnum class.

class darc.model.utils.JSONField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)

Bases: JSONField

JSON data field.

db_value(value)

Dump the value for database storage.

Parameters

value (Any) – Source JSON value.

Return type

Optional[str]

Returns

JSON serialised string data.

python_value(value)

Load the value from database storage.

Parameters

value (Optional[str]) – Serialised JSON string.

Return type

Any

Returns

Original JSON data.

class darc.model.utils.PickleField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)

Bases: BlobField

Pickled data field.

db_value(value)

Dump the value for database storage.

Parameters

value (Any) – Source value.

Return type

Optional[bytes]

Returns

Picked bytestring data.

python_value(value)

Load the value from database storage.

Parameters

value (Optional[bytes]) – SPicked bytestring data.

Return type

Any

Returns

Original data.

class darc.model.utils.Proxy(value)

Bases: IntEnum

Proxy types supported by darc.

FREENET = 5

Freenet proxy.

I2P = 3

I2P proxy.

NULL = 1

No proxy.

TOR = 2

Tor proxy.

TOR2WEB = 6

Proxied Tor (tor2web, no proxy).

ZERONET = 4

ZeroNet proxy.

darc.model.utils.table_function(model_class)

Generate table name dynamically.

The function strips Model from the class name and calls peewee.make_snake_case() to generate a proper table name.

Parameters

model_class (Model) – Data model class.

Return type

str

Returns

Generated table name.