Data Models

The darc.model module contains all data models defined for the darc project, including RDS-based task queue and data submission.

• Task Queues
  • Hostname Queue
  • Crawler Queue
  • Loader Queue
• Submission Data Models
  • Hostname Records
  • URL Records
  • robots.txt Records
  • sitemap.xml Records
  • hosts.txt Records
  • Crawler Records
  • Loader Records

Base Model

The darc.model.abc module contains abstract base class of all data models for the darc project.

class darc.model.abc.BaseMeta

Bases: object

Basic metadata for data models.

table_function()

Generate table name dynamically (c.f. table_function()).

Parameters

model_class (peewee.Model) –

Return type

str

database = <peewee.SqliteDatabase object>

Reference database storage (c.f. DB).

class darc.model.abc.BaseMetaWeb

Bases: darc.model.abc.BaseMeta

Basic metadata for data models of data submission.

database = <peewee.SqliteDatabase object>

Reference database storage (c.f. DB).

class darc.model.abc.BaseModel(*args, **kwargs)

Bases: peewee.Model

Base model with standard patterns.

Notes

The model will implicitly have a AutoField attribute named as id.

DoesNotExist

alias of BaseModelDoesNotExist

to_dict(keep_id=False)

Convert record to dict.

Parameters

keep_id (bool) – If keep the ID auto field.

Returns

The data converted through playhouse.shortcuts.model_to_dict().

Return type

None

Meta

Basic metadata for data models.

id = <AutoField: BaseModel.id>

class darc.model.abc.BaseModelWeb(*args, **kwargs)

Bases: darc.model.abc.BaseModel

Base model with standard patterns for data submission.

Notes

The model will implicitly have a AutoField attribute named as id.

DoesNotExist

alias of BaseModelWebDoesNotExist

Meta

Basic metadata for data models.

id = <AutoField: BaseModelWeb.id>

Miscellaneous Utilities

The darc.model.utils module contains several miscellaneous utility functions and data fields.

class darc.model.utils.IPField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)

Bases: peewee.IPField

IP data field.

db_value(val)

Dump the value for database storage.

Parameters

• value – Source IP address instance.

• val (Optional[Union[str, ipaddress.IPv4Address, ipaddress.IPv6Address]]) –

Returns

Integral representation of the IP address.

Return type

Optional[int]

python_value(val)

Load the value from database storage.

Parameters

• value – Integral representation of the IP address.

• val (Optional[int]) –

Returns

Original IP address instance.

Return type

Optional[Union[ipaddress.IPv4Address, ipaddress.IPv6Address]]

class darc.model.utils.IntEnumField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)

Bases: peewee.IntegerField

enum.IntEnum data field.

python_value(value)

Load the value from database storage.

Parameters

value (Optional[int]) – Integral representation of the enumeration.

Returns

Original enumeration object.

Return type

Optional[enum.IntEnum]

choices: enum.IntEnum

The original enum.IntEnum class.

class darc.model.utils.JSONField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)

Bases: playhouse.mysql_ext.JSONField

JSON data field.

db_value(value)

Dump the value for database storage.

Parameters

value (Any) – Source JSON value.

Returns

JSON serialised string data.

Return type

Optional[str]

python_value(value)

Load the value from database storage.

Parameters

value (Optional[str]) – Serialised JSON string.

Returns

Original JSON data.

Return type

Any

class darc.model.utils.PickleField(null=False, index=False, unique=False, column_name=None, default=None, primary_key=False, constraints=None, sequence=None, collation=None, unindexed=False, choices=None, help_text=None, verbose_name=None, index_type=None, db_column=None, _hidden=False)

Bases: peewee.BlobField

Pickled data field.

db_value(value)

Dump the value for database storage.

Parameters

value (Any) – Source value.

Returns

Picked bytestring data.

Return type

Optional[bytes]

python_value(value)

Load the value from database storage.

Parameters

value (Optional[bytes]) – SPicked bytestring data.

Returns

Original data.

Return type

Any

class darc.model.utils.Proxy(value)

Bases: enum.IntEnum

Proxy types supported by darc.

FREENET = 5

Freenet proxy.

I2P = 3

I2P proxy.

NULL = 1

No proxy.

TOR = 2

Tor proxy.

TOR2WEB = 6

//onion.sh/>`__, no proxy).

Type

Proxied Tor (`tor2web <https

ZERONET = 4

ZeroNet proxy.

darc.model.utils.table_function(model_class)

Generate table name dynamically.

The function strips Model from the class name and calls peewee.make_snake_case() to generate a proper table name.

Parameters

model_class (peewee.Model) – Data model class.

Returns

Generated table name.

Return type

str