Link Database¶

The darc project utilises Redis based database to provide tele-process communication.

Note

In its first implementation, the darc project used Queue to support such communication. However, as noticed when runtime, the Queue object will be much affected by the lack of memory.

There will be three databases, all following the save naming convension with queue_ prefix:

the hostname database – queue_hostname (HostnameQueueModel)
the requests database – queue_requests (RequestsQueueModel)
the selenium database – queue_selenium (SeleniumQueueModel)

For queue_hostname, queue_requests and queue_selenium, they are all Redis sorted set data type.

If FLAG_DB is True, then the module uses the RDS storage described by the peewee models as backend.

darc.db._db_operation(operation, *args, **kwargs)[source]¶

Retry operation on database.

Parameters

operation (Callable[[..], T]) – Callable / method to perform.
*args – Arbitrary positional arguments.

Keyword Arguments

**kwargs – Arbitrary keyword arguments.

Returns

Any return value from a successful operation call.

Return type

T

darc.db._drop_hostname_db(link)[source]¶

Remove link from the hostname database.

The function updates the HostnameQueueModel table.

Parameters: link (darc.link.Link) – Link to be removed.
Return type: None

darc.db._drop_hostname_redis(link)[source]¶

Remove link from the hostname database.

The function updates the queue_hostname database.

Parameters: link (darc.link.Link) – Link to be removed.
Return type: None

darc.db._drop_requests_db(link)[source]¶

Remove link from the requests database.

The function updates the RequestsQueueModel table.

Parameters: link (darc.link.Link) – Link to be removed.
Return type: None

darc.db._drop_requests_redis(link)[source]¶

Remove link from the requests database.

The function updates the queue_requests database.

Parameters: link (darc.link.Link) – Link to be removed.
Return type: None

darc.db._drop_selenium_db(link)[source]¶

Remove link from the selenium database.

The function updates the SeleniumQueueModel table.

Parameters: link (darc.link.Link) – Link to be removed.
Return type: None

darc.db._drop_selenium_redis(link)[source]¶

Remove link from the selenium database.

The function updates the queue_selenium database.

Parameters: link (darc.link.Link) – Link to be removed.
Return type: None

darc.db._gen_arg_msg(*args, **kwargs)[source]¶

Sanitise arguments representation string.

Parameters: *args – Arbitrary arguments.
Keyword Arguments: **kwargs – Arbitrary keyword arguments.
Returns: Sanitised arguments representation string.
Return type: str

darc.db._have_hostname_db(link)[source]¶

Check if current link is a new host.

The function checks the HostnameQueueModel table.

Parameters: link (darc.link.Link) – Link to check against.
Returns: A tuple of two bool values representing if such link is a known host and needs force refetch respectively.
Return type: Tuple[bool, bool]

darc.db._have_hostname_redis(link)[source]¶

Check if current link is a new host.

The function checks the queue_hostname database.

Parameters: link (darc.link.Link) – Link to check against.
Returns: A tuple of two bool values representing if such link is a known host and needs force refetch respectively.
Return type: Tuple[bool, bool]

darc.db._load_requests_db()[source]¶

Load link from the requests database.

The function reads the RequestsQueueModel table.

Returns: List of loaded links from the requests database.
Return type: List[darc.link.Link]

Note

At runtime, the function will load links with maximum number at MAX_POOL to limit the memory usage.

darc.db._load_requests_redis()[source]¶

Load link from the requests database.

The function reads the queue_requests database.

Returns: List of loaded links from the requests database.
Return type: List[darc.link.Link]

Note

At runtime, the function will load links with maximum number at MAX_POOL to limit the memory usage.

darc.db._load_selenium_db()[source]¶

Load link from the selenium database.

The function reads the SeleniumQueueModel table.

Returns: List of loaded links from the selenium database.
Return type: List[darc.link.Link]

Note

At runtime, the function will load links with maximum number at MAX_POOL to limit the memory usage.

darc.db._load_selenium_redis()[source]¶

Load link from the selenium database.

The function reads the queue_selenium database.

Parameters: check – If perform checks on loaded links, default to CHECK.
Returns: List of loaded links from the selenium database.
Return type: List[darc.link.Link]

Note

At runtime, the function will load links with maximum number at MAX_POOL to limit the memory usage.

darc.db._redis_command(command, *args, **kwargs)[source]¶

Wrapper function for Redis command.

Parameters

command (str) – Command name.
*args – Arbitrary arguments for the Redis command.

Keyword Arguments

**kwargs – Arbitrary keyword arguments for the Redis command.

Returns

Values returned from the Redis command.

Warns

RedisCommandFailed – Warns at each round when the command failed.

Return type

Any

See also

Between each retry, the function sleeps for RETRY_INTERVAL second(s) if such value is NOT None.

darc.db._redis_get_lock(name, timeout=None, sleep=0.1, blocking_timeout=None, lock_class=None, thread_local=True)[source]¶

Get a lock for Redis operations.

Parameters

name (str) – Lock name.
timeout (Optional[float]) – Maximum life for the lock.
sleep (float) – Amount of time to sleep per loop iteration when the lock is in blocking mode and another client is currently holding the lock.
blocking_timeout (Optional[float]) – Maximum amount of time in seconds to spend trying to acquire the lock.
lock_class (Optional[redis.lock.Lock]) – Lock implementation.
thread_local (bool) – Whether the lock token is placed in thread-local storage.

Returns

Return a new redis.lock.Lock object using key name that mimics the behavior of threading.Lock.

Return type

Union[redis.lock.Lock, contextlib.nullcontext]

Seel Also:: If REDIS_LOCK is False, returns a contextlib.nullcontext instead.

darc.db._save_requests_db(entries, single=False, score=None, nx=False, xx=False)[source]¶

Save link to the requests database.

The function updates the RequestsQueueModel table.

Parameters

entries (Union[darc.link.Link, List[darc.link.Link]]) – Links to be added to the requests database. It can be either a list of links, or a single link string (if single set as True).
single (bool) – Indicate if entries is a list of links or a single link string.
score (Optional[float]) – Score to for the Redis sorted set.
nx (bool) – Only create new elements and not to update scores for elements that already exist.
xx (bool) – Only update scores of elements that already exist. New elements will not be added.

Return type

None

darc.db._save_requests_redis(entries, single=False, score=None, nx=False, xx=False)[source]¶

Save link to the requests database.

The function updates the queue_requests database.

Parameters

entries (Union[darc.link.Link, List[darc.link.Link]]) – Links to be added to the requests database. It can be either a list of links, or a single link string (if single set as True).
single (bool) – Indicate if entries is a list of links or a single link string.
score (Optional[float]) – Score to for the Redis sorted set.
nx (bool) – Forces ZADD to only create new elements and not to update scores for elements that already exist.
xx (bool) – Forces ZADD to only update scores of elements that already exist. New elements will not be added.

Return type

None

darc.db._save_selenium_db(entries, single=False, score=None, nx=False, xx=False)[source]¶

Save link to the selenium database.

The function updates the SeleniumQueueModel table.

Parameters

entries (Union[darc.link.Link, List[darc.link.Link]]) – Links to be added to the selenium database. It can be either a list of links, or a single link string (if single set as True).
single (bool) – Indicate if entries is a list of links or a single link string.
score (Optional[float]) – Score to for the Redis sorted set.
nx (bool) – Only create new elements and not to update scores for elements that already exist.
xx (bool) – Only update scores of elements that already exist. New elements will not be added.

Return type

None

darc.db._save_selenium_redis(entries, single=False, score=None, nx=False, xx=False)[source]¶

Save link to the selenium database.

The function updates the queue_selenium database.

Parameters

entries (Union[darc.link.Link, List[darc.link.Link]]) – Links to be added to the selenium database. It can be either an iterable of links, or a single link string (if single set as True).
single (bool) – Indicate if entries is an iterable of links or a single link string.
score (Optional[float]) – Score to for the Redis sorted set.
nx (bool) – Forces ZADD to only create new elements and not to update scores for elements that already exist.
xx (bool) – Forces ZADD to only update scores of elements that already exist. New elements will not be added.

Return type

None

When entries is a list of Link instances, we tries to perform bulk update to easy the memory consumption. The bulk size is defined by BULK_SIZE.

Notes

The entries will be dumped through pickle so that darc do not need to parse them again.

darc.db.drop_hostname(link)[source]¶

Remove link from the hostname database.

Parameters: link (darc.link.Link) – Link to be removed.
Return type: None

See also

darc.db._drop_hostname_db()
darc.db._drop_hostname_redis()

darc.db.drop_requests(link)[source]¶

Remove link from the requests database.

Parameters: link (darc.link.Link) – Link to be removed.
Return type: None

See also

darc.db._drop_requests_db()
darc.db._drop_requests_redis()

darc.db.drop_selenium(link)[source]¶

Remove link from the selenium database.

Parameters: link (darc.link.Link) – Link to be removed.
Return type: None

See also

darc.db._drop_selenium_db()
darc.db._drop_selenium_redis()

darc.db.have_hostname(link)[source]¶

Check if current link is a new host.

Parameters: link (darc.link.Link) – Link to check against.
Returns: A tuple of two bool values representing if such link is a known host and needs force refetch respectively.
Return type: Tuple[bool, bool]

See also

darc.db._have_hostname_db()
darc.db._have_hostname_redis()

darc.db.load_requests(check=False)[source]¶

Load link from the requests database.

Parameters: check (bool) – If perform checks on loaded links, default to CHECK.
Returns: List of loaded links from the requests database.
Return type: List[darc.link.Link]

Note

At runtime, the function will load links with maximum number at MAX_POOL to limit the memory usage.

See also

darc.db._load_requests_db()
darc.db._load_requests_redis()

darc.db.load_selenium(check=False)[source]¶

Load link from the selenium database.

Parameters: check (bool) – If perform checks on loaded links, default to CHECK.
Returns: List of loaded links from the selenium database.
Return type: List[darc.link.Link]

Note

At runtime, the function will load links with maximum number at MAX_POOL to limit the memory usage.

See also

darc.db._load_selenium_db()
darc.db._load_selenium_redis()

darc.db.save_requests(entries, single=False, score=None, nx=False, xx=False)[source]¶

Save link to the requests database.

The function updates the queue_requests database.

Parameters

entries (Union[darc.link.Link, List[darc.link.Link]]) – Links to be added to the requests database. It can be either a list of links, or a single link string (if single set as True).
single (bool) – Indicate if entries is a list of links or a single link string.
score (Optional[float]) – Score to for the Redis sorted set.
nx (bool) – Only create new elements and not to update scores for elements that already exist.
xx (bool) – Only update scores of elements that already exist. New elements will not be added.

Return type

None

Notes

The entries will be dumped through pickle so that darc do not need to parse them again.

When entries is a list of Link instances, we tries to perform bulk update to easy the memory consumption. The bulk size is defined by BULK_SIZE.

See also

darc.db._save_requests_db()
darc.db._save_requests_redis()

darc.db.save_selenium(entries, single=False, score=None, nx=False, xx=False)[source]¶

Save link to the selenium database.

Parameters

entries (Union[darc.link.Link, List[darc.link.Link]]) – Links to be added to the selenium database. It can be either a list of links, or a single link string (if single set as True).
single (bool) – Indicate if entries is a list of links or a single link string.
score (Optional[float]) – Score to for the Redis sorted set.
nx (bool) – Only create new elements and not to update scores for elements that already exist.
xx (bool) – Only update scores of elements that already exist. New elements will not be added.

Return type

None

Notes

The entries will be dumped through pickle so that darc do not need to parse them again.

When entries is a list of Link instances, we tries to perform bulk update to easy the memory consumption. The bulk size is defined by BULK_SIZE.

See also

darc.db._save_selenium_db()
darc.db._save_selenium_redis()

darc.db.BULK_SIZE: int¶

Default: 100
Environ: DARC_BULK_SIZE

Bulk size for updating Redis databases.

See also

darc.db.save_requests()
darc.db.save_selenium()

darc.db.LOCK_TIMEOUT: Optional[float]¶

Default: 10
Environ: DARC_LOCK_TIMEOUT

Lock blocking timeout.

Note

If is an infinit inf, no timeout will be applied.

See also

Get a lock from darc.db.get_lock().

darc.db.MAX_POOL: int¶

Default: 1_000
Environ: DARC_MAX_POOL

Maximum number of links loading from the database.

Note

If is an infinit inf, no limit will be applied.

darc.db.REDIS_LOCK: bool¶

Default: False
Environ: DARC_REDIS_LOCK

If use Redis (Lua) lock to ensure process/thread-safely operations.

See also

Toggles the behaviour of darc.db.get_lock().

darc.db.RETRY_INTERVAL: int¶

Default: 10
Environ: DARC_RETRY

Retry interval between each Redis command failure.

Note

If is an infinit inf, no interval will be applied.

See also

Toggles the behaviour of darc.db.redis_command().

Link Database¶

darc

Navigation

Related Topics