Link Database

The darc project utilises file system based database to provide tele-process communication.

Note

In its first implementation, the darc project used multiprocessing.Queue to support such communication. However, as noticed when runtime, the multiprocessing.Queue object will be much affected by the lack of memory.

There will be two databases, both locate at root of the data storage path PATH_DB:

• the requests database – queue_requests.txt

• the selenium database – queue_selenium.txt

At runtime, after reading such database, darc will keep a backup of the database with .tmp suffix to its file extension.

darc.db.drop_hostname(link)

Remove link from the hostname database.

Parameters

link (darc.link.Link) – Link to be removed.

darc.db.drop_requests(link)

Remove link from the requests database.

Parameters

link (darc.link.Link) – Link to be removed.

darc.db.drop_selenium(link)

Remove link from the selenium database.

Parameters

link (darc.link.Link) – Link to be removed.

darc.db.get_lock(name, timeout=None, sleep=0.1, blocking_timeout=None, lock_class=None, thread_local=True)

Get a lock for Redis operations.

Parameters

• name (str) – Lock name.

• timeout (Optional[float]) – Maximum life for the lock.

• sleep (float) – Amount of time to sleep per loop iteration when the lock is in blocking mode and another client is currently holding the lock.

• blocking_timeout (Optional[float]) – Maximum amount of time in seconds to spend trying to acquire the lock.

• lock_class (Optional[redis.lock.Lock]) – Lock implementation.

• thread_local (bool) – Whether the lock token is placed in thread-local storage.

Returns

Return a new redis.lock.Lock object using key name that mimics the behavior of threading.Lock.

Return type

Union[redis.lock.Lock, contextlib.nullcontext]

Notes

If REDIS_LOCK is False, returns a contextlib.nullcontext instead.

darc.db.have_hostname(link)

Check if current link is a new host.

Parameters

link (darc.link.Link) – Link to check against.

Returns

If such link is a new host.

Return type

bool

darc.db.load_requests(check=False)

Load link from the requests database.

Parameters

check (bool) – If perform checks on loaded links, default to CHECK.

Returns

List of loaded links from the requests database.

Return type

List[darc.link.Link]

Note

At runtime, the function will load links with maximum number at MAX_POOL to limit the memory usage.

darc.db.load_selenium(check=False)

Load link from the selenium database.

Parameters

check (bool) – If perform checks on loaded links, default to CHECK.

Returns

List of loaded links from the selenium database.

Return type

List[darc.link.Link]

Note

At runtime, the function will load links with maximum number at MAX_POOL to limit the memory usage.

darc.db.save_requests(entries, single=False, score=None, nx=False, xx=False)

Save link to the requests database.

Parameters

• entries (List[darc.link.Link]) – Links to be added to the requests database. It can be either a list of links, or a single link string (if single set as True).

• single (bool) – Indicate if entries is a list of links or a single link string.

• score – Score to for the Redis sorted set.

• nx – Forces ZADD to only create new elements and not to update scores for elements that already exist.

• xx – Forces ZADD to only update scores of elements that already exist. New elements will not be added.

darc.db.save_selenium(entries, single=False, score=None, nx=False, xx=False)

Save link to the selenium database.

Parameters

• entries (List[darc.link.Link]) – Links to be added to the selenium database. It can be either an iterable of links, or a single link string (if single set as True).

• single (bool) – Indicate if entries is an iterable of links or a single link string.

• score – Score to for the Redis sorted set.

• nx – Forces ZADD to only create new elements and not to update scores for elements that already exist.

• xx – Forces ZADD to only update scores of elements that already exist. New elements will not be added.

darc.db.QR_LOCK: multiprocessing.Lock

I/O lock for the requests database _queue_requests.txt.

See also

• darc.db.save_requests()

darc.db.QS_LOCK: Union[multiprocessing.Lock, threading.Lock, contextlib.nullcontext]

I/O lock for the selenium database _queue_selenium.txt.

If FLAG_MP is True, it will be an instance of multiprocessing.Lock. If FLAG_TH is True, it will be an instance of threading.Lock. If none above, it will be an instance of contextlib.nullcontext.

See also

• darc.db.save_selenium()

• darc.const.FLAG_MP

• darc.const.FLAG_TH

darc.db.MAX_POOL: int

Default

1_000

Environ

DARC_MAX_POOL

Maximum number of links loading from the database.

Note

If is an infinit inf, no limit will be applied.