Link Database¶
The darc
project utilises file system based database
to provide tele-process communication.
Note
In its first implementation, the darc
project used
multiprocessing.Queue
to support such communication. However, as noticed
when runtime, the multiprocessing.Queue
object will be much affected by
the lack of memory.
There will be two databases, both locate at root of the
data storage path PATH_DB
:
At runtime, after reading such database, darc
will keep a backup of the database with .tmp
suffix
to its file extension.
-
darc.db.
load_requests
(check=False)[source]¶ Load link from the
requests
database.- Parameters
check (bool) – If perform checks on loaded links, default to
CHECK
.- Returns
List of loaded links from the
requests
database.- Return type
List[darc.link.Link]
Note
At runtime, the function will load links with maximum number at
MAX_POOL
to limit the memory usage.
-
darc.db.
load_selenium
(check=False)[source]¶ Load link from the
selenium
database.- Parameters
check (bool) – If perform checks on loaded links, default to
CHECK
.- Returns
List of loaded links from the
selenium
database.- Return type
List[darc.link.Link]
Note
At runtime, the function will load links with maximum number at
MAX_POOL
to limit the memory usage.
-
darc.db.
save_requests
(entries, single=False, score=None, nx=False, xx=False)[source]¶ Save link to the
requests
database.- Parameters
entries (Iterable[darc.link.Link]) – Links to be added to the
requests
database. It can be either an iterable of links, or a single link string (ifsingle
set asTrue
).single (bool) – Indicate if
entries
is an iterable of links or a single link string.score – Score to for the Redis sorted set.
nx – Forces
ZADD
to only create new elements and not to update scores for elements that already exist.xx – Forces
ZADD
to only update scores of elements that already exist. New elements will not be added.
-
darc.db.
save_selenium
(entries, single=False, score=None, nx=False, xx=False)[source]¶ Save link to the
selenium
database.- Parameters
entries (Iterable[darc.link.Link]) – Links to be added to the
selenium
database. It can be either an iterable of links, or a single link string (ifsingle
set asTrue
).single (bool) – Indicate if
entries
is an iterable of links or a single link string.score – Score to for the Redis sorted set.
nx – Forces
ZADD
to only create new elements and not to update scores for elements that already exist.xx – Forces
ZADD
to only update scores of elements that already exist. New elements will not be added.
-
darc.db.
QR_LOCK
: multiprocessing.Lock¶ I/O lock for the
requests
database_queue_requests.txt
.See also
-
darc.db.
QS_LOCK
: Union[multiprocessing.Lock, threading.Lock, contextlib.nullcontext]¶ I/O lock for the
selenium
database_queue_selenium.txt
.If
FLAG_MP
isTrue
, it will be an instance ofmultiprocessing.Lock
. IfFLAG_TH
isTrue
, it will be an instance ofthreading.Lock
. If none above, it will be an instance ofcontextlib.nullcontext
.