Link Database¶
The darc
project utilises file system based database
to provide tele-process communication.
Note
In its first implementation, the darc
project used
multiprocessing.Queue
to support such communication. However, as noticed
when runtime, the multiprocessing.Queue
object will be much affected by
the lack of memory.
There will be two databases, both locate at root of the
data storage path PATH_DB
:
At runtime, after reading such database, darc
will keep a backup of the database with .tmp
suffix
to its file extension.
-
darc.db.
load_requests
()¶ Load link from the
requests
database.After loading,
darc
will backup the original databasequeue_requests.txt
asqueue_requests.txt.tmp
and empty the loaded database.- Returns
List of loaded links from the
requests
database.- Return type
List[str]
Note
Lines start with
#
will be considered as comments. Empty lines and comment lines will be ignored when loading.
-
darc.db.
load_selenium
()¶ Load link from the
selenium
database.After loading,
darc
will backup the original databasequeue_selenium.txt
asqueue_selenium.txt.tmp
and empty the loaded database.- Returns
List of loaded links from the
selenium
database.- Return type
List[str]
Note
Lines start with
#
will be considered as comments. Empty lines and comment lines will be ignored when loading.
-
darc.db.
save_requests
(entries, single=False)¶ Save link to the
requests
database.- Parameters
entries (Iterable[str]) – Links to be added to the
requests
database. It can be either an iterable of links, or a single link string (ifsingle
set asTrue
).single (bool) – Indicate if
entries
is an iterable of links or a single link string.
-
darc.db.
save_selenium
(entries, single=False)¶ Save link to the
selenium
database.- Parameters
entries (Iterable[str]) – Links to be added to the
selenium
database. It can be either an iterable of links, or a single link string (ifsingle
set asTrue
).single (bool) – Indicate if
entries
is an iterable of links or a single link string.
-
darc.db.
QR_LOCK
: multiprocessing.Lock¶ I/O lock for the
requests
database_queue_requests.txt
.See also
-
darc.db.
QS_LOCK
: Union[multiprocessing.Lock, threading.Lock, contextlib.nullcontext]¶ I/O lock for the
selenium
database_queue_selenium.txt
.If
FLAG_MP
isTrue
, it will be an instance ofmultiprocessing.Lock
. IfFLAG_TH
isTrue
, it will be an instance ofthreading.Lock
. If none above, it will be an instance ofcontextlib.nullcontext
.