Link Database¶
The darc
project utilises file system based database
to provide tele-process communication.
Note
In its first implementation, the darc
project used
multiprocessing.Queue
to support such communication. However, as noticed
when runtime, the multiprocessing.Queue
object will be much affected by
the lack of memory.
There will be two databases, both locate at root of the
data storage path PATH_DB
:
At runtime, after reading such database, darc
will keep a backup of the database with .tmp
suffix
to its file extension.
-
darc.db.
load_requests
(check=False)[source]¶ Load link from the
requests
database.After loading,
darc
will backup the original databasequeue_requests.txt
asqueue_requests.txt.tmp
and empty the loaded database.- Parameters
check (bool) – If perform checks on loaded links, default to
CHECK
.- Returns
List of loaded links from the
requests
database.- Return type
List[str]
Note
Lines start with
#
will be considered as comments. Empty lines and comment lines will be ignored when loading.At runtime, the function will load links with maximum number at
MAX_POOL
to limit the memory usage.
-
darc.db.
load_selenium
(check=False)[source]¶ Load link from the
selenium
database.After loading,
darc
will backup the original databasequeue_selenium.txt
asqueue_selenium.txt.tmp
and empty the loaded database.- Parameters
check (bool) – If perform checks on loaded links, default to
CHECK
.- Returns
List of loaded links from the
selenium
database.- Return type
List[str]
Note
Lines start with
#
will be considered as comments. Empty lines and comment lines will be ignored when loading.At runtime, the function will load links with maximum number at
MAX_POOL
to limit the memory usage.
-
darc.db.
QR_LOCK
: multiprocessing.Lock¶ I/O lock for the
requests
database_queue_requests.txt
.See also
-
darc.db.
QS_LOCK
: Union[multiprocessing.Lock, threading.Lock, contextlib.nullcontext]¶ I/O lock for the
selenium
database_queue_selenium.txt
.If
FLAG_MP
isTrue
, it will be an instance ofmultiprocessing.Lock
. IfFLAG_TH
isTrue
, it will be an instance ofthreading.Lock
. If none above, it will be an instance ofcontextlib.nullcontext
.
-
darc.db.
MAX_POOL
: int¶ - Default
1_000
- Environ
Maximum number of links loading from the database.
Note
If is an infinit
inf
, no limit will be applied.