multi-threading web page import

[moku]

Originally posted by quant
ok, I see now what you meant ... I agree

Multi-threading would be nice, but probably much more difficult to maintain the database consistence. Don't know how the database that UR uses works. That's the question for them to ask.

(But to be honest, the database I have and use, is much more used to recall the data than to append )

I am not aware of how UR uses SQLite internally.

What I do know:

-- SQLite, per database, can only handle one writer at a time.

http://www.sqlite.org/lockingv3.html

SQLite v3 also provides some enhancements to help prevent what is called "writer starvation":

5.1 Writer starvation

In SQLite version 2, if many processes are reading from the database, it might be the case that there is never a time when there are no active readers. And if there is always at least one read lock on the database, no process would ever be able to make changes to the database because it would be impossible to acquire a write lock. This situation is called writer starvation.

SQLite version 3 seeks to avoid writer starvation through the use of the PENDING lock. The PENDING lock allows existing readers to continue but prevents new readers from connecting to the database. So when a process wants to write a busy database, it can set a PENDING lock which will prevent new readers from coming in. Assuming existing readers do eventually complete, all SHARED locks will eventually clear and the writer will be given a chance to make its changes.

-- Each UR database corresponds to one SQLite database.

Within the context of a single UR database, an import manager would allow sequential imports, each in turn locking the database to perform a write.

A background import/job manager would have to intelligently wait for its time to do its work. This is because a user might be doing something in the UI that updates the database. This is not as big of a problem as it sounds as import updates only update existing UI placeholder items. The import manager needs to do some basic validity checking (i.e. is the placeholder item still present and sane?) before running.

Of course any expensive operations should be done ahead of time so that SQLite is as efficient as possible.

One day, hopefully soon, SQLite may have row locking. And then the additional complexity of doing many background operations will be minimized. Until that day, it takes more thought due to SQLite's locking architecture.