Unofficial plugin / extension system design notes

[teratorn]

Here is a first draft at design notes for a Plugin / Extension system for Ultra
Recall. This is just technical, but there should also be thought put in to an
Online Directory, Forums, and Rating Systems for Scripts (produced by Kinook or
by 3rd parties).

You might want to allow automated script installation... Basically what you do
is just register a new File Extension and Mime Type in Windows, and then
provide download links to files that are served with that extension and mime
type. They then are automatically opened by UR (and after checking the
CRYPTOGRAPHIC SIGNATURE, they are allowed to be installed).

Otherwise it will show a Big Bad Warning before allowing the installation of
untrusted plugins.

So anyway... Here goes:

Do *not* try to be language agnostic. Pick the language that you want to use,
and then USE IT FULLY. Don't artificially handicap any of the power that the
language, and the STANDARD LIBRARY OF THAT LANGUAGE (this is key!) has to
offer.

-= DATABASE RELATED APIs =-

DB QUERY API. There should be an API that allows the user to QUERY the database
for sequences of ITEMS, just in the same way that the Advanced Search dialog
box allows. DO NOT PROVIDE DIRECT ACCESS TO SQL. You don't want to restrict
your freedom to change the DB schema in the future, and we want scripts to be
forward-compatible as much as possible.

Items should be returned as objects that have members that represent the
fields. Or perhaps there will simply be one member (e.g. "fields") that is a
Dictionary (associative array) that maps Field Names to Field Values. Field
Vales should be Strong Typed, meaning a Date field ought to have some
language-specific DateTime object that represents those values.

To change DB VALUES you then will either directly MUTATE (modify) a Value
Object that you already hold a reference to, OR (In the case of Immutable Value
Types) you create a new value, and insert it on the Item Object.

TRANSACTION SUPPORT. Provide a simple wrapper for the SQLite transaction
START/END calls, so that scripts can do their work inside nice transactions,
yay!

(Advanced Feature) SINGLE-COLUMN, or MULTI-COLUMN QUERY API. Provide an Query
API that allows the programmer to pull only certain columns out of the DB.

Say you only can what the Flag value is for the items that match your query...
The result of the query would then NOT be a sequence of Object Items, but
rather a sequence Flag Values (of whatever type the Flag is represented by).

If you specify more than one column, then the result would be a sequence of
sequences (tuples), with each sub-sequence containing the Values of the Columns
for that particular Item.

(Advanced Feature) LAZY QUERY ITERATION. Instead of returning full in-memory
sequences, you can instead return Iterable Objects (how you do this depends on
the language being used, of course).

ITEM ID QUERY. Every item in the database should have a unique, global ID (in a
particular .urd file) that can be used to retrieve it.

I'm borrowing a lot of these concepts from the excellent Axiom database
available for Python (it uses SQLite v. 3 under the hood)

-= THREADING CONCERNS =-

Without looking at your codebase I can't know what kind of threading model you
utilize in UltraRecall. So it's pretty hard for me to say what kind of concerns
you will have to deal with in keeping scripts Thread Safe.

My basic thoughts are that all Scripting APIs (where ever possible) should do
locking implicitly and automatically where needed. Script Programs should be
allowed to spawn multiple threads, because sometimes you have to use threads to
do certain things.

However, I don't personally like writing multi-threaded programs very much.
It's extremely difficult to do it well, and to not introduce horrible,
impossible to debug errors in to your programs.

That said, I would lean towards providing an asynchronous Reactor pattern
implementation (event loop) that operates inside a single thread, and that
multiple independent Scripts can use at the same time. This is the same
event-driven model used in GUIs.

If you use Python as the extension language, then Twisted is the obvious choice
for this. You don't have to do any work here, and you give access to a
tremendous amount of power. (I'm a committer on Twisted, so I'm biased, but it
really is the best...

-= DATA MODEL =-

So, you ask, how do you represent the wide variety of content that UR can
handle? Good question.

I don't know.

Lets take Rich Text as an example. I have no idea how you represent Item
Content internally (as a Microsoft RTF document???), so I'm not sure what you
will have to do to expose that content in a programmable fashion.

Ideally, Scripts would be able to manipulate Rich Content in all of its detail.

You may need to invent an XML Schema here, that you can map your internal
representation on to. (X)HTML, anyone?

Or if you could somehow use MS Word internally, then you can use the Word 2003
XML format, while though it is ugly, it is certainly usable and extremely
feature-rich.

So if you use XML, or something else, you will need to provide a DOM API,
similar in nature to the DOM APIs provided in Javascript and by various XML
parsers.

If you use XML, then you can just grab one of the open source DOMs for your
language of choice.

Now, of course, The Tree in UltraRecall is fundamentally a DOM, and you may
wish to provide some integration here, and use the same API for accessing the
ULTRA RECALL DOM, and the various CONTENT DOMs.

Maybe it call all be part of a single DOM (one big tree)?

I'm just thinking out loud here.. and without knowing anything about the
codebase I'm not sure what the best way forward would be...

The DOMs I'm talking about above are fundamentally TEXT BASED. But that may not
be entirely suitable for UR. You may want to provide a DOM that contains not
just generic NODES, but Strongly Typed Objects (that can contain Strongly Typed
Children Objects and Strongly Typed Attributes).

-= GUI ACCESS =-

So, scripts obviously need to be able to perform any actions that the user can
perform through using the various menu items, shortcut keys, and buttons that
are in the GUI. The script should also have access to the various Controls that
fundamentally make up the UR GUI.

E.g. The Script API should allow access to the various Panes, and it should be
able to e.g. Make them wider.

I don't talk here about *modifiying* anything that is *shown* in a pane,
because that fundamentally belongs to the DB / DATA MODEL APIs. There should
obviously be an API to read which Item is currently Selected in the Current
Tab. And thus you can change the data on that Item, and it will reflect in the
various Panes.

An API to add new Tabs, enumerate and remove existing Tabs, and so on.

Access to the clipboard ( though you could just let people use native OS-level
access to the clipboard) instead of providing your own Scripting API for it.

And so on...

How hard all this is, and how it may be done of course depends on how you have
written your GUI layers. Is it just straight MFC/win32 API?

Scripts also need to be able to create their own GUI Controls and have them be
integrated with the Standard Controls where needed. E.g. A Script should be
able to provide a new toolbar, and you could then place, and customize that
toolbar like any of the built-in ones.

Scripts also need to be able to create their own Form Dialogs and Popup
Windows, etc.

You don't really need to do anything special here except choose a language that
is powerful enough to access Windows APIs, and then Scripts can do anything
they need to do.

Some thought will need to go in to how, and at what level, GUI integration is
allowed between Controls created by the Script and those created by UR.

-= NETWORKING =-

Network access is huge. E.g. I want a Plugin that lets be enter a Movie Name (
I have a database of all the DVDs in my collection), and it then pulls down all
the metadata for that movie from IMDB in to the new Movie Item that I am
inserting.

Or I might write a plugin that syncs my Contacts with Gmail.

As above, just give access to a language that already has all this. (Twisted is
the most powerful way to do networking under Python, bar none

-= DEPENDENCIES =-

There ought to be some simple mechanism that one Script can use to tell UR that
it depends on another script. E.g. The script GmailSync might depend on
MetaScript >= 2.0. MetaScript could be a Script that provides various library
routines that are useful to multipe Scripts, and it may provide some mechanisms
to allow inter-script communication (over loopback sockets or Windows Pipes,
etc). Or it could also provide a simple mechanism for a given Script to
enumerate other loaded Scripts and get references to their public objects,
allowing direct communication and collaboration between scripts.

-= CONCLUSION =-

So basically I want to be able to do EVERYTHING. But I know that I won't get
that in the end... That's OK. Even a small fraction of these things, exposed
through a powerful language, would be very useful.

The word "Script" is kind of misleading... I probably should have used the word
Plugin or Extension above.

The very first thing you guys could do would be to allow direct SQLite access
to ENCRYPTED database files. There are plenty of things I could do on my own if
I just had that. Of course it's not pretty, and it's not very safe, and it's
not gonna be forward-compatible. But I'll take it!

I'll take, and be grateful for (and pay for!) anything you give us. But of
course, the more the better!

P.S. Thanks for taking the time to read this. I spent quite a lot of time
thinking and writing, however it is still only a first draft, and I look
forward to constructive comments.