UR using Too Many Resources

[gershonb]

Hello, I just got UR because the sample looked fun and I have some background in databases. I am enjoying being able to list all my website favorites and have dynamic links to things like google calendar, etc.

The problem is, UR uses a tremendous amount of resources, and its resource stack doesn't seem to be stable all the time. I'm on a quad machine with 8 gigs of ram. In Process Explorer (free download from Microsoft) I've tried setting the program's Affinity to run on only one processor. Still--it uses more resources than Skype, and Skype, bless its heart, is something my son the head programmer doesn't allow on the worksite because of resource problems.

Should I try importing just links rather than web content?

Any suggestions?
Gershon B

[kinook]

What resources are you seeing UR use a lot of? Right now on my box (Intel Core i7 w/ 8GB RAM), examining with Process Explorer, UR is using about 200MB virtual size, 30MB working set, 350 handles, 15 threads, 180 USER objects, 220 GDI objects, and negligible CPU, which is in the same ballpark as other GUI apps that are running (Word, Access, Process Explorer, Windows Live Mail, Firefox, Chrome). UR embeds Internet Explorer to display web pages that are imported or browsed to, and depending on the complexity of pages displayed by IE, IE itself might use more resources. Please send the info requested at http://www.kinook.com/Forum/showthread.php?t=3038. Thanks.

[armsys]

Hi Gershon,
Welcome aboard.
A simple answer: No.
It won't consume excessive resources unncessarily.
For maximizing Ultra Recall performance, try:
Ultra Recall is designed to handle large databases very efficiently (for instance, when populating the tree, only nodes that are expanded will actually be loaded from the database, the details for an item are only loaded when that item is selected, use of indexed searches, etc.).
But with items containing thousands of immediate child items or very large databases (hundreds of megabytes or multiple gigabytes in size), especially on older, slower computers, some operations in Ultra Recall can be slow.
Use a Local Drive
For best performance, store the .urd file on a local hard drive, not a network share or USB stick.
Items with Many Children
It is best to avoid expanding a node in the Data Explorer pane containing thousands of immediate child items, since loading many items into the tree can be slow and cannot be cancelled. Selecting the parent item is ok, because loading the Child Items pane can be cancelled. It's not very useful to expand that many items in the tree anyway. Instead, use a search to locate items of interest and select the item in the Search Results pane to use it. If you do expand a node with many children and UR becomes non-responsive, you can safely end the UltraRecall.exe process (using Task Manager) and restart UR without corrupting or losing any data.
Large Import Operations
When importing thousands of items or documents into a database (for instance, for archival purposes), rather than importing to Imported Items and moving all the items somewhere else later, import to the final location, since moving thousands of items in a large database can be slow.
Alternatively, cancel loading of the Child Items pane after a few hundred or thousand items are loaded (watch the Children: count in the status bar and press Esc to cancel loading), then select all (Ctrl+A) and move (Alt+L) those, and repeat until all items have been moved.
Updates in Large Databases
When performing update operations (importing, moving, deleting, etc.) on a large database, there are a couple of settings which can increase performance.
A) Adjust the SQLite synchronous level
Ultra Recall uses the SQLite database engine to store data in .urd files, which provides robust data management to ensure that your data is safe. Better performance (up to 50x faster for some operations) can be achieved by disabling the synchronous flag, but be aware that if Windows crashes or the computer loses power while UR is running, the database could become corrupted. Always perform regular backups of your database.
To use the fast SQLite synchronous level, extract and double-click UseFastSynchronousLevel.reg in the attached ZIP file, and restart UR. To restore the safe synchronous level, double-click UseSafeSynchronousLevel.reg and restart UR.
B) Disable Undo/Redo
Some update operations can be much faster if Ultra Recall's undo/redo feature is disabled. Undo/redo can be useful during normal use, so you may want to disable it only when performing large import or update operations.
To disable undo/redo in Ultra Recall, extract and double-click DisableUndoRedo.reg in the attached ZIP file, and restart UR. To enable undo/redo, double-click EnableUndoRedo.reg and restart UR.
Note: Use of these options requires Ultra Recall v3.2 or later.

[armsys]

Originally Posted by gershonb

...I've tried setting the program's Affinity to run on only one processor.

Tweaking the CPU affinity is a wrong direction to maximizing the UR performance.
Instead, disable prefetch and remove virtual memory.

The unusual heavy activities of UR may suggest the URD (ie database) is potentially corrupt. Relax. You won't likely lose any data. Just run Tools | Compact and Repair...
*** CAUTION *** Please read online help first re: Tools | Compact and Repair....
Let us know your result.

Import links instead of full content (aka webpages) whenever possible. Otherwise, you're overloading your UR unnecessarily.

Hope you're happy with your UR.

[schferk]

Very good advice, armsys, and only some of those I had read earlier. Very helpful indeed.

Now, for the risk of appearing dumb as far as I'm concerned, let's clarify for new users or people like me who haven't been too much interested in importing otherwise than text content, with some pictures / graphics here and there, but muse what could be possible, otherwise - I might be mistaken in what follows:

In UR, there are 3 ways for processing content:

a) Just link. In this case, content is not indexed, hence not searchable from within UR. This might be the case for linked .mht's, and linked .pdf's - or are just-linked pdf's UR-searchable? Or, is no such external file indexable here ? (but see b)

b) Embed. With embedding, an external file stays outside of the UR db, but is linked, and its content is (necessarily? or are there other indexed

c) Import. Here, the external file is stuffed into the UR db (which means it will bloat it, etc. - so in any case we don't miss something by just embedding, embedding is preferable to importing, so why ever import a file into UR to begin with? but see below).

In this case, everything (?) is searchable? Probably not, since UR presumably doesn't "know" special file formats, which it can import = stock, but not index. So, which file formats are indexable here? I suppose .mht (?), and certainly .pdf, right? Also, Word files, I suppose, .txt, .html, .rtf?

If I understand well, both b and c will have the same indexing characteristics: a file format that is indexable in b, is indexable (and that always means, searchable) in c, too, and vice versa - but is that so?

And, if I understand well, the only (?) advantage of c over b would be that the external file will be exported within the corresponding UR sub-tree, whenever you export that, since it's contained in it; on the other hand, embedded files would need to be manually moved / copied within a folder that would accompany the corresponding UR sub-tree (= exported into a neb UR file), and any such file you don't move / copy, will produce a broken link within the UR tree, and worse, whenever you search for their indexed content, it will probably produce false hits, when in fact the source for these indexe words isn't there anymore.

Sideline: Even when you manually move / copy the embedded files in question, you'll probably get broken links, since the name and / or path of the corresponding folders will not be the same anymore - which makes me muse that perhaps UR could have have implemented a comman which would update the path, on condition that you have the folder name (= last part of the path) unchanged; this solution would help UR users to embed a max of files, instead of having to import them for the sole reason of them staying "transportable". EDIT: And there might be a routine implemented into UR that does "verify the links to embedded files within the selected subtree and list any broken links".

And finally, the sole difference between a) and b) would be that b) automatically indexes linked files and otherwise presents these b) files as the a) files, within the tree? Both would appear as links there, and it's just your knowing that e.g. Word files are indexed, that would make you know the link for file abc.xyz is not indexed, meaning it's just a link, and the link for file abc.doc is indexed, meaning this link is an embedded file?

I suppose that there might be some people here in this forum that would be as thankful as I'd be to have all these questions answered, in a systemtic way, i.e. if somebody knowing the details of putting files into UR retook this schema a-b-c here, but with the respective corrected info in it.

[armsys]

Hi schferk,
I know one day inevitably I'll come across your mentally consuming verbose manifestos.
I'm terribly sorry: With due respect, I get lost once again. What are your points?
Please indulge my curiosity:
1. How much time do you spend to compose your prolific manifestos here?
2. Have you actually ever exercised due diligence in experimenting with products before publishing your magnificent manifestos here? For example, for RM, have you actually tested your RM dashboard(s) with topics/tasks embedded in 30 or more respective mission-critical mindmaps?

[Nobodo]

ROFL!

I still come to the kinook forums occasionally just to see if there are any new books that have been recently released.

[schferk]

RM
Have answered in due place.

Composing time
Above answer cost me 35 min. incl. the time to develop my points. When in focus, I often think rather fast, and - and here's the quirk that makes reading me strenuous -, while writing, I also catch all the relevant little details from my mind in order to "complete" the sub-subject as far as I'm able to do that, at the time. I also do not revise then, in order to bring it into better order, leaving the sub-details slightly misplaced - the reason for this is my knowing that further on, I'll get a better grasp of it all, which will then invalidate some of my points, and some day indeed, I should put it all into the right order - when I'm not any more in the groping-around stage. On the other hand, even this intermediate stage brings enough new insight that's worth to be shared; I judge this from what I otherwise find, in the web, and in publications available to me. It's also about ordering knowledge, and systematizing it, whilst anywhere in the web, the good things are enormously scattered, and with very big holes in between.

Back to topic
I know there is the help file. I admit I have developed sort of a phobia vÃ*v the UR help file, and I remember that of course there is some info that could inform me about some aspects of my question. But then, it has decidedly be my intention to ask like a newcomer to UR would ask: 1, 2, 3: 3 different kinds to put external things into UR, or to just link them to UR, and have it all together, one by one, from the lightest way of "connecting it to UR" down to the "most integrated" way (but which, for its blowing up the amount of date in UR, hence rising answering times, and even perhaps, at some point, stability probs, should be avoided whenever possible).

So a new user of UR, or a prospect for UR, would like to have such a COMPREHENSIVE account of this part of UR's functionality, from HIS point of view, i.e. deciding, which kind of external stuff can be put into UR in which way, and thus, what will I decide upon this kind of external stuff, and of that... and of course, such a comprehensive "simili-table" would also make evident any sensible but missing functionality that UR might have, at this point.

So here again, it's about systematization of knowledge, here about a CORE feature of any such sw, i.e. "how to store things in it, other than text". Please imagine a UR prospect: At this time, he's forced to get this basic, essential info from driving forth and back in several help file items, and even after this dreadful voyage, he probably will NOT be able to make the above-tried systematic "table", but will be in doubt about more than one detail here.

Hence the utmost interest of UR to provide that thing that I have tried to begin above, and where I certainly will have made numerous errors. But's it up to UR to clarify UR for new users, in order to make it a never-ending IQ test for every possible new UR user to gather his knowledge about UR on his own, and for every such prospect (or worse, new user) again, and again - that's a lot of extravagance UR shifts upon its (potential) users.

And since UR doesn't seem to see the urgent need for a better help file - of which my beginning of a systematization was only an example -, it would be so kind if experienced UR users helped all of us out here, took what I wrote above, put it into a new entry form, and weeded out any of my numerous mistakes and possible misconceptions there.