Search problems [resolved]

[Spliff]

My main database has got almost 60,000 items (mostly text, some pics in text, no external documents), almost 1.2 GB in all.

Problem a)

Most (simple) searches take a long or even very long time, albeit I have "enhanced full-text search" enabled. The same after "compact/repair database" with "enable enhanced full-text search features" DIS-abled, and then the same with EN-abled, i.e. after recreation of the index (I suppose this double "compact and repair" in this way, recreates the index).

Needs long time even for single word with option "just in titles", almost 60,000 items to search, but since there is an index, I'm surprised it takes this long.

Problem b)

(Those "Enhancements" always enabled, since I want them to be available in other searches, and switch between enhancements off and on needs compact-and-repair which for enabling the enhancements needs a long time.)

After stopping the search (because it takes too long), the "database is locked" problem. In order to unlock the database then, just closing it will not do (control-F4 does not work then), and closing several databases and UR (by alt-F4) will work, but then, UR will not open anymore (by clicking on the desktop symbol).

Just a complete Windows restart will help, since then, clicking on the desktop symbol will reopen UR (and the databases that were open), and they work as expected.


I suppose there will not be short-term solutions?

[kinook]

Searches should not be slow. We have a database with > 40K items and most searches are under 1 second (and we've never seen database locked when cancelling, although searches usually finish before they can be canceled). Please send as much info as possible from

https://www.kinook.com/Forum/showthread.php?t=3038

so we can investigate.

Also make sure Tools | Option | Search | Match anywhere in word by default is unchecked.

https://kinook.com/UltraRecall/Manual/searchdialog.htm

Thanks.

[Spliff]

Thank you, Kyle! In fact, that option was checked, when in practice, I would only need that very rarely, most searches being a full word or a word begin, and "then phrases". Will check if this problem will show again, even before my unchecking the option, it went away more or less.
__________

It seems that my searches are much faster now, it always takes some seconds instead of being immediate, but it's way better than stated above. On the other hand, I don't quite understand this speed progress, since before even my search stated above, I had done "Compact and Repair", together with "Enable enhanced full text", so this should build up the index, which would then be there, and without that index only built up gradually, over days or weeks?

I didn't encounter any more crashes (as described above), did not do any very "big" searches anymore, but I continued to encounter "off times", i.e. 20, 30 seconds or so, of UR remaining non responsive, after searches with hundreds of results. After those "off times", UR reacted normally again, though.

I don't see any option to prevent UR from starting the search before I press ENTER in the search line, or click the search button. This way, I again and again make the mistake to first enter the search string, and then while the search already has started, I specify the search, for example by "only the current item/subtree" or "only titles", and then I have to start a new search. Could you implement such an option? (I did not find any, the only "automatically start search" option I found being for saved searches.)

Also, the search seems to start in the way of "search as you type" which always irritate me since a shorter search string will obviously provide more hits than a longer one, and that means that the search pane partly fills with unwanted results which then will be deleted again. Could you implement an option by which any search only starts by pressing the button or the ENTER key? Obviously, this would only be needed for users who wanted some automatic start of the search to begin with.

I suppose that GLOBAL search for special characters is not possible. (?)

But is LOCAL search for special characters possible? With the default content / rtf component? Or with another rtf component available for users in UR? Is it possible to look out for tabs (by a special character combination? \t or ^t do not work, the routine does not "understand" that it should search for a special character here), for \r\n, \r, \n?

Especially in search-and-replace: Would it be possible to do such a LOCAL search-and-replace, for special characters? This would make sense for replacing multiple blank lines for instance, or to replace combinations of spaces and tabs by just tabs, and so on; currently, the user needs to export the content to MS Word or some other RTF editor, make the changes over there, then reimport into UR.

I get that global search-and-replace is not possible, especially because the rtf content is in blob format, whilst global search does not search in the original data but on plaintext copies of them, so that global replace in these copies would not change the original format. But would it be technically possible to implement some "group search-and-replace", which would only work on the currently listed search results? For exemple, when I would like to globally replace aaa by bbb, I could globally search for aaa, which would give a list of all items which, either in the title or in the content, would have aaa. Then only, this new search-and-replace would be triggered by the user, "replace aaa by bbb within the found / selected items", and this would call the regular, LOCAL search-and-replace dialogue, for the first item within the search results list, and this dialogue would ask for every replacement or not (choice of the user, since it's the current dialogue, as said, and the choice would obviously depend on the risk if there is some aaa in other contexts than the one where the (sub)string would have to be replaced, or if it may safely replaced since there is no such risk of unwanted replacements), but then, when the end of the current item is reached by this (i.e. when there is no further replacement to be made within the current item), the new routine to be implemented, would automatically apply the standard, local search and replace dialogue (be it left open if possible or automatically closed and reopened, if technically necessary to do so) to the NEXT item within the search results pane, and so on: This way, the user would not have a global search-and-replace, but they would be able to continue their Yes/No decisions over perhaps 20 or 50 items where the replace should take place, without having to manually open 50 items one-by-one, and then manually open, and close the local dialogue, i.e. the transition of the local search-and-replace, from one found item to the other, would be automated. Would that be possible? Since, the original content being stored in blob format, the user can not even replace (sub)strings within the content (just within the titles indeed), by using an external SQLite browser.
__________

EDIT:

Speaking of current version and "Match anywhere in word by default" off: Search results are immediate now!

[kinook]

To prevent find as you type, set Tools | Options | Search | Incremental search delay when typing to 0.

https://kinook.com/UltraRecall/Manual/searchdialog.htm

Replace is only available for the current text item (Edit | Replace). A localized replace like you're suggesting is not a bad idea and would be more feasible and less problematic than a global search/replace across the entire database.

[Spliff]

Thank you again so much for your help here, Kyle, problem solved!

And thank you indeed for considering my suggestion! In fact, not being able to search-and-replace (EDIT: beyond the current item) currently is the (one and only) "big fail" of UR, but in practical use, it's very rare that the user wants to "replace everywhere", so for practical means, if the user first does a "local search" (i.e. "this item and the subtree beneath it") in order to set up the scope for replace, and then gets some help, preventing the need for multiple, manual "open and close the local replace dialogue and go to the next in the list", this would a tremendous step forward.

That, with or even without a "global replace all within the list" within the "wrapper dialogue" for this, which should indeed be possible IF the user does use it correctly (understood that no "going back" will be possible):

Any "fully-automated" processing of the list (i.e. by consecutively applying the inner routine to all items within the list without user interaction) would presuppose "Match whole word only YES" and "Match case YES" and come with a "security dialogue" "Cannot be undone, do you want to proceed anyway?" or the like; whilst one or both of these "NO" would open the local "Replace dialogue" for any item in the list, but, as said before, this "inner" dialogue would automatically switch to the next item in the list whenever there is not any more (possible) replace within the current item (always top-down for the items in the list and for the occurrences within the items' contents).

In other words, in the first alternative, the "inner dialogue" would be served internally, and the user would only see the "outer dialogue", the "confirmation dialogue" and the "Done" message - here, with "whole word" and "match case", the user would take the risk of unwanted replacements (but could make a copy before, in case, of the whole subtree to be worked on), and in the second alternative, the user could change the settings ("whole word", "match case", "replace" or "replace all") for every item, but the "next" item, together with its replace dialogue, would appear automatically after more or less "manual" processing of the current item.

I think such an approach would be a very appealing solution for many a user, since "more or less global replaces" would, in almost all real cases, be upon whole words and case-sensitive anyway, or if not, the user could work manually upon the search results, one-by-one, but without the above-described fuss of manually switching between in case dozens of items to be processed.

[Spliff]

Please forgive my above error, I could have found it - and found it now - within the "Search Options" help: "Match anywhere in word by default ... Note that this does not use an indexed search and can be slow."

I obviously had not been aware of this, and with a database of 60,000 items, it's clear as day that this brought problems, unnecessarily indeed, since I don't even need this functionality.

Thank you very much again for your very kind help, and again, the Search has been working perfectly ever since! (Unfortunately, I can't change the thread title, but you could: > "Search problems (resolved)"!
__________

I would like to suggest implementing more shortkeys to the Search; these would be of tremendous help, some of them being of much more use to some users, others to others, and the implementation should not ask for much work I hope?

Please see below; according to your possible comments, or to comments of other users, I could re-think and come up with some other alternatives:

General Menu:
Alt-abefghiortvw = 12

Thus available for Alt-...: cdjklmnpqsuxyz = 14 (Total = 26)

currently used in "Search": Alt-l, n (in "Quick" only), s, u (in "Advanced" only)

("Stop" button does not need a shotcut, since it's also available by "Escape")

"Quick Search":


Fields:

"&Search for:"

Buttons:

Start (per "Enter")
Reset (why not Alt-z, similar to ^z "Undo"? could be written "Reset (&z)")
Copy (why not Alt-c?)
Adva&nced

Checkboxes:

Search titles only (I need to toggle this again and again; no "natural" Alt-key available here, but it would just be needed in "Quick"),
THUS, why not Alt-q (written as "Search titles only (&q)")
IF you are willing to change the &Limit search to selected item in Data Explorer pane to Alt-d ("&Data..."), the Alt-l would become
available for "Search tit&les only", but that's not really needed, I would be perfectly happy with "Search titles only (&q)",
the important thing is that there will be a shortcut, any shortcut, for this toggle
OR "Search titles only (&u)" if you don't want to change, in "Advanced", "Q&uick" to "&Quick", since any "Quick" shortcut will only
be used in "Advanced" and thus will be available for "Search titles only", a command only available in "Quick Search".

and, as in "Advanced":
Match whole words (why not Alt-m?)
&Limit search to selected item in Data Explorer pane (meaning the selected sub-tree, I also need to toggle this again and again)
Search only user-defined keywords (why not Alt-k?)
Include items in the Recycle Bin (why not Alt-y?)

"Advanced Search" (caption is always "Quick Search" though"):

Fields:

&Search for: (as in "Quick")

Buttons:

Start (as in "Quick")
Reset (as in "Quick") (see above)
Copy (as in "Quick") (see above)
Q&uick (Search) (could better be Alt-q instead, see "Store dates..." below)

Checkboxes:

Store dates relative to current date (suggested: Alt-u, i.e. c&urrent, with &Quick instead of Q&uick, is more mnemonic)
OR, if you don't want to change the Q&uick, Store &dates, i.e. Alt-d here

and, as in "Quick" (see above):
Match whole words (suggested as above: Alt-m)
&Limit search to selected item in Data Explorer pane
Search only user-defined keywords (suggested as above: Alt-k)
Include items in the Recycle Bin (suggested as above: Alt-y)