Search problems [resolved]

[Spliff]

My main database has got almost 60,000 items (mostly text, some pics in text, no external documents), almost 1.2 GB in all.

Problem a)

Most (simple) searches take a long or even very long time, albeit I have "enhanced full-text search" enabled. The same after "compact/repair database" with "enable enhanced full-text search features" DIS-abled, and then the same with EN-abled, i.e. after recreation of the index (I suppose this double "compact and repair" in this way, recreates the index).

Needs long time even for single word with option "just in titles", almost 60,000 items to search, but since there is an index, I'm surprised it takes this long.

Problem b)

(Those "Enhancements" always enabled, since I want them to be available in other searches, and switch between enhancements off and on needs compact-and-repair which for enabling the enhancements needs a long time.)

After stopping the search (because it takes too long), the "database is locked" problem. In order to unlock the database then, just closing it will not do (control-F4 does not work then), and closing several databases and UR (by alt-F4) will work, but then, UR will not open anymore (by clicking on the desktop symbol).

Just a complete Windows restart will help, since then, clicking on the desktop symbol will reopen UR (and the databases that were open), and they work as expected.


I suppose there will not be short-term solutions?

[kinook]

Searches should not be slow. We have a database with > 40K items and most searches are under 1 second (and we've never seen database locked when cancelling, although searches usually finish before they can be canceled). Please send as much info as possible from

https://www.kinook.com/Forum/showthread.php?t=3038

so we can investigate.

Also make sure Tools | Option | Search | Match anywhere in word by default is unchecked.

https://kinook.com/UltraRecall/Manual/searchdialog.htm

Thanks.

[Spliff]

Thank you, Kyle! In fact, that option was checked, when in practice, I would only need that very rarely, most searches being a full word or a word begin, and "then phrases". Will check if this problem will show again, even before my unchecking the option, it went away more or less.
__________

It seems that my searches are much faster now, it always takes some seconds instead of being immediate, but it's way better than stated above. On the other hand, I don't quite understand this speed progress, since before even my search stated above, I had done "Compact and Repair", together with "Enable enhanced full text", so this should build up the index, which would then be there, and without that index only built up gradually, over days or weeks?

I didn't encounter any more crashes (as described above), did not do any very "big" searches anymore, but I continued to encounter "off times", i.e. 20, 30 seconds or so, of UR remaining non responsive, after searches with hundreds of results. After those "off times", UR reacted normally again, though.

I don't see any option to prevent UR from starting the search before I press ENTER in the search line, or click the search button. This way, I again and again make the mistake to first enter the search string, and then while the search already has started, I specify the search, for example by "only the current item/subtree" or "only titles", and then I have to start a new search. Could you implement such an option? (I did not find any, the only "automatically start search" option I found being for saved searches.)

Also, the search seems to start in the way of "search as you type" which always irritate me since a shorter search string will obviously provide more hits than a longer one, and that means that the search pane partly fills with unwanted results which then will be deleted again. Could you implement an option by which any search only starts by pressing the button or the ENTER key? Obviously, this would only be needed for users who wanted some automatic start of the search to begin with.

I suppose that GLOBAL search for special characters is not possible. (?)

But is LOCAL search for special characters possible? With the default content / rtf component? Or with another rtf component available for users in UR? Is it possible to look out for tabs (by a special character combination? \t or ^t do not work, the routine does not "understand" that it should search for a special character here), for \r\n, \r, \n?

Especially in search-and-replace: Would it be possible to do such a LOCAL search-and-replace, for special characters? This would make sense for replacing multiple blank lines for instance, or to replace combinations of spaces and tabs by just tabs, and so on; currently, the user needs to export the content to MS Word or some other RTF editor, make the changes over there, then reimport into UR.

I get that global search-and-replace is not possible, especially because the rtf content is in blob format, whilst global search does not search in the original data but on plaintext copies of them, so that global replace in these copies would not change the original format. But would it be technically possible to implement some "group search-and-replace", which would only work on the currently listed search results? For exemple, when I would like to globally replace aaa by bbb, I could globally search for aaa, which would give a list of all items which, either in the title or in the content, would have aaa. Then only, this new search-and-replace would be triggered by the user, "replace aaa by bbb within the found / selected items", and this would call the regular, LOCAL search-and-replace dialogue, for the first item within the search results list, and this dialogue would ask for every replacement or not (choice of the user, since it's the current dialogue, as said, and the choice would obviously depend on the risk if there is some aaa in other contexts than the one where the (sub)string would have to be replaced, or if it may safely replaced since there is no such risk of unwanted replacements), but then, when the end of the current item is reached by this (i.e. when there is no further replacement to be made within the current item), the new routine to be implemented, would automatically apply the standard, local search and replace dialogue (be it left open if possible or automatically closed and reopened, if technically necessary to do so) to the NEXT item within the search results pane, and so on: This way, the user would not have a global search-and-replace, but they would be able to continue their Yes/No decisions over perhaps 20 or 50 items where the replace should take place, without having to manually open 50 items one-by-one, and then manually open, and close the local dialogue, i.e. the transition of the local search-and-replace, from one found item to the other, would be automated. Would that be possible? Since, the original content being stored in blob format, the user can not even replace (sub)strings within the content (just within the titles indeed), by using an external SQLite browser.
__________

EDIT:

Speaking of current version and "Match anywhere in word by default" off: Search results are immediate now!

[kinook]

To prevent find as you type, set Tools | Options | Search | Incremental search delay when typing to 0.

https://kinook.com/UltraRecall/Manual/searchdialog.htm

Replace is only available for the current text item (Edit | Replace). A localized replace like you're suggesting is not a bad idea and would be more feasible and less problematic than a global search/replace across the entire database.

[Spliff]

Thank you again so much for your help here, Kyle, problem solved!

And thank you indeed for considering my suggestion! In fact, not being able to search-and-replace (EDIT: beyond the current item) currently is the (one and only) "big fail" of UR, but in practical use, it's very rare that the user wants to "replace everywhere", so for practical means, if the user first does a "local search" (i.e. "this item and the subtree beneath it") in order to set up the scope for replace, and then gets some help, preventing the need for multiple, manual "open and close the local replace dialogue and go to the next in the list", this would a tremendous step forward.

That, with or even without a "global replace all within the list" within the "wrapper dialogue" for this, which should indeed be possible IF the user does use it correctly (understood that no "going back" will be possible):

Any "fully-automated" processing of the list (i.e. by consecutively applying the inner routine to all items within the list without user interaction) would presuppose "Match whole word only YES" and "Match case YES" and come with a "security dialogue" "Cannot be undone, do you want to proceed anyway?" or the like; whilst one or both of these "NO" would open the local "Replace dialogue" for any item in the list, but, as said before, this "inner" dialogue would automatically switch to the next item in the list whenever there is not any more (possible) replace within the current item (always top-down for the items in the list and for the occurrences within the items' contents).

In other words, in the first alternative, the "inner dialogue" would be served internally, and the user would only see the "outer dialogue", the "confirmation dialogue" and the "Done" message - here, with "whole word" and "match case", the user would take the risk of unwanted replacements (but could make a copy before, in case, of the whole subtree to be worked on), and in the second alternative, the user could change the settings ("whole word", "match case", "replace" or "replace all") for every item, but the "next" item, together with its replace dialogue, would appear automatically after more or less "manual" processing of the current item.

I think such an approach would be a very appealing solution for many a user, since "more or less global replaces" would, in almost all real cases, be upon whole words and case-sensitive anyway, or if not, the user could work manually upon the search results, one-by-one, but without the above-described fuss of manually switching between in case dozens of items to be processed.

[Spliff]

Please forgive my above error, I could have found it - and found it now - within the "Search Options" help: "Match anywhere in word by default ... Note that this does not use an indexed search and can be slow."

I obviously had not been aware of this, and with a database of 60,000 items, it's clear as day that this brought problems, unnecessarily indeed, since I don't even need this functionality.

Thank you very much again for your very kind help, and again, the Search has been working perfectly ever since! (Unfortunately, I can't change the thread title, but you could: > "Search problems (resolved)"!
__________

I would like to suggest implementing more shortkeys to the Search; these would be of tremendous help, some of them being of much more use to some users, others to others, and the implementation should not ask for much work I hope?

Please see below; according to your possible comments, or to comments of other users, I could re-think and come up with some other alternatives:

General Menu:
Alt-abefghiortvw = 12

Thus available for Alt-...: cdjklmnpqsuxyz = 14 (Total = 26)

currently used in "Search": Alt-l, n (in "Quick" only), s, u (in "Advanced" only)

("Stop" button does not need a shotcut, since it's also available by "Escape")

"Quick Search":


Fields:

"&Search for:"

Buttons:

Start (per "Enter")
Reset (why not Alt-z, similar to ^z "Undo"? could be written "Reset (&z)")
Copy (why not Alt-c?)
Adva&nced

Checkboxes:

Search titles only (I need to toggle this again and again; no "natural" Alt-key available here, but it would just be needed in "Quick"),
THUS, why not Alt-q (written as "Search titles only (&q)")
IF you are willing to change the &Limit search to selected item in Data Explorer pane to Alt-d ("&Data..."), the Alt-l would become
available for "Search tit&les only", but that's not really needed, I would be perfectly happy with "Search titles only (&q)",
the important thing is that there will be a shortcut, any shortcut, for this toggle
OR "Search titles only (&u)" if you don't want to change, in "Advanced", "Q&uick" to "&Quick", since any "Quick" shortcut will only
be used in "Advanced" and thus will be available for "Search titles only", a command only available in "Quick Search".

and, as in "Advanced":
Match whole words (why not Alt-m?)
&Limit search to selected item in Data Explorer pane (meaning the selected sub-tree, I also need to toggle this again and again)
Search only user-defined keywords (why not Alt-k?)
Include items in the Recycle Bin (why not Alt-y?)

"Advanced Search" (caption is always "Quick Search" though"):

Fields:

&Search for: (as in "Quick")

Buttons:

Start (as in "Quick")
Reset (as in "Quick") (see above)
Copy (as in "Quick") (see above)
Q&uick (Search) (could better be Alt-q instead, see "Store dates..." below)

Checkboxes:

Store dates relative to current date (suggested: Alt-u, i.e. c&urrent, with &Quick instead of Q&uick, is more mnemonic)
OR, if you don't want to change the Q&uick, Store &dates, i.e. Alt-d here

and, as in "Quick" (see above):
Match whole words (suggested as above: Alt-m)
&Limit search to selected item in Data Explorer pane
Search only user-defined keywords (suggested as above: Alt-k)
Include items in the Recycle Bin (suggested as above: Alt-y)

[kinook]

Many of these are already used. For instance, Alt+C is the default shortcut for Calendar, Alt+D for web address combo, Alt+Z to insert date/time, etc. And users can create other Alt+key shortcuts, which could conflict with search mnemonics.

[Spliff]

Sorry, I just discovered your answer now, my inadvertency.

I am aware of that, sorry for not commenting on this "problem" in my post above. But I think that within the special "context" of the Search Pane (not also: Search Results Pane) having focus - both user interface wise and technically (UR internals) -, the user would NOT be tempted to get to those functions, so their otherwise global shortcuts having another function within this special context would be acceptable; this being said, in order e.g. to toggle specific check-boxes there (e.g. "title only"), not only Alt-shortcuts would / could work, but also Control-shortcuts.

This being said, I suppose that with external macros, I could toggle "Button8" (for this "Search titles only" example), etc., will have to look into this; the latter not being a "general public" solution obviously.

[Spliff]

I would like to make a search scope suggestion.

When I am in a hoisted tab (i.e. 2 tabs, tab 1 comprising the whole tree, current tab 2 displaying a (in case, quite considerable) sub-tree), "Limit search to selected item in Data Explorer pane" does exactly that, so if I want to restrain my search to the hoisted sub-tree, I first must select the "source", the hoisted item for the items currently displayed, whilst the general search, even in the hoisted tab, works on the full, original tree, which in almost any use case is not what the user will want (I'm speaking for other users here, but I'm quite sure I'm not mistaken in that).

So my suggestion would be that general search within a hoisted tab should, "behind the scenes", apply the "Limit search to selected item in Data Explorer pane" functionality (= which is already there, so technically, it would be easy) to the source item of the hoisted tab, independently of which item in the hoisted tab currently is the current item.

Selecting the "Limit search to selected item in Data Explorer pane" would then, even in a hoisted tab, continue to apply to what it says, i.e. further limit the search scope to the current item and its sub-items.

And in the rare use cases where the user, from within a hoisted tab, would want to search by general scope, they would have to switch back to the un-hoisted tab, but then, if you want to work upon a search result beyond the scope of the current hoist, from within a hoisted tab, that causes problems anyway, so my suggestion would at least create a neat scope for the search results; currently, if you don't first select the source item before your "hoisted tab" search AND check "Limit search to...", you not only also get search results from outside of the hoisted sub-tree, but, even more of a problem, you cannot even identify those visually from those within the "hoist" (the possible column "Parent Title" just showing the immediate parent item, so it's rarely a reliable indication). Thus, I think that my suggestion will, after all, not harm any current use case but enhance the alleged "user experience" for any user.

EDIT: If really there was a risk that some users could not be happy about that change, there's always the possibility to create an option in Tools - Options - Search, "In hoisted tab, don't apply general search to the hoisted part only." or something like that... I'm sure nobody would check that option then. Or then the other way round, with everybody checking the option consequently.

Perhaps, and if an option really should be needed, a wording like "Search applies to full tree even in hoisted [better: "hoisting"?] tabs" would probably be best, with this option NOT being selected by default.

[Spliff]

"Perform phrase match by default" (in Tools - Options - Search" (set to ON)) is not always reliable.

I have got searches for "Art. 10" for example, any number here, "Art." short for "Article" but as "Art." in the text.

If I search for
"Art. 10"
I get correct results, but if I search for
Art. 10
(which, according to my option setting, should bring the same results), I get endless lists of items where just "Art. anynumber" occurs, AND where anywhere in the text, the number in question also occurs, so for
Art. 10
the option "Perform phrase search..." does not work.

The dot after "Art" may be the culprit here, haven't got the time to look deeper into this. Is there any known explanation, so that we can foresee the situations in which this will occur? Is the way the elements are indexed playing here?

As said, no problem with enclosing "".

[kinook]

I was unable to reproduce this behavior. Using the attached database, with 'Perform phrase match by default' checked, there are 3 results when searching on

Art. 10

(matching only items that have "Art. 10" in the item text) and 5 results with that option unchecked (also matching items that have Art. and 10 somewhere in the item text).

Please send the info from

https://www.kinook.com/Forum/showthread.php?t=3038

[Spliff]

Thank you for answering, Kyle! I must admit that after rising my main database's item count over 100,000 items (which caused real problems, see my thread "Reasonable database sizes"), and then setting it back to about 75,000 items (with the only (I thought) - persisting - problem left that after ^x, the item's icons are not greyed-out anymore - I decided I could live with that), I have NOT yet created a NEW main database, into which I then would have imported all my 75,000 items.

Thus, the behavior I described above, may be another "database having been too big" phenomenon indeed; before mentioning possible other glitches I may discover, I'll note them somewhere in-between, will re-create my main database, then check again if the problem persists, then only will mention it here in case!

(After the resizing back down to about 75,000 items, I obviously ran Tools - Compact and Repair - Enable enhanced full-text search features", so whilst this obviously - since you weren't able to re-create the phenomenon - is not an indexing problem (possibly connected to indexing of abbreviations and/or numbers); my command would have re-created the full index if I'm not mistaken), it should be connected to some "general instability residuals" after my blowing the database size up too much.)


EDIT: I now have created new databases, i.e. I copied most of my items from the "problem" database into a newly-created one, and some into another newly-created one, so my main database has got currently about 60,000 items. Since the flag formatting is database-specific, I also had to re-create the flag-formattings for both newly-created databases (I should perhaps create an empty database with those formattings, then copy that for filling it up with possible moves.)

Problem "^x will not grey-out the icon": Persists in newly-created main database with 60,000 item, does not persist in newly-created, tiny database. Thus: UR has problems with big databases in this respect; I can live with that, but it's useful to know about it. Obviously has nothing to do with previous over-resizing(s).

Problem "Default phrase search by option also finds the parts of the phrase": Is more general (in all three databases, the "source" one and the two newly created "target" ones): ANY phrase search gives unwanted results, any
wordone wordtwo
phrase (and, as said, "wordone wordtwo" works as expected):

Both parts are highlighted within result items, i.e. any occurrence of wordone, and and occurence of wordtwo, and even when wordone or wordtwo (I checked both) are in the middle of another word, e.g. abcwordtwoxyz, wordtwo in this example is highlighted.

On the other hand, it seems - I checked numerous such results - that in order to list an item as a search result, both parts, in correct order, have to be there as a phrase, i.e.
wordone wordtwo
exactly in this order, anywhere in the item in question; it's just that the part (and even within other words) are all highlighted this way then.

When I include the phrase within "", just the phrase occurrences are highlighted though, which is the expected behavior.

I can live with that, i.e. I just do phrase searches the traditional way, in "". (Here again, I had re-installed UR even before, so any trace of my "125,000 items in a single database adventure" should have been gone now. Perhaps it's "my system", in some way or another... as said, I can live with those glitches.

[Spliff]

Re Shortcuts for "Search titles only", etc. (see above): Non-mnemonic shortcuts are better than are no shortcuts at all. ;-)

And I had to realize that AutoHotkey "ControlClick, Button8, ahk_exe UltraRecall.exe" and variants of that are totally unreliable, while it's evident that for many UR users, "just titles" is an important switch, and for others, "whole words" may be...

As for an intermediate solution, UR users could use something like controlsend, Button8, {space}, ahk_exe UltraRecall.exe, which makes a reliable toggle.


Re the Search Results pane (which you will not like to stay beyond situational usefulness if you don't use the Child Items pane anyway), I tried some ways of coping with the problem (to have to first activate it by ^3, then close it by +F4, since a global "hide" command is missing; it's true that some users might also wish for such a thing for other panes, but then, they will probably want to see those anytime, whilst the search results are quite special: everybody needs that pane, but JUST after searching), including the command for automatic hiding of the pane in question (it's in the context menu for each pane), and I was not happy with the latter at all, so it's scripting again, but I'm sure a special "hide search results" command would arrange almost any user.

All the more so since in Autohotkey at least, I didn't find any solution for sending the necessary +{F4} to that pane, neither by any of several "send" commands, nor by controlsend. Thus, my macro has to retrieve the position of the pane, then add (about) 12 pix to the horizontal position and subtract (!) 12 pix to the vertical position (since I finally need the context menu of the caption, not of the list), then I send a rightclick to that position, and then, finally, I can send a "c", for "close (the pane)" to the pane's caption's context menu - that only works... Or then, some UR command "toggle the ^3 pane" (children, search results; ^c shows it in case, then sets focus to it: that's obviously different)... ;-)

Btw, I had also tried (manual) double-click on the ^3-pane caption, and that was overly (exceedingly) successful: Neither ^c nor a new search brought that pane up again, I had to close and re-open UR. I then did the same thing again, in order to verify, and again, the pane disappeared for good, and not even closing and re-opening UR did help: Perhaps the double-click, or some other means obviously had made the pane "floating", and it was now shifted beyond my screen (to the right of it), even for searches and child items display of other (!) databases. (Good to know it's beyond the screen (and thus is retrievable) if it has disappeared for good, then.)

And finally, when it's easy to close that window, the user might even resize its size and position - this comes handy with long search results lists -, e.g. placing it all over the Data Explorer, given the fact that navigation to some search result in the latter will kill the search results lists anyway; in order to situationally resize according to your respective needs, you'd need scripting though, again. (UR has a functionality for several "layouts", so if you can easily change them on-the-fly, that might be another way of doing things.)


Re the highlighting of the search results: Stable since a week or so, and not having done anything with any setting (or then, inadvertently perhaps?), the highlighted search results are black on RED, instead of black on yellow, and that of course infers with readability, besides the fact that it's aggressive on the eye. Any ideas as to possible causes?


Re highlighting the search results terms again: As we all know, any editing of ONE result item also breaks the highlighting of the terms in the other results. I don't know if doing away with this very unwanted behavior might be challenging, code-wise, or too much of recoding work?

[kinook]

Select another highlight color to change it to something else.

[Spliff]

Where would I do that?

I have looked into https://www.kinook.com/Forum/showthread.php?t=2267 ("How to change the fonts & colors in UR"), which speaks of Windows settings, but those have not been changed, and in my browser for example, selections are white on green, as always.

Also, in that link, it says, "Different variations of window colors and styles can also be chosen by changing the Paint theme at Tools | Customize | Options in Ultra Recall." - I looked into this, but didn't find any such setting in there.

It's just that in UR, search terms in search results are now black (not white e.g.) on red, instead of black on yellow, as before, and for no apparent reason.