Advanced Search with tree preservation (sort of) - HINT

[Spliff]

As I said some months before, I absolutely need Filtering in correct tree-order, and since Tree-Filtering just applies to flags, I also need some filtering by Search, for other item attributes.

Thus, in view of the problems we discussed, with "Search - Sort by Lineage, then by Tree Order" - here, the longer paths are not interwoven with the shorter ones, but placed behind the shorter ones, so tree order is not respected -, and since I need tree order preservation absolutely, I took out sub-trees to new UR DBs (so this augments the number of my UR DBs even more), then "flattened out" the hierarchy in that sense that in fact, I "flattened IN" the level 2 "core items" INTO the range of the level 1 "section" titles, preserving somewhat visual "hierarchy" with separator lines plus bolding (understood that only the "titles" are black-bolded, other items being only colored / italicised in case, but never bolded), i.e. previously:

0 - source item
1 - title
_2 - some item
__3 - some subitem(s) here
_2 - another item (any formatting)
1 - another title
_2 - some item
etc

now instead:

0 - source item
1 - title
1 - some item
_2 - some subitem(s) here
1 - another item (any formatting EXCEPT bold)
______________________________ (separator line (otherwise empty item))
1 - another title
1 - some item
etc
______________________________
1 - another title
1 - some item
etc
etc
______________________________ (bold line here, not replicated by the forum software)
Templates
Recycle Bin
INBOX
etc.

(Understood that you can further strenghten that (purely visual, not also technical) "hierarchy" by putting some "section titles" into CAPITALS, and / or can bolden (or not) the separator lines, as well as use shorter and longer separator lines (e.g. 20 underlines "_", or then 40 instead.)

NOW, since the "titles" are "bolded", i.e. they bear a specific flag, you can systematically search for them, in order to see the structure, AND you can search for your real target items, viewing the results, correctly "situated" within that "tree" structure, even with separator lines if you want, e.g.

Section 1 (some title)
item with "hit"
another "hit" item
____________________
Section 2 (some title, NO "hit" in here)
____________________
Section 3 (some title)
3 "hit" items here
...
...
____________________
Section 4 (some title)
etc
etc

Now, all this is in perfect "tree" order (since technically, it's not a tree but a flat list), but of course, this only works this neatly* as long as you only search for results within this flat list, i.e. within the "core" items which, to this effect, have been "promoted" to (the original "title" level,) level 1.

(*= whilst also including search results "deeper down" would ask for another search code, and then those results from "deeper down" would not be included in the (simili-) "tree" schema above, but listed below that, i.e. without any "location" indicator)

Now, for the above to work, you'd write an Advanced Search as follows:

Item - contains keywords - YourSearchTermHere
or Flag - equals - Yellow (or your respective "Bold" flag)
or Item Title - matches wildcard - _*
____and Indent Level - equals - 1 (you indent this "and" condition)

or you do it the other way round, indenting 3 of the 4 conditions:

Indent Level - equals - 1
____and Item - contains keywords - YourSearchTermHere
____or Flag - equals - Yellow (or your respective "Bold" flag)
____or Item Title - matches wildcard - _*


and finally, if you want to see ALL occurrences of your search term, of which at least the "core" item "hits" within the "tree" structure, the "rest" below:

Item - contains keywords - YourSearchTermHere
____or Indent Level - equals - 1
____or Flag - equals - Yellow (or your respective "Bold" flag)
____or Item Title - matches wildcard - _*


You see here ( i.e. in this blatant example of "shaping your work to the software", instead of it being the other way round ;-( ) that the (very first of several ones, in case) "contains keywords: ..." condition (and independently of it being for Item or Item Title, or possibly even another attribute / column) within the Advanced Search (AS) , should be available by some additional input field (and which is then accessible by Tab and, in particular, macro-accessible by its own control name, e.g. "Edit11" or similar), the respective value within the AS "list" then NOT being your specific search term anymore, but "[entry]" or similar: Since again and again, users need the same stored searches, but with an individual (i.e. ever changing) search terms, and fiddling with the value within the "Value" field in the "list" here is very time consuming, and copying your 10 or 12 most frequent search terms into 10 or 12 otherwise identical copies of the same search schema doesn't help but to some degree, beyond causing lots of unnecessary clutter. ;-)


EDIT:

I only mentioned it above ( referring to https://www.kinook.com/Forum/showthr...hlight=lineage ), but of course, that's the way it's to be done: for tree results, select any columns, but incl. Lineage (sort by: click the header) and Tree Order (sort by, secondary: control-click the header); and, of course (but with much less "clarity"), you could preserve that special "tree" within your original UR db (instead of creating a new one for it as I did), then adding a condition "Lineage must contain ..." (and adjusting the indentation level condition of course); this would also work for several such special "trees" per db, but I want it "neat", so I created additional DBs instead.

And, of course, making available QUICK entering of the MAIN STRING CONDITION, within ANY Advanced Search (= stored compound search SCHEMA, but for INDIVIDUAL main strings = search terms), would be a tremendous step ahead indeed:

Then, HOW to do it technically? Above, I mentioned the "first currence" of the condition, "contains keywords": this would be weird both for "the developer", Kyle, AND the user (for construction of their respective search schema). Much easier for both:

Allow, within any STRING value field in the list (wherever it is within the search construct), a special entry, e.g. [§§§] or similar (fixed by the UR code, then to be entered by the user exactly like that), and write the necessary code (easy) which "links" that special entry to the current content of the "variable string" field (above the "list"); and (also easy) don't do trigger the Search but an error message whenever the user has put in that "special string" in MORE than just one string value (or any other) "list" field; alternatively, add an entry "Individual value" to the drop-down in the "list" (by "list" I mean the "click-together" part of the AS) wherever a string (or number) entry is allowed (i.e. whenever the drop-down contains an entry "contains key word", or similar for a numeric value); then again, if the user has chosen that "Individual value" more than just once in their "list", upon "Search", instead of triggering the search, trigger the message "Just one value as "Individual" allowed. Please rectify".

That way, the code remains neat, AND the user can start to regularly use stored AS schemata for their individual, always changing search strings! ;-)

[kinook]

I'm not sure that I am completely following that, but could you use the quick search filter field within an Advanced Search to do that?

Can you provide an example .urd file with further explanation or demonstration of what you are requesting? Thanks.

[Spliff]

Thank you very much for your question: In fact, I had left out the problems arising with the combination of the "Search for" field, and then the "list conditions".

Unfortunately, the string condition within the field has to be fulfilled by each result sine qua non, AND then the "list conditions" (lc) have to be fulfilled, as a block, either, i.e. the implicit code is:
(yourstring) AND (whatever your other conditions here),
in other words, there is a big AND between the "Search for" field, and the "list", and that's why, e.g., at the start of the very first "list" line (and different from any other list line), there is NO "and/or" toggle, since the whole list is considered "and".

I admit that for some compound searches, this (very limiting) construct is fine, but for others it's not, e.g. whenever you want, as I do, in my "simili hierarchy within a flat list" Advanced Search, have any (or, as here, even several) "list" conditions alternatively to your (main) string condition, not "above on" it, so, if I want to also display the "construction" items (i.e. the inter-titling, as in my example above), I have to type the search term as just another list element, the "Search for" field being useless = left blank.

In this context, it's also of interest to remind that a simple toggle "and/or" between the field and the list would not resolve the problem, since then again, the whole list would be considered as ONE (not additional anymore, but alternative) condition, and thus would now display too many results, instead of too few. (Since in my example, the "indent level" condition is additional to anything else, whilst the other list conditions are alternative to the string condition.)

That's why I had suggested an additional field "Individual", to contain, in case, the individual string (or perhaps number, in case, for address, or year fields, or similar: individual data vs. "schematic" data, within the other "list" fields) for any (but just one) list field which (by the drop-down list, so that would be easy to add to the drop-down) allows for either string or numeric input: Then, in order to have an input field, the last column in the "list" would mention "IND." or somewhat (selected, as said, by the respective drop-down entry), and IF there is just ONE such list field (for which the user has selected the "Individual" field content), the input of the "Individual" field above, between the "Search for" field and the "Start" button (or even just above the "Search titles only", etc. block of check boxes), UR would replace the "IND." marker by the content of the "Individual" field above;

- user chose "ind." for more than one list field > Error message asking for "just 1 such list field"
- user chose "ind." for one list field, but without content in the "Individual" field > (either error message, or, and preferably,) value / condition is excluded from the sql search string
- there is (previous) input in the "Individual" field, but no condition with "Ind." > "Individual" string is simply not included into the search, since there is no condition which would include it


This arises the question if the "Search for" field could probably be used instead, with adding the usual "and/or" toggle also to the very first "list" row; default being "and" (as usual, too), and the "and/or" would only apply to that very first "list" condition; you would remove the global parentheses around the "list block".

The problem with this alternative would be twofold: Error-prone on the user-side (as is the current "package" indentation within the list, instead of parentheses, indentation regularly indicating subordination, which is not the case here where it's just grouping); and then, the "Search for" field currently only allows for "search string everywhere within the items", neither "title only" nor numeric fields, in other words and e.g., it doesn't allow for "title:Search term to be searched just in titles", which would be a frequent (!) search, especially if users do tagging in titles, and it's obvious that almost every such "tag search" would be for different tags or different tag values, so this would be quite inferior.

In contrast, my problem solution above - needing an additional field - would both be much neater (i.e. less ambiguous) for the user, and would also allow for fast (!), individual, compound searches of specific title data, and of data within numeric, or then specific string fields, i.e. its application scope would be much broader.

(And at the end of the day, the introduction of an additional field would probably even become the very first step for some day allowing sql "select" strings, allowing for quick individualization of more than just one condition? ;-) (User-side AHK input box, then a macro writing the necessary select strings. Would happily share.)

[kinook]

I'm still not grokking it. Can you provide an example with visual mockup? Thanks.

[Spliff]

A) First, you have the input field "Search for:": This is just for general = item-wide string search, can not be used for "search title", for "search item notes", or any other specific field search.


B) Then, you have buttons Start, Reset, Copy, Quick<,


B2) and to its right, there is room for a second input field, albeit not as large as the "Search for:" field.


C) The "list", with and/or (except for the very first row/element), Item/Attribute, Not-toggle, Comparison, and Value fields.
In here, the "Comparison" field is filled by a drop-down menu which just offer those elements / menu entries which make sense for the respective "Item/Attribute" selection, e.g. "Comparison: contains keywords" is present for "Item/Attribute: Item", and others, but not also for "Item/Attribute: Access Count" (since "contains keywords" would not make sens for that); on the other hand, "Comparison: less than" is available for "Item/Attribute: Access Count", but not for "Item/Attribute: Item" - and so on; when "Item/Attribute: Has Children" though, the "Comparison" drop-down just offers "exists" and "equals", and the "Value" column in the list doesn't allow user (string or number) input, but again provides a drop-down, for "yes/no" here.

Thus, it becomes evident that for adding an additional entry "Individual" (or similarly worded) to many (but not all of the) "Comparison" drop-down menus, would be easy, since that additional entry would apply to any "Comparison" drop-down even today, an entry inviting to enter either a string ("contains keywords") or a number ("greater than", etc) is present.

The drop-down "Individual" choice by the user would then automatically fill the "Value" field of that "list" row with a fixed "Individual", not allowing for user input, and if the user wanted to change that, and enter a manual value here, they would have to select another "Comparison" drop-down entry, making available the "Value" field available for manual input again.


Upon "Start" (by enter, or pressing the button), the code would check if there are more than just one "Individual" entries in the list B), and if there are, instead of triggering the search, it would ask the user to change those "Individual" rows to fixed ones (i.e. with a real value within the row's "Value" field, instead of invariant "Individual");

if there is only just one such row though, the code would retrieve the content of field B2) IF there is one, and quickly check if the format applies: just a number if the "Individual" row asks for number input, and any string if that row asks for string input; if there is a string for a number input row, the code would give an error message, along the lines of, "'Individual' input can't be a string here; enter a number, or change 'Item/Attribute' or 'Comparison'";

if there is (residual) input in B2), but no "Individual" row, the sql select is constructed without the B2) string;

if there is an "Individual" row, but no data in B2), then, similarly, the sql select is constructed without the "Individual" row condition.


- My example above shows it very clearly: Really "individual" searches have currently to be made by constructing the search within C), not by combining A) (since A doesn't allow but for too general search scope: the whole item, and for too few logic combinations: no grouping / parentheses allowed in A) with C), and that means often, the user must enter even the main search string of such compound searches within C (where it's cumbersome), instead of entering them in A).

- It might be of some interest, as an alternative to the above, to add the and/or even to C's first row, and then (instead of treating A) and C) as "A AND C" (block-wise)), to imply a "group" just between A) and the very first rows in C) while they are NOT indented, so

A) this
C)
and that
or other
___or these
___and those

would not be translated anymore to

(this) AND ((that or other) OR (these and those))

but to (OR having precedence over AND)

(this and (that or other)) OR (these and those)

In other words, on the logical level, A) would become a part of C), like any row 1, 2, 3... in C) while these rows are not indented.

As said above though, this alternative (avoiding B2) would not resolve the problem of A)'s too global scope, making e.g. individual title:search-string search impossible: the values for string-in-title search would again have to be entered in C), contrary to the first alternative, with B2):

In C), the user would select, for title-search, "Individual", and would then enter the individual value for each new title-search into B2); ditto for any other string or numeric attribute.

That way, the user could create several schema "advanced searches", for different search situations, BUT then enter the individual search string within B2).

Or then, they could enter it in A) IF A)'s functionality was greatly enhanced, allowing for input like
title:search-string notes:search-string,
the code then resolving such keyword:string combinations into the respective column search elements: Such compound searches within the same input field would be a first step to allowing parentheses in there, and would indeed, at some time, make C), the "list", redundant insofar as the "power user" was concerned.

Obviously, B2) is, code-wise and for the user, the simplest solution to the problem; the current "A AND C" (even with an added toggle for "A OR C") though, logically separates the "main search term" from the rest of the conditions, whilst - as shown above - more elaborate compound searches ask for more intimate integration of that "main search term" into other conditions:

My suggested solution, B2) (which, as A does, should also allow for AND and for OR), its value being retrieved by an "Individual" "value" marker in any C) row (as long as it is the only one of its kind), provides that: easy enter and change of the main search term(s), AND permitting that this "string condition" can be placed anywhere in the "select" code.

[Spliff]

It seems to me that the core problem with your Advanced Search construction currently lies in the fact that the whole "list" is treated as a block condition, and with the implicit "and" thus as a SUB-condition for the "Search for" string above the block; this fact makes that for many compound searches, the "string search" will have to be placed within the "list", while the "Search for" string remains empty, and then, adjusting the string to your current search needs - all other "list" rows remaining unchanged for that specific "Advanced Search schema" - is far from easy, i.e. time consuming.

Thus my suggestion above to logically treat the Search-for field as just another, UN-indented "row" of the "list", adding the and/or switch also to the first "list" row (string-field condition always "global" then, not for title-only and similar, which, as said, makes this solution non-optimal, but so much better than the current construction all the same):

string-field condition
and/or other non-indented condition(s)
__and/or indented condition(s)
____and/or (another block of) indented condition(s), and so on.

Obviously, it is possible to programmatically change string values even within the "list", here the value in the very first "list" row:

special-key:: ; to be pressed when the title of the advanced-search schema is selected
send {tab 5}
sleep, 200
send ^{home} ; to assure focus gets back to first row in case
; send {down} ; for second row, or {down 2} for third row, etc.
sleep, 100
send {end} ; set focus to row's value field
sleep, 100
send {f2} ; EDIT: you want to UPDATE this value, so leaving the F2 out would not have been the very best idea here ;-)
return

Trigger the special key (which in other apps can be used otherwise), then enter the string, then 2 "returns" will trigger the search; of course, that first row could be a title:string condition or similar.

It's ugly, but works: the user should adjust the width of the "list's" value column, obviously, making it as large as possible; fortunately, that width adjustment is persistent (if the user takes care of adjusting it when just 1 UR db is loaded).

In the end, certainly most searches could be done this way: Not using the "Search for" field, but using the very first "list" row for the main (global or specific) string condition instead, and then, of course, putting MUCH care into the "condition blocks", separated by different "list" indentations (parentheses would have been easier indeed).