Any way to create a compound (flag) search preserving the tree structure? Or in tree?

[kinook]

It's not clear what you are requesting. Can you explain it again, in plain English, with fewer words if possible. It seemed clear what you requested previously, but now you're saying what was added isn't useful. Thanks.

[Spliff]

I try to do a summary here, numbers as above:


1) TREE filtering instead of search/filter is not ideal (my bad in my first post) since user (and/or developer) would get into trouble by editing the filtered view, so no editing but endless forth and back between filtered view and complete view, and the complete view is the whole, expanded tree, so navigation problems in order to do the wanted edits: user's work would be possible but very cumbersome.


2) Thus, I suggest to add the necessary code to SEARCH. Can be done by an intermediate sql "view" with (virtual = just here, not stored) numeric path column by numeric path representation, then sorting by that (1.1.4.1 after 1.1.4 and before 1.1.5, currently it would come after 1.1.5 which breaks the tree preservation).


3) I present 3 ways of doing TREE filtering (see 1), a, b, c:

c being a flat list (possible if you really don't want to do the SEARCH thing (see 2),

b would be too demanding code-wise since for also graphically maintaining the tree, you would need to also show all "hidden" items (in light grey) but which would be necessary for the graphical tree representation of the non-hidden items

a (which you currently chose) is unacceptable since some non-hidden items ARE hidden though, since they are children / grandchildren / etc of hidden items (which are left out), so if you don't want to do c = flat list but graphical tree representation, you can't left out non-hidden items but must also preserve "hidden" items to "hold", to "situate" the non-hidden items = solution b = lots of code

To make this as clear as it gets: Currently, you do a = seemingly preserve the tree but hide "non-hidden" items which "depend" from hidden items, so it currently does NOT make sense, example:
hide everything except red and green
a is red, child b is green, you correctly show both
but a has another child, c, which is black, and which has a green child d on its turn;
you hide the non-to-be-hidden item d since you hide its black parent c

Hence the (complicated) solution b, or the (simple) solution c, here in TREE solution.


4) BUT as said, solution c in the tree (= flat list) would correspond to the SEARCH solution (from which you refrain because of the necessary, demanding coding), AND I say, since it's the similar code anyway, in the SEARCH results solution AND in the TREE solution, please do it (= the flat list, preserving the tree structure) per SEARCH results, since then

- the user can work concurrently on BOTH panes: for their deciding what to edit / move / etc they would consult the search results, and for their edits / moves / etc they would have the tree available in the state they need it to be for these edits, moves, etc

- the user can use ANY search for this work (e.g. searches for keywords, tags, whatever) for this editing-and-shuffling work, whilst if you do the flat list within the tree pane, this would only apply to flags, so the "usefulness scope" of filtering-while-preserving-the-tree-structure would be greatly enhanced, compared to the flat-list Data Explorer solution (with no extra developing work compared to the latter).


As said, "TREE a", your current "solution", is not a solution, since it hides items which are NOT to be hidden, just because they have the "wrong" parentage (which in a flat list would be easily hidden indeed).

So, "TREE b" (= also show the necessary "in-between" items) is complicated and not worthwhile, "TREE c" (=flat list in Data Explorer) would be possible but not easier to code than the flat-list "SEARCH" solution, and quite awful for the user, hence my suggestion to implement a SEARCH OPTION "preserve structure" (or "preserve hierarchy") (in the flat list, not also graphically of course), as another checkbox in the pack "Search titles only", "Match whole words", etc., and which would OVERRIDE any current column sort settings.

(I also said the additional code, with the creation of the intermediate SQL "view" from which then the "hierarchy sort" is done, should, of course, only triggered whenever the option "preserve structure" has been set for the search / filtering.)

This would then be available for ANY search / filter, not only for flags.

I hate to say it, but the developers of RightNote and MyInfo both implement quite nice features on a quite regular basis (especially MyInfo, whilst RightNote lately added the option to have TWO, concurrent Data Explorers, for alternative views, and which ironically, if we had them in UR, would invalidate my reservations re the TREE solutions (Tree b and Tree c, whilst current Tree a cannot be), whilst on some core functionality NOT being in the class of UR, so it's not a bad idea to ENHANCE and develop the STRONG points in UR's interface and functionality, unparalleled over there, and structure preservation, even as a default, might be considered a core functionality anyway - by which I don't wanna say that alternatively, all the numerous, but all structure NON preserving columns sorts weren't needed.

Again, "do it right" instead, in the Data Explorer, would be possible, but not less work than to put hierarchy preservation as an option into the search... which would be so much more useful for the user: much better workflow than Data Explorer toggle, and availability for ALL searches / filterings.

[Spliff]

It's not precisely a suggestion, but technically 2 bug reports, since it was said to be there in 2017, then again in 2021 (links above):


Needed: Filtering (can be called search when done by Search)

Schematically:

"Tree preservation" = bad, ambiguous expression since:

CP = Construction preservation = NEEDED = all items in their ORDER as in the tree (Data Explorer), even by flat list

GP = Graphical preservation = NOT needed = additional preservation of the indentation (instead of flat list)


2 Solutions:

TREE = in tree pane = Data Explorer, just possible for flags

SEARCH = in Search Results Pane, possible for ALL possible search criteria, thus preferable


TREE:

3 ways:

a) CP with GP, no intermediate items = current "solution", with (unavoidable) bug (proof above): non-hidden items are (incorrectly!) hidden though when intermediate elements (needed for the GP) are (correctly) hidden > NOT a solution

b) CP with GP, intermediate items necessary to "hold" the non-hidden items might be greyed out (but must be shown) > elegant but unnecessary solution with demanding code

c) CP only = flat-list = the solution to be preferred, but:


SEARCH:

a) current "solution" (presented in 2017 and again in 2021: sort results by columns lineage, then tree-order) is buggy since it doesn't do CP (proof above)

b) suggestion: implement another search option, "CP", overriding column sort, technical realization: build up numerical-path for intermediate sql view, then sort the search-pane output by that virtual column; code a little bit demanding indeed, but should be similar to solution TREE-c (flat-list in Data Explorer); preferable to that since 1. concurrent working in both panes possible (instead of TREE-c: endless forth-and-back between different content within the same pane), and 2. filtering by ANY possible criterion (even in combination of criteria, vs. just by flags in TREE-b or TREE-c)

(GP possible here, too, but neither needed in TREE nor in SEARCH)


RESULT:

UR becomes the only "outliner" with real, multi-factor tree filtering. (Some competition offers tag filtering, with construction not preserved.)

End of my bug report ;-)

[kinook]

What you're requesting negates one of the fundamental goals of UR, which is high performance. A lot of work is done to optimize populating the tree and search results, but the only way to implement what you're describing is to always load every single item in the database, in order to determine which ones should be excluded. This would add overhead of about 1 second for every 2.5K items in the database, for every search or refresh of the tree. Currently, loading the tree or search results is typically instant, even for databases with thousands of items. And the effort to implement this would be quite large.

Sorry, but it's not going to happen.

And it was never intended for fully replicate the tree in the search results, so I don't consider this a bug.

[Spliff]

Thank you for the info, so info in the two linked threads must be considered obsolete.

Thus, filtering in Data Explorer (now, thankfully, also possible for default items) has to be made carefully since "filter orphans" are not displayed; the user might not be aware of that.

Combination of two flags, first for construct, second for some core info or the like, remains possible IF the user makes sure they have no third flag / item "format" in the respective parentage of the flagged items to appear; this would mean that the user might be driven to FLATTEN out the hierarchy (which makes the navigation more cumbersome, but that's obviously the only way).

The really awful (sic!) aspect of this otherwise absolutely helpful feature is the fact that in-between, when I toggle between filter view and regular tree view, the tree is not anymore in the form it was before the filter view, but is completely expanded; perhaps the filter view toggle could store, then restore the tree in its previous state before filtering? Feasibility also depends on the existent code base, of course, which I obviously do not know.

[kinook]

Originally Posted by Spliff

...when I toggle between filter view and regular tree view, the tree is not anymore in the form it was before the filter view, but is completely expanded...

Can you elaborate on this? I am not seeing this behavior when switching between showing or hiding flagged items, if that's what you're referring to.

[Spliff]

Yes, that's what I meant, but I have to admit that now, by trying again, it's back to normal = expected behavior again, so that was my bad, sorry; I must have done something wrong.

On the other hand, I have spent some hours with trying to make the tree filtering functionality useful for me, to no avail; in order to see two flags and their occurrences in correct order, they would have to be all within the totally same "lineage" = the same long, very long siblings' list, and ditto for filtering by search, even with numerous tries with additional user attributes, it's all totally futile.

I then trialed some competitor, just for comparing, and what I found was hair-rising indeed:

Construction preservation seems to be absolute standard, default, both in flat-list form or in indentation-preserving = visual tree form.

Here's what I found - you will recognize the software, but I don't write this for advertising reasons:

"Global Search", then scope: In this section (would be a "tab" within the current db), In this notebook (=default, would be UR's current db), in open notebooks (= open DBs), in all notebooks; already scope-wise, this is perfect, and obviously, the developer took the additional effort to program the necessary code to gather the multiple SQLite indices, in order to make such scopes available to the user.

Then sorting, called "Group by": section (= default), title, date updated; title is titles by abc, date is dates by dates, and section is a flat list, preserving the construction order, in other words, not multiple sort orders, as in UR, are made available, just 3 fixed ones, but incl. the absolutely necessary construction order, which doesn't even have a name other than "section", so totally it's considered standard; then (not tried out), if you have scope db, open DBs or all DBs, the construction order in every section will be observed, just put one behind the other; also this would be user-side expected behavior.

I tried this with the search strings title:x (having added the x to just some titles), and with title:x OR title:y (having also added some y to other titles).

(Btw, I created my trial items in total disorder, in order to avoid a possible "by chance only" "tree order" sort which might have been produced by the creation order, and I added the x and y afterwards, also in disorder, so the program's "last change" date order doesn't play a role here, the default sort order is really the (current) construction order, nothing else.)

Then, I applied the same search strings (just x, or x or y, in the title) to the tree! Since the tree pane has a filter function, input field always visible, "Filter section notes", and here I was eager to see how the developer did the tree filtering (see a, b, c above): It's a simplified b over there, i.e. it's in tree form, together with the tree's indentations, and together with all necessary intermediates needed for showing ALL non-hidden items, i.e. there are no "filter orphans" over there, which, as we had to discover, are hidden-against-expectation in UR's tree filter.

Thus, the competitor does it all perfectly right, and even the tree filtering with any attribute or attributes combination, from the Filter field, so combinations could be automated by external macro, ditto for the Search field, whilst in UR, compound searches would have to be clicked-together (as described by me in another thread), which presents, for stored searches (which are possible in UR), the additional problem that it's then quite difficult to run those just with new values.


So now for the "bads" over there: I haven't discovered "flags" or any other way of formatting tree items (I mean the items' titles within the tree), and I absolutely love UR's way of doing that (now up to 20 formats ;-) ), in fact I can't do without, had tried for some time in some other software, NEED UR for this feature.

Also, with near half a million items in UR, I'm not ready to switch, that's for sure. On the other hand, I'm currently musing about switching to the other app just for that part of my work where I would need correct construction preservation in item filtering, not even filtering by more than one criterion, but by more than one VALUE or the same criterion (here: flag = red OR green, i.e. show both, and all of them, and in the correct order); I'm hesitating because I would then, for that part of my work, lose the aforementioned "tree formatting" for different kinds of items; I could (easily, as described) filter for them, by tags (which are available there) or simply by title tags (like my trial x and y), but UR's fine "distinguish'em by a glance" would be gone, and that would be, in my case i.e. from my needs, more than awful, even and also in that part of my work.

Btw, when I say "order preservation is (to be considered) standard", it's not just for what I just discovered within the competitor's search and tree filter, but it's how the search results are presented in any editor, any dedicated search tool (provided they come with a line view of all hits) by default: in their original occurrences' order, from first to last line of the text body, and any "grep" does work that way.

Thus, UR's omission of that "natural sort" obviously is very non-standard, AND a real problem; btw, with my (systematic) lineage-then-treeorder sort (which was deemed to be "tree order", since we believed the linked threads' allegations), I never succeeded in finding the relevant positions within the tree (Data Explorer) on my own, just from visually navigating from search result table to Data Explorer (which would have preserved the search result table), but had to press Enter on one of the results in the results table, in order to navigate there, and which then destroyed the results table (just navigating within the results table will only display the respective content, but will not "situate" the item within the tree): no wonder, since the results in the results table, deemed "in tree order", were sufficiently "mixed up", in order to lead me astray.

And that (not assuming the results were not in correct order, so no systematic check, verification, and the disappearance of the result table whenever I then navigated to the item in question) was responsible I lived with that for quite some time... whilst my current combination of TWO flag values, one for construction and one for the items-of-interest in that filter / search, I immediately discovered that tree-order is not.

I don't know if the competitor also applies (your) adjacency list model, or some path model, but I assume it's in your interest to check how he does it, SQLite-table and -columns-wise, in order to not encounter your coding problems for preserving tree order in search and filter results.

As for possible speed problems, I admit my trial db over there had less than 20 items, not 30,000 or more, as some of my UR DBs do, but above, I had said, in my first, and then again, more to the point, in my intermediate post, "(I also said the additional code, with the creation of the intermediate SQL "view" from which then the "hierarchy sort" is done, should, of course, only [be] triggered whenever the option "preserve structure" has been set for the search / filtering [in question].)" - so perhaps, speed problems would not need to spread all over the place, or then, some modification of the Adjacency List model might do it, even if that then (necessarily) slows down moves of whole sub-trees (which don't occur that often after all, once the users will have "organized themselves somewhat") a little bit?

Let me say here that I admire your solution how you have implemented (item, and whole sub-structures') cloning, from what I from your tables and your columns: That's really ingenious! So it would be quite a pity if you coudn't also find a viable solution for the - as alleged by me - standard tree preservation in search, and filter, results? ;-)