Any way to create a compound (flag) search preserving the tree structure? Or in tree?

[Spliff]

a) Within the TREE itself,

I can HIDE several flags, so that in the end, in theory, I have just a tree with the flags I want to appear. I can't hide regular (non-flagged) items though, so in order to realize what I need, I first have to assign a special, "regular" flag to all regular (non-flagged) items, then hide that "regular" flag, too.

It goes without saying that this comes with lots of side-effects, since your default item would now be a specially flagged item. Also, all this is entirely manual, since there are no STORED hide sets; a little menu with some 5, 6 (or even 12) entries for such sets would be welcome, with entry 1 = "everything is displayed", entry 2 = "hide flags 2, 4, 7", and the like.

This being said, it would be, or have been preferable to have such sets not in the form "hide flags x, y, z" but "SHOW flags x, y, z"...

BUT at the end of the day both versions would be practically the same IF the format "NO flag" (= regular = default item) could be INCLUDED in the flag list(s), be it for "show" or be it for "hide", in other words, in the (currently unique) flag list (currently for hide) it's the "NO flag" which is missing, so cannot be selected for hiding.

Adding this feature, "hide No-flags", within the unique hide-flags list, might be quite easy to implement? And, as said, some different such lists (sets), available by a menu, for different "views", would be welcome.


b) Within the SEARCH (compound searches, stored searches),

it currently seems impossible (?) to preserve the tree structure; several times in my 2 years with UR, I have tried, for hours, but I always hope I have overlocked the necessary "trick", or then, would it be possible to implement some system attribute giving access to tree preservation? (Of course, I ask for b) since current a) solution is so awkward, so if a) could be amended, b) would not be so important anymore, or then the other way round: if b) is / could be made possible, a) is not so important anymore.)

We now know that for a SIMPLE search, preservation of tree structure would be realized by sorting by lineage, then by tree order, e.g.
Item Title // Lineage (sort 1) // Tree Order (sort 2, means "child number") // other columns in case

This FAILS for compound searches, also when I try (perhaps mistakenly) with additional columns (IsObject, IndentLevel, or others):

Let's have a (stored) search for
FLAG = Red (for construction elements in different, higher-up indentation levels)
OR (which means "and" in the sense of "show items of either flag value")
FLAG = Green (for specific child/grandchild... elements for which I want to see the "distribution" within the structure)

Even in this really simple compound search the tree structure is NOT preserved: the "green" elements, by tree being direct or indirect children or grandchildren, etc of the "red" elements, do NOT appear "in place", even when there is NO missing = filtered-out element in-between, in other words, even if the green element is a direct child of a red element, it's not displayed directly beneath its parent but elsewhere = further down, and similar for the "construction" = red elements, which seem to be sorted by indentation level in-between themselves, and even some green elements appear in-between them IF their indentation level is high enough to appear in this upper part of the result list.


Of course, the b) SEARCH solution is of high interest here since they can be stored = be easily accessed then, and filtering out even regular (non-flagged) elements is a no-brainer here, whilst the (in theory, ideal) a) TREE solution (where filtering preserves the tree) currently has no such "stored" procedures (filter presets), and presents the additional "fail" of regular (default) elements not being filtered out even when necessary (interesting though, the filter code within the tree functions the way I would have expected it to work in search, too). But there hopefully might also be a current UR search solution, adapted to UR's specific search code internals?


EDIT

This might all be a big misunderstanding, since even a single flag search does NOT preserve the tree structure, with
ItemTitle, Lineage (sort 1), TreeOrder (sort 2)
listing all shorter lineages before longer lineages (i.e. ordering by indentation level even when that column is not even selected from within the "Choose columns" dialog; for finding elements (records), I might have not discovered it before, just assuming
ItemTitle, Lineage (sort 1), TreeOrder (sort 2)
displayed the search results in tree order (meaning: preserving the tree order from the data explorer), but for my needed filtering tasks, the fact that the structure of the tree is in part scrambled becomes obvious to the point of the result display not being of any use (the above described combination red-green becomes somewhat "better" visually (since then red and green are so much interwoven that you could think at first sight that it's all in tree order, but it's not, not at all) if I sort by TreeOrder (which in fact is "SiblingsNumber") first, then by Lineage, but it's just scrambled otherwise). Is there something wrong with my UR installation?

I am perfectly aware that for SIMPLE searches, the above does not correspond to what had been said in
https://www.kinook.com/Forum/showthread.php?t=5758
which shows that for me it's been a problem for more than 2 years now, and in
https://www.kinook.com/Forum/showthread.php?t=5462
which shows that it had been a problem for others even before.

So after 5758, we all thought that the problem, for SIMPLE searches, had been resolved. For COMPOUND searches, though, I then had thought this was a new problem, not having been treated before, whilst I now discover, on my system, that even for simple searches, the problem persists (btw, ALL ordering always in the way that the triangle shows UP (which means ABC), not down (which would mean ZYX)):

simple search for red flag, sort first by lineage, secondarily by "treeorder" (numbers 1 and 2 clearly visible, so I don't just assume but see on screen).

Real Tree Order (with indent levels):
0 (1) (first level below source-item)
p (2)
c (3)
z (3)
m (3)
y (3)
1 (2)
2 (2)
3 (2)
4 (2)
x (2)

Search result:
0 (1)
p (2)
1 (2)
2 (2)
3 (2)
4 (2)
x (2)
c (3)
z (3)
m (3)
y (3)

(For compound searches then, the (faulty) search results are as expected from the above simple-search results, but I now think that if the problem is solved, for simple searches, the compound search will produce flawless results, too.)

My data above representing (sic!) real data, i.e. longer strings, but with, in the real (!) tree, any greater indentation number (all 2's after the 1, all 3's after the 2) being a real child of the preceding lesser number (the 1 above them; the 2 above them), meaning the respective substrings before the respective "/" in the real lineage strings are identical, in other words, I just shortened the real data, but the example comes from real, real-world data.

[Spliff]

(Can't edit / add anymore to the above)
New (default) db, just 4 items. Obviously, our assumption (after the 2 linked forum threads above) it could be that easy, was wrong, but we just didn't notice, whilst it becomes evident with (but is not caused by) compound searches.

[kinook]

In the latest download (6.3.0.7), an option has been added at Tools | Flags to hide unflagged items when not showing flagged items.

Regarding fully replicating the tree content in the related items / search results pane, the available columns don't provide enough information to accomplish that.

[Spliff]

Well...


1)

I know /assumed there currently is not enough column info, that's why in-between I had suggested adding a new column; this was visible for some time here, but was part of a part I later edited out, I see now; I've been (and am) aware the necessary code will be very compact AND very demanding, the respective SQL is highly complicated (as I have known by my efforts to write such code myself, I just got to 90 p.c. of the wanted result); but then, since you do it (almost, see below) right in the tree ("Data Explorer"), it obviously can be done, albeit not on-spot.

I made a mistake by saying to do it within the tree itself was "ideal"; even before trying, just after reading you, my mistake occurred to me: In fact, we need the filtered-tree ASIDE to the tree, not in place of it, why? Because from the filtered-tree, we see what we (might) want to do within our tree, thus, if we have the two in two concurrent panes, we can toggle between filtered-tree, for (further) analysis, and tree, for doing changes;

technically, the same is true if we switch forth and back within one (the tree) pane, but if filtered-view, we (regularly, not systematically) need the complete = expanded (but filtered) tree, and thus the toggle "View - Show flagged items" will then show that complete, expended tree > "lost in hyperspace" > lots of searching / navigating after every one such toggle-back;

also, an exclusive (sic!) tree solution, = a) above instead of b) above (instead of both solutions, eventually), represents the inconvenient that tree preservation would only remain possible for filtering by flags, but not also for filtering by other criteria, e.g. "tags", e.g. some tag codes within the titles or elsewhere, or any other search element; it's obvious such functionality - tree preservation for ANY search / filtering would be more than welcome (all the more so since even, see the two links above, it was asked for by other users already in 2017, and assumed by all of us as "being there" while in fact it was not; it's just that currently, me trying a combination of two flags, one for the "construction" and the other one for "content specifics", it's become clear as day that the solutions we all believed in, in 2017 and 2021, don't give the results we're after.

Also, see 3) below in this respect, since in tree-preserving filtering, we will have problems within the tree which are absent within the (flat) search result list.


2)

Thus, I suggest to create the necessary code for tree preservation in searching / filtering, perhaps even then as an click-box, overriding the current column sorts, working as a toggle then (even if for the toggled view, the search has to be started again).

Btw, the new column could be a (= another) virtual one, and just be filled (in work memory) in cases the toggle "preserve tree structure" is set to "on", so as to simplify (and fasten) the proceedings when it's not needed - since in that intermediate SQL "view" (behind the scene), some tree column 1.1.1.1, 1.1.1.2, etc. would be necessary to be built I suppose.


3)

"since you do it (almost, see below) right in the tree" - why did I say that? Because I had assumed that a flag-filtered tree would show ALL NON-filtered items (I called them "elements" above, I mean "items", of course), and that's currently not the case in your current solution a) (I have checked.).

As I have already mentioned above, in the FLAT tree-preserving search results list, we don't encounter any problem to display ALL NON-filtered items, we just leave out the filtered items.

Otherwise in a filtered tree (which preserves the tree structure visually): Here, we have 3 possibilities (as the designer / developer):

a) We can simply leave out non-filtered items, too, if the necessary parenting (sub-) structure is filtered out currently, and that's been your current, quick-n-dirty "solution", which unfortunately makes the filtered view unusable, since a filtered view is about prominence (sic!) of the non-filtered items, not about aleatorically hiding them with the rest I'm afraid.

b) We can leave any partial substructure (= the intermediate-level parenting items), needed for correct tree positioning of the non-filtered items, untouched, perhaps (just for the screen representation) "formatting" them in light grey, and of course, the caption would become ("Data Explorer - Filtered View"); this obviously is, code-wise, the most elaborate solution (and would also imply that all un-necessary siblings of those intermediate parent items would be filtered out, of course).

Of interest here: According to the situation, this very elegant solution would not necessarily be the user's wanted one since if the "intermediates" are too numerous, they would possibly make necessary a lot of scrolling, and then, the question arises, both on the developer's as on the the user's side: To what "degree" should the "intermediary skeleton parts" be preserved? At least somewhat, at the higher-up levels, they would probably say, but then, whenever most of the non-filtered items are within some sub-structure, it would become evident that the "in-betweens" especially within that sub-structure should be preserved...

In other words, this would become highly-complicated stuff, for at the end of the day, little effect; of course, the only realistic (sic!) solution b) would be to implement a toggle (sic!), between "complete b)" (= show ALL necessary "intermediates" (in light grey) for and together with the non-filtered items), and c):

c) Even in the "Data Explorer" = tree pane, show a filtered flat list: easy, and also sufficient in itself, albeit a (toggled) combination with b) would be "ideal"...


4) BUT we've got additional PROBLEMS here, for our "tree solution", both for the developer as for the user:

How much "editing" will be allowed in those "tree solutions" a, b and c (except for simple renames that is: moving items?)? A code nightmare, AND a perfect occasion for the user to definitely (even if just partly) scramble their tree, even if the code is totally correct.

And that brings us back to the tree-preserving search/filter solution, b) on top: Concurrently (sic!) available: the (tree-preserving) "flat list" search results (and here, as in the "tree solution c", just the non-filtered items, but all of them), for deciding WHAT to do, AND the tree (in its current, and malleable, collapse-expand state) for DOING what we've decided to do, and whenever we then decide the two panes have got too much out of sync, we simply trigger the same search again.

It's clear as day that THAT would be a smooth workflow, AND it would not just apply to flag filtering, but to ALL searches / filterings.

The current "tree solution a) a)" (which even hides non-filtered items) being unusable as such, and the alternative "tree solutions" a)b) and a)c) coming with their problems of their own:

Please, Kyle, implement the necessary code for optionally preserving, by sort-overriding toggle, the tree structure in search results lists!

Here again, some might say, "not needed!", but as I have said, again and again, UR might NOT only be a "data repository", as which it's perceived by most of the general public, but could be the very best "write app" out there (Macworld included), and that would apply to ALL sorts of writing.

[kinook]

It's not clear what you are requesting. Can you explain it again, in plain English, with fewer words if possible. It seemed clear what you requested previously, but now you're saying what was added isn't useful. Thanks.

[Spliff]

I try to do a summary here, numbers as above:


1) TREE filtering instead of search/filter is not ideal (my bad in my first post) since user (and/or developer) would get into trouble by editing the filtered view, so no editing but endless forth and back between filtered view and complete view, and the complete view is the whole, expanded tree, so navigation problems in order to do the wanted edits: user's work would be possible but very cumbersome.


2) Thus, I suggest to add the necessary code to SEARCH. Can be done by an intermediate sql "view" with (virtual = just here, not stored) numeric path column by numeric path representation, then sorting by that (1.1.4.1 after 1.1.4 and before 1.1.5, currently it would come after 1.1.5 which breaks the tree preservation).


3) I present 3 ways of doing TREE filtering (see 1), a, b, c:

c being a flat list (possible if you really don't want to do the SEARCH thing (see 2),

b would be too demanding code-wise since for also graphically maintaining the tree, you would need to also show all "hidden" items (in light grey) but which would be necessary for the graphical tree representation of the non-hidden items

a (which you currently chose) is unacceptable since some non-hidden items ARE hidden though, since they are children / grandchildren / etc of hidden items (which are left out), so if you don't want to do c = flat list but graphical tree representation, you can't left out non-hidden items but must also preserve "hidden" items to "hold", to "situate" the non-hidden items = solution b = lots of code

To make this as clear as it gets: Currently, you do a = seemingly preserve the tree but hide "non-hidden" items which "depend" from hidden items, so it currently does NOT make sense, example:
hide everything except red and green
a is red, child b is green, you correctly show both
but a has another child, c, which is black, and which has a green child d on its turn;
you hide the non-to-be-hidden item d since you hide its black parent c

Hence the (complicated) solution b, or the (simple) solution c, here in TREE solution.


4) BUT as said, solution c in the tree (= flat list) would correspond to the SEARCH solution (from which you refrain because of the necessary, demanding coding), AND I say, since it's the similar code anyway, in the SEARCH results solution AND in the TREE solution, please do it (= the flat list, preserving the tree structure) per SEARCH results, since then

- the user can work concurrently on BOTH panes: for their deciding what to edit / move / etc they would consult the search results, and for their edits / moves / etc they would have the tree available in the state they need it to be for these edits, moves, etc

- the user can use ANY search for this work (e.g. searches for keywords, tags, whatever) for this editing-and-shuffling work, whilst if you do the flat list within the tree pane, this would only apply to flags, so the "usefulness scope" of filtering-while-preserving-the-tree-structure would be greatly enhanced, compared to the flat-list Data Explorer solution (with no extra developing work compared to the latter).


As said, "TREE a", your current "solution", is not a solution, since it hides items which are NOT to be hidden, just because they have the "wrong" parentage (which in a flat list would be easily hidden indeed).

So, "TREE b" (= also show the necessary "in-between" items) is complicated and not worthwhile, "TREE c" (=flat list in Data Explorer) would be possible but not easier to code than the flat-list "SEARCH" solution, and quite awful for the user, hence my suggestion to implement a SEARCH OPTION "preserve structure" (or "preserve hierarchy") (in the flat list, not also graphically of course), as another checkbox in the pack "Search titles only", "Match whole words", etc., and which would OVERRIDE any current column sort settings.

(I also said the additional code, with the creation of the intermediate SQL "view" from which then the "hierarchy sort" is done, should, of course, only triggered whenever the option "preserve structure" has been set for the search / filtering.)

This would then be available for ANY search / filter, not only for flags.

I hate to say it, but the developers of RightNote and MyInfo both implement quite nice features on a quite regular basis (especially MyInfo, whilst RightNote lately added the option to have TWO, concurrent Data Explorers, for alternative views, and which ironically, if we had them in UR, would invalidate my reservations re the TREE solutions (Tree b and Tree c, whilst current Tree a cannot be), whilst on some core functionality NOT being in the class of UR, so it's not a bad idea to ENHANCE and develop the STRONG points in UR's interface and functionality, unparalleled over there, and structure preservation, even as a default, might be considered a core functionality anyway - by which I don't wanna say that alternatively, all the numerous, but all structure NON preserving columns sorts weren't needed.

Again, "do it right" instead, in the Data Explorer, would be possible, but not less work than to put hierarchy preservation as an option into the search... which would be so much more useful for the user: much better workflow than Data Explorer toggle, and availability for ALL searches / filterings.

[Spliff]

It's not precisely a suggestion, but technically 2 bug reports, since it was said to be there in 2017, then again in 2021 (links above):


Needed: Filtering (can be called search when done by Search)

Schematically:

"Tree preservation" = bad, ambiguous expression since:

CP = Construction preservation = NEEDED = all items in their ORDER as in the tree (Data Explorer), even by flat list

GP = Graphical preservation = NOT needed = additional preservation of the indentation (instead of flat list)


2 Solutions:

TREE = in tree pane = Data Explorer, just possible for flags

SEARCH = in Search Results Pane, possible for ALL possible search criteria, thus preferable


TREE:

3 ways:

a) CP with GP, no intermediate items = current "solution", with (unavoidable) bug (proof above): non-hidden items are (incorrectly!) hidden though when intermediate elements (needed for the GP) are (correctly) hidden > NOT a solution

b) CP with GP, intermediate items necessary to "hold" the non-hidden items might be greyed out (but must be shown) > elegant but unnecessary solution with demanding code

c) CP only = flat-list = the solution to be preferred, but:


SEARCH:

a) current "solution" (presented in 2017 and again in 2021: sort results by columns lineage, then tree-order) is buggy since it doesn't do CP (proof above)

b) suggestion: implement another search option, "CP", overriding column sort, technical realization: build up numerical-path for intermediate sql view, then sort the search-pane output by that virtual column; code a little bit demanding indeed, but should be similar to solution TREE-c (flat-list in Data Explorer); preferable to that since 1. concurrent working in both panes possible (instead of TREE-c: endless forth-and-back between different content within the same pane), and 2. filtering by ANY possible criterion (even in combination of criteria, vs. just by flags in TREE-b or TREE-c)

(GP possible here, too, but neither needed in TREE nor in SEARCH)


RESULT:

UR becomes the only "outliner" with real, multi-factor tree filtering. (Some competition offers tag filtering, with construction not preserved.)

End of my bug report ;-)

[kinook]

What you're requesting negates one of the fundamental goals of UR, which is high performance. A lot of work is done to optimize populating the tree and search results, but the only way to implement what you're describing is to always load every single item in the database, in order to determine which ones should be excluded. This would add overhead of about 1 second for every 2.5K items in the database, for every search or refresh of the tree. Currently, loading the tree or search results is typically instant, even for databases with thousands of items. And the effort to implement this would be quite large.

Sorry, but it's not going to happen.

And it was never intended for fully replicate the tree in the search results, so I don't consider this a bug.

[Spliff]

Thank you for the info, so info in the two linked threads must be considered obsolete.

Thus, filtering in Data Explorer (now, thankfully, also possible for default items) has to be made carefully since "filter orphans" are not displayed; the user might not be aware of that.

Combination of two flags, first for construct, second for some core info or the like, remains possible IF the user makes sure they have no third flag / item "format" in the respective parentage of the flagged items to appear; this would mean that the user might be driven to FLATTEN out the hierarchy (which makes the navigation more cumbersome, but that's obviously the only way).

The really awful (sic!) aspect of this otherwise absolutely helpful feature is the fact that in-between, when I toggle between filter view and regular tree view, the tree is not anymore in the form it was before the filter view, but is completely expanded; perhaps the filter view toggle could store, then restore the tree in its previous state before filtering? Feasibility also depends on the existent code base, of course, which I obviously do not know.

[kinook]

Originally Posted by Spliff

...when I toggle between filter view and regular tree view, the tree is not anymore in the form it was before the filter view, but is completely expanded...

Can you elaborate on this? I am not seeing this behavior when switching between showing or hiding flagged items, if that's what you're referring to.

[Spliff]

Yes, that's what I meant, but I have to admit that now, by trying again, it's back to normal = expected behavior again, so that was my bad, sorry; I must have done something wrong.

On the other hand, I have spent some hours with trying to make the tree filtering functionality useful for me, to no avail; in order to see two flags and their occurrences in correct order, they would have to be all within the totally same "lineage" = the same long, very long siblings' list, and ditto for filtering by search, even with numerous tries with additional user attributes, it's all totally futile.

I then trialed some competitor, just for comparing, and what I found was hair-rising indeed:

Construction preservation seems to be absolute standard, default, both in flat-list form or in indentation-preserving = visual tree form.

Here's what I found - you will recognize the software, but I don't write this for advertising reasons:

"Global Search", then scope: In this section (would be a "tab" within the current db), In this notebook (=default, would be UR's current db), in open notebooks (= open DBs), in all notebooks; already scope-wise, this is perfect, and obviously, the developer took the additional effort to program the necessary code to gather the multiple SQLite indices, in order to make such scopes available to the user.

Then sorting, called "Group by": section (= default), title, date updated; title is titles by abc, date is dates by dates, and section is a flat list, preserving the construction order, in other words, not multiple sort orders, as in UR, are made available, just 3 fixed ones, but incl. the absolutely necessary construction order, which doesn't even have a name other than "section", so totally it's considered standard; then (not tried out), if you have scope db, open DBs or all DBs, the construction order in every section will be observed, just put one behind the other; also this would be user-side expected behavior.

I tried this with the search strings title:x (having added the x to just some titles), and with title:x OR title:y (having also added some y to other titles).

(Btw, I created my trial items in total disorder, in order to avoid a possible "by chance only" "tree order" sort which might have been produced by the creation order, and I added the x and y afterwards, also in disorder, so the program's "last change" date order doesn't play a role here, the default sort order is really the (current) construction order, nothing else.)

Then, I applied the same search strings (just x, or x or y, in the title) to the tree! Since the tree pane has a filter function, input field always visible, "Filter section notes", and here I was eager to see how the developer did the tree filtering (see a, b, c above): It's a simplified b over there, i.e. it's in tree form, together with the tree's indentations, and together with all necessary intermediates needed for showing ALL non-hidden items, i.e. there are no "filter orphans" over there, which, as we had to discover, are hidden-against-expectation in UR's tree filter.

Thus, the competitor does it all perfectly right, and even the tree filtering with any attribute or attributes combination, from the Filter field, so combinations could be automated by external macro, ditto for the Search field, whilst in UR, compound searches would have to be clicked-together (as described by me in another thread), which presents, for stored searches (which are possible in UR), the additional problem that it's then quite difficult to run those just with new values.


So now for the "bads" over there: I haven't discovered "flags" or any other way of formatting tree items (I mean the items' titles within the tree), and I absolutely love UR's way of doing that (now up to 20 formats ;-) ), in fact I can't do without, had tried for some time in some other software, NEED UR for this feature.

Also, with near half a million items in UR, I'm not ready to switch, that's for sure. On the other hand, I'm currently musing about switching to the other app just for that part of my work where I would need correct construction preservation in item filtering, not even filtering by more than one criterion, but by more than one VALUE or the same criterion (here: flag = red OR green, i.e. show both, and all of them, and in the correct order); I'm hesitating because I would then, for that part of my work, lose the aforementioned "tree formatting" for different kinds of items; I could (easily, as described) filter for them, by tags (which are available there) or simply by title tags (like my trial x and y), but UR's fine "distinguish'em by a glance" would be gone, and that would be, in my case i.e. from my needs, more than awful, even and also in that part of my work.

Btw, when I say "order preservation is (to be considered) standard", it's not just for what I just discovered within the competitor's search and tree filter, but it's how the search results are presented in any editor, any dedicated search tool (provided they come with a line view of all hits) by default: in their original occurrences' order, from first to last line of the text body, and any "grep" does work that way.

Thus, UR's omission of that "natural sort" obviously is very non-standard, AND a real problem; btw, with my (systematic) lineage-then-treeorder sort (which was deemed to be "tree order", since we believed the linked threads' allegations), I never succeeded in finding the relevant positions within the tree (Data Explorer) on my own, just from visually navigating from search result table to Data Explorer (which would have preserved the search result table), but had to press Enter on one of the results in the results table, in order to navigate there, and which then destroyed the results table (just navigating within the results table will only display the respective content, but will not "situate" the item within the tree): no wonder, since the results in the results table, deemed "in tree order", were sufficiently "mixed up", in order to lead me astray.

And that (not assuming the results were not in correct order, so no systematic check, verification, and the disappearance of the result table whenever I then navigated to the item in question) was responsible I lived with that for quite some time... whilst my current combination of TWO flag values, one for construction and one for the items-of-interest in that filter / search, I immediately discovered that tree-order is not.

I don't know if the competitor also applies (your) adjacency list model, or some path model, but I assume it's in your interest to check how he does it, SQLite-table and -columns-wise, in order to not encounter your coding problems for preserving tree order in search and filter results.

As for possible speed problems, I admit my trial db over there had less than 20 items, not 30,000 or more, as some of my UR DBs do, but above, I had said, in my first, and then again, more to the point, in my intermediate post, "(I also said the additional code, with the creation of the intermediate SQL "view" from which then the "hierarchy sort" is done, should, of course, only [be] triggered whenever the option "preserve structure" has been set for the search / filtering [in question].)" - so perhaps, speed problems would not need to spread all over the place, or then, some modification of the Adjacency List model might do it, even if that then (necessarily) slows down moves of whole sub-trees (which don't occur that often after all, once the users will have "organized themselves somewhat") a little bit?

Let me say here that I admire your solution how you have implemented (item, and whole sub-structures') cloning, from what I from your tables and your columns: That's really ingenious! So it would be quite a pity if you coudn't also find a viable solution for the - as alleged by me - standard tree preservation in search, and filter, results? ;-)

[Spliff]

It just occurs to me that the full core code, in other context, is already there, in order to implement at least an "ordered" (=in tree order), partial or total, but complete (=no "filter orphans" anymore), flat-list flag filter within the Data Explorer (=tree pane):

In tree, source item is selected = total filter; some other (parent of a sub-structure) item is selected = partial filter, same code.

File - Export - Item attribs to CSV - Attributes to export: Flag, IndentLevel, ItemTitle - targetfilename (Export child items recursive yes) path:\name.csv - Finish

gives: tree-ordered flat list in the form
"Item Title",FlagInCase,IndentLevel (or then "ItemTitle",,IndentLevel")

Ok, if it was in the form
IndentLevel,Flag,Item Title
you would even avoid the quotes around the item titles, but that's a minor flaw, fact is:

The above is in tree order

Flag info is there in order to format/color/etc the entry accordingly

(Even indent level info would be there in order to replicate the tree,
then code (outside sql: loop: "preserve if any child (=while indentlevel > current one) is to be preserved") could select the necessary intermediate items in order to replicate the tree form, but that is NOT necessary, so indent level info is not even needed, leave it out, do just Flag,ItemTitle above, which brings "Item Title",Flag)

Then just delete the array rows to be hidden (flag name/number not in the to-be-preserved-list), and put the remaining output into the Data Explorer, just as otherwise you would write it into the .csv file.

And since then, the Data Explorer content would be a flat (but formatted) list, without any indentation, even the caption would not to be changed, since the user immediately sees, from the content's "form" (=visual aspect), they are in "tree preserving flat list mode". ;-)


(I don't know about the possible (tree, not content) editing functionality of the current (sic!) flag filter, didn't dare try (item moves?), so admittedly, if you wanted to preserve those (if there are any) for the flat-list, that would complicate things, considerably in case. But for me it would be perfectly acceptable if the flat list would just SHOW the filtered tree correctly, perhaps with navigation = showing (and possibly editing) the respective content (as it is now, but even content editing in that "view" would not be absolutely necessary, ditto for item renames, not really necessary either)... but a Return (Enter) within the tree would LEAVE the flat view, and show the regular view, with the item from which the Return was done, being selected.)

[kinook]

It's not so easy to meld a totally different data structure and processing for export into the existing related items pane (which shows either child items or search results, and is a lot of complexity to begin with), but I will think about it and see what is possible.

[Spliff]

I understand each solution will arise its specific problems, and I don't know your code, especially how you fill the tree pane, then especially for (current) filtering, since whilst filling the tree pane (Data Explorer) with "regular" tree data will fetch something like "next 80 or so items by recursive find-children code", filtering will need to fetch the next 80 or so from the WHOLE tree, wherever those non-hidden items to be displayed may "hide", and that currently works sufficiently fast... it's just that non-to-be-hidden items are left out if their respective parenting structing is (correctly) left out.

Thus, I suppose that an optional flat list filter instead (but which will not hide not-to-hide items anymore), with just slightly changed code, will not represent a real speed problem, and that the code change implied will not be enormous? As said, the core difficulty that those filter items will have to be fetched from everywhere within the WHOLE tree, has already been worked out and resolved.

Perhaps it would be a good idea to also index the flag column (which thankfully is just one column since no two concurrent filters for the same item are allowed anyway).

Finding some solution would be more than great indeed, the sooner the better, since not only I need that thing, but as we have seen, UR here has been lacking what would be called standard functionality (since no tree-construct preserving filter / search, just export), and that should not last forever.

(For the time being, I use flat list (sic!) export plus some minimal scripting for import / display in a (plain text) editor window, with the (never more than 2, 3 concurrent) different flags being differentiated by (different) indentation levels (by tabs, so these indentation levels obviously don't correspond to the original tree's ones), or for a better formatted visualization (preservation of bold, italics, colors), I would have to look up some html code, then import the data set into some (free) html viewer instead of viewing it within an editor; respective print-outs would also be possible if needed.)

In fact, it would be one of those big steps ahead we would expect in major updates (v.7?), and let's face it, not only the competitor is lurking (market-wise that is, I'm not ready to switch, I repeat), AI is, and there's this plethora of little subscription "apps", which come and go, most of which have one or some good ideas implemented, the rest being so-so, but all of them do their respective denting into the (seemingly quite restrained) market, and it's obvious that these are demotivation factors for developers of "traditional" (but powerful) "outlining" applications. Thus:


The (above-mentioned) competitor (technically)

The competitor rewrote the code, with modern DevExpress component, but ironically, his tool installs not in Programs, but in Programs (x86), and, frankly, those very "modern" visual frameworks all look the same now, like skeletons, there's no "meat" - very different from UR where, with "Office 2010 blue" theme, I feel "at home", and that's important (and the reason why I would prefer to not even switch to the competitor for those, rather tiny, UR DBs where I absolute need "natural order preservation" or whatever we call it, and "just for flags" would be fine with me).

( After posting above, I discovered that over there, formatting within tree is possible, by "rules" which you put into some precedence order, e.g. "if comment contains b and u, then display it bold-blue", etc., for any attribute; I mention the comment one since that seems to be the only one which is accessible by shortcut, so that assigning the format is rather easy, not as cumbersome as for rules depending on other attributes, but all that is, or course, much more cumbersome than UR's direct (sic! and up to 20 now) format assignment if we put those (often-used) formats on their own shortkeys; over there, we could use an external macro to fill in the respective character or char combination indeed, but would have that dialog / char entry window popping up again and again; long word short, tree formatting isn't missing, so if I didn't like UR as much, whilst liking the competitor better - technically, there would be no obstacle to switch, the functionality being there. Btw, I invite fellow UR users to NOT mention the competitor's name here, since my ideas are aimed at UR, not directed toward the competition. ;-) )


AI

Then about AI, 2023 having been the year where this subject got into the awareness of the computer-affine masses, and speaking of the competition, TheBrain says they've got some AI implemented already, so that may be considered another "threat" to more traditional "outlining" software applications. I don't very much believe in AI re "outlining" since AI here would be about filing perhaps (from the inbox into which context(s)?) / grouping (multiple filing, permanent or on the spot), and finding (weighting the "hits", again for possible grouping).

Thus, it seems to me that AI in this software category will mainly serve to automate filing / grouping / search weighting rules, but allegedly to the cost of missing necessary connections, whilst establishing unneeded = unwanted ones (since those would just "clutter"), and as for search results' weighting, they would produce quite aleatoric results, since they would very much depend on your current search terms, and furthermore, in order for the AI to know HOW you would like it weight the results currently, i.e. in this very special situation

(in other words: AI might be really quite somewhat helpful in order to avoid manual processing in standards (sic!) proceedings, but what most of us do, is not "standard", but "from case to case", most of these situations being quite different from other (search situations) I suppose...),

you would probably need to enter lots of additional, "weight"-descriptive info... and you would never be sure anyway you got all the relevant results, at least among the more prominent ones ("prominence = relevance" just being AI's promise, but not what it really will be able to deliver, as far as it won't be able yet to look into your head, too... perhaps from 2050 on then? ;-) ).

(As for AI in TheBrain - which traditionally has the most expensive, in their case (I know what TB delivers, and what it does not) definitely overpriced, subscription / paid updates model -, the might apply some AI to some of the (real, big, deep!) problems of their graphical interface indeed, since their - quite spectacular - expanded view has been buried by themselves quite a long - both calendary and version - time ago, them informing their (over-?) loyal customer base (I don't make this up either, as I never make up anything) that the necessary code, to be rewritten in case together with all their codebase at the time, would be too complicated / time- or brain-consuming for their developers then, and since in those long years in-between, they have seen that (most of) their accepted this dire state of affairs, and accepted their "replacement" view, some (not-at-all equivalent) Mind map" view, this description has remained valid up to this day...

but then, it will have to be seen if AI can do some work for them (and their users) in this respect, and what I've said above will apply: while users had wanted (at the ancient time the "extended view" was there), more (and obviously manual, user-decisional) control over that "extended view" (and what it displayed thus, and also together with stored such views, which never were implemented either as far as I'm informed): AI will possibly bring back "extended view" (with a question mark here already), but the degree of user-sided malleability of these views will remain to be seen.

Anyway, TB's graphical concept (and just if it's implemented correctly, which had thus been the case up to version 7 or 8 if I remember well) lends to just some, special application cases, thus, except perhaps for another complete rehaul / recode (where it will again then lose core functionality? just asking...), with or without AI, will not easily become an earnest competitor too soon.

And, btw, filtering isn't so good over there either, just see "Tylast"'s "Sorry David, I becomes terribly burdensome to manually create a new thought as a surrogate filter. I have hundreds of thoughts with multiple tags. Manual just doesn't scale." in their forum thread "Plex Filters & Views" - admittedly, that was 3 years ago, and I don't follow their problems not as closely as that...)

Also, AI could help with those nasty homonyms, e.g. in German, sie (they) vs. Sie (you), and das (the) vs. dass (that) - for both pairs, you see, in almost every press article, multiple faults, and in other languages, there are certainly similar situations to be cleared by some, very simple, AI; also, I could imagine that easy AI components will soon become available for synonym search, especially for people who write in several languages and thus, later on, don't remember anymore in which language vocabulary they should search their search terms: that's AI which will become available for easy integration in almost any existing software, not something which will make some "new", "superior" applications some "winner" of some luring "war": as said, that, potentially fierce, AI will not enter this software field, since the connection user's "wishes / intentions - AI servicing those" will not be possible to be established on a really work-easying basis, for a quite long time to go, so the "big players" will not overtake (also) this market so soon, and we'll use their (much better-than-today) dictating instead of typing for input in the applications WE'll choose according to our needs, just as we do today.

Thus, from my point of view, that (almost mythical) AI threat (or more precisely and erroneously: AI as (sic!) threat) should not be taken as that perilous for, and thus by, "outliner" developers who focus and capitalize on their traditional applications' strengths.


The competitor again (licence-wise)

He has discarded the traditional "paid major updates" policy, but every (first or update) buy includes 12 months of updates, while he continuously produces updates indeed, so in the long run, he gets quite substantial sums (while he continues updates which are "worth it").

UR's update policy, in contrast, has been (quite rare) paid major updates, with traditionally not very much of "big steps ahead" in-between, but the latter aspect has changed recently (well, the developer got some good ideas out of the user base ;-) ), so there definitely might be some room for a slight (? more robust?) tightening of the major updates' frequence it seems. ;-) - It's obvious that the "major paid updates" policy, at first sight at least, implies - and that's the bad aspect for the user some (sic!) withholding strategy in-between, in order to motivate most of regular users to buy the update, BUT that problem loses much of its initial hazard whenever the users know, from experience - and this experience is clearly, fantastically established with UR now -, that even if the paid update in itself is a little bit sparse, they will get lots of "paid update items" in-between then and again, which, logically, they will not get if they don't buy the update, be it even a little bit "underwhelming" in itself, in case.

Hence, it might be "time", even if your little "withheld for next paid update" feature bouquet isn't currently that "overwhelming". ;-)


P.S. My AI passage above has become a little bit "uncoordinated", I admit, should have been better structured, but it's 1 o'clock in the morning, and my AI point is: There will be NO big AI revolution for "outliners" which would blast all current non-AI applications, but AI impact upon "outliners" will be selective, punctual and beneficial: it will just mean components to add, to enhance existent outlining applications; for a "big blast" which would annihilate existent software in THIS field, the necessary code-brain interface isn't there, and will not come soon, so resignation in face of what MS-Google-Amazon-etc might create would be premature by a whole generation here.

[Spliff]

It goes without saying that any filtering should work on the current sub tree (which may be the whole tree then of course, if the current item is the source item), because filtering is about de-cluttering, whilst otherwise (whole tree is filtered), there is, in most cases, enormous clutter before and after the filtered data you're after.

("you" being the user here)

Currently in UR, you can avoid that indeed, but the proceeds to get there have to be qualified as crazy.

Let's say you have a "bold" flag for the title items (=headings of sub-trees) of the first 2 or 3 indent levels of your tree, you want to filter one project sub tree, with its heading on tree level 1, and you want to see some special flags on project levels 1 and 2.

Now you first replace the bold title flags in your project (sub tree) by bold-scarlet title flags, then you simply filter out all bold (but regular black) title flags: Done... but in your project, you will need to maintain the new formatting for your headings, and that's not very intuitive it seems.

As for the de-clutter aim of filtering, I should perhaps explain HOW that is deemed to work, since many developers don't seem to see it must de-clutter, the above-mentioned competitor e.g., and as said, clutters the "filtered" view by all the "necessary" in-betweens, in order to also maintain the tree's graphics aspect = indentation; on the other hand, I must admit that over there, you would not use the (more or less useless then) "filter view" but the (correctly working) "search", as filter.

But I wanted to explain.

Let's have a project sub tree, with, on its level 1, 20 or 30 headings, and on its level 2, about 150 headings, distributed among the 20 or 30 level-1 headings, i.e. each of those bearing something between 5 and 30 level-2 headings.

Now, how much of those headings will you (HAVE TO!!!) preserve? In today's UR, all 170 or 180, all those of level 1, and all those of level 2; otherwise, you'll risk filter orphans (and, of course, not even speaking of possible special flags in project level 3 or below (i.e. above, numerically); or then, you pack all your level 2 headings into level 1:

instead of
1 - this
with 2 - that
1 - another one

you'd do
1 - this
1 - this - that
1 - another one
etc

which means you will have so much of clutter that "filtering" might not even make much sense anymore; of course, both solutions, since both will display 180 titling infos (30 plus 150, or 180 in just one level, doesn't matter) for your special flag items (which you're after, after all, your titling isn't but to "situate" them!), will give very good "situational" info for your specially-flagged items...

but that, your tree does it already, and filtering is about just seeing as much meta info as you WANT to get, for your current task, and here comes real filtering into play:

In the tree, you have precise locations for what you're after: 180 different ones, it's as fine-grained as you need it sometimes.

But to get an overview, you would NOT want this, you would just want all of your specially-flagged items indeed (and of ALL indentation levels that is, not only of project level 1 or 2), but situated into the above-mentioned 20, 30 "tree geographics" delimiters of project level 1, and then, it's not hundreds of meta info items anymore, with just some of the real info you're after, but it'll be dozens of - all the! - real info items, with just the amount of meta info you need, i.e. 20 or 30 "super groups" (but everything in correct tree order), and from there, whenever necessary, you switch back to the tree, in order to "situate" your special items even further, more "locally", or for whatever other reason.

Thus, it's also clear as day that (correctly ordered) filtering in the search pane is preferable, since then you can switch forth and back without having to use a visual toggle which will hide you one of the two views at every given moment, be it the one or the other then.