Any way to create a compound (flag) search preserving the tree structure? Or in tree?

[Spliff]

Thank you for the info, so info in the two linked threads must be considered obsolete.

Thus, filtering in Data Explorer (now, thankfully, also possible for default items) has to be made carefully since "filter orphans" are not displayed; the user might not be aware of that.

Combination of two flags, first for construct, second for some core info or the like, remains possible IF the user makes sure they have no third flag / item "format" in the respective parentage of the flagged items to appear; this would mean that the user might be driven to FLATTEN out the hierarchy (which makes the navigation more cumbersome, but that's obviously the only way).

The really awful (sic!) aspect of this otherwise absolutely helpful feature is the fact that in-between, when I toggle between filter view and regular tree view, the tree is not anymore in the form it was before the filter view, but is completely expanded; perhaps the filter view toggle could store, then restore the tree in its previous state before filtering? Feasibility also depends on the existent code base, of course, which I obviously do not know.

[kinook]

Originally Posted by Spliff

...when I toggle between filter view and regular tree view, the tree is not anymore in the form it was before the filter view, but is completely expanded...

Can you elaborate on this? I am not seeing this behavior when switching between showing or hiding flagged items, if that's what you're referring to.

[Spliff]

Yes, that's what I meant, but I have to admit that now, by trying again, it's back to normal = expected behavior again, so that was my bad, sorry; I must have done something wrong.

On the other hand, I have spent some hours with trying to make the tree filtering functionality useful for me, to no avail; in order to see two flags and their occurrences in correct order, they would have to be all within the totally same "lineage" = the same long, very long siblings' list, and ditto for filtering by search, even with numerous tries with additional user attributes, it's all totally futile.

I then trialed some competitor, just for comparing, and what I found was hair-rising indeed:

Construction preservation seems to be absolute standard, default, both in flat-list form or in indentation-preserving = visual tree form.

Here's what I found - you will recognize the software, but I don't write this for advertising reasons:

"Global Search", then scope: In this section (would be a "tab" within the current db), In this notebook (=default, would be UR's current db), in open notebooks (= open DBs), in all notebooks; already scope-wise, this is perfect, and obviously, the developer took the additional effort to program the necessary code to gather the multiple SQLite indices, in order to make such scopes available to the user.

Then sorting, called "Group by": section (= default), title, date updated; title is titles by abc, date is dates by dates, and section is a flat list, preserving the construction order, in other words, not multiple sort orders, as in UR, are made available, just 3 fixed ones, but incl. the absolutely necessary construction order, which doesn't even have a name other than "section", so totally it's considered standard; then (not tried out), if you have scope db, open DBs or all DBs, the construction order in every section will be observed, just put one behind the other; also this would be user-side expected behavior.

I tried this with the search strings title:x (having added the x to just some titles), and with title:x OR title:y (having also added some y to other titles).

(Btw, I created my trial items in total disorder, in order to avoid a possible "by chance only" "tree order" sort which might have been produced by the creation order, and I added the x and y afterwards, also in disorder, so the program's "last change" date order doesn't play a role here, the default sort order is really the (current) construction order, nothing else.)

Then, I applied the same search strings (just x, or x or y, in the title) to the tree! Since the tree pane has a filter function, input field always visible, "Filter section notes", and here I was eager to see how the developer did the tree filtering (see a, b, c above): It's a simplified b over there, i.e. it's in tree form, together with the tree's indentations, and together with all necessary intermediates needed for showing ALL non-hidden items, i.e. there are no "filter orphans" over there, which, as we had to discover, are hidden-against-expectation in UR's tree filter.

Thus, the competitor does it all perfectly right, and even the tree filtering with any attribute or attributes combination, from the Filter field, so combinations could be automated by external macro, ditto for the Search field, whilst in UR, compound searches would have to be clicked-together (as described by me in another thread), which presents, for stored searches (which are possible in UR), the additional problem that it's then quite difficult to run those just with new values.


So now for the "bads" over there: I haven't discovered "flags" or any other way of formatting tree items (I mean the items' titles within the tree), and I absolutely love UR's way of doing that (now up to 20 formats ;-) ), in fact I can't do without, had tried for some time in some other software, NEED UR for this feature.

Also, with near half a million items in UR, I'm not ready to switch, that's for sure. On the other hand, I'm currently musing about switching to the other app just for that part of my work where I would need correct construction preservation in item filtering, not even filtering by more than one criterion, but by more than one VALUE or the same criterion (here: flag = red OR green, i.e. show both, and all of them, and in the correct order); I'm hesitating because I would then, for that part of my work, lose the aforementioned "tree formatting" for different kinds of items; I could (easily, as described) filter for them, by tags (which are available there) or simply by title tags (like my trial x and y), but UR's fine "distinguish'em by a glance" would be gone, and that would be, in my case i.e. from my needs, more than awful, even and also in that part of my work.

Btw, when I say "order preservation is (to be considered) standard", it's not just for what I just discovered within the competitor's search and tree filter, but it's how the search results are presented in any editor, any dedicated search tool (provided they come with a line view of all hits) by default: in their original occurrences' order, from first to last line of the text body, and any "grep" does work that way.

Thus, UR's omission of that "natural sort" obviously is very non-standard, AND a real problem; btw, with my (systematic) lineage-then-treeorder sort (which was deemed to be "tree order", since we believed the linked threads' allegations), I never succeeded in finding the relevant positions within the tree (Data Explorer) on my own, just from visually navigating from search result table to Data Explorer (which would have preserved the search result table), but had to press Enter on one of the results in the results table, in order to navigate there, and which then destroyed the results table (just navigating within the results table will only display the respective content, but will not "situate" the item within the tree): no wonder, since the results in the results table, deemed "in tree order", were sufficiently "mixed up", in order to lead me astray.

And that (not assuming the results were not in correct order, so no systematic check, verification, and the disappearance of the result table whenever I then navigated to the item in question) was responsible I lived with that for quite some time... whilst my current combination of TWO flag values, one for construction and one for the items-of-interest in that filter / search, I immediately discovered that tree-order is not.

I don't know if the competitor also applies (your) adjacency list model, or some path model, but I assume it's in your interest to check how he does it, SQLite-table and -columns-wise, in order to not encounter your coding problems for preserving tree order in search and filter results.

As for possible speed problems, I admit my trial db over there had less than 20 items, not 30,000 or more, as some of my UR DBs do, but above, I had said, in my first, and then again, more to the point, in my intermediate post, "(I also said the additional code, with the creation of the intermediate SQL "view" from which then the "hierarchy sort" is done, should, of course, only [be] triggered whenever the option "preserve structure" has been set for the search / filtering [in question].)" - so perhaps, speed problems would not need to spread all over the place, or then, some modification of the Adjacency List model might do it, even if that then (necessarily) slows down moves of whole sub-trees (which don't occur that often after all, once the users will have "organized themselves somewhat") a little bit?

Let me say here that I admire your solution how you have implemented (item, and whole sub-structures') cloning, from what I from your tables and your columns: That's really ingenious! So it would be quite a pity if you coudn't also find a viable solution for the - as alleged by me - standard tree preservation in search, and filter, results? ;-)

[Spliff]

It just occurs to me that the full core code, in other context, is already there, in order to implement at least an "ordered" (=in tree order), partial or total, but complete (=no "filter orphans" anymore), flat-list flag filter within the Data Explorer (=tree pane):

In tree, source item is selected = total filter; some other (parent of a sub-structure) item is selected = partial filter, same code.

File - Export - Item attribs to CSV - Attributes to export: Flag, IndentLevel, ItemTitle - targetfilename (Export child items recursive yes) path:\name.csv - Finish

gives: tree-ordered flat list in the form
"Item Title",FlagInCase,IndentLevel (or then "ItemTitle",,IndentLevel")

Ok, if it was in the form
IndentLevel,Flag,Item Title
you would even avoid the quotes around the item titles, but that's a minor flaw, fact is:

The above is in tree order

Flag info is there in order to format/color/etc the entry accordingly

(Even indent level info would be there in order to replicate the tree,
then code (outside sql: loop: "preserve if any child (=while indentlevel > current one) is to be preserved") could select the necessary intermediate items in order to replicate the tree form, but that is NOT necessary, so indent level info is not even needed, leave it out, do just Flag,ItemTitle above, which brings "Item Title",Flag)

Then just delete the array rows to be hidden (flag name/number not in the to-be-preserved-list), and put the remaining output into the Data Explorer, just as otherwise you would write it into the .csv file.

And since then, the Data Explorer content would be a flat (but formatted) list, without any indentation, even the caption would not to be changed, since the user immediately sees, from the content's "form" (=visual aspect), they are in "tree preserving flat list mode". ;-)


(I don't know about the possible (tree, not content) editing functionality of the current (sic!) flag filter, didn't dare try (item moves?), so admittedly, if you wanted to preserve those (if there are any) for the flat-list, that would complicate things, considerably in case. But for me it would be perfectly acceptable if the flat list would just SHOW the filtered tree correctly, perhaps with navigation = showing (and possibly editing) the respective content (as it is now, but even content editing in that "view" would not be absolutely necessary, ditto for item renames, not really necessary either)... but a Return (Enter) within the tree would LEAVE the flat view, and show the regular view, with the item from which the Return was done, being selected.)

[kinook]

It's not so easy to meld a totally different data structure and processing for export into the existing related items pane (which shows either child items or search results, and is a lot of complexity to begin with), but I will think about it and see what is possible.

[Spliff]

I understand each solution will arise its specific problems, and I don't know your code, especially how you fill the tree pane, then especially for (current) filtering, since whilst filling the tree pane (Data Explorer) with "regular" tree data will fetch something like "next 80 or so items by recursive find-children code", filtering will need to fetch the next 80 or so from the WHOLE tree, wherever those non-hidden items to be displayed may "hide", and that currently works sufficiently fast... it's just that non-to-be-hidden items are left out if their respective parenting structing is (correctly) left out.

Thus, I suppose that an optional flat list filter instead (but which will not hide not-to-hide items anymore), with just slightly changed code, will not represent a real speed problem, and that the code change implied will not be enormous? As said, the core difficulty that those filter items will have to be fetched from everywhere within the WHOLE tree, has already been worked out and resolved.

Perhaps it would be a good idea to also index the flag column (which thankfully is just one column since no two concurrent filters for the same item are allowed anyway).

Finding some solution would be more than great indeed, the sooner the better, since not only I need that thing, but as we have seen, UR here has been lacking what would be called standard functionality (since no tree-construct preserving filter / search, just export), and that should not last forever.

(For the time being, I use flat list (sic!) export plus some minimal scripting for import / display in a (plain text) editor window, with the (never more than 2, 3 concurrent) different flags being differentiated by (different) indentation levels (by tabs, so these indentation levels obviously don't correspond to the original tree's ones), or for a better formatted visualization (preservation of bold, italics, colors), I would have to look up some html code, then import the data set into some (free) html viewer instead of viewing it within an editor; respective print-outs would also be possible if needed.)

In fact, it would be one of those big steps ahead we would expect in major updates (v.7?), and let's face it, not only the competitor is lurking (market-wise that is, I'm not ready to switch, I repeat), AI is, and there's this plethora of little subscription "apps", which come and go, most of which have one or some good ideas implemented, the rest being so-so, but all of them do their respective denting into the (seemingly quite restrained) market, and it's obvious that these are demotivation factors for developers of "traditional" (but powerful) "outlining" applications. Thus:


The (above-mentioned) competitor (technically)

The competitor rewrote the code, with modern DevExpress component, but ironically, his tool installs not in Programs, but in Programs (x86), and, frankly, those very "modern" visual frameworks all look the same now, like skeletons, there's no "meat" - very different from UR where, with "Office 2010 blue" theme, I feel "at home", and that's important (and the reason why I would prefer to not even switch to the competitor for those, rather tiny, UR DBs where I absolute need "natural order preservation" or whatever we call it, and "just for flags" would be fine with me).

( After posting above, I discovered that over there, formatting within tree is possible, by "rules" which you put into some precedence order, e.g. "if comment contains b and u, then display it bold-blue", etc., for any attribute; I mention the comment one since that seems to be the only one which is accessible by shortcut, so that assigning the format is rather easy, not as cumbersome as for rules depending on other attributes, but all that is, or course, much more cumbersome than UR's direct (sic! and up to 20 now) format assignment if we put those (often-used) formats on their own shortkeys; over there, we could use an external macro to fill in the respective character or char combination indeed, but would have that dialog / char entry window popping up again and again; long word short, tree formatting isn't missing, so if I didn't like UR as much, whilst liking the competitor better - technically, there would be no obstacle to switch, the functionality being there. Btw, I invite fellow UR users to NOT mention the competitor's name here, since my ideas are aimed at UR, not directed toward the competition. ;-) )


AI

Then about AI, 2023 having been the year where this subject got into the awareness of the computer-affine masses, and speaking of the competition, TheBrain says they've got some AI implemented already, so that may be considered another "threat" to more traditional "outlining" software applications. I don't very much believe in AI re "outlining" since AI here would be about filing perhaps (from the inbox into which context(s)?) / grouping (multiple filing, permanent or on the spot), and finding (weighting the "hits", again for possible grouping).

Thus, it seems to me that AI in this software category will mainly serve to automate filing / grouping / search weighting rules, but allegedly to the cost of missing necessary connections, whilst establishing unneeded = unwanted ones (since those would just "clutter"), and as for search results' weighting, they would produce quite aleatoric results, since they would very much depend on your current search terms, and furthermore, in order for the AI to know HOW you would like it weight the results currently, i.e. in this very special situation

(in other words: AI might be really quite somewhat helpful in order to avoid manual processing in standards (sic!) proceedings, but what most of us do, is not "standard", but "from case to case", most of these situations being quite different from other (search situations) I suppose...),

you would probably need to enter lots of additional, "weight"-descriptive info... and you would never be sure anyway you got all the relevant results, at least among the more prominent ones ("prominence = relevance" just being AI's promise, but not what it really will be able to deliver, as far as it won't be able yet to look into your head, too... perhaps from 2050 on then? ;-) ).

(As for AI in TheBrain - which traditionally has the most expensive, in their case (I know what TB delivers, and what it does not) definitely overpriced, subscription / paid updates model -, the might apply some AI to some of the (real, big, deep!) problems of their graphical interface indeed, since their - quite spectacular - expanded view has been buried by themselves quite a long - both calendary and version - time ago, them informing their (over-?) loyal customer base (I don't make this up either, as I never make up anything) that the necessary code, to be rewritten in case together with all their codebase at the time, would be too complicated / time- or brain-consuming for their developers then, and since in those long years in-between, they have seen that (most of) their accepted this dire state of affairs, and accepted their "replacement" view, some (not-at-all equivalent) Mind map" view, this description has remained valid up to this day...

but then, it will have to be seen if AI can do some work for them (and their users) in this respect, and what I've said above will apply: while users had wanted (at the ancient time the "extended view" was there), more (and obviously manual, user-decisional) control over that "extended view" (and what it displayed thus, and also together with stored such views, which never were implemented either as far as I'm informed): AI will possibly bring back "extended view" (with a question mark here already), but the degree of user-sided malleability of these views will remain to be seen.

Anyway, TB's graphical concept (and just if it's implemented correctly, which had thus been the case up to version 7 or 8 if I remember well) lends to just some, special application cases, thus, except perhaps for another complete rehaul / recode (where it will again then lose core functionality? just asking...), with or without AI, will not easily become an earnest competitor too soon.

And, btw, filtering isn't so good over there either, just see "Tylast"'s "Sorry David, I becomes terribly burdensome to manually create a new thought as a surrogate filter. I have hundreds of thoughts with multiple tags. Manual just doesn't scale." in their forum thread "Plex Filters & Views" - admittedly, that was 3 years ago, and I don't follow their problems not as closely as that...)

Also, AI could help with those nasty homonyms, e.g. in German, sie (they) vs. Sie (you), and das (the) vs. dass (that) - for both pairs, you see, in almost every press article, multiple faults, and in other languages, there are certainly similar situations to be cleared by some, very simple, AI; also, I could imagine that easy AI components will soon become available for synonym search, especially for people who write in several languages and thus, later on, don't remember anymore in which language vocabulary they should search their search terms: that's AI which will become available for easy integration in almost any existing software, not something which will make some "new", "superior" applications some "winner" of some luring "war": as said, that, potentially fierce, AI will not enter this software field, since the connection user's "wishes / intentions - AI servicing those" will not be possible to be established on a really work-easying basis, for a quite long time to go, so the "big players" will not overtake (also) this market so soon, and we'll use their (much better-than-today) dictating instead of typing for input in the applications WE'll choose according to our needs, just as we do today.

Thus, from my point of view, that (almost mythical) AI threat (or more precisely and erroneously: AI as (sic!) threat) should not be taken as that perilous for, and thus by, "outliner" developers who focus and capitalize on their traditional applications' strengths.


The competitor again (licence-wise)

He has discarded the traditional "paid major updates" policy, but every (first or update) buy includes 12 months of updates, while he continuously produces updates indeed, so in the long run, he gets quite substantial sums (while he continues updates which are "worth it").

UR's update policy, in contrast, has been (quite rare) paid major updates, with traditionally not very much of "big steps ahead" in-between, but the latter aspect has changed recently (well, the developer got some good ideas out of the user base ;-) ), so there definitely might be some room for a slight (? more robust?) tightening of the major updates' frequence it seems. ;-) - It's obvious that the "major paid updates" policy, at first sight at least, implies - and that's the bad aspect for the user some (sic!) withholding strategy in-between, in order to motivate most of regular users to buy the update, BUT that problem loses much of its initial hazard whenever the users know, from experience - and this experience is clearly, fantastically established with UR now -, that even if the paid update in itself is a little bit sparse, they will get lots of "paid update items" in-between then and again, which, logically, they will not get if they don't buy the update, be it even a little bit "underwhelming" in itself, in case.

Hence, it might be "time", even if your little "withheld for next paid update" feature bouquet isn't currently that "overwhelming". ;-)


P.S. My AI passage above has become a little bit "uncoordinated", I admit, should have been better structured, but it's 1 o'clock in the morning, and my AI point is: There will be NO big AI revolution for "outliners" which would blast all current non-AI applications, but AI impact upon "outliners" will be selective, punctual and beneficial: it will just mean components to add, to enhance existent outlining applications; for a "big blast" which would annihilate existent software in THIS field, the necessary code-brain interface isn't there, and will not come soon, so resignation in face of what MS-Google-Amazon-etc might create would be premature by a whole generation here.

[Spliff]

It goes without saying that any filtering should work on the current sub tree (which may be the whole tree then of course, if the current item is the source item), because filtering is about de-cluttering, whilst otherwise (whole tree is filtered), there is, in most cases, enormous clutter before and after the filtered data you're after.

("you" being the user here)

Currently in UR, you can avoid that indeed, but the proceeds to get there have to be qualified as crazy.

Let's say you have a "bold" flag for the title items (=headings of sub-trees) of the first 2 or 3 indent levels of your tree, you want to filter one project sub tree, with its heading on tree level 1, and you want to see some special flags on project levels 1 and 2.

Now you first replace the bold title flags in your project (sub tree) by bold-scarlet title flags, then you simply filter out all bold (but regular black) title flags: Done... but in your project, you will need to maintain the new formatting for your headings, and that's not very intuitive it seems.

As for the de-clutter aim of filtering, I should perhaps explain HOW that is deemed to work, since many developers don't seem to see it must de-clutter, the above-mentioned competitor e.g., and as said, clutters the "filtered" view by all the "necessary" in-betweens, in order to also maintain the tree's graphics aspect = indentation; on the other hand, I must admit that over there, you would not use the (more or less useless then) "filter view" but the (correctly working) "search", as filter.

But I wanted to explain.

Let's have a project sub tree, with, on its level 1, 20 or 30 headings, and on its level 2, about 150 headings, distributed among the 20 or 30 level-1 headings, i.e. each of those bearing something between 5 and 30 level-2 headings.

Now, how much of those headings will you (HAVE TO!!!) preserve? In today's UR, all 170 or 180, all those of level 1, and all those of level 2; otherwise, you'll risk filter orphans (and, of course, not even speaking of possible special flags in project level 3 or below (i.e. above, numerically); or then, you pack all your level 2 headings into level 1:

instead of
1 - this
with 2 - that
1 - another one

you'd do
1 - this
1 - this - that
1 - another one
etc

which means you will have so much of clutter that "filtering" might not even make much sense anymore; of course, both solutions, since both will display 180 titling infos (30 plus 150, or 180 in just one level, doesn't matter) for your special flag items (which you're after, after all, your titling isn't but to "situate" them!), will give very good "situational" info for your specially-flagged items...

but that, your tree does it already, and filtering is about just seeing as much meta info as you WANT to get, for your current task, and here comes real filtering into play:

In the tree, you have precise locations for what you're after: 180 different ones, it's as fine-grained as you need it sometimes.

But to get an overview, you would NOT want this, you would just want all of your specially-flagged items indeed (and of ALL indentation levels that is, not only of project level 1 or 2), but situated into the above-mentioned 20, 30 "tree geographics" delimiters of project level 1, and then, it's not hundreds of meta info items anymore, with just some of the real info you're after, but it'll be dozens of - all the! - real info items, with just the amount of meta info you need, i.e. 20 or 30 "super groups" (but everything in correct tree order), and from there, whenever necessary, you switch back to the tree, in order to "situate" your special items even further, more "locally", or for whatever other reason.

Thus, it's also clear as day that (correctly ordered) filtering in the search pane is preferable, since then you can switch forth and back without having to use a visual toggle which will hide you one of the two views at every given moment, be it the one or the other then.