Version 1 of Tablelist as treeview: A speed-up

Updated 2021-07-25 19:51:12 by hgiese

Warning

I just noticed a misbehaviour with my code. So you better don't use it yet. Sorry.

Background

I wanted to use a tablelist as a treeview to display infos about the internals of a microprocessor: its perpiherals, registers, etc. Not having used tablelist before it was the usual fighting with a new tool, but after some time I was done - or so I thought: During development I had been using a single peripheral, but when I fed my program the information of the processor as a whole

  • building the tree took forever,
  • resizing the widget was extremely jerky or didn't work at all and
  • the overall navigation was abysmal.

I then turned to using a ttk::treeview (which worked) but this problem kept nagging me.

Update: Below I provide a link to the "problem data" so you can judge for yourself.

The idea

Saving tablelist's content to a file is easy but when you re-load it all the "tree information" is lost. But then I had an idea: How about 'on demand' expanding?

  • Load the saved data, but don't feed it into the tree yet.
  • Only insert (and by this: display) the top level nodes
  • and on expanding any node insert just its direct children into the tree.
  • On collapsing reverse the operation.

The solution

The following is the solution I came up with plus a demo for you to try it out immediately.

  • If you start the file below with a number it creates this many (pseudo) peripherals (it defaults to 10). Enter a relatively high number (try 100). It will take a noticeable time to create the tree but once created it performs smoothly.
  • Play with it expanding and collapsing nodes, scrolling around, etc.
  • Press 'Restart': Everything will be destroyed, a new tree will be created and filled from the file which was created in the first step. This will happen instantaneously, practically independent from the number of peripherals created. Otherwise you won't probably notice any speed-up.

But using the link below you will see a significant improvement.

package require Tk
package require tooltip
package require tablelist_tile
package require scrollutil_tile

foreach ch [winfo children .] {destroy $ch}

#
# A node is to be expanded. Fill it with its /direct/ children
#
proc expandCmd {tree row} {
    variable gui

    set item [$tree rowcget $row -text]

    # get the column
    set colIdx [expr {[$tree depth $row] - 1}]
    set name [lindex $item $colIdx]

    # get our 'first level' parent's idx (in the tree)
    set parentIdx $row
    for {set i 1} {$i <= $colIdx} {incr i} {
        set parentIdx [$tree parentkey $parentIdx]
    }
    set parentName [$tree cellcget $parentIdx,0 -text]

    # get the subLst for 'parentName': find its list index
    set lst [dict get $gui lst]
    set lstIdx [lsearch -index 0 $lst $parentName]
    # ... and grab all the rest
    set subLst [lrange $lst $lstIdx end]

    # find the next sibling to 'parentName':
    set nxtIdx [lsearch -start 1 -index 0 -regexp $subLst {.+}]
    # ... and remove all lines from 'nxtIdx' on
    if {$nxtIdx == -1} {
        set subLst [lrange $subLst 0 end]
    } else {
        set subLst [lrange $subLst 0 [incr nxtIdx -1]]
    }
    # 'subLst' now contains all entries for the 'first level' 'parentName'

    # find the line which corresponds to 'name'
    set lineIdx [lsearch -index $colIdx $subLst $name]
    # ... and find its next sibling
    set stIdx [expr {$lineIdx + 1}]
    set nxtIdx [lsearch -start $stIdx -index $colIdx -regexp $subLst {.+}]
    # remove all lines from 'nxtIdx' on
    if {$nxtIdx != -1} {
        set subLst [lrange $subLst 0 [incr nxtIdx -1]]
    }

    # delete the dummy
    set 1stCh [lindex [$tree childkeys $row] 0]
    set 1stChTxt [$tree rowcget $1stCh -text]
    if {! [string match "*@*" $1stChTxt]} {
        # our dummy is already gone: this node has already been expanded
        return
    }
    $tree delete $1stCh

    # now find all /direct/ children: every line which has the /next/ column set
    set idxLst [lsearch -index [incr colIdx] -start $stIdx -all -regexp $subLst {.+}]

    # are we inserting into the last column? if so don't add a dummy
    set lastCol [expr {$colIdx == ([$tree columncount] - 1)}]

    # insert the new elements
    set newLst {}
    foreach idx $idxLst {
        set newIdx [$tree insertchild $row end [lindex $subLst $idx]]
        lappend newLst $newIdx
        if {! $lastCol} {
            $tree insertchild $newIdx end {"" "@"}
        }
        $tree collapse $newIdx
    }
}

#
# A cell of tablelist displays [lindex ... 0] of the text while a tooltip
# displays [lindex ... 1].
#
proc getText {txt} {
    return [lindex $txt 0]
}
proc tooltipAddCmd {t row col} {
    if {! [catch {$t cellcget $row,$col -text} txt]} {
        tooltip::tooltip $t $[lindex $txt 1]
    }
}

#
# Get the tree's data from 'fName'. Only create the top level nodes.
#
proc fillFromFile {fName} {
    variable gui

    set lst [split [contentOf $fName] "\n"]
    # remove a potentially empty element
    if {[lindex $lst end] eq ""} {
        set lst [lrange $lst 0 end-1]
    }

    set tree [dict get $gui tree]
    # find all 'first level' entries
    set idxLst [lsearch -all -index 0 -regexp $lst {.+}]
    foreach idx $idxLst {
        # insert each one into 'tree'
        set id [$tree insertchild root end [lindex $lst $idx]]
        # add a dummy
        $tree insertchild $id end {"" "@"}
    }
    $tree collapseall

    # save the data of tree
    dict set gui lst $lst

    # enable the 'on demand' expansion
    $tree configure -expandcommand expandCmd
}

#
# Save the tree to 'fName'. This proc should be called /once/ when the tree has been
# initially constructed.
#
proc saveTreeData {fName} {
    variable gui

    set tree [dict get $gui tree]
    set lst [set [$tree itemlistvar]]
    # strip off the item number
    foreach line $lst {
        lappend newLst [lrange $line 0 end-1]
    }
    string2File [join $newLst "\n"] $fName
}

#
# Create a tablelist as a child of 'parent'.
#
proc mkTree {parent} {
    variable gui

    set sa [scrollutil::scrollarea $parent.sa]
    set tree [tablelist::tablelist $sa.t -columns {0 "Peripherals" left
              0 "Registers" left 0 "Bit fields" left 0 "Enums" left} -stretch all \
              -tooltipaddcommand tooltipAddCmd \
              -tooltipdelcommand "tooltip::tooltip clear"]
    $sa setwidget $tree
    for {set idx 0} {$idx < [$tree columncount]} {incr idx} {
        $tree columnconfigure $idx -labelalign center
        $tree columnconfigure $idx -formatcommand getText
    }
    pack $sa -expand 1 -fill both
    # save 'tree'
    dict set gui tree $tree
    return $tree
}

#
# The following procs are only for the demo
#
proc contentOf {name} {
    set res [read [set f [open $name]]]
    close $f
    return $res
}
proc string2File {str name} {
  set outF [open $name w]
  puts $outF $str
  close $outF
}

#
# Fill the tree with 'numPeriphs' (pseudo) peripherals
#
proc mkPeriphs {numPeriphs fName} {
    variable gui

    set limit 8
    set tree [dict get $gui tree]
    for {set i 1} {$i <= $numPeriphs} {incr i} {
        set id [$tree insertchild root end periph$i]
        for {set j 1} {$j <= $limit} {incr j} {
            set id1 [$tree insertchild $id end [list "" [list p$i-reg$j p$i-reg$j]]]
            for {set k 1} {$k <= $limit} {incr k} {
                set id2 [$tree insertchild $id1 end [list "" "" [list p$i-r$j-f$k p$i-r$j-f$k]]]
                # for testing purposes we want some entries to not have children
                if {$k in {3 6}} {continue}
                for {set l 1} {$l <= $limit} {incr l} {
                    set id3 [$tree insertchild $id2 end [list "" "" "" [list p$i-r$j-f$k-e$l p$i-r$j-f$k-e$l]]]
                }
            }
        }
    }
    $tree collapseall
    saveTreeData $fName
}

#
# Create a tree, fill it with 'numPeriphs' (pseudo) peripherals and save
# the resulting tree's 'itemlistvar' to 'fName'.
#
proc start {numPeriphs fName} {
    set f [ttk::frame .f]
    mkTree $f
    set btn [button $f.btn -text "Restart" -command "restart $fName"]
    pack $btn -pady 20
    pack $f -expand 1 -fill both
    update
    mkPeriphs $numPeriphs $fName
}

#
# Start from a clean sheet and fill the tree with the content of 'fName'
#
proc restart {fName} {
    # destroy everything
    foreach ch [winfo children .] {destroy $ch}

    # now comes the real thing
    mkTree ""
    fillFromFile $fName
}

set fName exdata.txt
if {$argc == 0} {
    set numPeriphs 10
} else {
    set numPeriphs [lindex $argv 0]
}
start $numPeriphs $fName

The "problem data"

From link name https://www.dropbox.com/s/vhlal8kn0zbgokf/MXRT1062%5B1%5D.data?dl=0 download mimxrt1062.zip and unzip it. Then save and start the above script and then launch the following script.

foreach ch [winfo children .] {destroy $ch}
mkTree ""
fillFromFile MXRT1062.data
set t [dict get $gui tree]
update
# this call takes about 3 min on my machine so be prepared
$t expandall
update
# this one is fast: 10 sec
$t collapseall
saveTreeData tst.data

The result of the expanding is that the tablelist's 'itemlistvar' now contains every single item just as if I had created it from scratch.

Was it worth it?

For me in any case. Without this change tablelist - with the data I had - was plainly unusable while with it the performance is immediate. Why it is that with my data tablelist performed badly while in the above demo it is as fast as one can want I don't know. I can only speculate that it has to do with the enormous difference in size among my 'real' peripherals: From one with 4 registers with 7 (bitfields + enums) to one with 479 register with >7100 (bitfields + enums) - but that's only speculation.

Limitations

As implemented 'expandCmd' works only for - let's call them 'normalized' trees: A 'normalized' tree is a tree which when fully expanded holds in every row exactly one element. In the following examples the first one is 'normalized' while the second is not.

      parent
          child 1
              grandchild 1
          child 2
              grandchild 2
     parent
         child 1    child 2
                        grandchild 1    
                        grandchild 2

In the second example the two grandchildren will probably be assigned to 'child 1' - this would need to be adapted.

Oh, and don't attempt to save tablelist's data more than once. First of all you don't need to if you had called 'saveTreeData' as recommended above. If you do you are very likely to lose (potentially a lot of) data: It is the feature of 'expandCmd' that it populates tablelist's 'itemlistvar' only with expanded nodes - and if you then save the data you will lose all the unexpanded ones.