When building new flow control commands it is often useful to split the
string into words in the same way that Tcl parses the words to a command . List operations do not work as they throw away potentially significant quoting characters and also cannot cope with incorrectly formatted data. These two functions take a string and return the list of words. The second function also strips out comments which start with a # and end on a newline.  ''A 3rd function is added to the end of this document.  The 3rd function is an improved version of the first function below with a better running time for some inputs.''

----
 proc SplitIntoWords {block} {

    # We need to split the block up into words but cannot use
    # list operations as they throw away some significant
    # quoting, and [split] ignores braces as it should.
    # Therefore what we do is gradually build up a string out of
    # whitespace seperated strings. We cannot use [split] to
    # split the block into whitespace seperated strings as it
    # throws away the whitespace which maybe important so we
    # have to do it all by hand.

    set words {}
    set word ""

    while {[string length $block]} {
        # Look for the next group of whitespace characters.
        if {[regexp -indices "\[ \t\n\]+" $block all]} {
            # Remove the text leading up to and including the white space
            # from the block.
            set text [string range $block 0 [lindex $all 1]]
            set block [string range $block [expr {[lindex $all 1] + 1}] end]
        } else {
            # Take everything up to the end of the block.
            set text $block
            set block {}
        }

        # Add the text to the end of the word we are building up.
        append word $text

        if { [catch {llength $word} length] == 0 && $length == 1} {
            # The word is a valid list so add it to the list.
            lappend words [string trim $word]
            set word {}
        }
    }

    # If the last word has not been added to the list then there
    # is a problem.
    if { [string length $word] } {
        error "incomplete word \"$word\""
    }

    return $words
 }

 proc SplitIntoWordsStripComments {block} {

    # We need to split the block up into words but cannot use
    # list operations as they throw away some significant
    # quoting, and [split] ignores braces as it should.
    # Therefore what we do is gradually build up a string out of
    # whitespace seperated strings. We cannot use [split] to
    # split the block into whitespace seperated strings as it
    # throws away the whitespace which maybe important so we
    # have to do it all by hand.

    set words {}
    set word ""
    set comment 0

    while {[string length $block]} {
        # Look for the next group of whitespace characters.
        if {[regexp -indices "\[ \t\n\]+" $block all]} {
            # Remove the text leading up to and including the white space
            # from the block.
            set text [string range $block 0 [lindex $all 1]]
            set block [string range $block [expr {[lindex $all 1] + 1}] end]
        } else {
            # Take everything up to the end of the block.
            set text $block
            set block {}
        }

        # Add the text to the end of the word we are building up.
        append word $text

        # If the word is a comment then check to see whether it is
        # complete yet.
        if { $comment } {

            set index [string first "\n" $word]
            if { $index != -1 } {
                # The comment has been terminated.
                set comment 0
            }

            # Discard the part of the comment which has already been
            # found, even if a whole comment has been found only white space
            # could have come after the newline and that whitespace is not
            # significant.
            set word ""

        } elseif { [regexp -indices "^\[ \t\n\]*#" $word all] } {
            # The word starts with a hash so it is a comment, strip
            # off the matched portion which could contain newline
            # characters which would confuse the search for a terminating 
            # newline character.
            set word [string range $word [lindex $all 1] end]

            set index [string first "\n" $word]
            if { $index == -1 } {
                # The comment has not yet been terminated so keep looking
                # for the comment.
                set comment 1
            }

            # Discard the part of the comment which has already been
            # found, even if a whole comment has been found only white space
            # could have come after the newline and that whitespace is not
            # significant.
            set word ""
            
        } elseif { [catch {llength $word} length] == 0 && $length == 1} {
            # The word is a valid list so add it to the list.
            lappend words [string trim $word]
            set word {}
        }
    }

    # If the last word has not been added to the list then there
    # is a problem.
    if { [string length $word] } {
        error "incomplete word \"$word\""
    }

    return $words
 }

----
The function below is an improved version of the first function, SplitIntoWords.  
When the input to the first function is a string containing many words inside quotes, 
then the running time for the first function is O(N^2) where N is the number of words in the input.  
For example, if the input is:  {pattern {a b c d e}}, then the first function must parse the partial list:

   {a
   {a b
   {a b c
   ...
   {a b c d e}

This type of input can be very common because the input is often a pattern action pairs where the action is a Tcl script containing many words and the pattern is a single word.  The following function improves the running time by only making a list check when a word with a quote character ["{}] is seen.  
This can reduce the running time for the above example from O(N^2) to O(N).  
However, the worst case running time is still O(N^2) when every word inside the list also contains a quote character.

----
======
proc SplitIntoWords {block} {

    # We need to split the block up into words but cannot use
    # list operations as they throw away some significant
    # quoting, and [split] ignores braces as it should.
    # Therefore what we do is gradually build up a string out of
    # whitespace seperated strings. We cannot use [split] to
    # split the block into whitespace seperated strings as it
    # throws away the whitespace which maybe important so we
    # have to do it all by hand.

    set words {}
    set word ""

    while {[string length $block]} {
        # Look for the next word containing a quote: " { }
        if {[regexp -indices {[^ \t\n]*[\"\{\}]+[^ \t\n]*} \
                $block all]} {
            # Get the text leading up to this word, but not
            # including this word from the block.
            set text [string range $block 0 \
                    [expr {[lindex $all 0] - 1}]]
            # Get the word with the quote
            set wordWithQuote [string range $block \
                    [lindex $all 0] [lindex $all 1]]

            # Remove all text up to and including the word from the
            # block.
            set block [string range $block \
                    [expr {[lindex $all 1] + 1}] end]
        } else {
            # Take everything up to the end of the block.
            set text $block
            set wordWithQuote {}
            set block {}
        }

        if {$word != {}} {
            # If we saw a word with quote before, then there is a
            # partial list starting with that word.  In this case, add
            # the text and the current word to this partial list.
            append word $text $wordWithQuote
        } else {
            # Add the text to the result.  There is no need to parse
            # the text because it couldn't be a part of any list.
            # Then start a list with the word because we need to pass
            # this word to the Tcl parser
            append words $text
            set word $wordWithQuote
        }

        if { [catch {llength $word} length] == 0 && $length == 1} {
            # The word is a valid list so add it to the list.
            lappend words [string trim $word]
            set word {}
        }
    }

    # If the last word has not been added to the list then there
    # is a problem.
    if { [string length $word] } {
        error "incomplete word \"$word\""
    }

    return $words
}
======

----
''Who made this?  It's quite neat... -[FW]''

<<categories>> String Processing