Version 25 of Tcl Quoting

Updated 2020-01-18 16:04:18 by yorickthepoor

Tcl quoting is elegant, minimal, and consistent. However, it can be quite subtle when combined with language features like evaluation or with other domain-specific languages understood by routines like expr, regexp, or string match. It is almost guaranteed that someone new to Tcl will initially resort to unnecessarily complicated quoting schemes. The most common cause of Tcl quoting confusion is assumptions that are carried over from other languages. Tcl is not sh and it is not C. Likewise, it is also distinctly different from Lisp, Perl, or Python or nearly any other language. Its design is a novel synthesis of features and ideas from sh, C, Awk, and Fortran, among others. To appreciate its syntax, be prepared for something fundamentally different, minimal, and pure.

See Also

double substitution
How Tcl command evaluation affects routines that do their own interpretation.
escaping special characters
Provides routines to escape special characters in arguments to glob, string match, regexp, regsub, and subst.
Tcl syntax
everything is a string
The universal data interface provided by Tcl.
Tcl Minimal Escaping Style
Advocates using braces and quotes only when necessary.
Tcl_Merge
eval
Discusses proper quoting of scripts, particularly when they are programmatically composed.
Frequently-Made Mistakes
list
The key to programatically substituting a properly-quoted word into a script.
Is white space significant in Tcl?
Covers much of the material pertinent to Tcl quoting.
Move any widget
Contains an over-quoted script and a fixed version.
string map
The go-to routine for help generating properly-quoted Tcl code
Toplevel Watermark
Contains an over-quoted script and a fixed version.
Quoting Hell
You know you're a sinner when...
Quoting Heaven!
Can't have one without the other.
Quoting and function arguments
A pre-8.6 explanation of a few gotchas.
README.programmer , by Brent Welch, 1993
When and how to use list to avoid quoting hell. Predates {*}.

Description

In a Tcl script, most characters represent their literal value. The characters, $, [, {, \, and white space have functional meaning but can be quoted so that they lose that meaning and represent themselves instead. Whereas in other languages quoting may specify that a value is a string, in Tcl each values already is a string, so quoting is just another mechanism to choose which characters are interpreted, and which characters represent themselves.

What Not to Do

To stay out of trouble, follow this advice:

Don't pass multiple arguments to expr. Passing multiple arguments slows things down considerably because a new expression is effectively being generated and parsed each time instead a single expression being byte-compiled the first time. Secondly it might result in unintended double substitution.

Don't attempt to use braces to turn a value into a list or a script. It doesn't work that way. For example, enclosing a value like what does { do? in braces doesn't turn it into a list:

set value "what does { do?"
set script "set var {$value}"

#Here comes an error:
eval $value

The value of $script is clearly incomplete:

set var {what does { do?}

To do it correctly use list instead of enclosing a value in braces:

set value "what does { do? "
set script "set var [list $value]"
eval $value

or even better:

set value "what does { do? "
set script [list set var $value]
eval $value

Basic Quoting Techniques

For simple values there is no quoting needed:

puts Hello
set greeting Hello
puts $greeting

Substitutions can be performed on any word, including the name of the command:

set cmd puts
set greeting Hello
$cmd $greeting

Bare strings and substitutions can be directly adjoined:

set prefix antidis
set root establish
set suffix arian
set entry $prefix[set root]ment${suffix}ism

No quoting is needed here because $prefix[set root]ment${suffix}ism contains no literal whitespace outside of the script substitution and $ has its normal meaning as variable substitution.

variable substitution and script substitution can be used in a bare string:

set digits 245
puts 01[string replace $digits 1 1 34]67

output:

01234567

No quoting is required here because all characters in the script carry their normal syntactic meaning. [ and ] delimit a nested script but the whitespace in that script does not affect the outer script.

When a word includes literal whitespace, use {}

set greeting {Good morning}
puts $greeting

Variable substitution and script substitution are not performed within {}, so when those features are needed, " can be used instead of {}:

set name Bob
proc daystage {return Morning}
set greeting "Good [daystage], $name."
puts $greeting

output:

Good Morning, Bob.

Composing a Script for Evaluation

First, consider organizing the code such that the composed script consists of a single command:

proc reputation_cb {msg msg2} {
    puts $msg
    puts $msg2
    set time [clock seconds]
    puts [list {The time is now} [clock format $time]]
    set ::done 1
}

proc reputation {} {
    set msg {an idle and false imposition}
    set msg2 {got without merit} 
    eval [list reputation_cb $msg $msg2]
}
reputation

This is likely to be more performant as procedures are byte-compiled.

The apply variant of this example is:

proc reputation {} {
    set msg {an idle and false imposition}
    set msg2 {got without merit} 
    set script {
        puts $msg
        puts $msg2
        set time [clock seconds]
        puts [list {The time is now} [clock format $time]]
    }
    eval [list apply [list {msg msg2} $script [namespace current]] $msg $msg2]
}
reputation

To substitute a complete word into a script, use list to turn the word into a list containing only the word. This works because syntactically a command containing no substitutions is a list, so each word in the command has the same syntax as an item in a list. If a value that is not a list is substituted into a script but not quoted, the resulting script might be syntactically incorrect:

set msg {hello world}
set script "set msg2 $msg"
# Here comes an error:
eval $script

Here the value of $script is

set msg2 hello world

which results in the error message:

wrong # args: should be "set varName ?newValue?"

list quotes the substituted value correctly:

set msg {hello world}
set script "set msg2 [list $msg]"
eval $script

In this case the value of $script is

set msg2 {hello world}

which is what was intended.

A small but important variation which is more common is to make the entire command into a list:

set script [list set msg2 $msg]

This is preferred because even though everything is a value, Tcl is careful not to generate an actual string representation for the value until something requires it. In this case eval can use the internal list representation of the script directly, which has two advantages: First, a string representation of the script is not generated, which saves time and memory. Second, eval doesn't have to spend time parsing the string representation of the script. Instead, it uses the internal list representation as the parsed representation of the script, which is a big win in terms of efficiency. This has a rather far-reaching implication: Even though Tcl is known as a string-based language, it is really a list-processing language, and the low-friction way to program in Tcl is to embrace a list-oriented programming style.

string map can also be used to properly escape a word substituted into a script:

set msg {hello world}
set script [string map [list @msg@ [list $msg]] {
    set msg2 @msg@
}]
eval $script

To avoid ambiguity it's a good idea to give the placeholders a distinctly visible initial and final character. This is particularly true in large scripts, where the author might overlook the fact that a placeholder without clear delimiters occurs as a substring of another value. @ is common choice for the delimiter.

In a small script list is more convenient, but in a larger script where the additional escaping of other characters makes the script more difficult to read string map is cleaner.

ycl string template provides a more concise way to use string map.

Composing a Script for Later Evaluation

When composing a script for later evaluation, consider that the current evaluation environment might not be the environment the script eventually is evaluated in. In the following example, puts $msg will show $hello probably be an error because $msg

set msg goodbye
proc doitlater {} {
    set msg hello
    after 1000 {
        puts $msg
        set ::done 1
    }
}
doitlater
vwait done

To capture the local value of $msg, use apply instead:

set msg goodbye
proc doitlater {} {
    set msg hello
    after 1000 [list apply {msg {
        puts $msg
        set ::done 1
    }} $msg]
}
doitlater
vwait done

Passing $args to Another Command

In the following script, $args are any additional options to button, e.g., -fg blue

#! /bin/env tclsh
package require Tk
proc mybutton {parent name label args} {
    if {$parent eq {.}} {
        set myname $parent$name
    } else {
        set myname $parent.$name
    }
    button $myname -text $label -command [list puts stdout $label] {*}$args
    pack append $parent $myname {left fill}
}

mybutton . hello whadda -bg aquamarine

{*}$args is the right way to do it, but prior to Tcl 8.5, it was necessary to use eval:

eval {button $myname -text $label -command [list puts stdout $label]} $args

Alternatively, explicitly combine the lists first rather than relying on eval to concatenate its arguments:

set cmd [concat {button $myname -text $label -command [list puts stdout $label]} $args]
eval $cmd

Quoting at the C level

When composing an individual command to be evaluated from inside C code, use Tcl_Merge, which composes a list from argv. The result can then be passed to Tcl_Eval. Tcl_VarEval does not take pains to compose a list from its arguments, but simply concatenates them together. If its arguments happen to not be lists, the results could be unexpected. APN This advice is really applicable only when you have the command in the form of argv to begin with. Normally, at the C level arguments already exist as an array or list of Tcl_Obj structures and you should use Tcl_EvalObj* variants to evaluate commands, not Tcl_Eval or Tcl_VarEval.

Misc

AMG: Tcl by Ian Lance Taylor, 2011-03-31, discusses Tcl and alleges that its EIAS philosophy is its downfall. He argues that EIAS requires very precise quoting in order to get anything nontrivial to work right: "Even then I recently wrote some Tcl code with seven consecutive backslashes, admittedly in a complex use case. That's too much for easy reasoning, and in practice requires trial and error to get right." Sounds like a case of Quoting Hell, alright. I wish I had the chance to see the code in question and suggest an alternative, since in my experience there's always been a safe, clean way to avoid Quoting Hell.

AMG: Here's pooryorick's response to the article linked above:

This post mis-characterizes most of the aspects of Tcl that it attempts to describe. In particular, the comment about seven consecutive backslashes is a strong hint that Ian didn’t grasp the elegance of Tcl quoting. This happens when people come to Tcl steeped in other language traditions, and attempt to apply those traditions to to Tcl, which is a different sort of creature. With all due respect to Ian, each of the criticisms in this article indicate that he just didn’t stick with Tcl long enough, or perhaps look into it deeply enough to resolve his misunderstandings of the language. More info at [L1 ].

To Do