Templates and subst

APW 2007-06-07 The contents of this page has been moved from TemplaTcl: a Tcl Template engine to here, to be able to reference that code. It is an example on how to use a simple templating language which can have Tcl code interspersed.

jcw 2007-06-03: But I don't see the point in embedding an escape layer using XML. Here's a visually simpler approach

% if {$a != $b} {
    <h3> abc </h3>
    <li> [expr {$a != 1 ? $var1 : $var2}] </li>
% } else {
    <li> [myobj getResult] </li>
    <table>
        <tr> Hello there </tr>
    </table>
% }

What you get is the ability to use Tcl's [subst] where convenient, and an outer-level line-by-line escape for larger amounts of code. Just like in Tcl, where we use both if's and expr's.

The example is contrived and has relatively much flow control, so it doesn't come out as clearly as real-world examples would. The above isn't an arbitrary choice btw - it's what Mason uses. As it turns out, less than a dozen lines of Tcl are needed to convert the above into a normal Tcl script which consists mostly of subst calls. In fact, I use a second level of <%...%> escapes as well for structuring the entire files into "components". This is the sort of thing Mason has worked out long ago, driven by practical use, not mere preference of a specific technology or language construct.

Before taking this discussion further, my suggestion would be to take some notation and run with it for a while to see just well it works out. You can't discuss these design choices without using them in several decently-sized real-world website applications. Well, not meaningfully anyway - IMO.

APW: That approach looks very good to me and makes sense in that the final HTML layout is still visible in the template! Thanks to jcw for bringing that in.

jcw: Here's the code for that approach:

proc substify {in {var OUT}} {
    set pos 0  
    foreach pair [regexp -line -all -inline -indices {^%.*$} $in] {
        lassign $pair from to
        set s [string range $in $pos [expr {$from-2}]]
        append script "append $var \[" [list subst $s] "]\n" \
                                    [string range $in [expr {$from+1}] $to] "\n"
        set pos [expr {$to+2}]
    }
    set s [string range $in $pos end]
    append script "append $var \[" [list subst $s] "]\n"
}

You give it a template, and it returns a script you can eval to apply that template. I turn such scripts into procs, but that's just one way to use this.

APW: Great work jcw, that's exactly, what I am looking for.

pyk: AMG highly recommends JCW's [substify], and uses a variant of it to drive his Web site [L1 ], including a custom search engine made for a friend. See his code here: [L2 ].

  • First look at template [L3 ]
  • Then look at [emit_template] in site.tcl [L4 ]
  • Which is called near the bottom of script [L5 ].

see also: CGI script for directory listing, simple posts and templates

FF I noticed a little problem with switch:

% switch what {
% case1 {
something
% }
% what {
something other
% }
% }

It seems that two adjacent % lines generate an empty append line, and a construct like switch don't like that.

APW: Here is a slightly modified version of substify which should handle switch correctly:

proc substify {in {var OUT}} {
    set script ""
    set pos 0
    foreach pair [regexp -line -all -inline -indices {^%.*$} $in] {
        foreach {from to} $pair break
        set s [string range $in $pos [expr {$from-2}]]
        if {[string length $s] > 0} {
            append script "append $var \[" [list subst $s] "]\n"
        }
        append script "[string range $in [expr {$from+1}] $to]\n"
        set pos [expr {$to+2}]
    }
    set s [string range $in $pos end]
    if {[string length $s] > 0} {
        append script "append $var \[" [list subst $s] "]\n"
    }
    return $script
}

jcw: Good catch, thanks!

FYI, fixed a missing init of $script in the above (when $in is empty).

merlin 2007-11-02 17:00 GMT: I think, it is simple to use a native tcl interpreter to parse templates. We have a simple template syntax, which is easy to convert ("compile" a template) into proper and correct tcl code. Doing subst on that code ("fetch" a template), we get a rendered page.

Compiling can be done without any variables set, which are used in the template. "Fetching" requires these variables to be set in the current scope (else we get a tcl error, about using unset variables).

# ====== Compiler ======
# compilng section block
proc parse_section {arg} {return "\[_section [lindex $arg 0] [parse [lindex $arg 1]] \{"}
proc parse_end_section {} {return "\}\]"}

# compiling if-else blocks
proc parse_if {arg} {return "\[if [parse $arg] \{_echo \""}
proc parse_else {arg} {return "\"\} else \{_echo \""}
proc parse_end_if {} {return "\"\}\]"}

# curly braces
proc parse_lbr {arg} {return "\\\{"}
proc parse_rbr {arg} {return "\\\}"}

# Determines block {...} type, passed to this proc as str parameter
# and calls a proc to change that content into complement tcl-code.
# Returns a new line to substitute {...} block
proc parse {str} {
    if [string equal -length 1 $str \$] {
        # Block starting with $ is a command to put variable value
        if [regexp {\$(.+)\[(\d+)\]} $str trash vh vi] {
            # If there are square brackets with an index in there, like [3],
            # replacing that recursively with call to list index
            set str "\[lindex [parse "$$vh"] $vi\]"
        }
    } elseif [string equal -length 1 $str /] {
        # Block starting with / is an end for some block.
        # It can't contain any parameters
        set cmd [string range $str 1 end]
        set str [parse_end_$cmd]
    } else {
        if [regexp {^(\w+)(?: (.+))?$} $str trash cmd arg] {
            # If the block begins with the word, that word is considered as
            # command name, the rest is parameters of that command
            set str [parse_$cmd $arg]
        }
        # !TODO!: allow "else" with expressions parsing
    }
    return $str
}

# Find and replace any {...} to [parse ...], then subst the result
proc compile {tpl} {
    upvar template template

# --- Save ", ], [, $ between blocks ---
    # Find coordinates of that "betweens"
    set beg 0
    set escapes {}
    while {[regexp -start $beg -indices {\{([^\}]+)\}} $tpl res]} {
        set escapes [concat $beg [expr [lindex $res 0]-1] $escapes]
        set beg [expr [lindex $res 1]+1]
    }
    set escapes [concat $beg [expr [string length $tpl]-1] $escapes]
    # $escapes now contain a reversed list of blocks to escape

    # Escape rules
    set escamap {\" \\\\\\\" \$ \\\\\\\$ \[ \\\\\\\[ \] \\\\\\\]}
    # " (extra \ for subst)

    foreach {i j} $escapes {
        set tpl [string replace $tpl $i $j [string map $escamap [string range $tpl $i $j]]]
    }

    set template $tpl

    # Regexp below finds any {...} and replaces them to [parse ...].
    # then subst calls there parses, and strips some backslashes
    uplevel {return [subst [regsub -all {\{([^\}]+)\}} $template {[parse {\1}]}]]}
# !TODO! rewrite a compile to more clean code,
# allowing a nested {} in the {...} blocks,
# which will enable syntax like {section i {a b c}}<li>{$i}</li>{/section}
}

# ====== "Fetcher" ======
proc _echo {val} {return $val}
proc _section {name list block} {
    set result {}
    foreach $name $list {append result [subst $block]}
    return $result
}

proc fetch {ctpl} {
    upvar template template
    set template $ctpl
    uplevel {return [subst $template]}
}

Usage of this code is demonstrated below (assuming code is in the file cp.tcl):

source "cp.tcl"

# A template
set tpl {Some{if $e}<ul>{section i $cycle}<li>"{$i}"</li>{/section}</ul>{else}"{$cycle[1]}"{/if}tails}

# Data for that template
set e 1
set cycle {one two three}

# ====== Call of compiler and fetcher ======
puts "Source template: $tpl"
puts "Compiled template: [compile $tpl]"
puts "Filled-in block: [fetch [compile $tpl]]"

Everything (fetch, compile) is done with subst =)


AMG: [substify] is great stuff. Thanks, JCW! See Web Templating for more information on how I use it. (Actually, comments on how Andy is using this was moved to this page above, just after "AMG highly recommends...")

AMG again: Here is [substify] adapted to work as a simple Unix-style text filter.

package require Tcl 8.5
proc applytemplate {template} {
    set script ""
    set pos 0
    foreach pair [regexp -line -all -inline -indices {^%.*$} $template] {
        lassign $pair from to
        set str [string range $template $pos [expr {$from - 2}]]
        if {$str ne ""} {
            append script "puts \[" [list subst $str] \]\n
        }
        append script [string range $template [expr {$from + 1}] $to]\n
        set pos [expr {$to + 2}]
    }
    set str [string range $template $pos end]
    if {$str ne ""} {
        append script "puts -nonewline \[" [list subst $str] \]
    }
    uplevel 1 $script
}
while {[lindex $argv 0] eq "-file"} {
    set chan [open [lindex $argv 1]]
    set argv [concat [lrange $argv 2 end] [read $chan]]
    close $chan
    unset chan
}
dict with argv {applytemplate [read stdin]}

Example usage:

$ cat enums.c.tmpl
#include <stdlib.h>
typedef enum {
% foreach enum $enums {
    $enum,
% }
} ${name}_t;
char *${name}_names\[\] = {
% foreach enum $enums {
    "$enum",
% }
    NULL
};
% # This is a comment.
$ ./template.tcl enums "a b c d e" name "letter" < enums.c.tmpl > enums.c
$ cat enums.c
#include <stdlib.h>
typedef enum {
    a,
    b,
    c,
    d,
    e,
} letter_t;
char *letter_names[] = {
    "a",
    "b",
    "c",
    "d",
    "e",
    NULL
};

Lordmundi: [substify] is great. I'm using it to serve web templates like I'm guessing many others. However, I ran into an issue where the pages I'm serving have javascript in them, and therefore, lots of dollar signs("$") in them, which are getting interpreted by subst. I can backslash them, but this tends to make it very messy. Is there an easy way to edit substify such that another char, possible "#" at the beginning denotes pure text much in the way that % marks pure tcl? I tried editing it a couple of times and am not having much success.


Bezoar : I ran in to same problem as Lordmundi above. I was able to just declare tcl variables to resubstitute themselves back in to the script. Thank goodness the {} is a valid array name. For example:

% array set {} { document "$(document)" "'#sqlresult'" "$('#sqlresult')" }
    <script>
      $(document).ready(function() {
       $('#sqlresult').dataTable();
      } );
     $(document).ready( function () {
       $('#sqlresult').dataTable( {
        "bPaginate": false
       } );
      } );
     $(document).ready( function () {
     $('#sqlresult').dataTable( {
        "bLengthChange": true
       } );
      } );
   </script>

Will result in the script block going through unchanged.

jmn 2023-08-28 TIP 672 proposes to use the $(..) syntax for expr evaluation. https://core.tcl-lang.org/tips/doc/trunk/tip/672.md

The empty array name is still allowed - but presumably the trick above won't work if this goes ahead.

I love clever tricks like Bezoar's one above - but there's always the risk they won't work forever if they're a little off the beaten track.

On the other hand - there is an alternative expr syntax proposal in TIP 676 https://core.tcl-lang.org/tips/doc/trunk/tip/676.md

That one will affect people who are doing interesting/tricky things with = as a command.

I don't imagine that both will go through - but who knows. I put this here as a reminder to people to keep an eye on the TIPs to see what's in the pipeline that may affect features that are important to you.