Updated 2017-05-25 14:26:10 by harmolthof

See http://www.tcl.tk/man/tcl/TclCmd/scan.htm for the formal man page for the Tcl command scan.
scan string format ?varName ...?

If varNames are supplied, will store values parsed (from string according to format) in the named variables and return the number of variables written to. If no varNames are present, returns the list of values parsed.

One important use is to possibly strip off leading zeroes from decimals:
 scan $possiblyZeroedDecimal %d cleanInteger

Context: Leading zeroes induce octal parsing in the expr command and in if and similar commands that invoke expr. Use of scan to remove leading zeros can avoid this often-reported problem.

One very frequently-asked question is how one converts between display and numeric format for ASCII characters. Scan provides the usual answer:
    foreach character {a A b B c} {
        scan $character %c numeric
        puts "ASCII character '$numeric' displays as '$character'."

This makes the table
    ASCII character '97' displays as 'a'.
    ASCII character '65' displays as 'A'.
    ASCII character '98' displays as 'b'.
    ASCII character '66' displays as 'B'.
    ASCII character '99' displays as 'c'.

In more recent Tcl's (8.3 or so up) you also can call scan inline:
 set numeric [scan $character %c]

And the %c format is not limited to ASCII, but can handle any Unicode. (RS)

LV Can someone explain what this means - calling scan inline? Where is scan getting its input in this case? From stdin?

Lars H: It refers to the result of scan, not its input. Rather than setting some variables, it will return a list of the parsed values. (RS's example is rather ugly, since it really sets $numeric to a list with one element, but as quick character code look-up at a prompt, a scan char %c is hard to beat.)

Alastair Davies: When using in-line scan, note that the list that is returned will contain blank elements for format specifiers that are not satisfied. If there is only one format specifier, a list with one blank element is returned, and note especially that this is not the same as an empty string. For example, scan foo %d will return {}. Now, noting the statement on the man page:
 In the inline case, an empty string is returned when the end of the input string
 is reached before any conversions have been performed.

you might think that scan foo %d would return an empty string. But Don Porter carefully explains this special case (quoting from c.l.t.)

Back to the example:
 % scan foo %d

We're going to scan the string "foo", trying to parse a decimal integer from it, and when we're done, we're going to return the results as an inline list, since we provided no variables for field value storage.

While scanning, we see "f", and "f" can't be part of a decimal integer. In the non inline case, we would assign no value to the variable associated with this field. In the inline format, an empty string is stored in the corresponding list location to represent that situation.

Now we've exhausted the format spec string. But note that we haven't reached "the end of the input string." We're still sitting at the "f". So the "underflow" case where we run out of input before we run out of format spec does not apply, and we should not expect scan to return the empty string.

To see the underflow case in action, consider these examples:
 % scan a ab
 % scan {} %d
 % scan {   } %d

In all these cases we run out of string to parse before we run out of format spec string to guide our parsing. There are more cases where this happens, but they're difficult to construct. Consult the sources and the test suite for details and more examples.

Part of the reason this does tend to be confusing is that the detection of the "underflow" case and returning the special magic empty value when it is detected is just about 100% worthless in Tcl. It's a feature of sscanf() in C that was apparently slavishly copied over without full consideration of whether it had any value in Tcl. (A common problem in built-in commands and features from Tcl's earliest days. Octal, anyone?)

scan's rules are arcane and weird, but they are what they are.

To match specified characters use %[chars], to match un-specified characters use %[^chars]:
% scan 12a34 {%d%[abc]%d} x - y
% list $x $y
12 34

Notice that the above scan's format string says "some decimal number, followed by either an a, or a b or a c, followed by another decimal number". Then, the last 3 arguments are x, the name of the variable to be assigned the first decimal number, - , the name of the variable to be assigned the letter, and y, the name of the second decimal number. The name - was chosen with the intent to convey "oh, this value isn't going to be used".

Also see "Dump a file in hex and ASCII", as well as "u2x" in the "Bag of algorithms".

Binary has a scan which complements use of this scan.

AMG: Even a simple numeric scan always results in shimmering.
% set var [expr 123]; list
% tcl::unsupported::representation $var
value is a int with a refcount of 2, object pointer at 0317BDF8,
internal representation 0000007B:0317BD68, no string representation
% scan $var %d; list
% tcl::unsupported::representation $var
value is a int with a refcount of 2, object pointer at 0317BDF8,
internal representation 0000007B:0317BD68, string representation "123"

I'd think the single numeric conversions could be optimized to recognize when the input already has numeric representation, then simply return the input.

AMG: Tcl's [scan] doesn't integrate terribly well with file I/O. In C, scanf() and fscanf() try to read as many characters as needed for the format, but in Tcl, you need to know in advance how many characters to read. I find this limitation to be very unfortunate, but I don't know a good solution.

Perhaps [scan] could be given a special option to get its data from a channel rather than a string, then the C-level guts of the [scan] and file I/O implementations can work everything out much more efficiently and correctly than can be done at the Tcl level.

Netstrings are one scenario where this would be helpful. You need to read a length prefix (not knowing in advance how many digits form the length), then confirm the next character is colon (for framing), then read the data (length in bytes indicated by the prefix), then confirm the final character is comma (for framing).

In C, you just do scanf("%d", &len) to get the length prefix. But in Tcl, you have to [read] one character at a time until you find a non-digit. That's a lot of looping and script which would be much better handled at the C level.

(Pedantic aside: the scanf() approach isn't technically correct because it allows negative lengths and leading zeroes, but you can check for negatives and can forgive leading zeroes.)

(And a question: with scanf("%d:", &len), how can you tell whether or not the colon was found in the input? I think the answer is to use %n. Trouble with that is inconsistency in whether successful scanf("%d:%n", &len, &pos) returns 1 or 2. So set pos to 0 first then check if it changed. What a mess.)

Let's say you have all the data read in advance, e.g. it came from a UDP datagram, thereby making Tcl [scan] suitable for reading the length prefix. [scan $datagram %d len] works, but how do you know how many digits were consumed? Guess you'd need %n for that too: [scan $datagram %d:%n len pos]. Now the data is [string range $datagram $pos [expr {$pos + $len - 1}], and [string index $datagram [expr {$pos + $len}]] should be a comma. So far, so good.

Buuuuuuuuut how do you now process the next netstring in the sequence? You have to use [string range] to chop off the consumed portion, which involves a lot of copying. I recently had a great deal of trouble when doing similar processing with strings many megabytes in size, wishing to scan a few fields from the front at a time. So if we're going around adding options to [scan], I also suggest adding a -start option to skip leading characters. [regexp] has such an option. Or go the [binary scan] route and offer %[email protected] to skip to an arbitrary location, though I prefer the option approach.

(By the way, [scan] also has a serious slowdown with large input strings, possibly due to conversion from UTF-8 to UCS-2 or whatever for random access, even when random access is not required. Fix that too.)