Updated 2014-06-02 09:56:15 by dbohdan

JSON is a lightweight, text-based, language-independent data interchange format.

See Also  edit

JSON-RPC
JSON is also used as the data format for a lightweight RPC protocol
Tcllib JSON
Tclon
yajl-tcl, by Karl Lehenbauer
a Tcl binding for yajl (yet another json library). Purportedly much faster than Tcllib json or huddle.
JSON, Rosetta Code
includes code for translating Tcl data structures to json using 8.6's ::tcl::unsupported::representation feature:
A Tcl extension written in C for parsing JSON, by Mikhail Teterin
announcement, comp.lang.tcl, 2014-01-14. This extension made its way into Tcllib, with slight modifications.

Documentation  edit

RFC 7158, The Javascript Object Notation (JSON) Data Interchange Format
JSON Example

Reference  edit

What is JSON, JSON-RPC and JSON-RPC-Java?, Slashdot, 2005-01-24

Description  edit

The JSON site probably explains it best:
"JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate. It is based on a subset of the JavaScript Programming Language, Standard ECMA-262 3rd Edition - December 1999. This feature can also be found in Python. JSON is a text format that is completely language independent but uses conventions that are familiar to programmers of the C-family of languages, including C, C++, C#, Java, JavaScript, Perl, TCL, and many others. These properties make JSON an ideal data-interchange language."

string is double gotcha  edit

[gchung] 2012-11-06: Tcl's string is double does not conform to JSON specification. For testing valid numbers, you should use the following regexp:
proc is_valid_json_number {value} {
    regexp -- {-?(?:[1-9][[:digit:]]*|0)(?:\.[[:digit:]]+)?(?:[eE][+-]?[[:digit:]]+)?} $value
}

Using string is double allows non-valid numbers to slip through. E.g. "00", "0x0", "0.".

AMG: That regular expression may be fine for limiting the acceptable number forms, but also continue to use string is double to check for over/underflow.

JSON in OpenACS  edit

OpenACS has its own take on handling JSON data, see [1]. From the intro:
   Utility ad_procs for Tcl <-> JSON conversion.

   This is based on the tcllib json package written by Andreas Kupries, and
   later rewritten to parse via regular expressions by Thomas Maeder.

   The tcllib version suffers from generating Tcl structures from JSON strings
   with no type (JSON array or object) information.  The resulting structures can't
   be converted back to JSON strings, you have to munge them with type information
   first.  And the code making use the Tcl structure also needs to know whether each
   field is an object or array.

   It also depends on the DICT package or Tcl 8.5.

   This rewrite doesn't depend on DICT, declares procs using ad_proc (so the API
   will be picked up by our API browser), and is symmetrical (you can convert from
   JSON to the Tcl representation and back again).

Misc  edit


JMN 2009-01: The way I see it, a JSON generator is a little more useful than a JSON parser. The primary usecase I've had for JSON is pumping a chunk of structured data to the client where there is a JavaScript engine available to consume it.

Whilst in some scenarios JSON-formatted structured data might also be sent to the server where another language such as Tcl or PHP might need to parse it - it seems more common that the server-side language gets its data as simple values already split into the fields of a query string or form POST data.

TCV 2009-03-08: I've been solving this problem lately myself, the primary difficulty being how to encode the multiple types which are available in JSON in an ambiguously typed Tcl value. The solution for me is to make all values lists where the first element is the type name and the rest are the data for that type. Objects are tagged "obj" and then followed by alternating keys and values (the keys are strings so there's no reason to tag them, but values must be). Lists are tagged "list" and every subsequent element is an item in the list. Strings are "string", numbers are "num", and booleans are "bool". The special "null" when appearing alone is null. I have some preliminary code, not quite ready for release, on my website [2].

slebetman 2009-04-01: The huddle package in tcllib does the tagged data thing quite well. And huddle has a built-in jsondump subcommand. I've been working on a different approach though. Having tagged data means I either have to jump through hoops to access the data in Tcl or rely on third party APIs (can't use regular list and dict commands for example). What I've done instead is to implement a JSON compiler. Given a specification of the data structure, and the tcl data, generate a JSON string:
# data is plain old tcl values
# spec is defined as follows:
# {string} - data is simply a string, "quote" it if it's not a number
# {list} - data is a tcl list of strings, convert to JSON arrays
# {list list} - data is a tcl list of lists
# {list dict} - data is a tcl list of dicts
# {dict} - data is a tcl dict of strings
# {dict xx list} - data is a tcl dict where the value of key xx is a tcl list
# {dict * list} - data is a tcl dict of lists
# etc..
proc compile_json {spec data} {
    while [llength $spec] {
        set type [lindex $spec 0]
        set spec [lrange $spec 1 end]
        
        switch -- $type {
            dict {
                lappend spec * string
                
                set json {}
                foreach {key val} $data {
                    foreach {keymatch valtype} $spec {
                        if {[string match $keymatch $key]} {
                            lappend json [subst {"$key":[
                                compile_json $valtype $val]}]
                            break
                        }
                    }
                }
                return "{[join $json ,]}"
            }
            list {
                if {![llength $spec]} {
                    set spec string
                } else {
                    set spec [lindex $spec 0]
                }
                set json {}
                foreach {val} $data {
                    lappend json [compile_json $spec $val]
                }
                return "\[[join $json ,]\]"
            }
            string {
                if {[string is double -strict $data]} {
                    return $data
                } else {
                    return "\"$data\""
                }
            }
            default {error "Invalid type"}
        }
    }
}

# Usage:
% compile_json {dict * list} {a {1 2 3} b {4 5}}
{"a":[1,2,3],"b":[4,5]}

# Data may be nested:
% compile_json {dict * {list {dict d list}}} {a {{c 1} {d {2 2 2} e 3}} b {{f 4 g 5}}}
{"a":[{"c":1},{"d":[2,2,2],"e":3}],"b":[{"f":4,"g":5}]}

For the last example, the specification reads as:
{
    dict <---------------- a dict
        * {list     <------- where all (*) values are lists
            {dict     <----- of dicts
                d list  <----- where the value of "d" is a list
            }                and all other values are strings (default)
          }
}

Steve Bennett 2010-10-13: I independently came up with something almost identical. My version correctly escapes string values using:
return \"[string map [list \\ \\\\ \" \\" \n \\n / \\/ \b \\b \r \\r \t \\t] $value]\"

Also, rather than automatically deciding whether to encode a string or a number, I use 'num' as a type. This also allows the special values false, true and null to be encoded as bare values rather than being quoted.

kanryu 2009-04-15: It's good way! I implement one for huddle as "huddle compile" :)
% huddle jsondump [huddle compile {dict * {list {dict d list}}} {a {{c 1} {d {2 2 2} e 3}} b {{f 4 g 5}}}] {} {} 
{"a":[{"c":1},{"d":[2,2,2],"e":3}],"b":[{"f":4,"g":5}]}

slebetman 2009-04-12: That's great kanryu. However I'd prefer this to be implemented as part of the json package rather than huddle. It would avoid having to do the conversion twice (dict->huddle->json). Could the maintainer of the json package in tcllib please steal my code and make a nice json::compile command? Pretty please..? ;-)

Some tests of json package using examples given on the JSON page.
if {[info script] eq $argv0} {
    set examples {
        {
            "glossary": {
                "title": "example glossary",
                "GlossDiv": {
                    "title": "S",
                    "GlossList": [{
                        "ID": "SGML",
                        "SortAs": "SGML",
                        "GlossTerm": "Standard Generalized Markup Language",
                        "Acronym": "SGML",
                        "Abbrev": "ISO 8879:1986",
                        "GlossDef": 
                        "A meta-markup language, used to create markup languages such as DocBook.",
                        "GlossSeeAlso": ["GML", "XML", "markup"]}]}}
        }
        
        {"menu": {
            "id": "file",
            "value": "File:",
            "popup": {
                "menuitem": [
                    {"value": "New", "onclick": "CreateNewDoc()"},
                    {"value": "Open", "onclick": "OpenDoc()"},
                    {"value": "Close", "onclick": "CloseDoc()"}
                ]
            }
        }
        }
        
        {"widget": {
            "debug": "on",
            "window": {
                "title": "Sample Konfabulator Widget",
                "name": "main_window",
                "width": 500,
                "height": 500},
            "image": { 
                "src": "Images/Sun.png",
                "name": "sun1",
                "hOffset": 250,
                "vOffset": 250,
                "alignment": "center"},
            "text": {
                "data": "Click Here",
                "size": 36,
                "style": "bold",
                "name": "text1",
                "hOffset": 250,
                "vOffset": 100,
                "alignment": "center",
                "onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"
            }
        }
        }            
    
        {"menu": {
            "header": "SVG Viewer",
            "items": [
                {"id": "Open"},
                {"id": "OpenNew", "label": "Open New"},
                null,
                {"id": "ZoomIn", "label": "Zoom In"},
                {"id": "ZoomOut", "label": "Zoom Out"},
                {"id": "OriginalView", "label": "Original View"},
                null,
                {"id": "Quality"},
                {"id": "Pause"},
                {"id": "Mute"},
                null,
                {"id": "Find", "label": "Find..."},
                {"id": "FindAgain", "label": "Find Again"},
                {"id": "Copy"},
                {"id": "CopyAgain", "label": "Copy Again"},
                {"id": "CopySVG", "label": "Copy SVG"},
                {"id": "ViewSVG", "label": "View SVG"},
                {"id": "ViewSource", "label": "View Source"},
                {"id": "SaveAs", "label": "Save As"},
                null,
                {"id": "Help"},
                {"id": "About", "label": "About Adobe CVG Viewer..."}]}
        }
    }
    
    foreach example $examples {
    puts "convert: $example"
    puts [json::json2dict "\{$example\}"]
    }
}

RLH: I do a package require JSON on OSX and the library isn't there or my setup is broken?

JH: The package name is "json".

RLH: Is there any kind of "rule" about package names? I should have tried the lowercase json but didn't think of it. Just wondering.

RLH: I found it after doing a find. I am stupid sometimes.

HolgerJ 2008-08-14: when I do a package require json, I get a message "can't find package dict". I checked the Tcl version (Kubuntu Hardy 8.04) and found out it was 8.4.16; after installing 8.5.0 and calling the interpreter with tclsh8.5, it seems to work.

glennj: you can get the dict package for tcl 8.4: see the dict page

"tom.rmadilo" writes on comp.lang.tcl:
I put up a JSON text to Tcl list converter for my own testing, if anyone wants to see what the Tcl format looks like:

wttp://www.rmadilo.com/json/

It also prints out debugging info on a char-by-char basis

There is also http://code.google.com/p/tcl-json/, a package using Yeti.

An example of http and JSON doing PHP page queries IP-geolocation

[Antender] 2010-12-25 13:51:59:

I'll post here my 1 line JSON parser which has such advantages:

  • Parses JSON directly in dict structure
  • Handles all JSON datatypes: arrays, objects, strings, other values
  • Has O(N) difficulty, because of the use of strict replaces

Reformatted version by Antender and AMG. It's indented for readability, it combines many of the regsubs into a string map, and uses a single string range instead of two executions of string replace.
proc json2dict JSONtext {
string range [
    string trim [
        string trimleft [
            string map {\t {} \n {} \r {} , { } : { } \[ \{ \] \}} $JSONtext
            ] {\uFEFF}
        ]
    ] 1 end-1
}

Antender: Added dict2json. Have no support of JSON arrays due to Tcl's dynamical typing.
proc dict2json {dictionary} {
    dict for {key value} $dictionary {
        if {[string match {\[*\]} $value]} {
            lappend Result "\"$key\":$value"
        } elseif {![catch {dict size $value}]} {
            lappend Result "\"$key\":\"[dict2json $value]\""
        } else {
            lappend Result "\"$key\":\"$value\""
        }
    }
    return "\{[join $Result ",\n"]\}"
}

dbohdan 2014-06-01: Unfortunately, because of how it uses string map the above implementation of json2dict will corrupt URLs stored in JSON values. A minimal example that demonstrates the problem:
eltclsh > set j {{"results": [{"artworkUrl512":"http://a1379.phobos.apple.com/long/path/mzl.lqtjexwh.png"}]}}
{"results": [{"artworkUrl512":"http://a1379.phobos.apple.com/long/path/mzl.lqtjexwh.png"}]}
eltclsh > puts [::json::json2dict $j]
results {{artworkUrl512 http://a1379.phobos.apple.com/long/path/mzl.lqtjexwh.png}}
eltclsh > puts [json2dict $j]
"results"  {{"artworkUrl512" "http //a1379.phobos.apple.com/long/path/mzl.lqtjexwh.png"}}

dbohdan 2014-06-01: I found the following function useful for accessing JSON data converted with ::json::json2dict.
proc jsonget {json args} {
    foreach key $args {
        if {[dict exists $json $key]} {
            set json [dict get $json $key]
        } elseif {[string is integer $key]} {
            if {$key >= 0 && $key < [llength $json]} {
                set json [lindex $json $key]
            } else {
                error "can't get item number $key from {$json}"
            }
        } else {
            error "can't get \"$key\": no such key in {$json}"
        }
    }
    return $json
}

The motivation behind it is that right now ::json::json2dict converts JSON arrays into lists. As a result you need to chain calls to lindex and dict get to access values stored inside complexly structured JSON blobs. There is an issue on the Tcllib tracker to add the option -indexlists to ::json::json2dict that would make it index arrays into dicts of the form {0 data1 1 data2...} but it is not yet merged: https://core.tcl.tk/tcllib/tktview?name=2967134fff. With that in mind the above function allows you to say
jsonget [json::json2dict $j] results 0 artworkUrl512

instead of
dict get [lindex [dict get [json2dict $j] results] 0] artworkUrl512

dbohdan 2014-06-02: Fixed unnecessary exprs.