tDOM

What:	tDOM
Where:	http://tdom.github.com/ (NOTE: Replacement for www.tdom.org as of 11/2010) http://groups.yahoo.com/group/tdom
Repository:	git://github.com/tDOM/tdom.git (public read-only access)
Issue Tracker:	https://github.com/tDOM/tdom/issues
Description:	C based XML extension for Tcl. Based on the Expat parser, with uses Tcl namespaces, allows you to access the DOM trees as Tcl DOM objects. Includes an HTML reader that reads HTML and generates a DOM tree.
	Currently at v0.8.3.
Updated:	11/2010
Contact:	See web site

This tightly-coded extension emphasizes speed and memory economy. In contrast to the "Pure Tcl" TclDOM, it bests leading Java DOM implementations by an order of magnitude (!) in both processing speed and memory demand. Jochen also exploits tDOM's expressivity to offer a nice HTML reader, XML validator--called tnc--and XSLT engine. tDOM is in production at several commercial installations.

The official home of Jochen Loewer's and Rolf Ade's tDOM XML engine was http://www.tdom.org/

(Dead as of 10/2010). [bovine] recently setup a github organization with a mirror of the old website and a fork from the repository makr kept as mirror of the old CVS repository.

0.8.3 doesn't compile for Tcl version 8.6. Use trunk or a later version instead.

Documentation edit

Overview
dom: Create an in-memory DOM tree from XML
domDoc: Manipulates an instance of a DOM document object
domNode: Manipulates an instance of a DOM node object
expat: Creates an instance of an expat parser object
expatapi: Functions to create, install and remove expat parser object extensions.
tdom: tdom is an expat parser object extension to create an in-memory DOM tree from the input while parsing.
tnc: tnc is an expat parser object extension, that validates the XML stream against the document DTD while parsing.

Tutorials and articles edit

A tDOM tutorial: available here on the Wiki.
examples on this page
tDOM – A fast XML/DOM/XPath package for Tcl written in C1: by Jochen Loewer, a valuable description of tDOM's origins and uses presented at the first European Tcl Conference.
Processing XML documents with Tcl and tDOM: Carsten Zerbst, 2003, illustrating how practical and easy tDOM is. And here's the same article in german: XML-Dokumente mit Tcl und tDOM bearbeiten

Examples edit

Downloading your utility usage from Pacific Gas and Electric using TCL
Scraping timeentry.kforce.com

An XPath query

a quick way to get values from XML documents:

PS 2004-11-04:

set xml {
<agents>
    <agent id="007">
        <name type="first">James</name>
        <name type="last">Bond</name>
        <age>Still attractive</age>
        <sex>Male</sex>
    </agent>
    <agent id="013">
        <name type="first">Austin</name>
        <name type="last">Powers</name>
        <age>Depends on your timeline</age>
        <sex>Yes, please</sex>
    </agent>
</agents>
}


set dom [dom parse $xml]
set doc [$dom documentElement]
puts "Agent: [$doc selectNodes {string(/agents/agent[@id='013']/@id)}]"
puts "First Name: [$doc selectNodes {string(/agents/agent[@id='013']/name[@type='first'])}]"
puts "Last Name: [$doc selectNodes {string(/agents/agent[@id='013']/name[@type='last'])}]"
puts "Age: [$doc selectNodes {string(/agents/agent[@id='013']/age)}]"

Will output:

Agent: 013
First Name: Austin
Last Name: Powers
Age: Depends on your timeline

An XPath query with a namespace

Trying to retrieve elements from a namespace was confusing me until I found a post

demonstrating the use of -namespaces with selectNodes.

% set fh [open small.gpx]
file13bbfa8
% set xmldata [read $fh]
<?xml version="1.0"?>
<gpx
version="1.0"
creator="ExpertGPS 1.1 - http://www.topografix.com"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.topografix.com/GPX/1/0"
xsi:schemaLocation="http://www.topografix.com/GPX/1/0 http://www.topografix.com/GPX/1/0/gpx.xsd">
<time>2002-02-27T17:18:33Z</time>
<bounds minlat="42.401051" minlon="-71.126602" maxlat="42.468655" maxlon="-71.102973"/>
<wpt lat="42.438878" lon="-71.119277">
<ele>44.586548</ele>
<time>2001-11-28T21:05:28Z</time>
<name>5066</name>
<desc><![CDATA[5066]]></desc>
<sym>Crossing</sym>
<type><![CDATA[Crossing]]></type>
</wpt>
<wpt lat="42.439227" lon="-71.119689">
<ele>57.607200</ele>
<time>2001-06-02T03:26:55Z</time>
<name>5067</name>
<desc><![CDATA[5067]]></desc>
<sym>Dot</sym>
<type><![CDATA[Intersection]]></type>
</wpt>
</gpx>

% close $fh

This is edited down from http://www.topografix.com/fells_loop.gpx

for brevity's sake.

Get the doc and the root.

% set doc [dom parse $xmldata]
domDoc013CAE80
% set root [$doc documentElement]
domNode013B1FDC

Try to get a list of waypoints

Define a namespace and use it with selectNodes

% set ns {gpxns http://www.topografix.com/GPX/1/0}
gpxns http://www.topografix.com/GPX/1/0

Sample XPath queries with and without the namespace

% $root selectNodes -namespaces $ns //wpt
% $root selectNodes -namespaces $ns //gpxns:wpt
domNode013B2060 domNode013B2194
% $root selectNodes -namespaces $ns {//gpxns:wpt[1]}
domNode013B2060

--skm 2006/12/10

kpv Another approach is to set the namespaces to be searched once in the beginning.

% $doc selectNodesNamespaces $ns
% $root selectNodes gpxns:wpt

Parse HTML

Here's an example of using tdom to parse some html:

package require tdom
package require http

# html - the source of the html page
proc pullOutTheURLs {html} {

# Parse your HTML document into a DOM tree structur
set doc [dom parse -html $html]

# root will be the root element of your HTML document,
# ie. the HTML element
set root [$doc documentElement]

# The following finds all anchor links <a>. It isn't clear to me,
# if you also interested in the urls of <area>, <link> and <base>
# elements.
set nodeList [$root selectNodes {descendant::a}]

# init the result list
set urlList {}

# Pull out the Values of the href attributes
foreach node $nodeList {
    set attList [$node attributes *]
    foreach attribute $attList {
        if {[string tolower $attribute] == "href"} {
            lappend urlList [$node getAttribute $attribute]
            break
        }   
    }
}

# Get rid of the DOM representation of your HTML document
$doc delete

# finished
return $urlList
}

# Test it set urlList [pullOutTheURLs ][1

]]

foreach url $urlList {
    puts $url
}

MHN 2012-11-08: I have not tested the performance, but the nested foreach looks like an overhead (compared to byte-code tdom)... How about the following?

# pull out the values of the href attributes
foreach attr [$root selectNodes {descendant::a/@href}] {
    lappend urlList [lindex $attr 1]
}

# if case insensitive matching is required, use {descendant::a/@*['href' = translate(name(),'HREF','href')]}

# if no URL matches "href" then it can be achieved in one line
set urlList [lsearch -all -inline -exact -not [concat {*}[$root selectNodes descendant::a/@href]] href]

XSLT Example

I am an XSLT newbie. I didn't find an XSLT example for tDOM on the wiki, so I thought I would provide one. It is almost too easy to warrant one, but I think it helps anyway.

I took the sample XML and simple XSL from the " What kind of language is XSLT?" [2] essay. For paste here, I will use a shorter version of the xml file.

% package require tdom
080
% set gamedata {<results group="A">
<match>
    <date>10-Jun-1998</date>
    <team score="2">Brazil</team>
    <team score="1">Scotland</team>
</match>
<match>
    <date>23-Jun-1998</date>
    <team score="0">Scotland</team>
    <team score="3">Morocco</team>
</match>
</results>}
% set gamedoc [dom parse $gamedata]
domDoc012E5690

xsldata was set to the first xsl example from the What kind of language is XSLT? essay. That is:

% set xsldata {
         <xsl:transform xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
            <xsl:template match="results">
            <html>
                    <head>
                    <title>
                        Results of Group <xsl:value-of select="@group"/>
                    </title>
                    </head>
                    <body>
                            <h1>
                                Results of Group <xsl:value-of select="@group"/>
                            </h1>
                            <xsl:apply-templates/>
                    </body>
            </html>
            </xsl:template>
            <xsl:template match="match">
                    <h2>
                        <xsl:value-of select="team[1]"/> versus <xsl:value-of select="team[2]"/>
                    </h2>
                    <p>Played on <xsl:value-of select="date"/></p>
                    <p>Result: 
                            <xsl:value-of select="team[1] "/> 
                            <xsl:value-of select="team[1]/@score"/>,
                            <xsl:value-of select="team[2] "/> 
                            <xsl:value-of select="team[2]/@score"/>
                    </p>
            </xsl:template>
    </xsl:transform>
}

% set soccerstyle [dom parse $xsldata]
domDoc012DB570
% $gamedoc xslt $soccerstyle gamehtml
domDoc012E6CC0
% $gamehtml asXML
<html>
    <head>
        <title>
        Results of Group A</title>
    </head>
    <body>
        <h1>
        Results of Group A</h1>
        <h2>Brazil versus Scotland</h2>
        <p>Played on 10-Jun-1998</p>
        <p>Result: 
            Brazil2,
            Scotland1</p>
        <h2>Scotland versus Morocco</h2>
        <p>Played on 23-Jun-1998</p>
        <p>Result: 
            Scotland0,
            Morocco3</p>
    </body>
</html>

Loading XML from file with the right encoding settings

tDOM provides some helpers for this in the tdom.tcl lib distributed with the tDOM. Cited from a comp.lang.tcl answer by Rolf Ade

tDOM::xmlOpenFile expects a filename and returns a file channel handle, which is readily fconfigure'd and seek'ed to get feeded into a dom parse -channel ... Please note, that the proc open a channel and returns that. That channel will not magically go away, if you're done with it. It's your responsibility to close that channel, if you don't need them anymore. So, a typical use pattern (sure, not the only) is

set xmlfd [tDOM::xmlOpenFile $filename]
set doc [dom parse -channel $xmlfd]
close $xmlfd

tDOM::xmlReadFile is just a wrapper around tDOM::xmlOpenFile. The pattern is

set doc [dom parse [tDOM::xmlReadFile $filename]]

and you're done. No leaking file channels, filename in, DOM tree out.

Obtaining and compiling tDOM edit

The current version is 0.8.3. Grab it from http://tdom.github.com/

The tDOM GIT repository allows read-only anonymous access. To get the latest status of the project just do

git clone git://github.com/tDOM/tdom.git

A collection of binaries for Win/Linux/Mac, in a single package is available at irrational-numbers project

At runtime, the right binary library will be used.

License issue: Since my employer applies stricter license checks now, I was told that I cannot use tdom in software for customers any more, because of a license conflict:

The library itself is under MPL (Mozilla Public License)
Two of the source files are under LGPL: generic/DOMhtml.c and generic/XMLsimple.c

According to the FSF, these two licenses are contradicting. Can this be helped? RS 2011-02-28

schlenk 2011-02-28 Those two LGPL headers seem to be from drh probably taken from tkhtml2? If yes it should be easy enough to fix, as DRH has release the tkhtml2 code into public domain and tkhtml3 is BSD licensed (was LGPL before). Probably just needs asking drh and Rolf Ade to fix it.

escargo 28 Feb 2008 - The "official home" does not provide any way to submit bug reports. I did report a problem to the Yahoo group, since seemed to be the only resort available.

snichols I recently compiled tdom 0.8.0 on Windows XP successfully with threads enabled, but when doing a package require from within Tcl 8.4.7 I get the following error, "too many nested evaluations (infinite loop?)" when doing the package require tdom command. Any ideas? snichols Thanks in advance. - RS: Oh yes, a silly bug - in the file tdom.tcl, comment out the line

package require tdom

a file can't require a package during providing it :)

snichols Thank you very much. That fixed the issue.

AK Aug 2, 2006. When was recently ? According to Rolf Ade this problem was fixed Sep 29, 2004. Both the package index generated by the TEA Makefile, and the one found in the 'win' directory load the shared library (i.e. DLL) first, then source tdom.tcl. As the DLL runs the C equivalent of 'package provide tdom' the 'package require tdom' executed by 'tdom.tcl' is satisfied and will not loop.

I've build a binary package of tDOM 0.7.4 for Debian GNU/Linux 3.0 (Woody) [3] -- Jochem Huhmann

Various combinations of mirrors, backups, and experimental work are often announced through the mailing list at http://groups.yahoo.com/group/tdom/

. This is particularly important, as sdf.lonestar.org seems relatively erratic in reliability.

The authors of tdom announces their new releases on the tdom yahoo group.

Phaseit provides a mirror

(This link is to a VERY old version of tdom).

An article on IBM's developerWorks pages

in Nov 2001 profiles a company (Ideogramic) that uses tDOM for XMI processing in its product.

PS 04Nov2004Deadlink Is it still available somewhere? Another article in February 2002

highlights tDOM's performance, while also supplying a recipe for your own start with the package. More recently,

In the midst of one chat conversation, Rolf Ade explained that, "The most valuable thing in tDOM is his [Jochen]'s C-coded xpath engine."

PT - I'll second that. tDOM's XPath processing is extremely competent.

A tDOM tutorial could include these points (proposed by Rolf Ade):

how to use the SAX interface (setting up event handler scripts, stacking event handler scripts, error handling etc)
how to parse (parse a string, or read the XML data out of a channel, with notes about the encoding problems)
how to get a DOM tree representation of XML data
the XPointer, DOM 1 and DOM 2 and XPath commands to find or to navigate to some nodes of interest
how to use tDOM's XSLT engine (to build an XSLT processor or in server application)
how to serialize DOM trees (as XML, HTML or text)
how to validate the XML data while parsing
how to create DOM trees from scratch
how to create additional, tcl scripted DOM methods
how to create additional, tcl scripted XPath functions

June 24th: tDOM-0.7.1 released. Enhancements to XSLT/XPath support -- now full compliant (thanks Rolf), Tcl-Thread support (thanks Zoran), a lot of bug fixes (Rolf), some HTML parser enhancements, even simple XML parser is namespace-aware.

Uhm, this "full compliant" XSLT support is a little bit to much said - Jochen would have done better, if he had used my "almost compliant".

Don't get this wrong. It is true, that tDOM's XSLT support was greatly improved over the last releases. I'm pretty sure, you could use every 'real live' XSLT stylesheet with it, with correct result. I won't confuse you with outlying nifty difty XSLT details, therefor I omit the list of things that are not quit right.

There are (prominent) tools out there, that don't do it better than tDOM's XSLT engine, due to my extensive testing, but nevertheless claim since a couple of months 100% XSLT complicance, which is simply not completely true, and nobody bothers.

So just use tDOM's XSLT and you will be happy with it. It's only, that I know my business and "full compliant" is not 100% true. Even the missing outlying details will be added, in the next months, for sure. I wonder, what Jochen then will write, in the announcement ;-).

The XPath support is now indeed really very compliant and complete. de.

LV finally, with some editing, was able to get this package to compile.

MSW now understands why tDOM crashed for him ... obviously you shouldn't be generating variables more than once (like two consecutive [<domDoc> documentElement root] in different functions make tDOM trip). He's still intrigued by appended nodes not being addressable (no localName, no URI, not reachable via [selectNodes])..