OttoCompiler

What: Tcl-Compiler
Intro: http://nhi1.selfhost.co/wiki/Tcl-Compiler
Description: The Tcl-Compiler compiles Tcl-Scripts into native C-Code
Updated: 26 feb 2021 (reopen this project)
Contact: mailto:[email protected] (Andreas Otto)

Links

HOWTO PAGES

Updates

09 June 2021 -> only small updates

  1. spend 2 days on simple CORE-DUMP, later figure out that 'package unload Tcl' was the problem
  2. add new POWER-Tcl package code (old code was 20 years old :-) -> now much smarter

Start to rename parts of the software, the POWER-Tcl-!!PACKAGE!! is now called LIBRARY

  1. a LIBRARY has usually a project.cct file, a file system structure and one or more files to compile or to install.
  2. a LIBRARY is the distribution format of POWER-tcl code and binaries.
  3. a LIBRARY is installed into the POWER-tcl runtime with cct pkg --add ... (the pkg may be changed later into lib)

01 June 2021 -> the BUILD tool get more power, today the TEST power ...

Compiler3> cct build --test ALL 
BUILD MSG    : start in directory                                 -> COMPILER3_SOURCE/.
BUILD MSG    : Tests began at                                     -> Tue Jun 01 22:59:09 CEST 2021
BUILD MSG    : Tcl 8.3.4 tests running in interp                  -> COMPILER_HOME:./exe/linuxi386/bin/cctsh8.3

BUILD MSG    : CctTclsh - test running in working dir             -> ./cmp/CctTclsh/tests
Only sourcing test files that match:  *.test
tclsh.test
++++ tclsh-1 PASSED
++++ tclsh-1-b PASSED
++++ tclsh-2 PASSED
++++ tclsh-2-b PASSED
++++ tclsh-3 PASSED
++++ tclsh-3-b PASSED
++++ tclsh-3-c PASSED
++++ tclsh-4 PASSED
++++ tclsh-5 PASSED

BUILD MSG    : Md5sum - test running in working dir               -> ./cmp/Md5sum/tests
Only sourcing test files that match:  *.test
md5sum.test
++++ md5sum-1-1 PASSED
++++ md5sum-1-2 PASSED
++++ md5sum-1-3 PASSED
++++ md5sum-1-4 PASSED
++++ md5sum-2-1 PASSED
++++ md5sum-2-2 PASSED
++++ md5sum-2-3 PASSED
++++ md5sum-2-4 PASSED
++++ md5sum-3-1 PASSED
++++ md5sum-3-2 PASSED
++++ md5sum-3-3 PASSED
++++ md5sum-3-3 PASSED

BUILD MSG    : Blowfish - test running in working dir             -> ./cmp/Blowfish/tests
Only sourcing test files that match:  *.test
blowfish.test
++++ blowfish-1-1 PASSED
++++ blowfish-1-2 PASSED
++++ blowfish-1-3 PASSED
++++ blowfish-1-4 PASSED
++++ blowfish-1-5 PASSED
++++ blowfish-1-6 PASSED
++++ blowfish-1-7-1 PASSED
++++ blowfish-1-7-2 PASSED

BUILD MSG    : Tests ended at                                     -> Tue Jun 01 22:59:10 CEST 2021

TEST RESULTS

: Library          Total    Skipped  Passed   Failed  
: ---------------- -------- -------- -------- --------
: CctTclsh         9        0        9        0       
: Md5sum           12       0        12       0       
: Blowfish         8        0        8        0       

31 Mai 2021 -> my today problem, something eat's a byte ...

test blowfish-1-7-1 {IN stdin, OUT stdout, io in ascii and string} {                                                           
  exec cct blowfish --dec --key 3456 \                                                                                         
      --str [exec echo "HelloWorld" | cct blowfish --enc --in stdin --key 3456 ]                                               
} {HelloWorld}
        
test blowfish-1-7-2 {IN stdin, OUT stdout, io in binary and string} {                                                          
  exec cct blowfish --dec --binary --key 3456 \
      --str [exec echo "HelloWorld" | cct blowfish --enc --binary --in stdin --key 3456 ]                                      
} {HelloWorld}                                                                                                                 

hmmmm

Compiler3> cct build --test Blowfish -- -match "blowfish-1-7-*"
Tcl 8.3.4 tests running in interp:  .../Compiler3.BUILD/lib/JavaKiller/exe/linuxi386/bin/cctsh8.3
Tests running in working dir:  .../Compiler3/cmp/Blowfish/tests
Only running tests that match:  blowfish-1-7-*
Only sourcing test files that match:  *.test
Tests began at Mon May 31 21:07:38 CEST 2021
blowfish.test
++++ blowfish-1-7-1 PASSED

==== blowfish-1-7-2 IN stdin, OUT stdout, io in binary and string FAILED
---- Result was:
HelloWor&
---- Result should have been:
HelloWorld
==== blowfish-1-7-2 FAILED

Tests ended at Mon May 31 21:07:38 CEST 2021
TestDriver.tcl: Total   8       Passed  1       Skipped 6       Failed  1
Sourced 1 Test Files.
Files with failing tests: blowfish.test

The problem is the fconfigure, if I delete it it works but other tests fail. good the default ascii feature (convert all binary in hex) works, the --binary option is more an addon and there seems to be a old tcl-bug.

    set OUT [ ::FileLib::Open $VALUE(--out) w ]                                                                                
    
    if { $VALUE(--binary) } { 
      fconfigure $OUT -encoding binary -translation binary                                                                     
      if { $VALUE(--enc) } {
        set ret [ ::crypt::bf_encrypt_bin $STR $KEY ]                                                                          
      } else {
        set ret [ ::crypt::bf_decrypt_bin $STR $KEY ]                                                                          
      }
      puts -nonewline $OUT  $ret                                                                                               
    } else {
      if { $VALUE(--enc) } {
        set ret [ ::crypt::bf_encrypt $STR $KEY ]                                                                              
      } else {
        set ret [ ::crypt::bf_decrypt $STR $KEY ]                                                                              
      }
      puts $OUT $ret                                                                                                           
    }                                                                                                                          
    
    flush $OUT
    ::FileLib::Close $OUT                                                                                                      

It seemst to be no problem with the tcl exec because the bash show the problem too

Compiler3> cct blowfish --dec --binary --key 3456 --str "$(echo "HelloWorld" | cct blowfish --enc --binary --in stdin --key 3456)" | xxd
00000000: 4865 6c6c 6f57 6f72 26                   HelloWor&

and more simple...

Compiler3> cct blowfish --dec --binary --key 3456 --str "$(cct blowfish --enc --binary --str "HelloWorld" --key 3456)" | xxd
00000000: 4865 6c6c 6f57 6f72 26                   HelloWor&

now final simple… the --enc seems to be ok.

Compiler3> cct blowfish --enc --str "HelloWorld" --key 3456
a3302be41886d310c68c
Compiler3> cct blowfish --enc --str "HelloWorld" --key 3456 --binary | xxd
00000000: a330 2be4 1886 d310 c68c                 .0+.......

I thing the problem is not the --binary, because the ''binary-only-test' is always ok

Compiler3> echo $(cct blowfish --enc --str "HelloWorld" --key 3456 --binary | cct blowfish --dec --key 3456 --binary)
HelloWorld

The problem is the binary to --str, it seems that tcl always expect utf as string and do some conversion. If I delete the fconfigure than tcl is doing output (binary) to utf and this fit into the --str. But without fconfigure the binary only test fail because the output (binary) to utf not fit the expected binary.

every bug create a new feature, and this feature is the --binary-to-utf option

Compiler3> echo $(cct blowfish --dec --binary --key 3456 --str "$(cct blowfish --enc --binary-to-utf --str "HelloWorld" --key 3456)")
HelloWorld

30 Mai 2021 -> nice update for cct tclsh ...

example : Simple tcl one-liner -> but the background is important.

cat /some/file | cct tclsh '::FileLib::Read stdin'

The cct tclsh eval the argument ::FileLib::Read stdin as script and call tclsh-unknown if something is missed.

The tclsh-unknown uses the following evaluation test order :

  1. test if it is a file (script) or a power-tcl special-build-in
  2. test if it is a command from a package which is not already loaded
  3. test if it is a command from a auto_index database (auto_load)
  4. test if it is a OS command (auto_execok)
  5. final call original unknown to handle default lookup like (auto_load)

Important is that the command ::FileLib::Read is in a package not loaded by default. The cct tclsh replace the default unknown to a tclsh-unknown which is able to load missing package first.

By default the package-name in power-tcl is the same name as the first namespace, in this case FileLib.

After the package is loaded successful than the ::FileLib::Read stdin read the data from stdin and return the data as large string. The cct tclsh trimright the final '\n' from output and print the result to stdout.

some examples from the default plugin test, called with: cct tclsh --test

set cct   [info nameofexecutable]
set echo  [auto_execok echo]

test tclsh-1 {eval string from commandline, return result} {
  exec $cct tclsh {set otto 78bb}
} {78bb}

test tclsh-1-b {eval string from commandline as args, return result} {
  exec $cct tclsh set otto 88oo
} {88oo}

test tclsh-2 {eval file from argument, return stdout from file} {
  makeFile {puts 12345} tclsh-2.txt
  exec $cct tclsh tclsh-2.txt
} {12345}

test tclsh-2-b {eval file from argument with additional argument} {
  makeFile {puts fff666-$argc-$argv} tclsh-2-b.txt
  exec $cct tclsh tclsh-2-b.txt xxx yyy zzz
} {fff666-3-xxx yyy zzz}

# attention expect SCRIPT not DATA from stdin
test tclsh-3 {eval script from stdin, return stdout from script} {
  exec $echo {puts abcd} | $cct tclsh --eval-stdin
} {abcd}

test tclsh-3-b {eval script from stdin, return result from script} {
  exec $echo {set otto abcdxyz} | $cct tclsh --eval-stdin
} {abcdxyz}

test tclsh-3-c {eval script from stdin, return stdout AND result from script} {
  exec $echo {puts yyt ; set otto 444} | $cct tclsh --eval-stdin
} {yyt
444}

test tclsh-4 {autoload package, read text from stdin and return result from ::FileLib::Read} {
  exec $echo ghij | $cct tclsh {::FileLib::Read stdin}
} {ghij}

test tclsh-5 {exec external command} {
  exec $cct tclsh {echo pp23}
} {pp23}

29 Mai 2021 -> testing, testing, testing ...

last 12 days was more testing and bug fixing, not much code added.

Compiler3> git log --format= --numstat --after="mai 17" | awk '{add+=$1; del+=$2} END {print "add=",add," : ","del=",del}'
add= 9688  :  del= 15356

Today I found the first real bug in old Compiler source, this happen in :

catch { uplevel #0 { ... } }

if { ... } return with an error. The problem was that #0 set the varFramePtr = NULL and was NOT reset.

int
Ot_UplevelObjCmd(interp, level, refv)
    Tcl_Interp      *interp;            /* Current interpreter. */
    int             level;              /* the call level */
    Ot_X_Cmd_Proc   *refv;              /* Dynamic pointer Structure */
{
    register Interp *iPtr = (Interp *) interp;
    int result;
    CallFrame *savedVarFramePtr, *framePtr;

    Tcl_ResetResult(interp); 
    iPtr->cmdCount++;

    /*
     * Find the level to use for executing the command.
     */

    result = Ot_TclGetFrame(interp, level, &framePtr);
    if (result == TCL_ERROR) {
        return TCL_ERROR;
    }

    /*
     * Modify the interpreter state to execute in the given frame.
     */

    savedVarFramePtr = iPtr->varFramePtr;
    iPtr->varFramePtr = framePtr;  // iPtr->varFramePtr=NULL if level=0

    /*
     * Execute the residual arguments as a command.
     */

    result = (*refv)(interp);

    if (result == TCL_ERROR) {
        // old bug: 'return TCL_ERROR;' -> return does NOT set iPtr->varFramePtr back to savedVarFramePtr
        result = TCL_ERROR;
    }

    /*
     * Restore the variable frame, and return.
     */

    iPtr->varFramePtr = savedVarFramePtr;
    return result;
}

27 Mai 2021 -> test add missing features

some small updates are ongoing…

add plugin view as general purpose document viewer

> cct view --mlist

mime  : viewer     : args  : regexp          : description
----- : ---------- : ----- : --------------- : -----------------
html  : firefox    :       : .*[.]html?      : html files
pdf   : firefox    :       : .*[.]pdf        : pdf files
info  : info       : -f    : .*[.]info       : info files
man   : man        : -l    : .*[.][1-9n]     : man files
plain : less       :       : [^.]+           : plain files

add plugin man as more special document viewer for man pages

Compiler3> cct man tcl
tcldesjr.n          tcldocstrip.n       tcllib_dns.n        tcllib_fifo.n       tcllib_ip.n         tcllib_msgcat.n     tcllib_random.n     tcllib_try.n        tcllib_zero.n       tcl_parse.n         tclreadline         tcltest.n
tcldes.n            tcllib_coroutine.n  tcllib_fifo2.n      tcllib_interp.n     tcllib_memchan.n    tcllib_null.n       tcllib_string.n     tcllib_variable.n   tcllib_zlib.n       tcl_prefix.n        tclsh.1  

possible good news about tcl, able to add type definition without break old code.

.../Compiler3> cct arg.test 
list element in braces followed by "arg1" instead of space
    while executing
"proc otto { {L}arg1 {S}arg2 } {

  puts hallo

}"
    (file "arg.test" line 16)

also possible

declare {I}counter {W}bigcounter {{I}L}ListOfInterger {A}keyBook

26 Mai 2021 -> add new technology.

Every day new code, Every day a new feature. On top of the already existing power-tcl license code I add something new, I don't even have a good name for it.

    possible: power-tcl-guard

The power-tcl-guard technology is something to control HOW and WHERE a software is used. It is more like an extension of the already available operating-system-security but full under control of the customer and not under control of something or somebody else.

The power-tcl-guard technology is the first technology in duty on application startup (ref power-tcl-boot) and provide the possibility to stop or to log the software usage.

What is possible ?

  1. allow execution of the software ONLY if a special condition is met
  2. protocol every start of a software to a logging server
  3. really grantee in-house security by force, stolen software is useless
  4. no user interaction required.
  5. probably more...

morning problem

I locked myself out.

../Compiler3/src/CctRt> OtBuild.sh test


LICENCE ERROR (Power-Tcl):  licence file not available

please order a new licence with the public key :

>>> linuxi386/Power-Tcl#d7c316233e44c4a853348ef6194cc473 <<<

please contact : xxx@yyy
more info at   : http://nhi1.selfhost.co/Tcl-Compiler

ok, got it

> cat tmp.sig | cct key --action decrypt 


LICENCE OK (Power-Tcl):

>>> Power-Tcl.linuxi386#3ad1d6510298718a559f403a7bf09ec1 <<<

    VALID: Fri May 28 20:56:28 CEST 2021

25 Mai 2021 -> I really like my error messages

This is nothing special just a file, mentioned in the project.cct, is missing. I really like the information in the message, the stack and the layout.

> cct make c
CONFIG ERROR [::Config::Is::File] : PROJECT FILE_TCL -> 'internal.tcl' file does not exists

  STACK >>>
    ::Config::Is::File {PROJECT FILE_TCL} internal.tcl ...
    ::Config::Is::Test {PROJECT FILE_TCL} internal.tcl {} ...
    ::Config::Values::Read FILE_TCL internal.tcl {} ...
    ::Config::Interp::Source .../Compiler3/cmp/Key/project.cct ...
    ::Config::Values::Source .../Compiler3/cmp/Key/project.cct ...
    ::Config::read .../Compiler3/cmp/Key/project.cct 1 ...
    ::Config::Main @project.cct -cache yes ...
    ::IConfig::ProjectL * @project.cct ...
    ::CctMake::Make ...
    ::CctMake::Main ...
    ::Main ...
    file: Cct/Main.tcl

24 Mai 2021 -> more updates on "Key"

In the past the Key feature was only the license-key but now the Key feature expanded.

  1. The Key feature is a mixture of tcl and C code compiled into a power-tcl-binary.
  2. The Key feature is always available as FIRST power-tcl code.
  3. A lot of real basic features are moving into the Key feature to be available very early and without a package require ....

The boot-order of power-tcl with boot feature enabled

function decription old new
Cct_Main called from main, parse arguments generic/tclMain.c generic/cctMain.c
Tcl_AppInit init tcl & packages unix/tclAppInit.c unix/cctBootInit.c
Key key check & power-tcl init - cmp/Key/KeyCct
Tcl_Init call for every new interp - -
InitScript setup tcl environment generic/tclInitScript.h cmp/InitScript
TclInit_init setup tcl, former init.tcl lib/tcl8.3 cmp/TclInit
PATTERN_boot_Init init package ineeded & auto_index - cct build --pattern PATTERN --boot .../cct8.3.4/unix

22 Mai 2021 -> BOOT loader technology expanded

The boot-loader now link into the TCL Makefile.in with just a few lines at the end:

#
# add support for 'boot'
# > cct build --pattern CCT --boot-dir ...
#

TCL_BUILD_LIB_SPEC      = @TCL_BUILD_LIB_SPEC@
TCL_LD_SEARCH_FLAGS     = @TCL_LD_SEARCH_FLAGS@

include export/boot.make

# DO NOT DELETE THIS LINE -- make depend depends on it.

What it does?

The POWER-Tcl-boot-loader technology does the following thing :

  1. the pattern define the tcl packages be part of the bootloader
  2. compile the tcl packages into power-tcl-binary-archive (libPackageX.Y.a)
  3. compile a new tclsh with name pattern and static link the power-tcl-binary-archive into the shell
  4. add all code to either package require or auto_load the code.
  5. finally a single executable with name pattern (lowercase) is available with tcl and all pattern packages included

The following makefile-targets are added

boot-binaries:extend binaries and add all pattern (multiple pattern are allowed)
boot-install:extend install-binaries
boot-clean:extend clean
pattern-clean:extend boot-clean
pattern-install:extend boot-install
pattern:extend boot-binaries

The pattern binary is using the POWER-Tcl file cctBootInit.c as replacement for tclAppInit.c.

typical Morning-Job

Compiler3> git commit -a -m "more on build and boot"
[master b3d3189] more on build and boot
 37 files changed, 1452 insertions(+), 938 deletions(-)
 create mode 120000 bin/cctsh
 create mode 100644 cmp/Build/db.tcl
 create mode 100644 cmp/Build/list.tcl
 create mode 100644 cmp/Build/make.tcl
 create mode 100644 cmp/Build/pkg.tcl
 create mode 100644 cmp/Build/prj.tcl
 rewrite cmp/Build/project.cct (72%)
 create mode 100644 src/CctRt/cct8.3.4/generic/cctMain.c
 rename src/CctRt/cct8.3.4/unix/{tclBootInit.c => cctBootInit.c} (99%)

19 Mai 2021 -> BOOT loader update

The POWER-Tcl-boot-loader is close to finish, cct startup-time is close to ZERO.

small example : startup POWER-Tcl with boot-loader

task: start shell, initialize, setup the plugin-database and finally print help-screen 0.006 sec.

The shell from the example is a shell with the basic POWER-Tcl features compiled in.

HOWTO build the shell ?

  1. create the boot-loader with: cct build --pattern CCT --boot-dir ...
  2. using the build pattern: CCT = KeyCct TclInit Cct StdLib
  3. the pattern is using the following POWER-Tcl-projects ... list below ...
> cct build --pattern CCT --list project 
package      : projects
------------ : ----------------------------------------------------
StdLib       : StdLib FileLib
Cct          : Cct Args CctCmd CctComplete
KeyCct       : Key
TclInit      : TclInit_auto TclInit_initScript TclInit_history msgcat1.1 TclInit_word opt0.4 tcltest1.0 TclInit_ldAout http1.0 http2.4 TclInit_parray TclInit_safe TclInit_package TclInit_init

One thing is important to know :

  1. Every project is a tcl script(s) or a shared library or a static archive
  2. if it is a static archive and compiled into the executable the binary code is added but NOT initialized
  3. the initialize is done if a package require... is done OR the auto_index request a proc/command from the static archive to load
  4. the startup-time is very low because only the code required is initialized

One of the problems was to add the code from .../lib/tcl8.3 into the binary.

> time ./cct -h

usage: cct [master option ...] command ...

  master options :

    -m             ARCH     platform target                          [linuxi386]
    -d             FLAG     use debug mode                           [no]
    -v             FLAG     use verbose mode                         [no]
    -f             FLAG     use force mode                           [no]

  commands :

    blowfish       blowfish encryption
    build          build and install the distribution
...
    tkdemo         start the tk demo
    view           view plugin interface
    wish           the POWER-Tk shell

  default options :

    --help         FLAG     print the help (-h)                      [no]
    --info         FLAG     get information about command            [no]
    --edit         FLAG     open project and edit files              [no]
    --test         FLAG     run tests in 'tests' subdirectory        [no]
                    > all remaining arguments are passed to tcltest
    --args         LIST     command-line arguments                   [NOT-SET]


real    0m0,006s
user    0m0,006s
sys     0m0,000s

just to compare, the grep :

> time grep --help
real    0m0,002s
user    0m0,002s
sys     0m0,001s

The POWER-Tcl build up the plugin-database for display the plugin-list, the startup trace show this:

> strace ./cct -h |& grep access
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (Datei oder Verzeichnis nicht gefunden)
access("/tcl8.3", R_OK)                 = -1 ENOENT (Datei oder Verzeichnis nicht gefunden)
access(".../Compiler3.BUILD/lib/JavaKiller/var/SysIndex/linuxi386_cct_pkgIndex.tcl", F_OK) = 0
access(".../Compiler3/example/CctMan.CVS/Cct.cfg", R_OK) = 0
access(".../Compiler3/example/CctTest.CVS/Cct.cfg", R_OK) = 0
access(".../Compiler3/example/TclHttpd/Cct.cfg", R_OK) = 0
access(".../Compiler3/example/CctKeyGen.CVS/Cct.cfg", R_OK) = 0
access(".../Compiler3/example/CctFind.CVS/Cct.cfg", R_OK) = 0
access(".../Compiler3/cmp/Build/Cct.cfg", R_OK) = 0
access(".../Compiler3/example/CctDiff.CVS/Cct.cfg", R_OK) = 0
access(".../Compiler3/cmp/CctIndex/Cct.cfg", R_OK) = 0
access(".../Compiler3/cmp/PkgLib/Cct.cfg", R_OK) = 0
access(".../Compiler3/example/CctProfile.CVS/Cct.cfg", R_OK) = 0
access(".../Compiler3/example/BWidget/Cct.cfg", R_OK) = 0
access(".../Compiler3/example/CctLabel/Cct.cfg", R_OK) = 0
access(".../Compiler3/example/Crm/Cct.cfg", R_OK) = 0
access(".../Compiler3/example/CctMail.CVS/Cct.cfg", R_OK) = 0
access(".../Compiler3/cmp/TMakeLib/Cct.cfg", R_OK) = 0
access(".../Compiler3/cmp/Bras/Cct.cfg", R_OK) = 0
access(".../Compiler3/cmp/CctView/Cct.cfg", R_OK) = 0
access(".../Compiler3/example/CgiPoll.CVS/Cct.cfg", R_OK) = 0
access(".../Compiler3/example/CctPerf.CVS/Cct.cfg", R_OK) = 0
access(".../Compiler3/example/CctKey.CVS/Cct.cfg", R_OK) = 0
access(".../Compiler3/cmp/Md5sum/Cct.cfg", R_OK) = 0
access(".../Compiler3/cmp/Config/Cct.cfg", R_OK) = 0
access(".../Compiler3/example/CgiGbook.CVS/Cct.cfg", R_OK) = 0
access(".../Compiler3/example/CctPlugin/Cct.cfg", R_OK) = 0
access(".../Compiler3/cmp/Wish/Cct.cfg", R_OK) = 0
access(".../Compiler3/example/Blowfish/Cct.cfg", R_OK) = 0
access(".../Compiler3/cmp/Agent/Cct.cfg", R_OK) = 0
access(".../Compiler3/cmp/Tclsh/Cct.cfg", R_OK) = 0
access(".../Compiler3/example/TclIDE/Cct.cfg", R_OK) = 0

It is important to notice that the entire lib/tcl8.3 directory is gone, it's now total in C

18 Mai 2021

yesterday I add a small daily programmer helper, section tclsh

> cct tclsh file dirname want/tcl
want

To call a tcl shell use cct tclsh, but some things where expanded by default, this is good for shell scripts etc… The code is quite simple:

# main proc called by "cct tclsh ..."
proc ::CctTclsh::Main {} {
    ::Args::CheckInit
    if { $SET(--args) } {
        ::StdLib::Push STDIN stdin STDOUT stdout STDERR stdout  ERROR EXIT
        puts  [ ::ITclSh::Exec $VALUE(--args) ]
        ::StdLib::Pop  STDIN STDOUT STDERR ERROR
    } else {
        set ::tcl_interactive 1
        if {[info exists ::env(TCL_RCFILE)]} {
          uplevel #0 source $::env(TCL_RCFILE)
        }
    }
}

# interface proc called for TCL code/files
proc ::ITclSh::Exec { { Args {} } } {
    variable EXEC
    variable NOVERSION
    if {[llength $Args] == 1} {
      set Args [lindex $Args 0]
    }
    set cmd [lindex $Args 0]
    ::Env::Push $NOVERSION 1
    if {[ file exists $cmd ] || "$cmd" == "package" || "$cmd" == "load" } {
      # real file -> do an exec
      set ret [ ::StdLib::Exec [ concat [list $EXEC] $Args ] ]
    } elseif {[llength [info commands $cmd]] != 0} {
      set ret [eval $Args]
    } elseif {[auto_load $cmd]} {
      set ret [eval $Args]
    } elseif { [set execL [auto_execok $cmd]] != "" } {
      set ret [ ::StdLib::Exec [concat $execL [lrange $Args 1 end]] ]
    } else {
      ::StdLib::ErrorN ITclSh "don't know how to exec: $Args"
    }
    ::Env::Pop  $NOVERSION
    set ret
}

17 Mai 2021

  1. last weekend I add a new technology called the Indexed-Binary-File-technology

The indexed-binary-file is just a POWER-Tcl-compiled-library and in addition a tclIndex file with the definition for every proc in the binary.

example using the CctPlugin.tcl file

step-1, install the binary

> cct make c.is 
## working on project 'CctPlugin'
# making `c_exe.is'
install      : c_exe                                              -> ./CctPlugin.tpkg/arch/linuxi386/lib/pkg
# making `c.is'
install      : c.is                                               -> OK

step2, show the index

> ls -al ./CctPlugin.tpkg/arch/linuxi386/lib/pkg
-rwxr-xr-x 1 XXXXXX users 16032 17. Mai 12:23 libCctPlugin1.0.so
-rw-r--r-- 1 XXXXXX users   528 17. Mai 12:23 libCctPlugin1_0.tclIndex

> cat ./CctPlugin.tpkg/arch/linuxi386/lib/pkg/libCctPlugin1_0.tclIndex
set auto_index(::CctPlugin::Main) [list load [file join $dir libCctPlugin1.0.so]]
set auto_index(::CctPlugin::Args::Init) [list load [file join $dir libCctPlugin1.0.so]]
set auto_index(::CctComplete::MySpecialComplete) [list load [file join $dir libCctPlugin1.0.so]]
set auto_index(::CctPlugin::Args::Help) [list load [file join $dir libCctPlugin1.0.so]]
set auto_index(::CctPlugin::ErrorN) [list load [file join $dir libCctPlugin1.0.so]]
set auto_index(::CctPlugin::Args::Main) [list load [file join $dir libCctPlugin1.0.so]]

Ok, what is the goal ?

  1. autoload of libraries, even if compiled into binary. traditional tcl only able to make tclIndex from pure tcl files.
  2. currently a lot of work is done for the POWER-Tcl-boot-loader this mean to put POWER-Tcl wery early into duty, even befor all the package stuff is available.

The technology is quite simple and only works for POWER-Tcl-compiled-binaries. The technology is using the repository of compiled tcl-code. In the repository is already an index called .IS_PROC-inputFileName and this index already has all proc's defined in the POWER-Tcl-compiled-binary

> cat ~/.cct/repository/Master/CctPlugin_1_0/.IS_PROC-CctPlugin 
@X0-C13@ {::CctPlugin::Main {}}
@X0-C8@ {::CctPlugin::Args::Init {}}
@X0-C10@ {::CctComplete::MySpecialComplete S,s}
@X0-C11@ {::CctPlugin::Args::Help {}}
@X0-C6@ {::CctPlugin::ErrorN s,*}
@X0-C12@ {::CctPlugin::Args::Main {}}

statistics

Compiler3> git log --format= --numstat | awk '{add+=$1; del+=$2} END {print "add=",add," : ","del=",del}'
add= 2998894  :  del= 533366

nice usage screen-shot - distribution check with error

==> cct -f pkg --check-dist |& less
PKG CHECK MSG : check part 1 -> check package database
PKG CHECK MSG : check part 2 -> find missing or broken files
PKG CHECK MSG : check part 3 -> find unknown files
PKG CHECK WARN : found non package directory '.../Compiler3.BUILD/lib/JavaKiller/bin'
PKG CHECK WARN : found non package directory '.../Compiler3.BUILD/lib/JavaKiller/exe/linuxi386/man'
PKG CHECK WARN : found non package directory '.../Compiler3.BUILD/lib/JavaKiller/exe/linuxi386/man/man3'
PKG CHECK WARN : found non package file '.../Compiler3.BUILD/lib/JavaKiller/exe/linuxi386/lib/tclreadline3.0/tclIndex'
PKG CHECK WARN : found non package directory '.../Compiler3.BUILD/lib/JavaKiller/exe/linuxi386/lib/tcl8.3/ccttest1.0'
PKG CHECK WARN : found non package file '.../Compiler3.BUILD/lib/JavaKiller/exe/linuxi386/bin/tcl2meta0_8.3'
PKG CHECK WARN : found non package file '.../Compiler3.BUILD/lib/JavaKiller/bin/cct'
PKG CHECK MSG : check part 4 -> find unchecked files
PKG CHECK MSG : check part 5 -> deep check
PKG CHECK MSG : check package: T2mRt_3_0
PKG CHECK MSG : check package: CctRt_3_0
PKG CHECK MSG : check package: TclRl_3_0
PKG CHECK MSG : check package: Readline_1_0
PKG CHECK MSG : check package: PkgLib_2_0
PKG CHECK MSG : check package: MetaToC_3_0
PKG CHECK MetaToC_3_0 [::PkgLib::Check::Test-md5sum] : check failed for file:
    '.../Compiler3.BUILD/lib/JavaKiller/./template/Env/COMPILER.c'
    the existing md5sum 'ceb92f97bc9f84a3f12925a279b421dd'
    is not the original md5sum '356e74147f9839bb8a2b4a08339ace5f'
    please delete the package 'cct pkg --del MetaToC_3_0'
    and    add    the package 'cct pkg --add MetaToC_3_0'
    to rebuild the package database

  STACK >>>
    ::PkgLib::Check::Test-md5sum .../Compiler3.BUILD/lib/JavaKiller/./template/Env/COMPILER.c 356e74147f9839bb8a2b4a08339ace5f ...
    ::PkgLib::Check::Main MetaToC_3_0 ...
    ::PkgLib::Check::Dist ...
    ::PkgLib::Exec CHECKDIST 1 ...
    ::CctPkg::Main ...
    ::Main ...
    file: Cct/Main.tcl
PKG CHECK MSG : check package: Config_2_0
PKG CHECK MSG : check package: CctPlugin_1_0
PKG CHECK MSG : check package: CctIndex_2_0
PKG CHECK MSG : check package: Base_2_0
PKG CHECK MSG : check package: Agent_2_0

15 Mai 2021

  1. The Early-Bird-Dist now has a detailed package list
  2. I rename the core tcl executable into cctsh, cct and ccttest just to avoid unwanted interaction with traditional tcl
    1. Unfortunately I CANNOT rename libtcl8.3.so to libcct8.3.so because every tcl extension still needs the traditional tcl name.
  3. Lots of updates to the dist including message-layout, StdLib, FileLib etc.

10 Mai 2021

The last couple of days I add a new technology called Power-Tcl-Kit.

The strategic goal is quite simple: "put everything under one roof"

OLD

PowerTcl was a directory-tree with a modified tclsh as starter and a filesystem filled with application specific files. Every application was a collection of Power-Tcl-projects installed into the Power-Tcl-runtime. Everything together, the runtime, the application and supporting data files was the Power-Tcl-application.

NEW

Now I add a new technology to install most of the files into ONE binary. This binary is called Power-Tcl-Kit, a fat pure C binary. The goal was NOT to create a fat binary, the goal was to make this fat binary faster as the original Power-Tcl.

The original Power-Tcl has a 10-1000 times faster startup as all the "traditional" languages developed/improved in the last ~20 Years, including all the scripting languages like perl, python, ruby, php etc but also the so called "high" languages like java, C#, go etc. The Power-Tcl-Kit goes further with a more aggressive optimization and a binary-internal-package-management. Everything is still 100% tcl compatible.

But every new technology also introduce a new problem. The biggest problem was the software-in-duty problem. Now there a 3 sources of application-code:

  1. the traditional tcl-script.
  2. the POWER-Tcl compiled tcl-script as shared-library.
  3. NEW: the Power-Tcl-Kit code compiled into the fat binary.

It takes some time to get all the package-management-tools used to this triple but now it seems to be stable. The Power-Tcl-Kit project is still not finished because now an avalanche of new opportunities arrive.

I'm not only the developer of POWER-Tcl I'm also the developer of the NHI1, Managed-Object and Programming-Language-Micro-Kernel technology. All projects are based on the Token-Stream technology, which was developed in the 1990++ and probably some near time in future this projects will join :-)

05 Mai 2021

I'm not lazy - these are the stats from my POWER-Tcl-git repository since the project restarted. Usually I have between 500 and 5000 update lines per day :-)

Compiler3> git log --format= --numstat | awk '{add+=$1; del+=$2} END {print "add=",add," : ","del=",del}'
add= 2981426  :  del= 408284

The above number is really everything, this is probably not a fair number as it also contains a lot of external references, but the code from the POWER-Tcl-Core is more like the lower limit.

Compiler3/cmp> git log --format= --numstat . | awk '{add+=$1; del+=$2} END {print "add=",add," : ","del=",del}'
add= 92792  :  del= 22722

The last two days I update :

  1. The project code now allow multiple project-files for a single project.
  2. The autoconf code to add more POWER-Tcl specific tests.
  3. The StdLib to be more compact
  4. And cleanup some definitions to be more clear to understand

The project-definition-tree for a single project is now :

  1. The build-in defaults defined in the Config package.
  2. The toplevel project-file at COMPILER3_SOURCE/project.cct
  3. Any project-file between the current directory and the toplevel directory
  4. finally the local project file defined with --project-file option or project.cct as default.

04 Mai 2021

This days a new tool go into production and the POWER-Tcl got a lot of cleanup

  • the new tool is cct label ... this tool add a header to a file which need a legal information.
  • the cct make *.exp... or cct export ... was updated to create and export archive files
  • the --edit feature got an update to integrate into screen command-line multi window tool.
  • internal I update my web site to use ONLY http protocoll, ssl support was removed. To access POWER-Tcl use http://nhi1.selfhost.co/POWER-Tcl

The export feature of POWER-Tcl is used to integrate into other software.

  • exp.meta -> export intermediate .meta file(s)
  • exp.tcl -> export compiled .tcl file(s)
  • exp.c -> export .c file(s)
  • exp.o -> export binary .o file(s), link with a single object file, use int Ot_ProjectFileInit(interp) for initialization of a single file, be aware that ONLY a single file is initialized and not the project/extension at all.
  • exp.a -> export archive .a file(s) filled with all .o files of a project, add int Project_Init(interp) into the C-Code for initialization.
  • exp.so -> export shared .so file(s), use load libProjectX.Y.so to load this library into 'TCL'.

The label feature is used to add text into a file with focus on different aspects, I already use this tool in the NHI1 project but now a rewrite was done for POWER-Tcl :

  • legal: trademark/author/contact/disclamer etc - always choose the right legal header.
  • git: use git log ... to update file with commit information like date-time, hash, user etc.
  • code: based on file-type and local setup create and add static code into the file to avoid error-prone and booring code updates.

29 Apr 2021

This is a large release adding multiple improvements and features to POWER-Tcl :

  • automake, autoconf and make integration -> VPATH build.
  • cct make options for fast prototyping and debugging : --project-file, --test-file and -d.
  • a lot of internal code-updates and bug-fixes.

25 Apr 2021

This is a nice example of an "Morning-Error" but this is also a nice example of an error in a POWER-Tcl compiled file. As you see the STACK always works, even the errorInfo shows the stack with proc-names but without code. But one thing is missing, the good old __FILE__ and __LINE___ from the wonderfull C debugging. Tcl misses some really usefull debugging-infos from day one, and the most important miss is the line and file information.

  • I thing adding more debugging features is one point of the future in POWER-Tcl.
Compiler3> cct make --edit 
ENVIRONMENT ERROR [::Env::Check] : variable '' is not available (ERR08) -> please check 'Compiler' setup (ERR-07)

  STACK >>>
    ::Env::Check {} {} ...
    ::FileLib::Resolve {} ...
    ::PkgLib::COMPILER_HOME ...
    ::PkgLib::Db::INDEXFILE . ...
    file: pkg/libPkgLib2.0.so

23 Apr 2021

Today a pretty nice technology was added: I call it POWER-starter

In the past the POWER-Tcl was a normal tclsh with some enhancements, able to load and execute a POWER-Tcl-share-library. But now something NEW was added… instead creating a shared-library I add a feature called objexp

example: objexp on StdLib

> cd cmp/StdLib
> cct make c.objexp
... compiles: tcl->c->o
> find export/ -name "*.o"
export/StdLib/COMPILER_STDLIB.o
export/StdLib/StdLib.o
export/FileLib/COMPILER_FILELIB.o
export/FileLib/FileLib.o
export/StackLib/COMPILER_STACKLIB.o
export/StackLib/StackLib.o

and now I use the "*.o" files to create a special tclsh just called cct. This cct is now the POWER-Tcl starter:

OLD: cct (BASH script) -> set some environment -> exec tclsh8.3 package Cct ... (from POWER-Tcl) -> do the processing

NEW: cct (BINARY with Cct and other stuff) -> do the processing

This is a pretty quick start as there is NO single line of BASH or TCL script involved and everything is in pure C.

20 Apr 2021

  1. update HOWTO compile
  2. update HOWTO make
  3. add TPACKAGE project option to link a PROJECT with a installation PACKAGE
  4. add cct make --detail to print out the project data:
.../Compiler3/example/CctPlugin.CVS> cct make --details

PROJECT CctPlugin {
  DEBUGMODE         0
  FILE_TCL CctPlugin.tcl {
    HEADER            {}
    META2C {
      DEBUG             0
      OPTIMIZATION      1
      PRINT             0
      SKIP              {}
    }
    META2TCL {
      PRINT             0
    }
    TCL2META {
      CMD               {}
      PKG_REQUIRE       Tcl
      PRINT             0
      SAFE              0
      TEST              0
    }
  }
  INSTDIR_C         .../Compiler3/example/CctPlugin.CVS/CctPlugin.tpkg/arch/linuxi386/lib/pkg
  INSTDIR_TCL       .../Compiler3/example/CctPlugin.CVS/DIR-Tcl
  MAJOR             1
  MINOR             0
  STRIP             1
  TPACKAGE          CctPlugin.tpkg
}

19 Apr 2021

This is a nice overview about the POWER-Tcl-development-workplace packages :

.../Compiler3.CVS> cct build --list
  name            tclV  pkgV  restriction    pattern            dir
  --------------- ----- ----- -------------- ------------------ ------------------------
  Base                  2.0   --pkg          ALL EARLY RT       ./cmp/Base
  StdLib          2.0   2.0                  ALL EARLY RT       ./cmp/StdLib
  External        2.0   2.0                  ALL EARLY RT       ./cmp/External
  CctIndex        2.0   2.0                  ALL EARLY RT       ./cmp/CctIndex
  Config          2.0   2.0                  ALL EARLY RT       ./cmp/Config
  PkgLib          2.0   2.0                  ALL EARLY RT       ./cmp/PkgLib
  TMakeLib        2.0   2.0                  ALL CMP            ./cmp/TMakeLib
  MetaToTcl       3.0   3.0                  ALL CMP            ./cmp/MetaToTcl
  MetaToC         3.0   3.0                  ALL CMP            ./cmp/MetaToC
  Wish                  3.0                  ALL TK             ./cmp/Wish
  Agent           2.0   2.0                  ALL CMP            ./cmp/Agent
  System          2.0   2.0                  ALL EARLY RT       ./cmp/System
  InitScript      9.9   2.0                  ALL RT             ./cmp/InitScript
  Target          2.0   2.0                  ALL CMP EARLY RT   ./cmp/Target
  Md5sum          2.0   2.0                  ALL EARLY RT       ./cmp/Md5sum
  Bras                  2.1                  ALL CMP            ./cmp/Bras
  Cct             2.0   2.0                  ALL EARLY RT       ./cmp/Cct
  CctView         1.0   1.0                  ALL EARLY RT       ./cmp/CctView
  CctRep          2.0   2.0                  ALL CMP            ./cmp/CctRep
  CctBuild        1.0   1.0                  ALL                ./cmp/Build
  TclToMeta             3.0   --pkg          ALL CMP            ./src/t2m8.3.4
  CctRt                 3.0   --pkg          ALL EARLY RT       ./src/cct8.3.4
  CctPerf         2.0   2.0                  ALL                ./example/CctPerf.CVS
  CctTest         2.0   2.0                  ALL                ./example/CctTest.CVS
  CctDiff         2.0   2.0                  ALL                ./example/CctDiff.CVS
  CctFind         2.0   2.0                  ALL                ./example/CctFind.CVS
  CctPlugin       2.0   2.0                  ALL                ./example/CctPlugin.CVS
  CctMan          2.0   2.0                  ALL                ./example/CctMan.CVS
  BWidget               1.31                 ALL                ./example/BWidget
  TclIDE          2.0   2.0                  ALL                ./example/TclIDE
  TclHttpd              2.0                  ALL                ./example/TclHttpd
  TclLib                2.0                  ALL                ./example/TclLib
  Blowfish        1.0   1.0                  ALL                ./example/Blowfish
  Cgi             1.0   1.0                  ALL CGI            ./example/Cgi.CVS
  CgiGbook              2.0                  ALL CGI            ./example/CgiGbook.CVS
  CgiFormmail     1.0   2.0                  ALL CGI            ./example/CgiFormmail.CVS
  CgiPoll         1.0   1.0                  ALL CGI            ./example/CgiPoll.CVS
  CgiKey          1.0   1.0                  ALL CGI            ./example/CgiKey.CVS
  CgiUserkey      1.0   1.0                  ALL CGI            ./example/CgiUserkey.CVS
  CgiHidden       1.0   1.0                  ALL CGI            ./example/CgiHidden.CVS
  CctMail         1.0   1.0                  ALL CGI            ./example/CctMail.CVS
  CctCrm          1.0   1.0                  ALL                ./example/Crm
  OtCC                  2.0   --pkg          ALL CC             ./src
  OtTcl                 8.34  --pkg          ALL                ./src
  OtTk                  8.34  --pkg          ALL TK             ./src
  OtMetaKit             1.0   --pkg          ALL CGI            ./src
  Readline              1.0   --pkg          ALL EARLY RT       ./src
  TclRl                 3.0   --pkg          ALL EARLY RT       ./src
  TcLib                       --make         ALL                ./cmp/TcLib
  TkLib                       --make         ALL TK             ./cmp/TkLib

18 Apr 2021

This week a lot of code-cleanup and feature-adding was done.

The new features are:

package lockcct pkg --lock pkgname (also --unlock)
distribution upgradecct build --pkg upgrade
plugin testing:cct md5sum --test ...

HOWTO lock/unlock

The package-lock is important because in POWER-Tcl every code exist TWO times :

  1. once as ordinary TCL code
  2. once as POWER-Tcl binary code

The goal is that only ONE code is in duty. To solve this problem the binary can be locked and unlocked. To lock mean just renaming the binary to binary.lock and rebuild the package index files.

Example: edit plugin code without lock

.../Compiler3.CVS> cct plugin --edit 
CctPlugin ARGUMENT ERROR [::Args::Edit-Arg] : The package 'CctPlugin' uses the BINARY from the directory:
    > .../Compiler3.CVS/lib/JavaKiller/exe/linuxi386/lib/pkg
    To edit the package, the SOURCE must be available and the BINARY must be locked. 
    > Use 'cct pkg --lock CctPlugin' to lock the BINARY.

  STACK >>>
    ::Args::Edit-Arg ...
    ::Args:: ...
    ::Args::Invoke2 --edit ...
    ::CctPlugin::Args::Main ...
    ::Args::CmdInvoke CctPlugin ...
    ::CctCmd::Invoke CctPlugin ...
    ::Args::Invoke2 plugin ...
    ::CctCmd::Args::Main ...
    ::Args::Main ...
    ::Main ...
    file: Cct/Main.tcl

After edit is finished one simple command 'cct build --target CctPlugin --pkg upd' recompile and install the new package:

> cct build --target CctPlugin --pkg upd
> CctPlugin       ./example/CctPlugin.CVS
BUILD MSG : ./example/CctPlugin.CVS        -> cct make -s c.is
BUILD MSG : ./example/CctPlugin.CVS        -> cct pkg --dir COMPILER_HOME --noidx --upd CctPlugin
package: CctPlugin_2_0
PKG UPD MSG: update package                           -> CctPlugin_2_0
PKG ADD MSG: work on package                          -> CctPlugin_2_0
PKG ADD MSG: setup arch                               -> linuxi386
PKG ADD MSG: check destination directory              -> .../Compiler3.CVS/lib/JavaKiller
PKG ADD MSG: check destination directory              -> .../Compiler3.CVS/lib/JavaKiller/exe
PKG ADD MSG: write package to destination directory   -> .../Compiler3.CVS/lib/JavaKiller/var/pkg/install/CctPlugin_2_0
PKG ADD MSG: cleanup installation directory           -> .../Compiler3.CVS/lib/JavaKiller/var/pkg/install/CctPlugin_2_0
PKG ADD MSG: cleanup installation directory           -> .../Compiler3.CVS/lib/JavaKiller/var/pkg/arch/CctPlugin_2_0
PKG ADD MSG: write package finish                     -> CctPlugin_2_0
PKG ADD MSG: delete build/inst/arch directory         -> done
PKG UPD MSG: clean package                            -> CctPlugin_2_0
BUILD MSG : .                              -> cct index --dir COMPILER_HOME --sys package
PKG INDEX : work in directory                        -> .../Compiler3.CVS/lib/JavaKiller/exe/linuxi386/lib/pkg
PKG INDEX : create pkgIndex.tcl (MSG01)              -> START
PKG INDEX : create pkgIndex.tcl (MSG02)              -> END
BUILD MSG : .                              -> cct index --dir COMPILER_HOME --sys cct
CCT INDEX : work in directory                        -> .../Compiler3.CVS/lib/JavaKiller/exe/linuxi386/lib/pkg
CCT INDEX : create Cct.cfg (MSG01)                   -> START
  > found : CctPkg CctPlugin CctRep 
CCT INDEX : create Cct.cfg (MSG02)                   -> END
BUILD MSG : .                              -> cct index --dir COMPILER_HOME --sys cgi
CGI INDEX : work in directory                        -> .../Compiler3.CVS/lib/JavaKiller/exe/linuxi386/lib/pkg
CGI INDEX : create Cgi.cnf (MSG01)                   -> START
  > found : 
CGI INDEX : no valid package found (WAR01)           -> END
BUILD MSG : .                              -> cct index --dir COMPILER_HOME --sys create
SYS INDEX : work in directory                        -> .../Compiler3.CVS/lib/JavaKiller
SYS INDEX : create arch_pkgIndext.tcl (MSG01)        -> START
SYS INDEX :  create file                             -> linuxi386_pkgIndex.tcl
  add package : AgProc AgVar Agent Args Blowfish Cct CctBlowfish CctBuild CctBwdemo CctCc CctCc CctCheck CctCmd CctComplete CctCrm CctDiff CctExport CctFind CctGbook CctHttpd CctIde CctIndex CctInfo CctInit CctKey CctKeygen CctMail CctMake CctMan CctMd5sum CctPerf CctPkg CctPlugin CctPoll CctProfiler CctRep CctTclsh CctTest CctTkdemo CctView CctWish Cgi CgiChat_main CgiCmd CgiFormmail CgiGbook_add_db CgiGbook_add_form CgiGbook_list_form CgiHidden CgiKey CgiPoll CgiPoll_res CgiUserkey Config External FileLib Find IAgent IBras IFind IIndex IMan IMetaToC IMetaToTcl ITMakeLib ITclHttpd ITclIDE ITclIDEserver ITkDiff IView IndexCct IndexCgi IndexPkg IndexSys IndexTcl InitScript LibMetaToTcl Md5sum MetaToC MetaToTcl PerfLib PkgLib Poll_db Poll_read StackLib StdLib System TMakeLib Target TclIDE TclIDEclient TclIDEserver TestDriver TkCon TkConProc TkDiff base64 bras cmdline counter csv fileutil ftp ftpd graph html htmlparse http http::admin http::auth http::cgi http::config http::counter http::debug http::demo http::direct http::dirlist http::doc http::eval http::imagemap http::include http::ismaptcl http::ismaptk http::log http::logstd http::mail http::mtype http::opentrace httpd httpd__admin httpd__auth httpd__cgi httpd__config httpd__counter httpd__debug httpd__demo httpd__direct httpd__dirlist httpd__doc httpd__eval httpd__imagemap httpd__include httpd__ismaptcl httpd__ismaptk httpd__log httpd__logstd httpd__mail httpd__mtype httpd__opentrace httpd__passcheck httpd__prodebug httpd__redirect httpd__safecgio httpd__session httpd__snmp httpd__srvui httpd__status httpd__stdin httpd__telnet httpd__threadmgr httpd__upload httpd__url httpd__utils httpd__version javascript log math matrix md5 mime ncgi nntp pop3 profiler queue report sha1 smtp stack start_httpd start_httpdthread stats struct tclreadline textutil tree typedcmdline uri 
SYS INDEX : create arch_pkgIndext.tcl (MSG01)        -> END

POWER-Tcl-Ide

  • A lot of improvement got the POWER-Tcl-Ide.

The Ide is a total new development-concept, nothing equal was ever done. The goal is to have a full featured Ide but only as command-line-tool with auto-complete.

  • YES - it is working

The auto-complete feature always give a hint to the user what to-do and the user interact with a POWER-Tcl-distrubution as target.

A typical IDE has a permanently running process, GUI, etc., but the POWER-Tcl-ide has no running process. The only thing that POWER-Tcl has is a single command called cct and a tremendous startup speed that exceeds anything known in modern programming.

example: if the user hit the Tab button the bash start the autocomplete cct complete ... and the cct return everything possible what a user can do with the current command-line.

Because always a full new cct is started (no server or something else is used) the cct need the tremendous startup-speed to provide immediate response.

13 Apr 2021

Test the Early-Bird distribution, for example the package verification utility:

> /build/compiler/EarlyBird> cct pkg --check2 
PKG CHECK MSG: check part 1 -> find unused install template
PKG CHECK MSG: finish part 1
PKG CHECK MSG: check part 2 -> find unknown files
PKG CHECK MSG: finish part 2

In POWER-Tcl an application has always the TCL interpreter and all dependency packages included.
The goal is that an installed application has no external dependencies. This is possible because
TCL ist very small compared to JAVA or other "modern" software and TCL is FREE.

To put everything together the POWER-Tcl has an package-manager cct pkg ... which is also part of every application.
To check if an application is ok the package-manager uses a package-database with an md5sum+info about every
file installed. The command cct pkg --check2 checks the entire application. The package-database is also a part of every application.

To build an application a new tool was added this week called cct build ... . This tool is part of the POWER-Tcl-Developer-Workplace.
To build the Early-Bird application I use : cct build --dir /build/compiler --dist Early --make c.is --pkg add

usage: cct [master-option...] build [user-option ...] [--] args
  short: build and install the distribution

  master options :

    -m             ARCH     platform target                          [linuxi386]
    -d             FLAG     use debug mode                           [no]
                    > simulate the build, do NOT exec
    -v             FLAG     use verbose mode                         [no]
    -f             FLAG     use force mode                           [no]
                    > use 'force' for package installation
  user options :

    -i             FLAG     interactive mode                         [no]
    -e             TARGET   edit target                              [NOT-SET]

    --list         FLAG     list all targets                         [no]
    --proc         FLAG     check the procs : cct check -o...        [no]
    --dir          DIR      set DISTDIR root                         [/my/dist]

    --make         ACTION   compile the files :                      [NOT-SET]
                    > m.exp, m.rb, m.rm, tcl.all, tcl.exp, tcl.is
                    > tcl.rb, tcl.rm, c.all, c.cmp, c.exp, c.is
                    > c.rb, c.rm, c.wk, clean

    --prj          ACTION   work on 'project.cct' file :             [check]
                    > edit  : vim 'project.cct'
                    > save  : tar -czvf SAFE/DATE/target.tgz .project.cct
                    > check : cct check -p ...

    --pkg          ACTION   work on package :                        [upd]
                    > add upd del edit incr lock unlock index
                    > edit  : edit VERSION & pkg.tcl
                    > incr  : increment MAJOR from VERSION
                    > index : create package index

    --dist         ACTION   select distribution, set '--pattern' :   [NOT-SET]
                    > WinRT LinRT LinCgi LinComp WinComp Early Dev

    --pattern      LIST     select pattern (combine with '+-') :     [NOT-SET]
                    > ALL CC CGI CMP EARLY RT TK

    --target       LIST     specific target (disable '--pattern') :  [NOT-SET]
                    > Agent, BWidget, Base, Blowfish, Bras, Cct, CctBuild
                    > CctCrm, CctDiff, CctFind, CctIndex, CctMail, CctMan, CctPerf
                    > CctPlugin, CctRep, CctRt, CctTest, CctView, Cgi, CgiFormmail
                    > CgiGbook, CgiHidden, CgiKey, CgiPoll, CgiUserkey, Config, External
                    > InitScript, Interface, Md5Sum, MetaToC, MetaToTcl, OtCC, OtMetaKit
                    > OtTcl, OtTk, PkgLib, Readline, StdLib, System, TMakeLib
                    > Target, TcLib, TclHttpd, TclIDE, TclLib, TclRl, TclToMeta
                    > TkCon, TkLib

  default options :

    --help         FLAG     print the help (-h)                      [no]
    --info         FLAG     get information about command            [no]
    --edit         FLAG     open project and edit files              [no]
    --args         LIST     command-line arguments                   [*]

9 Apr 2021

  • today I put back the famous tclreadline package.

tclreadline gives you a bash like interactive shell like TkCon for the commandline. tclreadline is started with cct tclsh and no file as argument.

Note
A typical UNIX command like ls -al is evaluated directly without exec etc
Example
using find together with tclreadline empowerd TCL :
tclsh8.3 [~/Project/Compiler3.CVS] find . -name README.md -ls
442018264      8 -rw-r--r--   1  dev1usr  users        4459 Apr  7 21:48 ./README.md

The tclreadline package was part of a project called POWER-os (a subproject of POWER-Tcl) which is an entire LINUX without bash but using TCL as system-programming-language.

To install the tclreadline feature two POWER-Tcl-packages are involved:

Readline.tpkg
Compile and install the libreadline into the POWER-Tcl-runtime
TclRl.tpkg
Compile and install the tclreadline into the POWER-Tcl-runtime

All the configuration compiling and installation is done by POWER-Tcl with just ONE command :

Readline.tpkgcct pkg --add Readline
TclRl.tpkgcct pkg --add TclRl

7 Apr 2021

3 Apr 2021

2 Apr 2021

28 Mar 2021

26 Mar 2021

In the last few days I setup the tools for distribution:

  • switch documentation tool from wiki to doxygen
  • add GNU automake for setup
  • work on general code-cleanup
  • compile the libraries

20 Mar 2021

In the last few days I have been working on the following tasks:

  • Frontend library improvement.
    • much less code with more functions
  • bash-readline-auto-complete-plugin
    • the frontend command line tool cct generates the code for the autocomplete function of bash automatically.
  • package and project file improvements
    • the POWER-Tcl-package is used to distribute code and the final application
    • the POWER-Tcl-project is used to configure the build-process

One thing that Tcl-Compiler does very well is locating BUGS (proof by myself)

  • If proc-names are changed or the number and type of arguments then Tcl-Compiler finds the broken references.
  • If proc is used without a package require etc..
  • Broken code, unexecutable branch of code etc..

There is a good chance to have end of next week a working early-bird edition.

  • but with one throw-back (tcl-8.3.4) (the 20 years old tcl)
  • this tcl has all modern features like namespace, objects etc but missing all the oo code

15 Mar 2021

The compiling of TCL comes closer, step by step...,

  • propably the largest technology-improvement for a scripting-language in 20 years.
  • More details at: HOWTO use the make tool
    compiling ......... c_MRep.cr
    .../Compiler3/lib/JavaKiller/exe/linuxi386/bin/tclsh8.3 package MetaToC
    loading script Meta2C
    loading script LibMeta2Tcl
    loading script Agent
    Compiling Step 1.  (Parse)
    Working on procedure ... <::Agent::MRep::Warning>
    Working on procedure ... <::Agent::MRep::PkgList>
    Working on procedure ... <::Agent::MRep::LFileList>
    Working on procedure ... <::Agent::MRep::Read>
    Working on procedure ... <::Agent::MRep::ReadArray>
    Working on procedure ... <::Agent::MRep::LReadArray>
    Working on procedure ... <::Agent::MRep::ReadList>
    Working on procedure ... <::Agent::MRep::LReadList>
    Working on procedure ... <::Agent::MRep::Write>
    Working on procedure ... <::Agent::MRep::Close>
    Working on procedure ... <::Agent::MRep::Delete>
    Working on procedure ... <::Agent::MRep::DeletePkg>
    Working on procedure ... <::Agent::MRep::ListWrite>
    Working on procedure ... <::Agent::MRep::ArrayWrite>
    Working on procedure ... <::Agent::MRep::List>
    Compiling Step 2.  (Header)
    Compiling Step 3.  (Include)
    Compiling Step 4.  (Static)
    Compiling Step 5.  (CmpLoc)
    Compiling Step 6.  (Prototype)
    Compiling Step 7.  (Template)
    Compiling Step 8.  (Procs)
    Compiling Step 9.  (Init)
    create object ..... c_MRep_linuxi386
    /usr/bin/gcc -DVERSION=\"3.0\" -D BUILD_TclToMeta -D OTTARGET_linuxi386 -I.../Compiler3/lib/JavaKiller/exe/linuxi386/include -I../../C -O -fPIC -Wall -c ../../C/MRep.c -o MRep.o

13 Mar 2021

  • get the make tool back to work.
  • today I had my first build :-)
    Compiler3/cmp/Bras> cct -t IBras make m
    compiling ......... m_IBras
    PROTOTYPE not defined <PROC | PkgProvide          , NS | >
    PROTOTYPE not defined <PROC | Push                , NS | ::StdLib>
    PROTOTYPE not defined <PROC | ErrorN              , NS | ::StdLib>
    PROTOTYPE not defined <PROC | Pop                 , NS | ::StdLib>
    PROTOTYPE not defined <PROC | COMPILER_HOME       , NS | ::Env>
    PROTOTYPE not defined <PROC | COMPILER_ARCH       , NS | ::Env>
    PROTOTYPE not defined <PROC | configure           , NS | ::bras>
    PROTOTYPE not defined <PROC | pinclude            , NS | ::bras>
    PROTOTYPE not defined <PROC | include             , NS | ::bras>
    PROTOTYPE not defined <PROC | dumprules           , NS | ::bras>
    PROTOTYPE not defined <PROC | consider            , NS | ::bras>
    PROTOTYPE not defined <PROC | trimErrorInfo       , NS | ::bras>
    meta .............. m

11 Mar 2021

  • work on documentation
  • improve the cct front-end API

08 Mar 2021

Today is a good day because today the cct tool is back.

  • The main problem was that after 20 years of silence, many code references are broken and parts of the tool need to be revised to make it work again.
  • As I mentioned earlier, there are three Tcl-Compiler revisions and the current revision (3) depends on part of the previous revision (2) and so on.

example: cct

 usage: cct [master option ...] command ...

  master options :

    -t        STRING   compiler package               [NOT SET]
    -a        FLAG     all "*" compiler packages      [no]
    -m        STRING   platform target                [linuxi386]
    -d        FLAG     use debug mode                 [no]

  commands :

    blowfish  blowfish encryption
    bwdemo    start the BWidget demo
    cc        c/c++ cross-compiler
    check     checks various parts of the installation
    crm       customer relationship
    diff      the diff tool
    export    export internal code
    find      find/grep/change tool
    gbook     manage guestbook cgi
    httpd     the tcl web-server
    ide       the "Compiler" IDE
    index     create all kinds of indexes
    info      info about various details
    init      setup a new project
    mail      send emails
    make      interface to the make tool
    man       interface to the man pages
    md5sum    compute the md5 digest
    perf      interface to the tcl "perf" tool
    pkg       interface to the cct "package" tool
    poll      manage poll cgi
    profiler  plugin profiler
    rep       interface to the repository
    tclsh     the POWER-Tcl shell
    test      interface to the tcl "test" tool
    tkdemo    start the tk demo
    view      view plugin interface
    wish      the POWER-Tk shell

  default options :

    -h        print the help
    --help    print the help

07 mar 2021

first signs of life, the current status of Tcl-Compiler is ongoing.

I need some time to get "used-to" 20J old code using tcl-8.3.4 interpreter :-)

Infact there are THREE Tcl-Compiler

  • Compiler1 the initial one, very rudimentary
  • Compiler2 the one which was sold, GOOD but not PERFECT
  • Compiler3 the current one, more PERFECT but UNFINISHED :-)

status: I work on Compiler3

The Compiler3 is more than just an compiler, it is more like an entire infrastructure including

  • package: management, signing, distribution
  • frontend: command-line, gui, web…
  • speed: compile, anaylyze

All tcl features are available in one command -> cct.

  • the tclsh is available as cct tclsh

The goal of the cct is to give the user a perfect environment for tcl without doing any setup mess.

example: PkgLib

 > cct -t TkCon pkg --add -h
 loading script Cct

 usage: cct [master option ...] pkg [user-option ...]

  master options :

    -t        STRING   compiler package               [TkCon]
    -a        FLAG     all "*" compiler packages      [no]
    -m        STRING   platform target                [linuxi386]
    -d        FLAG     use debug mode                 [no]

  user-options :

    --add     FLAG     install the package            [yes]
    --del     FLAG     remove  the package            [no]
    --upd     FLAG     update  the package            [no]
    --list    FLAG     short info about packages      [no]
    --info    FLAG     long info about a package      [no]
    --check   FLAG     check a given package          [no]
    --check2  FLAG     check installation             [no]
    --init    FLAG     create a package from template [no]
    --rec     FLAG     recover package database       [no]
    --clean   FLAG     cleanup COMPILER_HOME          [no]
    --find    FILE     find package for file          [NOT SET]
    --noidx   FLAG     don't create index files       [no]

    -f        FLAG     force mode, don't ask          [no]
    -d        DIR      installation base              [DEFAULT]

  default options :

    -h        print the help
    --help    print the help

  additional informations :

  1) the "-t" option provides the name of the package:
     --del, --list, --info, --check -> PACKAGE_MAJOR_MINOR
     --add --upd -> PACKAGE 
  2) the "--list" "--info" and "--check" option accept a
     pattern for "-t" as argument
  3) the "--rec" option does not recover all the lost data :
     recover: PACKAGE, MAJOR, MINOR
     lost: all other data, but set to useful defaults
  4) with the master "-d" option no directory cleanup's
     are done
  5) "--noidx" is mostly used during mass package 
     installation

04 mar 2021

26 feb 2021

Yesterday I had a look into the source-code of tcl-compiler and I was really surprised how beautiful the source-code is :-)

  • I (Andreas Otto) as the developer have complete forgot this code.

There was a discussion that I don't publish all of the source-code

  • I can say the sourceforge should be complete!

As I mentioned 20 years ago the tcl-compiler can be used as plain tcl script and you can compile the tcl-compiler itself as executable.

> be aware that tcl-compiler uses a minimal modified tcl distribution with TWO features added/improved

  1. a p-unkown tcl function called if a unknown function is used
  2. a v-unkown tcl function called if a unknown variable is used

the modified tcl is part of the code at sourceforge

mfg


OLD stuff below

At the First European Tcl/Tk Users Meeting in 2001, Andreas Otto presented a system he called simply, "Tcl Compiler".

Attributes

website
sourceforge.net
website (original)
compiler-factory.com via archive.org
website (even more original)
t-online.de via archive.org
current version
2.2
release time
2001-10-16

Reading

Compiler 1.3 Release Announcement ,comp.lang.tcl ,2001-04-17
What Is Token-Stream
tutorial (dead)
Anyone who has a copy of this please shout out!
How to Compile (dead link)
PYK would really like to see this document if anyone still has a copy of it.
How to Cross Compile
ditto.
TCL Compiler - Code update ,comp.lang.tcl ,2000-03-01
Tcl Compiler ,comp.lang.tcl ,2006-03-14

Resources

repository

See Also

GSoc Idea: Evaluate and Update Tcl Compiler

Description

arjen 2014-01-08 08:48:52:

The SourceForge project is still available, That might be a good starting point, if anyone wants to revive this tool.

SEH: I have been combing through the sourceforge code and (sparse) docs. It appears that the code base is not complete, the author held back a C runtime library that is necessary actually to run the compiled Tcl program. If someone has up-to-date contact information for the author, perhaps we could persuade him to make a full release. The project looks to be worth saving.

SEH 2021-03-15: I'm following your progress with interest... I encourage you to continue. I look forward to trying the compiler.


rasputnik - 2021-04-05 08:26:07

Does this all make any sense? Read the page about Power-Tcl but cant understand a thing, maybe i'm too stupid but there's no information what is the tool, what it does, how it works. And no source code, no examples. The only link to source code is to something 15 year old, but even that doesn't seem to provide any useful info about what it is.


JAL - 2021-06-08 23:48:45

I'm also interested/curious. I do have a large application that could benefit greatly from >3x speedup, where our strategy at the moment is to gradually convert performance critical parts into C. If we didn't have to hand-code these in C, it would be great.

I've had a look through the information I could find, but could not quite understand some basic points. Please correct what I say below.

  1. Is this fully an "ahead of time" compiler? Being that it requires handlers for unknown variable and command, it suggests the need to be able to execute the program in order to compile it.
  2. Is this a tracing / caching JIT type of approach? E.g. doing a stack analysis and examining how the command at each level is invoked, then performing aggressive inlining (or caching) and then optimization?
  3. Examining the C code output, does this compiler always just translate down to TCL core calls? Or can it go deeper?
  4. Is this just stripping away the TCL-level parsing and emitting the equivalent C code? (via tracing during execution)?

I'd be willing to take a deeper look / play around / help if I could get a better idea of what's going on.


aotto1968 2021-06-09: The basic Idea is quite simple:

  1. every TCL command has an adequat C command, example: info -> Tcl_InfoObjCmd
  2. the next step is adding optimization for special cases like expr or uplevel etc.
  3. finally to put everything together to be a tcl extension able to load pure binary tcl package.

Basically the new C extension has the same structure as the tcl code but in C plus additional optimization.


JAL 2021-06-14 22:28:37

Thanks AOtto1968 for the reply. So, if I'm understanding correctly (and please correct me), OttoCompiler will essentially "inline" the equivalent C code, possibly specializing it according to the call pattern (e.g. constant folding? eliminating branches?). Can it also perform math operations (or maybe even string ops) in an "unboxed" way (convert TCL->native , do the op, then native -> TCL for returning results) ?