[JR]: This is a simple [critcl] function to read an entire file a line at a time and return it as a list.

    package require critcl 0.33

    critcl::ccode {
      #include <stdio.h>
    }

    critcl::cproc loadf {char* fn} Tcl_Obj* {
        FILE *fd;
        Tcl_Obj *r,*s;
        char line[1024];
        r=Tcl_NewListObj(0,0);
        Tcl_IncrRefCount(r);
        s=Tcl_NewStringObj("",0);
        fd=fopen(fn,"r");
        if (!fd) {
            return r;
        }
        while(!feof(fd)) {
            int l;
            if (!fgets(line,1024,fd)) {
                break;
            }
            l=strlen(line);
            if (line[l-1]=='\n') {
                line[--l]='\0';
                Tcl_AppendToObj(s,line,l);
                Tcl_ListObjAppendElement(NULL,r,s);
                s=Tcl_NewStringObj("",0);
            } else {
                Tcl_AppendToObj(s,line,l);
            }
        }
        fclose(fd);
        if (Tcl_GetCharLength(s) > 0) {
            Tcl_ListObjAppendElement(NULL,r,s);
        }
        return r;
    }

    # tcl version for comparison
    proc loadf_tcl {file} {
        set f [open $file]
        fconfigure $f -translation binary
        set d [read -nonewline $f]
        close $f
        split $d \n
    }

    # use 
    set lines [loadf something]

The critcl version is as much as 3 times faster than the tcl version on a large file.

----
I just noticed that on files with lots of short lines (e.g., /usr/share/dict/words) the critcl version is actually slower than the tcl version.  This bears looking into.

[AMG]: Maybe your fgets() is buggy.  I suggest using [strace] to see if it's making unreasonable numbers of read syscalls.

----
With the conventional wisdom that [[read [[open $file]] [[file size $file]]]] is faster than [[read [[open $file]]]], I wonder if the critcl function is still 3x faster. 

''Wow. I've never ever used critcl before. Now I'm convinced. I didn't even know I had critcl on my system, but the above cut'n'pasted and Just Worked. Nifty. And yes, the critcl version is still faster (after the initial compile)''

[AMG]: Conventional wisdom?  Why would it be faster to tell [read] how many bytes to get?

[JMN] 2023-07: With Tcl 8.6.13 and critcl 3.2 I was unable to find any speedup from the critcl version (on windows).  I only tested with text files up to about 458MB or so and up to around 1.9M lines I don't know what constitutes a 'large' file. Perhaps it needs to be large with respect to available memory, or perhaps Tcl script performance has improved.

<<categories>> Command | File