I recently needed to use a few (slightly) non-standard encodings in a tclkit and wondered how I can register them at runtime. The solution that I eventually found was to copy the *.enc file into the encoding subdir of $tcl_library, like this:
file copy shiftjis.enc [file join $tcl_library encoding]
For tclkits the $tcl_library directory is a virtual one (in memory), so it's ok to copy files into it. They aren't saved on disk.
The encoding man page might lead you to believe it would be enough to just append my-encodings-dir to the tcl_libPath variable but that doesn't work because the variable is set at startup time and never re-read. You could change the library path with C function calls.
Couldn't one have an encoding subcommand for this task, maybe something like "encoding create <name> <channel>" ?