GraphQL

GraphQL with Tcl

Napier October 9, 2017

This is not currently an attempt to build a fully compliant GraphQL package, but I did want to be a parse the GraphQL general syntax and use it to unify the syntax that I use to communicate between various parts of my application. I figured others may find this useful to use or extend. I built this without reading the spec closely so it should be considered when using this code. I will likely improve it over time, and if it ends up being used a lot in our production app you can expect it will grow out from here.

This should be pretty quick. On my system is parses the example query in 1 -2.5 milliseconds. I am sure there are places we can optimize performance. Probably the place with the most optimization capabilities would be with directives since we have to continue parsing their structure even if not included currently.

GraphQL can be an awesome way of querying and controlling various aspects and can be considered a replacement of the "REST" protocol.

It would be awesome to have full GraphQL compliance at some point... just not in the scope of these utilities!


GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data. GraphQL provides a complete and understandable description of the data in your API, gives clients the power to ask for exactly what they need and nothing more, makes it easier to evolve APIs over time, and enables powerful developer tools.


Current Support

Not Yet Supported

While most of these things would actually be pretty easy to support, it is not really something that our team personally requires at this time. Feel free to help out and add support and send pull requests to the tcl-modules repo

Again, this is pretty much just a parser for the syntax. It's missing all of the main functionality such as calling functions and returning results in the requested structure. However, this should be simple enough to do on your own once you have the parsed data structure.

Example GraphQL Query

  • Example JS GraphQL Project - This is a quick example of GraphQL used in an AWS Lambda environment combined with dataLoader request caching. This is what we personally use for our API.

GraphQL allows us to make a query and request the exact data we want. Our server can then respond with only that data without all the extra values that normally might be included with a rest call. When this starts to get more and more complex you quickly realize it can save an amazing amount of REST calls by turning them all into one.

query MyGraphQLQuery(
  $projectIdentityID: String!
  $size: Int!
  $includeColor: Boolean = false $skipBanner: Boolean = false
) {
  project(projectIdentityID: $projectIdentityID) {
    events( 
       size: $size
    ) {
      timestamp
      event {
        from
        banner @skip(if: $skipBanner)
        title
        message
        priority
        icon
        color @include(if: $includeColor)
      }
    }
  }
}

The reason the syntax works like this when it comes to the variable values is so that a client can save a static string value (the query) and, without manipulating the value, simply dispatch that string with a set of variables without mutation.

Currently this will parse the syntax and return a simple data structure to parse through so that the query/mutation can be handled properly. We may receive a JSON query that looks like this:

{
    "query":     {
        "query": "$THE_QUERY_ABOVE"
    },
    "variables": {
        "skipBanner":        true,
        "projectIdentityID": "PROJECT",
        "size":              10
    }
}

We expect the value above be parsed once it has been transformed (via your method of parsing json to dict) by your favorite json package (RL_JSON ;-)). The tcllib json package should do just fine here.

Note: This structure is the way graphql compliant servers will generally expect the data to be formatted. This is inline with GraphiQL programs method of parsing.

In this case, the data returned would be:

type query 
name MyGraphQLQuery 
variables {
  includeColor 0
  skipBanner 1 
  projectIdentityID PROJECT 
  size 10
} 
definitions {
  projectIdentityID {type string required true} 
  size {type integer required true} 
  includeColor {type boolean required false default false} 
  skipBanner {type boolean required false default false}
} 
requests { 
  project {
    name project 
    fn project
    args {
      projectIdentityID PROJECT
    } 
    props {
      {
        name events 
        fn events 
        args {
          size 10 
        } 
        props {
          { name timestamp } 
          { 
            name event 
            fn event 
            props {
              {name from} 
              {name title} 
              {name message} 
              {name priority} 
              {name icon}
            }
          }
        }
      }
    }
  }
}

Walk the Result

Walking the result of a parsed query is really quite simple. At any level we may invoke functions that should inherit the properties of the function it is called within, if the value is a function call, it will have the "fn" key and the "props" key. Directives are removed automatically and will not be included in this structure unless the directive did not pass.

Below is a little helper to show how you may parse a request then walk it, in this case printing out the resulting request.

It is important to note that while the "requests" object is a tcl dict, it should generally be parsed as a list (and its representation will be a list if you want to prevent shimmering). This is because it is technically legal to define two values with the same key. While this may not make sense for queries, it does mark sense for mutations. For example, say we wanted to run two functions during a mutation:

mutation {
  onCluster(slug: "one") {
    status(value: "My New Status")
  }
  onCluster(slug: "two") {
    status(value: "My New Status 2")
  }
}
proc printProp prop {
  upvar 1 lvl plvl
  set lvl [expr { $plvl + 1 }]
  set prefix [string repeat " " $lvl]
  puts "$prefix -- PROP -- [dict get $prop name]"
  if {[dict exists $prop args]} {
    puts "$prefix Args: [dict get $prop args]"
  }
  if {[dict exists $prop fnargs]} {
    puts "$prefix FN Args: [dict get $prop fnargs]"
  }
  if {[dict exists $prop props]} {
    puts "$prefix - Total Props [llength [dict get $prop props]]"
    foreach cprop [dict get $prop props] {
      printProp $cprop
    }
  }
}

proc print {} {
  set lvl 0
  foreach {k v} $::result {
    switch -- $k {
      requests {
        foreach {query schema} $v {
          puts "
            --- QUERY $query ---
          "
          printProp $schema
        }
      }
      default {
        puts "-$k -> $v"
      }
    }
  }
  puts "
    Time to Parse: [expr {$::stop - $::start}] microseconds
  "
}

proc parse {} {
  set data [json get $::PACKET]
  set ::start [clock microseconds]
  set ::result [::graphql::parse $data]
  set ::stop [clock microseconds]
  print
}

With the query that we show at the start:

Type query
projectIdentityID PROJECT
-variables -> skipBanner 1 projectIdentityID PROJECT size 10 includeColor false
-type -> query
-name -> MyGraphQLQuery
-definitions -> projectIdentityID {type string required true} size {type integer required true} includeColor {type boolean required false default false} skipBanner {type boolean required false default false}

            --- QUERY project ---

  -- PROP -- project
  Args: projectIdentityID PROJECT
  - Total Props 1
   -- PROP -- events
   Args: size 10
   - Total Props 2
    -- PROP -- timestamp
    -- PROP -- event
    - Total Props 5
     -- PROP -- from
     -- PROP -- title
     -- PROP -- message
     -- PROP -- priority
     -- PROP -- icon

Expected Response

In the case of this query, it would expect a response that resembles. This is actually the part that is missing from the graphql specification. With a full implementation, each function would have an associated schema and would define what it will return for each. This is actually pretty powerful since it allows tools to introspect the API and give things like autocompletion... perhaps one day :-). :

{
    "project": {
        "events": [
            {
                "timestamp": 1507595940054,
                "event":     {
                    "title":    "Event Title",
                    "from":     "From Value",
                    "message":  "My Message",
                    "priority": 1,
                    "icon":     "alert"
                }
            },
            {
                "timestamp": 1507595940054,
                "event":     {
                    "title":    "Event Title",
                    "from":     "From Value",
                    "message":  "My Message",
                    "priority": 1,
                    "icon":     "alert"
                }
            },
            {
                "timestamp": 1507595940054,
                "event":     {
                    "title":    "Event Title",
                    "from":     "From Value",
                    "message":  "My Message",
                    "priority": 1,
                    "icon":     "alert"
                }
            },
            {
                "timestamp": 1507595940054,
                "event":     {
                    "title":    "Event Title",
                    "from":     "From Value",
                    "message":  "My Message",
                    "priority": 1,
                    "icon":     "alert"
                }
            },
            {
                "timestamp": 1507595940054,
                "event":     {
                    "title":    "Event Title",
                    "from":     "From Value",
                    "message":  "My Message",
                    "priority": 1,
                    "icon":     "alert"
                }
            },
            {
                "timestamp": 1507595940054,
                "event":     {
                    "title":    "Event Title",
                    "from":     "From Value",
                    "message":  "My Message",
                    "priority": 1,
                    "icon":     "alert"
                }
            },
            {
                "timestamp": 1507595940054,
                "event":     {
                    "title":    "Event Title",
                    "from":     "From Value",
                    "message":  "My Message",
                    "priority": 1,
                    "icon":     "alert"
                }
            },
            {
                "timestamp": 1507595940054,
                "event":     {
                    "title":    "Event Title",
                    "from":     "From Value",
                    "message":  "My Message",
                    "priority": 1,
                    "icon":     "alert"
                }
            },
            {
                "timestamp": 1507595940054,
                "event":     {
                    "title":    "Event Title",
                    "from":     "From Value",
                    "message":  "My Message",
                    "priority": 1,
                    "icon":     "alert"
                }
            },
            {
                "timestamp": 1507595940054,
                "event":     {
                    "title":    "Event Title",
                    "from":     "From Value",
                    "message":  "My Message",
                    "priority": 1,
                    "icon":     "alert"
                }
            }
        ]
    }
}

The Code

This will likely be uploaded to the tcl-modules repo shortly at which point it will become the place for the most recent revisions of the script below. Although I will try to keep it up-to-date.

Tip: This regular expression sure is easier to work with when using our syntax highlighting for tcl ;-) check it out on atom "language-dashtcl" (https://www.dropbox.com/s/t3zhdbr1mfxtkle/Screenshot%202017-10-09%2017.43.23.png?dl=0%|%screenshot%|% )

namespace eval graphql {}
namespace eval ::graphql::parse {}

namespace eval ::graphql::regexp {
  variable graphql_re {(?xi) # this is the _expanded_ syntax
    ^\s*
    (?:  # Capture the first value (query|mutation) -> $type
         # if this value is not defined, query is expected.
      ([^\s\(\{]*)
      (?:(?!\()\s*)?
    )?
    (?:          # A query/mutation may optionally define a name
      (?!\()
      ([^\s\(\{]*)
    )?
    (?:\(([^\)]*)\))? # Capture the values within the variables scope
    \s*
    (?:{\s*(.*)\s*})
    $
  }

  # The start of the parsing, we capture the next value in the body
  # which starts the process of capturing downwards.
  variable graphql_body_re {(?xi)
    ^\s*
    (?!\})
    ([^\s\(\{:]*)  # capture the name of the query
    (?:        # optionally may define a name and fn mapping
      (?=\s*:)
      \s*:         # if a colon, then we need the fn name next
      \s*
      (?:
        ((?:
          (?=\[)      # WARNING: Breaking Schema Syntax Here for Tcl specific sugar
          \[[^\]]*\]  # allow passing tcl calls directly as a sugaring to GraphQL
                      # when providing a name (name: [myProc arg arg])
        ) |
        (?:
          [^\s\(\{]* # capture the name of the fn
        ))
      )
    )?
    \s*
    (?:\(([^\)]*)\))? # optionally capture var definitions
    \s*
    (.*)  # grab the rest of the request
    $
  }

  variable graphql_next_query_re {(?xi)
    ^\s*(?:\{\s*)?
    (?!\})
    ([^\s\(\{:]*)  # capture the name of the query
    (?:        # optionally may define a name and fn mapping
      (?=\s*:)
      \s*:         # if a colon, then we need the fn name next
      \s*
      (?:
        ((?:
          (?=\[)      # WARNING: Breaking Schema Syntax Here for Tcl specific sugar
          \[[^\]]*\] # allow passing tcl calls directly as a sugaring to GraphQL
        ) |
        (?:
          [^\s\(\{]* # capture the name of the fn
        ))
      )
    )?
    (?:
      (?=\s*\()    # check if we have arg declarations
      \s*
      \(
        ([^\)]*)   # grab the arg values
      \)
    )?
    (?:           # directives @include(if: $boolean) / @skip(if: $boolean)
      (?=\s*@)
      \s*(@[^\(]*\([^\)]*\)) # capture the full directive to be parsed
                             # we only capture the full directive and
                             # run another query to get the fragments
                             # if needed.
    )?
    (?:
      (?=\s*\{)    # this is a object type, capture the rest so we know
                   # to continue parsing
      \s*\{
      (.*)
      $
    )?
    \s*           # this will only have a value if we are done parsing
    (.*)          # this value, otherwise its sibling will.
  }

  variable graphql_directive_re {(?xi)
    ^@
    ([^\s\(]*)  # the directive type - currently "include" or "skip"
    \s*\(if:\s*
    ([^\s\)]*)  # capture the variable to check against
    \s*\)
    $
  }
}

proc ::graphql::parse data {
  set parsed [dict create]
  set definitions {}
  set type {}

  if {[dict exists $data variables]} {
    dict set parsed variables [dict get $data variables]
  }

  if {[dict exists $data query query]} {
    set query [string trim [dict get $data query query]]
  }

  regexp -- $regexp::graphql_re $query \
    -> type name definitions body

  dict set parsed type $type

  dict set parsed name $name

  set body [string trim $body]

  if {$definitions ne {}} {
    ::graphql::parse::definitions $definitions
  }

  ::graphql::parse::body $body

  return $parsed
}

proc ::graphql::parse::definitions definitions {
  upvar 1 parsed parsed

  foreach {var type} $definitions {
    if {$var eq "="} {
      set default [string trim $type " \"'\{\}"]
      # we are defining a default value to the previous variable
      if {![dict exists $parsed variables $lastParsedVar]} {
        dict set parsed variables $lastParsedVar $default
      }
      dict set parsed definitions $lastParsedVar default $default
      continue
    }

    set var [string trimright $var :]
    set var [string trimleft $var \$]

    if {[string match "*!" $type]} {
      set type [string trimright $type !]
      set required true
      if {![dict exists $parsed variables $var]} {
        tailcall return \
          -code error \
          -errorCode [list GRAPHQL PARSE VAR_NOT_DEFINED] \
          " variable $var is required but it was not provided within the request"
      }
    } else {
      set required false
    }

    if {[string index $type 0] eq "\["} {
      set isArray true
      set type [string range $type 1 end-1]
    } else {
      set isArray false
    }

    if {[dict exists $parsed variables $var]} {
      set varValue [dict get $parsed variables $var]
    } else {
      unset -nocomplain varValue
    }

    set type [string tolower $type]

    switch -- $type {
      float {
        set type double
        set checkType true
      }
      boolean {
        set checkType true
      }
      int {
        set type integer
        set checkType true
      }
      default {
        set checkType false
      }
    }

    if {$checkType && [info exists varValue]} {
      if {$isArray} {
        set i 0
        foreach elval $varValue {
          if {![string is $type -strict $elval]} {
            tailcall return \
              -code error \
              -errorCode [list GRAPHQL PARSE VAR_INVALID_TYPE IN_ARRAY] \
              " variable $var element $i should be ${type} but received: \"$elval\" while checking \"Array<${varValue}>\""
          }
          incr i
        }
      } elseif {![string is $type -strict $varValue]} {
        tailcall return \
          -code error \
          -errorCode [list GRAPHQL PARSE VAR_INVALID_TYPE IN_ARRAY] \
          " variable $var should be type \"${type}\" but received: \"$varValue\""
      }
    }

    set lastParsedVar $var

    dict set parsed definitions $var [dict create \
      type     [string tolower $type] \
      required $required
    ]
  }
}

proc ::graphql::parse::arg arg {
  upvar 1 parsed parsed
  if {[dict exists $parsed variables]} {
    set variables [dict get $parsed variables]
  }
  if {[string index $arg 0] eq "\$"} {
    set name [string range $arg 1 end]
    if {[dict exists $variables $name]} {
      set arg [dict get $variables $name]
    } else {
      # our parsing should have already given an error if it detected
      # this value shoudl be defined - we will simply set it to {}
      set arg {}
      #return -code error " variable $name not found for arg $arg"
    }
  }
  return $arg
}

proc ::graphql::parse::fnargs fnargs {
  upvar 1 parsed parsed
  set data [dict create]

  # set argName  {}
  # set argValue {}
  foreach arg $fnargs {
    set arg [string trim $arg]
    if {$arg eq ":"} {
      continue
    }
    if {[info exists argValue]} {
      # Once defined, we can set the value and unset our vars
      dict set data [arg $argName] [arg $argValue]
      unset argName
      unset argValue
    }
    if {![info exists argName]} {
      set colonIdx [string first : $arg]
      if {$colonIdx != -1} {
        if {[string index $arg end] eq ":"} {
          set argName [string trimright $arg :]
        } else {
          lassign [split $arg :] argName argValue
        }
      } else {
        # this is probably not right?
        set argName $arg
      }
    } else {
      set argValue $arg
    }
  }

  if {[info exists argName] && [info exists argValue]} {
    dict set data [arg $argName] [arg $argValue]
  }

  return $data
}

proc ::graphql::parse::directive directive {
  upvar 1 parsed parsed

  if {[dict exists $parsed variables]} {
    set variables [dict get $parsed variables]
  } else {
    set variables [dict create]
  }

  regexp -- $::graphql::regexp::graphql_directive_re $directive \
    -> type var

  if {[string index $var 0] eq "\$"} {
    set name [string range $var 1 end]
    if {[dict exists $variables $name]} {
      set val [dict get $variables $name]
    }
  } else {
    set val $var
  }

  switch -nocase -- $type {
    include {
      if {![info exists val] || ![string is true -strict $val]} {
        return false
      }
    }
    skip {
      if {[info exists val] && [string is true -strict $val]} {
        return false
      }
    }
    default {
      return tailcall \
        -code error \
        -errorCode [list GRAPHQL BAD_DIRECTIVE] \
        " provided a directive of type $type ($directive).  This is not supported by the GraphQL Syntax."
    }
  }

  return true

}

proc ::graphql::parse::body remaining {
  upvar 1 parsed parsed
  # set lvl 1

  while {$remaining ne {}} {
    set props [list]

    regexp -- $::graphql::regexp::graphql_body_re $remaining \
      -> name fn fnargs remaining

    if {![info exists name] || $name eq {}} {
      break
    }

    if {[string index $name 0] eq "\$"} {
      if {[dict exists $parsed variables [string range $name 1 end]]} {
        set name [dict get $parsed variables [string range $name 1 end]]
      }
    }

    if {$fn eq {}} {
      set fn $name
    }

    if {$fnargs ne {}} {
      set fnargs [::graphql::parse::fnargs $fnargs]
    }

    set remaining [nextType $remaining]

    dict lappend parsed requests $name [dict create \
      name  $name \
      fn    $fn \
      args  $fnargs \
      props $props
    ]

    set remaining [string trimleft $remaining "\}\n "]
  }
}

proc ::graphql::parse::nextType remaining {
  upvar 1 props pprops
  # upvar 1 lvl plvl
  upvar 1 parsed parsed
  # set lvl [expr {$plvl + 1}]

  while {[string index $remaining 0] ne "\}" && $remaining ne {}} {
    unset -nocomplain name
    set skip false
    set props [list]

    regexp -- $::graphql::regexp::graphql_next_query_re $remaining \
      -> name fn fnargs directive schema remaining

    if {![info exists name] || $name eq {}} {
      break
    }

    if {[string index $name 0] eq "\$"} {
      if {[dict exists $parsed variables [string range $name 1 end]]} {
        set name [dict get $parsed variables [string range $name 1 end]]
      }
    }

    if {$directive ne {}} {
      # directive will tell us whether or not we should be
      # including the value.
      if {![::graphql::parse::directive $directive]} {
        set skip true
      }
    }

    set prop [dict create name $name]

    if {[info exists fnargs] && $fnargs ne {}} {
      set fnargs [::graphql::parse::fnargs $fnargs]
      dict set prop args $fnargs
    }

    if {[info exists schema]} {
      set schema [string trim $schema]
      if {$schema ne {}} {
        if {$fn eq {}} {
          set fn $name
        }
        dict set prop fn $fn
        set schema [nextType $schema]
        set schema [string trim $schema]
        # remove the trailing curly bracket
        if {[string index $schema 0] eq "\}"} {
          set remaining [string range $schema 1 end]
        } else {
          set remaining $schema
        }
      }
    }

    if {[string is false $skip]} {
      if {[llength $props] > 0} {
        dict set prop props $props
      }
      lappend pprops $prop
    }

    set remaining [string trim $remaining]
  }

  # At this point, $schema will have content if we need to continue
  # parsing this type, otherwise it will be within remaining
  return $remaining
}