Nicolas Hery

Decoding JSON sum types in Go without panicking

2025-03-14T00:00:00Z

Table of contents

Whether we find them useful or not, sum types exist

Sum types (aka tagged unions, discriminated unions, variants, or sometimes enums) are unavoidable as a software developer. If you've been writing code for any length of time, you've almost certainly encountered them.

Many languages support sum types natively: Zig, TypeScript, OCaml, Rust, just to name a few. Even OpenAPI has them, the de-facto standard and language-agnostic way to define HTTP APIs using JSON. So even if the programming language you're using doesn't natively support sum types, you may still need to handle a JSON payload over the wire that is modeled as one. This requires deciding how to decode that payload in your language of choice.

Personal feelings about sum types aside, I think most people would agree they effectively model data structures that can be "one of these (potentially very different) things, and nothing else". And once you've experienced sum types with a switch statement or match expression combined with exhaustiveness checking, it's hard to go back.

Let's take a primitive type, a boolean or bool in Rust. It has 2 possible values: true or false (also called "cardinality"). A struct or record is called a "product type" because you can count the number of possible values (or cardinality) by multiplying the number of possible values of each field. So if I have a struct with 2 boolean fields (example here in Rust):

struct UserStatus {
    signed_up: bool,
    subscribed: bool,
}

The number of possible values for this struct (or "product type") is: 2x2 = 4.

Now I didn't choose this struct example completely at random. Some of the possible values are not valid in this particular domain: a user can't be subscribed if they are not signed up as well. You'll also hear the phrase "make illegal states unrepresentable" when talking about sum types.

A "sum type" is called that way because... you guessed it. You can count the number of possible values (or cardinality) by summing the number of possible values of each branch. So if I have the following sum type (example still in Rust, where they are called "enums"):

enum UserStatus {
    Anonymous,
    SignedUp { subscribed: bool },
}

The number of possible values for this "sum type" is: 1+2 = 3.

I'll leave it as an exercise to the reader to discuss and decide which of these two data structures is better adapted for representing this particular domain.

My first nil pointer panic in Go was due to lack of sum types

Ok, that section title is a bit cheeky and probably not entirely true. But when I figured out what caused the panic in my code, the thought "sum types would've caught this at compile time" did cross my mind. I'm sure the astute reader could find better ways to structure my first implementation, even without sum types. But humor me for the sake of this article.

Let me say it now: This is not one of those "Go should have sum types" post. A lot has already been written on the topic and I don't want to get into the debate (although you'll probably guess where I stand). Let's just assume I want to emulate something like sum types in Go, then:

How do I do so without straying too far from what's idiomatic in the language?
How do I encode and decode it, to and from JSON, with the structure we'll see below?

This post is also not a criticism of Go. I came across this issue in my first Go project, and I actually enjoyed working with the language. Having shied away from Go for a while (notably because of lack of sum types), I finally gave it a try because it seemed a good fit for this project. The fast compile times, robust standard library, simplicity of the language, and great developer tooling all delivered on their promise.

For the anecdote, the first time I ran go build was on the sample codebase from Alex Edward's "Let's Go Further" book (excellent book by the way), and I had to run it again because it was so much faster than what I was used to (cough Haskell cough), I thought nothing had happened.

Back to the historical context: I'm feeling very productive with Go on this particular project. The feedback loop is amazing, and I have a working proof-of-concept in just a couple of days. Code seems to just slip from my fingers, everything works on the first try, zero values and pointers do not scare me anymore, I just need to add this last thing and... then it hits me:

2024/12/07 12:16:53 http: panic serving [::1]:60984:
runtime error: invalid memory address or nil pointer dereference
goroutine 4 [running]:
net/http.(*conn).serve.func1()
    /usr/local/go/src/net/http/server.go:1947 +0xb0
panic({0x100a00f00?, 0x100f19b00?})
    /usr/local/go/src/runtime/panic.go:785 +0x124
example/main.TransformAction(0x14000115e08)
    /Users/nicolashery/dev/example/main.go:110 +0x1c
example/main.(*Server).handleTransformActions(0x140001cad80, {0x100ad6358, 0x14000160380}, 0x140001597c0)
    /Users/nicolashery/dev/example/main.go:157 +0x20c
[...]

Ouch. Having done a lot of Haskell and (strict) TypeScript recently, I had forgotten one could get such runtime errors. But I don't panic (pun intended), and I carefully look at the code mentioned in the stack trace.

Below is a simplified version of the code for the sake of this article (the actual implementation had bigger structures and more cases). Can you spot the error? You have 5 seconds.

func TransformAction(a *Action) string {
	var result string

	switch a.Type {
	case ActionType_CreateObject:
		result = fmt.Sprintf(
			"create_object %s %s %s", a.Object.Type, a.Object.ID, a.Object.Name,
		)
	case ActionType_UpdateObject:
		result = fmt.Sprintf(
			"update_object %s %s %s", a.Object.Type, a.Object.ID, a.Object.Name,
		)
	case ActionType_DeleteObject:
		result = fmt.Sprintf("delete_object %s", a.Object.ID)
	case ActionType_DeleteAllObjects:
		result = "delete_all_objects"
	}

	return result
}

Ok, obviously you'll want to Cmd/Ctrl+Click on Action to see what it is:

type Action struct {
	Type   ActionType `json:"type"`
	Object *Object    `json:"object,omitempty"`
	ID     *string    `json:"id,omitempty"`
}

func NewActionCreateObject(object *Object) Action {
	return Action{
		Type:   ActionType_CreateObject,
		Object: object,
	}
}

func NewActionUpdateObject(object *Object) Action {
	return Action{
		Type:   ActionType_UpdateObject,
		Object: object,
	}
}

func NewActionDeleteObject(id string) Action {
	return Action{
		Type: ActionType_DeleteObject,
		ID:   &id,
	}
}

func NewActionDeleteAllObjects() Action {
	return Action{
		Type: ActionType_DeleteAllObjects,
	}
}

Did you see the error? If yes, then you can stop reading now and get back to work. I'm joking. Didn't see it in the allowed time limit? Don't worry, the Go type checker couldn't either.

Decoding JSON sum types in Go, take one

How did I get to the code above, you might wonder? Well, imagine our service is receiving a JSON payload that looks like this:

[
  {
    "type": "create_object",
    "object": {
      "type": "user",
      "id": "1",
      "name": "user1"
    }
  },
  {
    "type": "update_object",
    "object": {
      "type": "user",
      "id": "1",
      "name": "user1 updated"
    }
  },
  {
    "type": "delete_object",
    "id": "1"
  },
  {
    "type": "delete_all_objects"
  }
]

These are all different types of "actions", and this JSON representation is not unreasonable. The OpenAPI specification has a discriminator "pet" example, and the Redocly documentation a "vehicle" example, that are similar to this. (I have yet to come across an API with pets so apologies my example will be the less fun, but maybe more realistic.)

My naive attempt to decode this JSON, because I was in a rush (and maybe also because Copilot suggested it, if I'm being honest), was to create a struct which I call "bag of all the fields". This is a struct with all possible fields for every action type merged, and using pointers. The zero-value of pointers is nil which will be set for fields that are "unused" by a particular action type. Here it is in all its glory:

type Action struct {
	Type   ActionType `json:"type"`
	Object *Object    `json:"object,omitempty"`
	ID     *string    `json:"id,omitempty"`
}

type ActionType string

const (
	ActionType_CreateObject     ActionType = "create_object"
	ActionType_UpdateObject     ActionType = "update_object"
	ActionType_DeleteObject     ActionType = "delete_object"
	ActionType_DeleteAllObjects ActionType = "delete_all_objects"
)

This works because json.Unmarshal doesn't care if there are missing fields in the JSON payload, it will just set the zero-value for them:

actions := []Action{}
if err := json.Unmarshal(data, &actions); err != nil {
	return err
}

We can also go the other way and call json.Marshal to encode the struct into the same JSON representation as the snippet above. The omitempty struct tag option will remove fields unused by each action type from the resulting JSON.

So we're off to the races, what can go wrong with a bag of pointers? Subtle bugs when trying to access a field that is nil because unused by that action type, that's what:

switch a.Type {
case ActionType_CreateObject:
	result = fmt.Sprintf(
		"create_object %s %s %s", a.Object.Type, a.Object.ID, a.Object.Name,
	)
case ActionType_UpdateObject:
	result = fmt.Sprintf(
		"update_object %s %s %s", a.Object.Type, a.Object.ID, a.Object.Name,
	)
case ActionType_DeleteObject:
	// the bug was here!
	// for this action type `a.Object` is `nil`
	// the correct code should be: `*a.ID`
	result = fmt.Sprintf("delete_object %s", a.Object.ID)
case ActionType_DeleteAllObjects:
	result = "delete_all_objects"
}

How do OpenAPI and Protobuf handle this?

I pick myself up after this runtime panic, and have the following genius idea: There are code generators for OpenAPI, if I give them the specification for the JSON discriminated union above, what do they output for Go? Also, Protocol Buffers is a popular wire format that is based on code generation, and the Oneof field looks a lot like a sum type, so what do they generate for Go?

The OpenAPI schema for an action would look like this:

Action:
  type: object
  discriminator:
    propertyName: type
    mapping:
      "create_object": "#/components/schemas/CreateObject"
      "update_object": "#/components/schemas/UpdateObject"
      "delete_object": "#/components/schemas/DeleteObject"
      "delete_all_objects": "#/components/schemas/DeleteAllObjects"
  oneOf:
    - $ref: "#/components/schemas/CreateObject"
    - $ref: "#/components/schemas/UpdateObject"
    - $ref: "#/components/schemas/DeleteObject"
    - $ref: "#/components/schemas/DeleteAllObjects"
CreateObject:
  type: object
  properties:
    type:
      type: string
    object:
      $ref: "#/components/schemas/Object"
# ...

If I feed this to the OpenAPI Generator (note that I'm using the useOneOfDiscriminatorLookup=true option for better output), I get what I'll call a "bag of all the branches":

type Action struct {
	createObject     *CreateObject
	updateObject     *UpdateObject
	deleteObject     *DeleteObject
	deleteAllObjects *DeleteAllObjects
}

type CreateObject struct {
	Object Object `json:"object"`
}

// ...

It generates an UnmarshalJSON method for Action that:

first decodes the JSON to check the "type" field (this is thanks to the useOneOfDiscriminatorLookup=true codegen option)
according to the value of "type", it chooses the appropriate branch and decodes the JSON using the corresponding struct (CreateObject, UpdateObject, etc.)

Edited for clarity, it looks something like this:

func (a *Action) UnmarshalJSON(data []byte) error {
	var tagged struct {
		Type ActionType `json:"type"`
	}

	if err := json.Unmarshal(data, &tagged); err != nil {
		return err
	}

	var err error
	switch tagged.Type {
	case ActionType_CreateObject:
		err = json.Unmarshal(data, &a.createObject)
	case ActionType_UpdateObject:
		err = json.Unmarshal(data, &a.updateObject)
	case ActionType_DeleteObject:
		err = json.Unmarshal(data, &a.deleteObject)
	case ActionType_DeleteAllObjects:
		err = json.Unmarshal(data, &a.deleteAllObjects)
	}

	return nil
}

To get the actual underlying value, the generator creates a method (which I'll name Value() here) that returns the first non-nil pointer:

func (a *Action) Value() any {
	if a.createObject != nil {
		return a.createObject
	}

	if a.updateObject != nil {
		return a.updateObject
	}

	if a.deleteObject != nil {
		return a.deleteObject
	}

	if a.deleteAllObjects != nil {
		return a.deleteAllObjects
	}

	return nil
}

So this is already a big improvement on my "bag of all the fields" approach. Since the accessor method to the underlying value returns any, I'm now checking the .(type) which can be one of the more precise structs (CreateObject, UpdateObject, etc.):

func TransformAction(action *Action) string {
	var result string

	switch v := action.Value().(type) {
	case *CreateObject:
		result = fmt.Sprintf(
			"create_object %s %s %s", v.Object.Type, v.Object.ID, v.Object.Name,
		)
	case *UpdateObject:
		result = fmt.Sprintf(
			"update_object %s %s %s", v.Object.Type, v.Object.ID, v.Object.Name,
		)
	case *DeleteObject:
		// can't make the same mistake here!
		// trying to do `v.Object.ID` will cause a compiler error:
		// "type *DeleteObject has no field or method Object"
		result = fmt.Sprintf("delete_object %s", v.ID)
	case *DeleteAllObjects:
		result = "delete_all_objects"
	}

	return result
}

Some issues remain though:

I "trust" the any return value of the accessor method to be one of the action structs (CreateObject, UpdateObject, etc.) and nothing else
If I add a "branch" (i.e. another action type), I can easily forget to update the switch statement in TransformAction

Another generator that I tried out, oapi-codegen, uses a slightly different approach. It holds on to a json.RawMessage and delays the decoding until we call an equivalent of the action.Value() accessor method:

type Action struct {
	union json.RawMessage
}

type CreateObject struct {
	Type   string `json:"type"`
	Object Object `json:"object"`
}

// ...

func (a *Action) Value() (any, err) {
	// JSON decoding happens here now
}

The decoding works essentially the same, first decode enough to check the "type" field, then according to its value unmarshal into one of the action structs (CreateObject, UpdateObject, etc.). The json.RawMessage documentation actually has a similar example.

Since delaying JSON decoding wasn't particularly useful in my case, I didn't choose this route. But I wanted to mention it for completeness' sake.

What about Protocol Buffers (aka "Protobuf")? I found the following particularly interesting in their Go generated code guide:

For a oneof field, the protobuf compiler generates a single field with an interface type isMessageName_MyField. It also generates a struct for each of the singular fields within the oneof. These all implement this isMessageName_MyField interface.

Let's try it out. Even though we're working with a JSON API, a Protobuf definition for our data model could look like this:

message Action {
  oneof value {
    CreateObject create_object = 1;
    UpdateObject update_object = 2;
    DeleteObject delete_object = 3;
    DeleteAllObjects delete_all_objects = 4;
  }
}

message CreateObject {
  Object object = 1;
}

// ...

The generated code indeed creates an interface isAction_Value with a single method, as well as an Action struct that holds a field with that interface type:

type Action struct {
	Value isAction_Value `protobuf_oneof:"value"`
	// other protobuf-specific fields omitted
}

type isAction_Value interface {
	isAction_Value()
}

type Action_CreateObject struct {
	CreateObject *CreateObject `protobuf:"bytes,1,opt,name=create_object,json=createObject,oneof"`
}

func (*Action_CreateObject) isAction_Value() {}

// ...

Now these two code generators, OpenAPI and Protobuf, will be the inspiration for my second attempt at decoding the JSON sum type in a more type-safe way...

Decoding JSON sum types in Go, take two

After a bit of searching on the topic of "Go sum types", I stumbled across this: go-check-sumtype. From the README:

A typical proxy for representing sum types in Go is to use an interface with an unexported method and define each variant of the sum type in the same package to satisfy said interface. This guarantees that the set of types that satisfy the interface is closed at compile time.

This "interface with an unexported method" (also called "sealed interface", or "marker interface") sounded like a reasonable way to do it. And it's also what the Protobuf codegen seems to be using.

I replaced my single "bag of all the fields" struct with a sealed interface IsAction and a struct for each variant (CreateObject, UpdateObject, etc.). Each variant struct implements the interface:

//sumtype:decl
type IsAction interface {
	// sealed interface to emulate sum type
	isAction()
}

type CreateObject struct {
	Object Object `json:"object"`
}

func (*CreateObject) isAction() {}

type UpdateObject struct {
	Object Object `json:"object"`
}

func (*UpdateObject) isAction() {}

type DeleteObject struct {
	ID string `json:"id"`
}

func (*DeleteObject) isAction() {}

type DeleteAllObjects struct{}

func (*DeleteAllObjects) isAction() {}

Now I am quite pleased. Not only do these action-specific structs provide more type-safety, but if I forget to handle a variant in my switch statement (or if I add a new one that implements the sealed interface), the go-check-sumtype linter will catch it instead of getting an error at runtime!

func TransformAction(action *Action) string {
	var result string

	// if we miss a case here, `go-check-sumtype` will catch it!
	// for example, omitting `case *DeleteAllObjects` will cause a linter error:
	// "exhaustiveness check failed for sum type IsAction: missing cases for DeleteAllObjects"
	switch v := action.Value().(type) {
	case *CreateObject:
		result = fmt.Sprintf(
			"create_object %s %s %s", v.Object.Type, v.Object.ID, v.Object.Name,
		)
	case *UpdateObject:
		result = fmt.Sprintf(
			"update_object %s %s %s", v.Object.Type, v.Object.ID, v.Object.Name,
		)
	case *DeleteObject:
		// better type-safety for each branch!
		// for example, trying to do `v.Object.ID` will cause a compiler error:
		// "type *DeleteObject has no field or method Object"
		result = fmt.Sprintf("delete_object %s", v.ID)
	case *DeleteAllObjects:
		result = "delete_all_objects"
	}

	return result
}

I still needed to figure out how to decode the JSON sum types payload into this interface and structs. You can't unmarshal into an interface value directly, you need to pass a concrete type. So I created a wrapper struct like so:

type Action struct {
	value IsAction
}

I also found the exhaustive linter, so why stop at sum types when you can also have enums! I defined one for action types, which are used as "tags" in my tagged union, along with the proper methods for JSON and string representations:

type ActionType int

const (
	ActionType_CreateObject ActionType = iota
	ActionType_UpdateObject
	ActionType_DeleteObject
	ActionType_DeleteAllObjects
)

func (t ActionType) MarshalJSON() ([]byte, error) {
	// ...
}

func (t *ActionType) UnmarshalJSON(data []byte) error {
	// ...
	// note: this will return an error for any invalid action type string
}

func (t ActionType) String() string {
	// ...
}

I then defined UnmarshalJSON for my Action wrapper struct like so:

func (a *Action) UnmarshalJSON(data []byte) error {
	var tag struct {
		Type ActionType `json:"type"`
	}

	if err := json.Unmarshal(data, &tag); err != nil {
		return err
	}

	var v IsAction
	// note: `exhaustive` linter will catch if we miss a case here
	switch tag.Type {
	case ActionType_CreateObject:
		v = new(CreateObject)
	case ActionType_UpdateObject:
		v = new(UpdateObject)
	case ActionType_DeleteObject:
		v = new(DeleteObject)
	case ActionType_DeleteAllObjects:
		v = new(DeleteAllObjects)
	}

	if err := json.Unmarshal(data, v); err != nil {
		return err
	}

	a.value = v
	return nil
}

This works similarly to what we saw in the OpenAPI generated code:

first decode only what is needed in the JSON to check the "type" field
second, according to the value of "type", choose the appropriate variant struct of the sum type (CreateObject, UpdateObject, etc.) and use it to decode the JSON payload

For the other way around, I also defined MarshalJSON for the wrapper struct:

func (a *Action) MarshalJSON() ([]byte, error) {
	v := a.value

	data, err := json.Marshal(&v)
	if err != nil {
		return nil, err
	}

	var tagged map[string]any
	if err := json.Unmarshal(data, &tagged); err != nil {
		return nil, err
	}

	// note: `go-check-sumtype` linter will catch if we miss a case here
	switch v.(type) {
	case *CreateObject:
		tagged["type"] = ActionType_CreateObject
	case *UpdateObject:
		tagged["type"] = ActionType_UpdateObject
	case *DeleteObject:
		tagged["type"] = ActionType_DeleteObject
	case *DeleteAllObjects:
		tagged["type"] = ActionType_DeleteAllObjects
	}

	return json.Marshal(&tagged)
}

In this method we:

first encode the wrapped interface as JSON (unlike decoding, we can do this because the interface here will be initialized with an underlying concrete type: CreateObject, UpdateObject, etc.)
second, to add the tag in the "type" field, we do a roundtrip: decode into a map[string]any, add the tag to that map, and re-encode the map into JSON

Notice that I use the exhaustive linter in UnmarshalJSON to make sure I handle all possible tags, and I use the go-check-sumtype linter in MarshalJSON to make sure I handle all possible variant structs. So given I keep the "enum" and "sum type" up-to-date, I will have exhaustiveness checking in both these methods (in addition to other methods or functions, such as TransformAction we saw earlier).

That's it! Yes, there is a bit of boilerplate, but if one is using Go they are probably already OK with a little boilerplate here and there. Also, between AI coding assistants and other codegen tools, the cost of boilerplate can be mitigated. And finally there is that thing we say, "code is read (and maintained) much more often than written"? So I'd argue the added type-safety and the fact that we catch issues at compile time instead of runtime may be worth the tradeoff.

Alternative implementations

Of course, the implementation described above is only one possible way of decoding JSON sum types in Go. Below are a couple alternatives, some of which we've already mentioned.

There is the "bag of all branches" approach (full example here):

type Action struct {
	createObject     *CreateObject
	updateObject     *UpdateObject
	deleteObject     *DeleteObject
	deleteAllObjects *DeleteAllObjects
}

There is the "delayed decoding" approach (full example here):

type Action struct {
	payload json.RawMessage
}

With the "sealed interface" approach I ended up using, I also considered an implementation of MarshalJSON that doesn't require an encode/decode roundtrip to add the tag, at the cost of a bit more boilerplate. It uses struct embedding instead (full example here):

func (a *Action) MarshalJSON() ([]byte, error) {
	var data []byte
	var err error

	switch v := a.value.(type) {
	case *CreateObject:
		tagged := struct {
			Type ActionType `json:"type"`
			CreateObject
		}{
			Type:         ActionType_CreateObject,
			CreateObject: *v,
		}
		data, err = json.Marshal(&tagged)
	// ...
	}

	return data, err
}

Finally, it is worth mentioning that there are different ways to represent sum types in JSON, notably:

internally tagged (the one used in this article): {"type": "delete_object", "id": "1", "soft_delete": true}
adjacently tagged: {"type": "delete_object", "value": {"id": "1", "soft_delete": true}}
externally tagged: {"delete_object": {"id": "1", "soft_delete": true}}

The naming is taken from the Rust library Serde's documentation, which provides a good explanation and examples for each representation.

All JSON representations are possible with the Go implementation of sum types described in this post (you can find the adjacently tagged full example here).

What Go could have been: V lang?

While I was working on this, this little gem appeared in my social news feed: the V programming language.

I'll let the project's website do its own marketing:

V is very similar to Go. If you know Go, you already know ≈80% of V.

Source: vlang.io

And:

[...] V is very similar to Go, and its domain is similar to Rust's [...]

Source: vlang.io/compare

Wait... The simplicity of Go, but with enums and sum types? Yes, please!

I tried porting my example to V, and I have to admit it works out quite nicely (full source here):

enum ObjectType {
	user
	group
}

struct Object {
	type ObjectType
	id   string
	name string
}

struct CreateObject {
	object Object
}

struct UpdateObject {
	object Object
}

struct DeleteObject {
	id string
}

struct DeleteAllObjects {}

type Action = CreateObject | UpdateObject | DeleteObject | DeleteAllObjects

fn transform_action(action Action) string {
	return match action {
		CreateObject {
			'create_object ${action.object.type} ${action.object.id} ${action.object.name}'
		}
		UpdateObject {
			'update_object ${action.object.type} ${action.object.id} ${action.object.name}'
		}
		DeleteObject {
			'delete_object ${action.id}'
		}
		DeleteAllObjects {
			'delete_all_objects'
		}
	}
}

The match expression has exhaustiveness checking, of course. And the Action sum type decodes from/encodes to JSON right out-of-the box (with the caveat that, at the time of writing, it uses the adjacently tagged representation with no way of configuring it).

Before getting too excited it is worth noting that V is very much a niche language, and can't be compared to Go's popularity and ecosystem. Our industry works in mysterious ways, who knows why some languages gain traction while others don't. Also, this might have some truth to it:

There are only two kinds of languages: the ones people complain about and the ones nobody uses.

Bjarne Stroustrup, The C++ Programming Language

Nevertheless, I found the V language interesting! It's a garbage-collected language that seems to have found a sweet spot between Go's simplicity and Rust's powerful type system.

Examples in other languages

If you made it this far, I'll leave you with a link to this repository. It contains an implementation of the example sum type from this article, with JSON encoding/decoding (where applicable), in the following languages:

Go
Haskell
OpenAPI
Protocol Buffers
Rust
TypeScript
V

Nesting APIs and ReaderT environments with Servant

2023-12-13T00:00:00Z

Table of contents

Introduction

Many of the HTTP APIs I've worked on tended to have a nested nature. At each level or sub-API, the request handlers often shared some amount of context. And the deeper you went into the API's nested structure, the more this context grew.

For example, at the top level of the API, all routes could have access to the TraceId for telemetry. Going one level down, we could split the API between non-authenticated (aka "public") routes and authenticated (aka "private") routes. Each branch of this split defines a sub-API. All authenticated routes would have access to a User object. Inside the authenticated API, another nested API could have routes such as /projects/:projectId and /projects/:projectId/tickets. The routes of this last sub-API could have access to a Project object.

Notice that what goes into the shared context at each API level often comes from the HTTP headers and the URL path items. In the example above, the traceparent HTTP header would help us create a TraceId, the Authorization header would allow us to load a User, and the :projectId in the URL path would be used to fetch the Project.

This context also typically contains server dependencies such as database connection pools, HTTP clients, loggers, etc. As these resources do not depend on the request, they can and should be created ahead of time.

Servant gives us powerful tools to build nested APIs in a type-safe manner with its type-level DSL, even if at the cost of a bit of a learning curve. Newer versions of the web framework also introduce NamedRoutes. This feature allows us to structure APIs using records, making it easier to work with more complex route hierarchies.

The "ReaderT design pattern" gives us tools to define a shared context that request handlers can easily access. While it is not the only way to do so, it is popular and approachable enough that we'll use it in this post. The Reader or ReaderT's "environment" represents the request context within this pattern. By nesting environments, we can create different levels of context.

In this article, I'll assume some familiarity with Servant and the ReaderT design pattern. We'll also use Servant's hoistServer to run our custom ReaderT monad and nest our environments. If you need a refresher, the Servant example in this post could be a good item to review.

Server vs. request environment

Before we dive into building nested APIs and defining different levels of context for each of them, let's focus on "level 0": the server environment. All other levels of context will be request environments.

The server environment is different because it is created only once upon application startup, and the resources or attributes it holds will be shared across requests. On the other hand, a request environment is recreated on every request, and the attributes that it holds are scoped to that particular request.

This is an important distinction and was actually the source of a minor bug on a codebase I worked on.

Let's take a look at an example. Imagine we have an API, only one level for now, defined as:

type Api =
  "v1"
    :> Header "traceparent" TraceParentHeader
    :> Header "Authorization" AuthorizationHeader
    :> "tickets"
    :> NamedRoutes TicketApi

data TicketApi mode = TicketApi
  { createTicket
      :: mode
        :- ReqBody '[PlainText] CreateTicketRequest
        :> Post '[PlainText] CreateTicketResponse
  , getTicket
      :: mode
        :- Capture "ticketId" TicketId
        :> Get '[PlainText] GetTicketResponse
  }

We want to run our request handlers in a custom monad App that follows the ReaderT design pattern:

newtype App a = App
  { unApp :: ReaderT AppEnv IO a
  }

Our environment holds the resources and context that most of our request handlers will need. In this example, the environment contains a pool of database connections, an HTTP connection manager used by our HTTP client, a TraceId for telemetry, and an authenticated User object:

data AppEnv = AppEnv
  { dbPool :: Pool Connection
  , httpManager :: Manager
  , traceId :: TraceId
  , user :: User
  }

We define a helper function, runAppServant, that we'll use later. Given an AppEnv, it runs an App action in Servant's Handler monad:

runAppServant :: AppEnv -> App a -> Handler a
runAppServant env action =
  Handler . ExceptT . try $ runReaderT (unApp action) env

To create the server, we first wire up the request handlers in the TicketApi record defined earlier:

createTicketHandler :: CreateTicketRequest -> App CreateTicketResponse
createTicketHandler = -- ...

getTicketHandler :: TicketId -> App GetTicketResponse
getTicketHandler = -- ...

ticketServer :: ServerT (NamedRoutes TicketApi) App
ticketServer =
  TicketApi
    { createTicket = createTicketHandler
    , getTicket = getTicketHandler
    }

Now we can put it all together in the root server:

server
  :: Maybe TraceParentHeader
  -> Maybe AuthorizationHeader
  -> Server (NamedRoutes TicketApi)
server maybeTraceParentHeader maybeAuthHeader =
  hoistServer (Proxy @(NamedRoutes TicketApi)) run ticketServer
  where
    run :: App a -> Handler a
    run action = do
      let appEnv = -- ...
      runAppServant appEnv action

Since our request handlers are defined in the custom App monad, we use Servant's hoistServer. We pass it a run function that converts an action from App to Servant's Handler. This function is also called "natural transformation" in the documentation. Inside run, we use the helper function runAppServant that we defined earlier. We must also create an AppEnv to pass to runAppServant as an argument.

Let's look at a first implementation of run:

server
  :: Maybe TraceParentHeader
  -> Maybe AuthorizationHeader
  -> Server (NamedRoutes TicketApi)
server maybeTraceParentHeader maybeAuthHeader =
  hoistServer (Proxy @(NamedRoutes TicketApi)) run ticketServer
  where
    run :: App a -> Handler a
    run action = do
      -- Bad: These get recreated on every request
      dbPool <- liftIO $ createDbPool "app:app@localhost:5432/app" 10
      httpManager <- liftIO $ createHttpManager 20
      -- Good: These need to be created on every request
      traceId <- liftIO $ getOrGenerateTraceId maybeTraceParentHeader
      user <- liftIO $ authenticateUser httpManager maybeAuthHeader
      let appEnv =
            AppEnv
              { dbPool = dbPool
              , httpManager = httpManager
              , traceId = traceId
              , user = user
              }
      runAppServant appEnv action

One thing that is not immediately apparent when calling hoistServer, at least not for me at first, is that the natural transformation function you give it (run in our case) gets called on every request.

The comments in the code snippet above gave it away, but server-wide resources such as database connection pools or HTTP connection managers are not something you want to recreate for every request. It is either wasteful or misses out on the optimizations brought by resource pools and keeping connections alive.

On the other hand, the TraceId and the User object are definitely request-specific. Indeed, we need to use the request object to create them. In this case, we used the request's HTTP headers, but it could also have been the URL path parameters.

The minor bug mentioned earlier was an HTTP connection Manager being recreated on every request because it was done in the transformation function passed to hoistServer. Nothing critical, but definitely not how the HTTP connection Manager is meant to be used:

If possible, you should share a single Manager between multiple threads and requests.

hackage.haskell.org/package/http-client/docs/Network-HTTP-Client.html#t:Manager

Creating a new Manager is a relatively expensive operation, you are advised to share a single Manager between requests instead.

hackage.haskell.org/package/http-client/docs/Network-HTTP-Client.html#v:newManager

In our example, let's add logging to the functions that create the resources and attributes for AppEnv. Then, we'll run the first server implementation above and make two requests. Here is the log output in the terminal, with some comments added for clarity:

# Server startup
# (nothing)

# Request 1
[Info] Created new database connection pool of size 10 for app:app@localhost:5432/app
[Info] Created new HTTP client manager with number of connections kept-alive per host of 20
[Info] Using existing trace ID 208327fb-d2ca-473f-9e15-85ce49db7493
[Info] Authenticated user with ID d42ed530-adba-41f0-99af-60bd6c476617

# Request 2
[Info] Created new database connection pool of size 10 for app:app@localhost:5432/app
[Info] Created new HTTP client manager with number of connections kept-alive per host of 20
[Info] Generating new trace ID 849a577b-7137-4738-9314-3bf9658d883d
[Info] Authenticated user with ID d42ed530-adba-41f0-99af-60bd6c476617

Indeed, we see that the database connection pool and the HTTP client manager are being recreated for each request, which we don't want. Let's fix this with a second server implementation.

Since we called our request-specific environment AppEnv, let's define another record AppServerEnv that holds the server-wide resources shared across requests:

data AppServerEnv = AppServerEnv
  { dbPool :: Pool Connection
  , httpManager :: Manager
  }

Now, instead of creating these server resources inside the run transformation function, we add them as a parameter to the server function:

server
  :: AppServerEnv
  -> Maybe TraceParentHeader
  -> Maybe AuthorizationHeader
  -> Server (NamedRoutes TicketApi)
server AppServerEnv {dbPool, httpManager} maybeTraceParentHeader maybeAuthHeader =
	hoistServer (Proxy @(NamedRoutes TicketApi)) run ticketServer
  where
    run :: App a -> Handler a
    run action = do
      -- Good: Only these get created on every request
      traceId <- liftIO $ getOrGenerateTraceId maybeTraceParentHeader
      user <- liftIO $ authenticateUser httpManager maybeAuthHeader
      let appEnv =
            AppEnv
              { dbPool = dbPool
              , httpManager = httpManager
              , traceId = traceId
              , user = user
              }
      runAppServant appEnv action

Note: Most functions in a web service run in the context of a request. This is why I haven't defined a monad for the server context AppServerEnv and called it "level 0". But if we have some functions that only require these server-wide resources and are not request-specific, we could define a custom monad AppServer to run them in.

Now we create AppServerEnv and its resources upon application startup instead of inside the hoistServer transformation function:

main :: IO ()
main = do
  -- Good: These get created only once, at server startup
  dbPool <- createDbPool "app:app@localhost:5432/app" 10
  httpManager <- createHttpManager 20
  let port = 3000
      appServerEnv =
        AppServerEnv
          { dbPool = dbPool
          , httpManager = httpManager
          }
      waiApp = serve (Proxy @Api) (server appServerEnv)
  Warp.run port waiApp

Let's run this second server implementation and make our two requests to make sure resource and environment creation is now happening the way we want it:

# Server startup
[Info] Created new database connection pool of size 10 for app:app@localhost:5432/app
[Info] Created new HTTP client manager with number of connections kept-alive per host of 20

# Request 1
[Info] Using existing trace ID 208327fb-d2ca-473f-9e15-85ce49db7493
[Info] Authenticated user with ID d42ed530-adba-41f0-99af-60bd6c476617

# Request 2
[Info] Generating new trace ID 849a577b-7137-4738-9314-3bf9658d883d
[Info] Authenticated user with ID d42ed530-adba-41f0-99af-60bd6c476617

Indeed, we see that the database connection pool and the HTTP client manager only get created once instead of being recreated for each request.

If you'd like to read through this section's complete and runnable examples, you can do so in this gist.

An example nested API

Now that we've underlined the difference between the server and request environments, and when to create them, let's focus on nesting and creating different request environments.

We'll imagine we're building an API for a ticket and issue tracker, similar to Jira but greatly simplified for this example. The layout of the API looks like this (adapted from the layout helper function from Servant):

/v1
└─ Header "traceparent"
   ├─ GET /health
   ├─ GET /layout
   |
   └─ Header "Authorization"
      ├─ GET /organizations
      |
      └─ /organizations/:organizationId
         ├─ POST /projects
         ├─ GET /projects/:projectId
         |
         └─ /projects/:projectId
            ├─ POST /tickets
            └─ GET /tickets/:ticketId

Here, we have four levels of nesting. The handlers for each level will run in their own custom monad based on the ReaderT design pattern (ex: AppTicket for the last level) and with their own environment (ex: AppTicketEnv). These environments will also include all of the contexts from the levels above them.

In this way, each level of handlers creates a "sub-API" (ex: TicketApi). Everything needed to implement the handlers of a sub-API comes from the environment of the monad they run in or the definition of the sub-API itself.

For example, the :ticketId URL path parameter in the last endpoint is included in the TicketAPI definition and allows us to retrieve the ticket. But suppose we need to check user access control for this ticket's project. In that case, we'll use the AppTicketEnv environment. It contains the information we need, captured by the API levels above this one: the user from the "Authorization" header and the project from the :projectId URL path parameter.

From top to bottom, or parent to child, the different levels in this example are as follows:

Level 1: App (ReaderT AppEnv IO)
- Includes all server-wide context (Logging function and Database connection pool in this example)
- Captures the "traceparent" HTTP header and uses it to create an OpenTelemetry Tracing context (setting the active span in this example)
Level 2: AppAuthenticated (ReaderT AppAuthenticatedEnv IO)
- Includes everything from the level above it
- Captures the "Authorization" HTTP header and uses it to authenticate the current user and create an Auth context (containing the UserId in this example)
Level 3: AppProject (ReaderT AppProjectEnv IO)
- Includes everything from the levels above it
- Captures the :organizationId URL path parameter and uses it to fetch the Organization object that the projects belong to
Level 4: AppTicket (ReaderT AppTicketEnv IO)
- Includes everything from the levels above it
- Captures the :projectId URL path parameter and uses it to fetch the Project object that the tickets belong to

And here is what the first couple of levels look like in Haskell, using Servant's NamedRoutes to declare each sub-API as a record:

type Api =
  "v1"
    :> Header "traceparent" TraceParentHeader
    :> NamedRoutes RootApi

-- Level 1: App
data RootApi mode = RootApi
  { health
      :: mode
        :- "health"
        :> GetNoContent
  , layout
      :: mode
        :- "layout"
        :> Get '[PlainText] LayoutResponse
  , authenticatedApi
      :: mode
        :- Header "Authorization" AuthorizationHeader
        :> NamedRoutes AuthenticatedApi
  }

-- Level 2: AppAuthenticated
data AuthenticatedApi mode = AuthenticatedApi
  { listOrganizations
      :: mode
        :- "organizations"
        :> Get '[PlainText] ListOrganizationsResponse
  , projectApi
      :: mode
        :- "organizations"
        :> Capture "organizationId" OrganizationId
        :> "projects"
        :> NamedRoutes ProjectApi
  }

-- Level 3: AppProject
data ProjectApi mode = ProjectApi
	{ -- ...
	}

For the full Servant API definition, see Api.hs of the gist containing the example used in this post.

Nested ReaderT environments

Most codebases I've seen run all request handlers in the same custom monad, for example App. At the top level, Servant's hoistServer is used to translate App back to a Servant Handler and run the server with serve, as we saw in the first section of this article.

That is perfectly fine for a lot of applications. But as the API becomes more complex and nested, we might notice one of two things. Either the AppEnv context contained in the App monad becomes big and holds attributes only used by a subset of handlers and ignored by the rest. Or we find ourselves repeating many of the same logic in the handlers, such as authenticating the User or fetching the Organization that the resources belong to.

One way to approach this can be to define different environments and monads for each sub-API, as described in the previous section. But how do we translate everything back to the Handler that Servant understands? We could do it manually by wrapping each handler individually, but that can quickly become cumbersome.

What wasn't apparent to me initially was that you can make multiple calls to hoistServer. Moreover, the documentation presents hoistServer as a utility to bring handlers running in a custom monad such as App back to Servant's Handler. But we can use it to translate a server from any arbitrary monad (ex: AppAuthenticated) to another arbitrary monad (ex: App).

Let's take a look at an example. We create different custom monads based on the ReaderT design pattern (App, AppAuthenticated, AppProject, etc.) corresponding to each level of our API definition above. Each monad has an associated environment, suffixed with *Env, that holds the environment from the level above it and any additional context specific to that level. We also have a transformation function, prefixed with run*, that translates each monad to the monad from the level above it. The topmost function finally translates back to Servant's Handler. We'll look at the implementation of these transformation functions later.

-- Level 1
newtype App a = App
  { unApp :: ReaderT AppEnv IO a
  }

data AppEnv = AppEnv
  { appLogger :: LogFunc
  , dbPool :: Pool Connection
  , tracer :: Tracer
  -- Above are server-wide dependencies, below are request-specific
  , activeSpan :: IORef Span
  }

runApp :: AppEnv -> App a -> Handler a
runApp = -- ...

-- Level 2
newtype AppAuthenticated a = AppAuthenticated
  { unApp :: ReaderT AppAuthenticatedEnv IO a
  }

data AppAuthenticatedEnv = AppAuthenticatedEnv
  { appEnv :: AppEnv -- Environment from Level 1
  , userId :: UserId
  , appOrganizationService :: OrganizationService
  }

runAppAuthenticated :: AppAuthenticatedEnv -> AppAuthenticated a -> App a
runAppAuthenticated = -- ...

-- Level 3
newtype AppProject a = AppProject
  { unApp :: ReaderT AppProjectEnv IO a
  }

data AppProjectEnv = AppProjectEnv
  { appAuthenticatedEnv :: AppAuthenticatedEnv -- Environment from Level 2
  , projectOrganization :: Organization
  }

runAppProject :: AppProjectEnv -> AppProject a -> AppAuthenticated a
runAppProject = -- ...

-- etc.

Each sub-API from the definition has its own "sub-server", and that sub-server's handlers run in the appropriate monad. For example:

-- Level 1
rootServer :: AppDeps -> ServerT (NamedRoutes RootApi) App
rootServer deps =
  RootApi
    { health = healthHandler
    , layout = layoutHandler
    , authenticatedApi = -- ...
    }

healthHandler :: App NoContent
layoutHandler :: App Text

-- Level 2
authenticatedServer
  :: Maybe AuthorizationHeader
  -> ServerT (NamedRoutes AuthenticatedApi) AppAuthenticated
authenticatedServer maybeAuthHeader =
  AuthenticatedApi
    { listOrganizations = listOrganizationsHandler
    , projectApi = -- ...
    }

listOrganizationsHandler :: AppAuthenticated ListOrganizationsResponse

-- Level 3
projectServer :: OrganizationId -> ServerT (NamedRoutes ProjectApi) AppProject
projectServer organizationId =
  ProjectApi
    { createProject = createProjectHandler
    , getProject = getProjectHandler
    , ticketApi = -- ...
    }

createProjectHandler :: CreateProjectRequest -> AppProject CreateProjectResponse
getProjectHandler :: ProjectId -> AppProject GetProjectResponse

-- etc.

At the top level, we define the server that runs in Servant's Handler. We use hoistServer as we did at the beginning of the article to create the AppEnv and translate rootServer running in App to server running in Handler:

server
  :: AppDeps
  -> Maybe TraceParentHeader
  -> Server (NamedRoutes RootApi)
server deps maybeTraceParentHeader =
  hoistServer (Proxy @(NamedRoutes RootApi)) run (rootServer deps)
  where
    run :: App a -> Handler a
    run action = do
      activeSpan <- -- Use `maybeTraceParentHeader` to create
      let appEnv =
            AppEnv
              { appLogger = depsLogger deps
              , dbPool = depsDbPool deps
              , tracer = depsTracer deps
              , activeSpan = activeSpan
              }
      runApp appEnv action

rootServer :: AppDeps -> ServerT (NamedRoutes RootApi) App
rootServer = -- ...

Note: AppDeps contains any dependencies created at application startup, such as database connection pools and loggers. It is the server-wide environment we described earlier.

For each of the other levels, we can do something similar. We use hoistServer again to translate one level's monad to the monad from the level above it. In the run function passed to hoistServer, we create the environment for the lower level. We use information from the request, such as URL parameters or headers, and make necessary HTTP or database calls to populate the environment's attributes.

For example, to go from AppAuthenticated to App:

rootServer :: AppDeps -> ServerT (NamedRoutes RootApi) App
rootServer deps =
  RootApi
    { health = healthHandler
    , layout = layoutHandler
    , authenticatedApi = authenticatedServer' deps
    }

authenticatedServer'
  :: AppDeps
  -> Maybe AuthorizationHeader
  -> ServerT (NamedRoutes AuthenticatedApi) App
authenticatedServer' deps maybeAuthHeader =
  hoistServer (Proxy @(NamedRoutes AuthenticatedApi)) run (authenticatedServer maybeAuthHeader)
  where
    run :: AppAuthenticated a -> App a
    run action = do
    	appEnv <- ask
    	userId <- -- Use `maybeAuthHeader` to authenticate user
    	let appAuthenticatedEnv =
    				AppAuthenticatedEnv
              { appEnv = appEnv
              , userId = userId
              , appOrganizationService = depsOrganizationService deps
              }
      runAppAuthenticated appAuthenticatedEnv action

authenticatedServer
  :: Maybe AuthorizationHeader
  -> ServerT (NamedRoutes AuthenticatedApi) AppAuthenticated
authenticatedServer = -- ...

Note: We introduce an intermediary authenticatedServer' for clarity and to keep the code similar to the previous level.

You can see in the snippet above how authenticatedServer, running in AppAuthenticated, is mounted in rootServer, running in App. We use hoistServer a second time like we did the first time to translate rootServer from App to Handler.

The main logic happens in the run :: AppAuthenticated a -> App a transformation function passed to hoistServer. In this function, we must construct an AppAuthenticatedEnv to pass to runAppAuthenticated.

To do so, we first retrieve the AppEnv context from the App monad using ask from ReaderT, and embed it in AppAuthenticatedEnv. This is how we nest the different ReaderT environments: each environment has an attribute that holds the environment from the level above it.

Next, we extend the AppAuthenticatedEnv context by adding other attributes. For instance, we can use the "Authorization" header to authenticate and fetch the user's information, such as the userId. We can do this with HTTP calls, database queries, or other IO actions since these custom monads are based on ReaderT env IO.

Let's take a look at another example, going from AppProject to AppAuthenticated:

authenticatedServer
  :: Maybe AuthorizationHeader
  -> ServerT (NamedRoutes AuthenticatedApi) AppAuthenticated
authenticatedServer _ =
  AuthenticatedApi
    { listOrganizations = listOrganizationsHandler
    , projectApi = projectServer'
    }

projectServer' :: OrganizationId -> ServerT (NamedRoutes ProjectApi) AppAuthenticated
projectServer' organizationId =
  hoistServer (Proxy @(NamedRoutes ProjectApi)) run (projectServer organizationId)
  where
    run :: AppProject a -> AppAuthenticated a
    run action = do
    	appAuthenticatedEnv <- ask
    	projectOrganization <- -- Use `organizationId` to fetch organization object
    	let appProjectEnv =
    				AppProjectEnv
              { appAuthenticatedEnv = appAuthenticatedEnv
              , projectOrganization = projectOrganization
              }
      runAppProject appProjectEnv action

projectServer :: OrganizationId -> ServerT (NamedRoutes ProjectApi) AppProject
projectServer = -- ...

We use hoistServer a third time to embed projectServer in authenticatedServer. In the run transformation function for this level, we also grab the previous environment, AppAuthenticatedEnv, and nest it in the new environment, AppProjectEnv. This time, we enhance the new environment by using the organizationId in the URL path parameters to fetch the organization that owns the projects in the current request.

Now let's go back and implement the helper functions for each custom monad that we've been using inside the run functions passed to hoistServer. As a reminder, here are their signatures:

runApp :: AppEnv -> App a -> Handler a
runAppAuthenticated :: AppAuthenticatedEnv -> AppAuthenticated a -> App a
runAppProject :: AppProjectEnv -> AppProject a -> AppAuthenticated a
-- etc.

The top-level runApp gets us to a Servant Handler, as we've seen earlier. There are already existing examples and explanations that break it down, such as this article. Here is the implementation:

runApp :: AppEnv -> App a -> Handler a
runApp env action =
  Handler . ExceptT . try $ runReaderT (unApp action) env

Let's take one of the sub-API transformation functions, for instance runAppProject. Since both AppProject and AppAuthenticated are ReaderT env IO monads, one way to get from one to the other is by unwrapping AppProject with runReaderT and reconstructing AppAuthenticated with ReaderT:

runAppProject :: AppProjectEnv -> AppProject a -> AppAuthenticated a
runAppProject appProjectEnv action =
  AppAuthenticated
    . ReaderT
    $ \_appAuthenticatedEnv -> runReaderT (unAppProject action) appProjectEnv

We're only going from one ReaderT to another and translating the target monad's environment (AppAuthenticatedEnv) into the source monad environment (AppProjectEnv).

It turns out withReaderT does exactly that. So we can simplify a little and write:

runAppProject :: AppProjectEnv -> AppProject a -> AppAuthenticated a
runAppProject appProjectEnv action =  do
  let mapEnv _appAuthenticatedEnv' = appProjectEnv
  AppAuthenticated $ withReaderT mapEnv (unAppProject action)

-- For reference:
withReaderT :: (r' -> r) -> ReaderT r m a -> ReaderT r' m a

In our case, the environment transformation function mapEnv :: r' -> r replaces the environment altogether. But since all our sub-API environments have their parent's environment embedded in them, we might be able to use withReaderT more effectively. Let's start by in-lining runAppProject in projectServer' where it was used:

projectServer' :: OrganizationId -> ServerT (NamedRoutes ProjectApi) AppAuthenticated
projectServer' organizationId =
  hoistServer (Proxy @(NamedRoutes ProjectApi)) run (projectServer organizationId)
  where
    run :: AppProject a -> AppAuthenticated a
    run action = do
    	appAuthenticatedEnv <- ask
    	projectOrganization <- fetchOrganization organizationId
    	let appProjectEnv =
    				AppProjectEnv
              { appAuthenticatedEnv = appAuthenticatedEnv
              , projectOrganization = projectOrganization
              }
          mapEnv _appAuthenticatedEnv' = appProjectEnv
      AppAuthenticated $ withReaderT mapEnv (unAppProject action)

We can simplify run and remove the ask for appAuthenticatedEnv since withReaderT passes it to the mapEnv function:

run :: AppProject a -> AppAuthenticated a
run action = do
  projectOrganization <- fetchOrganization organizationId
  let mapEnv appAuthenticatedEnv =
        AppProjectEnv
          { appAuthenticatedEnv = appAuthenticatedEnv
          , projectOrganization = projectOrganization
          }
  AppAuthenticated $ withReaderT mapEnv (unAppProject action)

Finally, since most of the work of going from an AppProject to an AppAuthenticated is building the AppProjectEnv, we can encapsulate all that and re-introduce the runAppProject function. The parameters of runAppProject provide it with everything it needs to build AppProjectEnv, in this particular case the OrganizationId:

projectServer' :: OrganizationId -> ServerT (NamedRoutes ProjectApi) AppAuthenticated
projectServer' organizationId =
  hoistServer
    (Proxy @(NamedRoutes ProjectApi))
    (runAppProject organizationId)
    (projectServer organizationId)

runAppProject :: OrganizationId -> AppProject a -> AppAuthenticated a
runAppProject organizationId action = do
  projectOrganization <- fetchOrganization organizationId
  let mapEnv appAuthenticatedEnv =
        AppProjectEnv
          { appAuthenticatedEnv = appAuthenticatedEnv
          , projectOrganization = projectOrganization
          }
  AppAuthenticated $ withReaderT mapEnv (unAppProject action)

If you'd like to see the complete source code for this section, Server.hs in the gist is a good place to start.

Implementing the request handlers

Let's now look at using these different monads and nested environments by implementing the request handlers. As an example, we'll write the handler for the "create project" endpoint from level 3:

POST /v1/organizations/:organizationId/projects

This handler will:

Create a new telemetry span using the tracer and activeSpan from the AppEnv parent context
Grab the current userId from the AppAuthenticatedEnv parent context
Grab the current projectOrganization from the AppProjectEnv handler context
Save a new project record in the database using the dbPool connection information from the AppEnv parent context
Log a message using the appLogger from the AppEnv parent context
Send a response, but we'll skip that part for this example

Since all the handler monads are newtypes over ReaderT and have a MonadReader derived instance, we can use asks with the path through the nested environment records up to the specific context attribute we need. Here is a first implementation:

createProjectHandler :: CreateProjectRequest -> AppProject CreateProjectResponse
createProjectHandler projectName = do
  tracer <- asks (tracer . appEnv . appAuthenticatedEnv)
  activeSpan <- asks (activeSpan . appEnv . appAuthenticatedEnv)
  tracedWith tracer activeSpan "create_project" $ do
    userId <- asks (userId . appAuthenticatedEnv)
    organizationId <- asks (organizationId . projectOrganization)
    dbPool <- asks (dbPool . appEnv . appAuthenticatedEnv)
    _ <-
      runDatabaseWith dbPool
        $ query
          "insert into projects (name, organization_id) values (?, ?) returning id"
          (projectName, organizationId)
    logFunc <- asks (appLogger . appEnv . appAuthenticatedEnv)
    _ <-
      flip runLoggingT logFunc
        $ logInfo
        $ "created project"
        :# [ "user_id" .= userId
           , "organization_id" .= organizationId
           ]
    -- ...

This implementation works perfectly fine, and we could stop here. However, the selector functions passed to asks have a bit of repetitiveness. They do have the advantage of being very clear and explicit. But it will get worse if we have a lot of nested environments. Indeed, if we look at the "create ticket" handler one level deeper:

createTicketHandler :: CreateTicketRequest -> AppTicket CreateTicketResponse
createTicketHandler ticketName = do
  tracer <- asks (tracer . appEnv . appAuthenticatedEnv . appProjectEnv)
  activeSpan <- asks (activeSpan . appEnv . appAuthenticatedEnv . appProjectEnv)
  tracedWith tracer activeSpan "create_project" $ do
    userId <- asks (userId . appAuthenticatedEnv . appProjectEnv)
    organizationId <- asks (organizationId . projectOrganization . appProjectEnv)
    projectId <- asks (projectId . ticketProject)
    dbPool <- asks (dbPool . appEnv . appAuthenticatedEnv . appProjectEnv)
    -- ...

To improve the code, we can create small helper functions to reuse across handlers of the same level. For the AppProject level, we can define:

traced :: Text -> AppProject a -> AppProject a
traced spanName action = do
  tracer <- asks (tracer . appEnv . appAuthenticatedEnv)
  activeSpan <- asks (activeSpan . appEnv . appAuthenticatedEnv)
  tracedWith tracer activeSpan spanName action

getUserId :: AppProject UserId
getUserId = asks (userId . appAuthenticatedEnv)

getProjectOrganization :: AppProject Organization
getProjectOrganization = asks projectOrganization

runDatabase :: Database a -> AppProject a
runDatabase action = do
  dbPool <- asks (dbPool . appEnv . appAuthenticatedEnv)
  runDatabaseWith dbPool action

runLogging :: LoggingT AppProject () -> AppProject ()
runLogging action = do
  logFunc <- asks (appLogger . appEnv . appAuthenticatedEnv)
  runLoggingT action logFunc

Our "create project" handler now becomes:

createProjectHandler :: CreateProjectRequest -> AppProject CreateProjectResponse
createProjectHandler projectName = traced "create_project" $ do
  userId <- getUserId
  organizationId <- organizationId <$> getProjectOrganization
  _ <-
    runDatabase
      $ query
        "insert into projects (name, organization_id) values (?, ?) returning id"
        (projectName, organizationId)
  runLogging
    $ logInfo
    $ "created project"
    :# [ "user_id" .= userId
       , "organization_id" .= organizationId
       ]
  -- ...

Some of these helper functions remind me of the "embed" pattern described by Matt Parsons in his book Production Haskell:

The general pattern that I recommend is to embed things into your App type.
runSql :: Database a -> App a
runRedis :: Redis a -> App a
Matt Parsons, Production Haskell (2023), "5.5 Embed, don’t Stack"

The drawback of this approach is that we need to re-implement the helper functions for each monad and level that needs them (App, AppAuthenticated, AppProject, etc.).

Additionally, if we change the structure of the nested environments and cause the path in asks to change, we might need to refactor a lot of code (although the type system would guide us through the whole process).

In the next section, we'll see one way to address these issues.

Note: Since we're using monad-logger-aeson for logging, we can already remove runLogging by implementing a MonadLogger instance for AppProject and all the other request handler monads. For details on how to do that, please refer to this post.

The Has type class pattern

For functionality reused across different levels of the nested API (ex: traced, getUserId, runDatabase, etc.), instead of hard-coding to a specific monad and environment (ex: App, AppAuthenticated, etc.), we can generalize and introduce Has* type classes.

Similar to the "ReaderT" design pattern we've been using, the "Has type class" pattern seems popular. It is also recommended by the rio library (see the Has type classes section of the tutorial).

What does it look like in our example? Let's take the tracing functionality with the traced function, a good example of reuse since all our handlers call it. First, we wrap in a single record TracingEnv everything the function needs from the environment: the tracer and the active span. Then, we define the HasTracing type class to retrieve this tracing context:

data TracingEnv = TracingEnv
  { tracer :: Tracer
  , activeSpan :: IORef Span
  }

class HasTracing env where
  getTracing :: env -> TracingEnv

Note: The Has type class pattern is sometimes used in combination with lenses, notably by the rio library and documentation. I do not use them here because they add another concept to learn, and the type errors can be challenging to understand. But all of these examples would work perfectly fine with lenses as well.

To generalize, we'll use these Has* type classes in combination with the MonadReader type class. Our custom monads are based on ReaderT env IO and can automatically derive MonadReader.

This was the concrete implementation of traced we wrote earlier for AppProject:

traced :: Text -> AppProject a -> AppProject a
traced spanName action = do
  tracer <- asks (tracer . appEnv . appAuthenticatedEnv)
  activeSpan <- asks (activeSpan . appEnv . appAuthenticatedEnv)
  tracedWith tracer activeSpan spanName action

We can now generalize it to:

traced :: (MonadReader env m, HasTracing env, MonadIO m) => Text -> m a -> m a
traced spanName action = do
	tracer <- tracer <$> asks getTracing
  activeSpan <- activeSpan <$> asks getTracing
  tracedWith tracer activeSpan spanName action

Each Has* type class and their helper functions can be moved to their own module, in this case Tracing.hs. This module can be used by request handlers from all levels of the nested API, regardless of the monad they run in, as long as the monad's environment defines an instance of HasTracing.

For example, we defined the AppProject monad as:

newtype AppProject a = AppProject
  { unApp :: ReaderT AppProjectEnv IO a
  }

data AppProjectEnv = AppProjectEnv
  { appAuthenticatedEnv :: AppAuthenticatedEnv
  , projectOrganization :: Organization
  }

For the environment of AppProject, we can define the HasTracing instance as:

instance HasTracing AppProjectEnv where
  getTracing = tracingEnv . appEnv . appAuthenticatedEnv

Note: The AppEnv record now wraps the tracer and active span in the TracingEnv record we introduced.

We can go a little further. Since we'll probably define a HasTracing instance for all levels (AppEnv, AppAuthenticatedEnv, etc.), and since our environments are nested, we can use getTracing from the instance of the level above:

instance HasTracing AppProjectEnv where
  getTracing = getTracing . appAuthenticatedEnv

The AppTicket environment instance is similar:

instance HasTracing AppTicketEnv where
  getTracing = getTracing . appProjectEnv

In the same manner, for authentication information, we can create a reusable Authentication.hs module and introduce a HasAuth type class:

data AuthEnv = AuthEnv
  { userId :: UserId
  }

class HasAuth env where
  getAuth :: env -> AuthEnv

This was the concrete helper function to get the current user we defined in the previous section:

getUserId :: AppProject UserId
getUserId = asks (userId . appAuthenticatedEnv)

We can now replace it with a generalized version using the HasAuth type class:

getUserId :: (MonadReader env m, HasAuth env) => m UserId
getUserId = userId <$> asks getAuth

We then define instances for the environments of each handler monad:

instance HasAuth AppAuthenticatedEnv where
  getAuth = authEnv

instance HasAuth AppProjectEnv where
  getAuth = getAuth . appAuthenticatedEnv

instance HasAuth AppTicketEnv where
  getAuth = getAuth . appProjectEnv

The implementation of the request handlers stays the same as the previous section, except now we import the shared helper functions that leverage the Has* type classes instead of using concrete functions specific to that API level:

import Authentication (getUserId)
import Database (query, runDatabase)
import Tracing (traced)

createProjectHandler :: CreateProjectRequest -> AppProject CreateProjectResponse
createProjectHandler projectName = traced "create_project" $ do
  userId <- getUserId
  organizationId <- organizationId <$> getProjectOrganization
  _ <-
    runDatabase
      $ query
        "insert into projects (name, organization_id) values (?, ?) returning id"
        (projectName, organizationId)
  logInfo
    $ "created project"
    :# [ "user_id" .= userId
       , "organization_id" .= organizationId
       ]
  -- ...

Note that introducing the "Has type class" pattern is optional. Keeping the previous section's implementation is completely fine and can work for many projects and teams.

One may call out that defining all of the Has* type classes and instances still feels a bit like boilerplate. I think the rio tutorial has some good arguments on why that might be acceptable:

The boilerplate here, amortized across a project, is really small

This is the “safe” kind of boilerplate: the compiler will almost always prevent you from making a mistake

The code above is obvious and easy to follow

The code above compiles really quickly

I'll also add that, like any generalization at the type level, the compiler errors might be a little more difficult to parse, notably for less experienced Haskell developers:

We can write a type class HasAppEnvironment and require that instead of a concrete AppEnvironment [...] I generally recommend against this approach. Type class polymorphism is a great way to introduce confusing type errors into a project.

Matt Parsons, Production Haskell (2023), "5.3 AppEnvironment"

Although in practice and on the codebases I've worked on, I didn't notice any notable issues with this particular pattern.

Wrapping up

If you'd like to browse the complete source code for the example used in this post, you can find it in this gist. The modules are organized as such:

Api.hs: The nested API definition using Servant NamedRoutes
Server.hs: Attach request handlers to the API definition and create nested Servant servers using hoistServer
Main.hs: Create server-wide dependencies and run the root server
For each level of the nested API, the ReaderT env IO custom monads, their environments, and the request handler implementations:
Reusable logic and services, using the Has* type class pattern:

If you're curious, I also created a rio port of the same gist.

To summarize, some of the key takeaways for this article are:

Use NamedRoutes and records to define nested Servant APIs. Although not a requirement, they help with code clarity and have better type errors than anonymous routes.
Be wary of recreating server-wide resources on every request if you instantiate them in the transformation function of hoistServer. Instead, create them in your main function and pass them to the server functions as dependencies.
Use hoistServer multiple times to combine request handlers running in different custom monads, matching the nested structure of the API definition.
Use withReaderT to map from one request environment to another, embedding the previous environment into the next and adding new context attributes.

Be mindful of what you choose to put in the different ReaderT request environments. Remember that the transformation function passed to hoistServer will run on every request. For example, fetching the user object could be relatively cheap and used by most handlers, so it would be a good candidate to add to the environment. But avoid more expensive operations or loading data only used by one or two handlers, that would be better done in the handlers themselves.

The pattern described in this article with nested calls to hoistServer is not a "middleware" pattern, although it might seem similar. For instance, we can throw an error if authentication fails and short-circuit all downstream handlers this way. However, we can't use other middleware features like setting the response headers.

For proper middleware functionality, we can wrap the top-level server with any WAI middleware, such as the ones from wai-extra. Or, for more advanced users, you can also create your own Servant combinators (see William Yao's article Writing Servant combinators for fun and profit for a detailed introduction on how to do so).

Finally, this is only one possible way to structure a web application. Regardless of which patterns and conventions are used, what's more important is that they are documented and well-understood by the team. It's also good practice to let the application structure grow organically. Start simple with only a single environment and split things up when and if obvious sub-domains appear.

Using MonadLogger without LoggingT

2023-09-26T00:00:00Z

This post is likely more useful for people beginning with Haskell than those with more experience. When I was less comfortable with Haskell and its ecosystem, it wasn't obvious to me how to use code with a MonadLogger constraint without using the LoggingT transformer. Now that it makes sense, I thought I'd write up my notes. They might help someone who is asking themselves the same question.

Table of contents

What are MonadLogger and LoggingT?

I'll assume you are already familiar with MonadLogger and LoggingT, but here is a quick overview if you need it.

The MonadLogger typeclass from the monad-logger package provides a generic logging interface that other libraries and applications can use. It is then up to the caller to decide how logging actually happens. For example, the logs could go to stdout in JSON format, be collected in a list in memory, we could suppress logging altogether, etc. I like to think of it as dependency injection.

The LoggingT monad transformer is a helper that allows you to wrap a monad and give it logging capability thanks to its MonadLogger instance. It comes with useful utility functions, such as runStdoutLoggingT, to dispatch the transformer by giving it a concrete logging function.

From what I gathered, MonadLogger is quite popular and widespread in the Haskell library ecosystem and application code. Libraries like monad-logger-aeson and Blammo build upon it and provide additional functionality such as structured logging in JSON, pretty-printing logs during development, and configuration via environment variables.

Background

I was working on a codebase that was using monad transformer stacks. There were between 4 and 6 transformers for each custom monad, sometimes more. Something that looked like this:

newtype App a = App
  {unApp :: ReaderT AppConfig (ExceptT AppError (TracingT (LoggingT IO a)))}

I had been reading up on the "ReaderT design pattern" and liked its simplicity and pragmatism.

I was also reading the book Production Haskell by Matt Parsons, which I recommend. It had a few passages on the topic that I found interesting. One discussing the ReaderT pattern:

Fortunately, ReaderT Env IO is powerful enough that we don’t need any additional transformers. Every additional transformer provides more choices and complexity, without offering significant extra power. [...] We can write an instance of MonadLogger directly to avoid needing the LoggingT type.

Matt Parsons, Production Haskell (2023), "5.5 Embed, don't Stack"

And another one discussing logging libraries:

A common mispattern with this library [monad-logger] is incorporating the LoggingT transformer directly into your own code. LoggingT is a “minimal” transformer that should be used only when providing MonadLogger behavior to types you don’t control. Instead, consider writing a manual instance of MonadLogger, allowing you more flexibility in how the messages are used and formatted.

Matt Parsons, Production Haskell (2023), "17.4 Libraries in Brief"

I wanted to see what it would take to refactor this particular codebase to use the ReaderT pattern and what the result would look like. In other words, to end up with something like this:

newtype App a = App
  {unApp :: ReaderT AppEnv IO a}

The codebase had a lot of functions polymorphic on the monad m and using interfaces such as MonadLogger. Functions that looked like this:

action :: (MonadConfig m, MonadLogger m, MonadIO m) => m ()
action = do
  itemUrl <- configItemUrl <$> getConfig
  logInfo $ "Fetching item" :# ["itemUrl" .= itemUrl]
  item <- liftIO $ fetchItem itemUrl
  -- ...

To get to a single ReaderT, I had to remove LoggingT. But all of the examples of logging I had seen so far used LoggingT.

Haskell has somewhat of a reputation that it lacks tutorial-style and beginner-friendly documentation. And that you must often figure things out by reading the types from the generated Haddock documentation. In all fairness, it's impressive that you can figure things out from the types most of the time, which I did for this particular matter. Few programming languages can boast of having such property. Nevertheless, there is still room for more tutorials. The rest of this post might help fill that gap.

With LoggingT

First, let's briefly look at how using MonadLogger with LoggingT looks like. If you are already familiar with this, feel free to skip to the next section.

We'll include LoggingT in our concrete monad App. For this example, our monad transformer stack contains only ReaderT and LoggingT. But it could have more transformers, as mentioned in the introduction above.

newtype App a = App
  {unApp :: ReaderT AppEnv (LoggingT IO) a}
  deriving newtype
    ( -- ...
    , MonadIO
    , MonadReader AppEnv
    , MonadLogger
    )

Note that there is a MonadLogger instance for LoggingT. There is also an instance for a ReaderT containing a monad with a MonadLogger instance. That is the idea behind monad transformers. So we can derive the MonadLogger instance for App, we don't need to write it ourselves.

In our runApp function, we can use one of the "run logging" functions provided by monad-logger to dispatch the LoggingT, such as runStdoutLoggingT (or the runStdoutLoggingT from monad-logger-aeson):

runApp :: AppEnv -> App a -> IO a
runApp env action =
  runStdoutLoggingT $ runReaderT (unApp action) env

Alternatively, we can inject it into runApp as a LoggingT IO a -> IO a parameter. We then defer to the caller, such as our main function, to decide which "run logging" function to use. This is consistent with injecting other dependencies, like AppEnv:

runApp :: AppEnv -> (LoggingT IO a -> IO a) -> App a -> IO a
runApp env runLogging action =
  runLogging $ runReaderT (unApp action) env

Let's say we have a function that uses logging, something like:

action :: (MonadReader AppEnv m, MonadLogger m, MonadIO m) => m ()
action = do
  itemUrl <- asks appItemUrl
  logInfo $ "Fetching item" :# ["itemUrl" .= itemUrl]
  item <- liftIO $ fetchItem itemUrl
  if isInvalidItem item
    then
      logWarn
        $ "Skipping invalid item"
        :# [ "itemUrl" .= itemUrl
           , "itemId" .= itemId item
           ]
    else processItem item

And it is used in our app:

app :: App ()
app = do
	-- ...
  action

We can run this using our runApp function and passing it the required dependencies as arguments:

import Control.Monad.Logger.CallStack (runStdoutLoggingT)

main :: IO ()
main = do
  let appEnv =
        AppEnv
          { appName = "example"
          , appItemUrl = "https://www.example.com/item"
          }
      runLogging = runStdoutLoggingT
  runApp appEnv runLogging app

The example above also works for monad-logger-aeson, which is meant to be a drop-in replacement for monad-logger to get JSON structured logging.

Blammo is another useful logging library. It is a wrapper around monad-logger-aeson, providing features such as environment variable configuration and pretty printing logs during development. Blammo has its own "run logging" functions to dispatch a LoggingT. We can use runSimpleLoggingT, which has an equivalent signature to runStdoutLoggingT:

import Blammo.Logging.Simple (runSimpleLoggingT)

main :: IO ()
main = do
  let -- ...
      runLogging = runSimpleLoggingT
  runApp appEnv runLogging app

For more customization, we can also use runLoggerLoggingT. We pass it a Blammo.Logger or an app environment record that contains one. See the Blammo documentation for more details.

Without LoggingT

If we want to call code with a MonadLogger m constraint without using LoggingT, we'll still need to run it in a concrete monad with a valid MonadLogger instance. Let's keep our custom monad App from the example above, but we'll remove LoggingT and only keep ReaderT (also known as the "ReaderT design pattern"):

newtype App a = App
  {unApp :: ReaderT AppEnv IO a}
  deriving newtype
    ( -- ...
    , MonadIO
    , MonadReader AppEnv
    )

Note that we can't derive a MonadLogger instance for App as we did earlier. We need to implement it ourselves.

Let's look at the definition of the MonadLogger type class:

class Monad m => MonadLogger m where
    monadLoggerLog :: ToLogStr msg => Loc -> LogSource -> LogLevel -> msg -> m ()

Now let's look at how LoggingT implements it:

instance MonadIO m => MonadLogger (LoggingT m) where
    monadLoggerLog a b c d = LoggingT $ \f -> liftIO $ f a b c (toLogStr d)

Or, rewritten for clarity:

instance MonadIO m => MonadLogger (LoggingT m) where
    monadLoggerLog :: ToLogStr msg => Loc -> LogSource -> LogLevel -> msg -> LoggingT m ()
    monadLoggerLog loc logSource logLevel msg =
        LoggingT $ \logFunc ->
            liftIO $ logFunc loc logSource logLevel (toLogStr msg)

If we look at how LoggingT is defined:

newtype LoggingT m a = LoggingT
  {runLoggingT :: (Loc -> LogSource -> LogLevel -> LogStr -> IO ()) -> m a}

It looks similar to a ReaderT, with r swapped for the logging function Loc -> LogSource -> LogLevel -> LogStr -> IO ():

newtype ReaderT r m a = ReaderT
  {runReaderT :: r -> m a}

Since our App is a ReaderT AppEnv, we should be able to do something similar as long as AppEnv contains a logging function that we can pull out. And since App has IO as the underlying monad, we can call the logging function, which runs in IO.

Let's first add the logging function to our AppEnv, aliasing it to LogFunc for convenience:

type LogFunc = Loc -> LogSource -> LogLevel -> LogStr -> IO ()

data AppEnv = AppEnv
  { -- ...
  , appLogFunc :: LogFunc
  }

Now, we can implement our custom MonadLogger instance for App:

instance MonadLogger App where
  monadLoggerLog loc logSource logLevel msg =
    App . ReaderT $ \appEnv -> do
      let logFunc = appLogFunc appEnv
      logFunc loc logSource logLevel (toLogStr msg)

Notice how similar this is to the MonadLogger instance for LoggingT shown earlier. On the outside, we reconstruct our monad with App and ReaderT. On the inside, the body is the ReaderT function r -> m a, in this case, AppEnv -> IO (). We extract the logging function from AppEnv, and we can call it directly since the body is in IO.

An alternative implementation would be to use the instances we have for App. The MonadReader AppEnv instance gives us asks, and the MonadIO instance gives us liftIO. Using these instances would avoid explicitly rebuilding App with ReaderT. If our custom monad was more complex than the "ReaderT pattern", this might make things easier. Our new implementation would look like this:

instance MonadLogger App where
  monadLoggerLog loc logSource logLevel msg = do
    logFunc <- asks appLogFunc
    liftIO $ logFunc loc logSource logLevel (toLogStr msg)

Now that we have a valid MonadLogger instance for App, we need to find a way to construct a logging function to put into our AppEnv when initializing the app.

For inspiration, we can look at the source of runStdoutLoggingT from monad-logger, which we used when we had LoggingT as part of our stack:

runStdoutLoggingT :: MonadIO m => LoggingT m a -> m a
runStdoutLoggingT = (`runLoggingT` defaultOutput stdout)

We notice it uses defaultOutput from the section "utilities for defining your own loggers", which sounds like what we want. The stdout handle is from base.

Putting it together, defaultOutput stdout fits the type of the logging function we want in AppEnv:

import Control.Monad.Logger.CallStack (defaultOutput)
import System.IO (stdout)

main :: IO ()
main = do
  let logFunc = defaultOutput stdout
      appEnv =
        AppEnv
          { -- ...
          , appLogFunc = logFunc
          }
  runApp appEnv app

The example above works for both monad-logger and monad-logger-aeson, after changing the imports appropriately. The latter also provides a defaultOutputWith if we need more customization.

For Blammo, I needed more work to figure out how to use it. We saw in the previous section that Blammo provides functions such as runSimpleLoggingT if we use a LoggingT. But it does not have utilities to construct our own logging function, similar to defaultOutput from monad-logger. It also defines a Blammo.Logger type that is initialized and configurable, by default via environment variables.

Blammo's README has good tutorial-style documentation, with examples of how to customize and integrate it with various frameworks. However, at the time there weren't explicit examples of how to use it without LoggingT.

I looked at the source of runLoggerLoggingT and saw it was a bit more involved than runStdoutLoggingT from monad-logger:

runLoggerLoggingT
  :: (MonadUnliftIO m, HasLogger env) => env -> LoggingT m a -> m a
runLoggerLoggingT env f = (`finally` flushLogStr logger) $ do
  runLoggingT
    (filterLogger (getLoggerShouldLog logger) f)
    (loggerOutput logger $ getLoggerReformat logger)
 where
  logger = env ^. loggerL

It uses non-exported internals such as loggerOutput and getLoggerReformat to construct a MonadLogger logging function. It also does more than build a logging function. For instance, it flushes logs with a finally and wraps the LoggingT m action f with filterLogger.

The next place I looked was the examples for RIO and Amazonka, where I noticed the usage of askLoggerIO from MonadLoggerIO. This looked promising.

class (MonadLogger m, MonadIO m) => MonadLoggerIO m where
    -- | Request the logging function itself.
    askLoggerIO :: m (Loc -> LogSource -> LogLevel -> LogStr -> IO ())

LoggingT IO has an instance of MonadLoggerIO, which makes sense since we give it the logging function when we run it.

So we can use Blammo's runLoggerLoggingT or runSimpleLoggingT to run a small LoggingT IO action just for the purpose of extracting the logging function with askLoggerIO:

import Blammo.Logging.Simple (runSimpleLoggingT)

type LogFunc = Loc -> LogSource -> LogLevel -> LogStr -> IO ()

getBlammoLogFunc :: IO LogFunc
getBlammoLogFunc =
  runSimpleLoggingT $ do
    logFunc <- askLoggerIO
    pure logFunc

Or, more concisely:

getBlammoLogFunc :: IO LogFunc
getBlammoLogFunc =
  runSimpleLoggingT askLoggerIO

Putting it all together, we can run this IO action when the app starts to retrieve our logging function and insert it into AppEnv:

main :: IO ()
main = do
  logFunc <- getBlammoLogFunc
  let appEnv =
        AppEnv
          { -- ...
          , appLogFunc = logFunc
          }
  runApp appEnv app

Let's verify that Blammo's configuration works as expected when running it this way.

Pretty-print by default:

$ cabal run without-loggingt
2023-09-26 18:58:38 [info     ] App started                     appName=example-2c
2023-09-26 18:58:38 [info     ] Fetching item                   itemUrl=https://www.example.com/item
2023-09-26 18:58:38 [warn     ] Skipping invalid item           itemId=652412308 itemUrl=https://www.example.com/item

Use JSON output:

$ LOG_FORMAT=json cabal run without-loggingt
{"time":"2023-09-26T18:59:32.092596139Z","level":"info","location":{"package":"main","module":"Main","file":"WithoutLoggingT.hs","line":111,"char":3},"message":{"text":"App started","meta":{"appName":"example-2c"}}}
{"time":"2023-09-26T18:59:32.092616247Z","level":"info","location":{"package":"main","module":"Main","file":"WithoutLoggingT.hs","line":97,"char":3},"message":{"text":"Fetching item","meta":{"itemUrl":"https://www.example.com/item"}}}
{"time":"2023-09-26T18:59:32.09262781Z","level":"warn","location":{"package":"main","module":"Main","file":"WithoutLoggingT.hs","line":101,"char":7},"message":{"text":"Skipping invalid item","meta":{"itemUrl":"https://www.example.com/item","itemId":"652412308"}}}

Only display above a certain log level:

$ LOG_LEVEL=warn cabal run without-loggingt
2023-09-26 19:00:22 [warn     ] Skipping invalid item           itemId=652412308 itemUrl=https://www.example.com/item

Since working on this, Blammo's maintainers have accepted my pull request to add an example of using Blammo without LoggingT to the documentation.

Wrapping up

The TL;DR for running code with a MonadLogger constraint without using the LoggingT transformer would be:

Assuming we are using a custom monad App that is a ReaderT AppEnv IO, or some monad that has MonadReader AppEnv and MonadIO instances defined
Add the logging function Loc -> LogSource -> LogLevel -> LogStr -> IO () to the environment type AppEnv
Implement a MonadLogger instance for App, using asks from MonadReader AppEnv to grab the logging function from the environment, and liftIO from MonadIO to call it
Construct the logging function when initializing the app, and put it in the AppEnv created
- For monad-logger and monad-logger-aeson, use the utility functions such as defaultOutput with a handle such as stdout
- For Blammo, use runSimpleLoggingT or runLoggerLoggingT to run a LoggingT IO block that returns the logging function by calling askLoggerIO from MonadLoggerIO

You can find the complete working examples with and without LoggingT used for the article in this Gist.

LoggingT was only one of the many transformers I removed while refactoring from a monad transformer stack to a single ReaderT over IO. I removed ExceptT by throwing exceptions in IO. The other transformers were equivalent to ReaderT, some using an IORef or similar for any mutable state. For those transformers, I merged their environments into AppEnv.

Hopefully, some will find this post helpful. Writing it at least helped me sharpen my understanding of how this works.

Finally, this is only one possible way of doing things. My goal was to standardize a codebase on the ReaderT design pattern. However, some may find it easier to add the LoggingT wrapper into the App type and dispatch it in runApp. Of course, that works perfectly fine as well. It is a matter of personal preference or the style adopted by the team.

Comparing request handlers in Scotty, Yesod, and Servant

2023-08-31T00:00:00Z

Table of contents

Context

When someone asks on the internet "which Haskell web framework should I use?", popular answers are Scotty, Yesod, or Servant. Each answer gives different arguments and tradeoffs.

On one end of the spectrum, Scotty is easy to start with. It is similar to Ruby's Sinatra or Node.js' Express, which I've used extensively in previous jobs. Therefore, when learning Haskell, Scotty was a natural pick for my first small project. I wanted to learn the language Haskell, not a web framework and its specifics.

I had a positive experience using Scotty for this first project, so I was curious when reading some comment threads that pointed out some of its shortcomings. For example:

Sure, [Scotty] looks simple, but as soon as you try to do anything more flexible or powerful, you need to know how to dig into how monad transformers work. The documentation on extending it isn't great either.

[...]

ActionM and ScottyM are okay, but then you want a ReaderT or you want to forkIO or you want to do a bracket and suddenly you're thrown into a hell of monad transformer Weirdness.

r/haskell/comments/l7q1dx/using_scotty_in_production/

Also:

scotty has basically 0 features, which makes it seem simple, but it's actually rather complex under the hood, and that complexity leaks out as soon as you need to do anything non-trivial (eg make your handlers in something other than IO).

r/haskell/comments/v7ryqt/minimal_web_framework_ie_flask_for_haskell/

I don't mean this to be a criticism of Scotty. Users, including myself, agree it fills a specific space in the Haskell web framework ecosystem.

These fair remarks prompted me to try doing something "non-trivial" with Scotty and see how easy or complicated that would be. Then, I would compare it to doing the same with Yesod, which I've used a little, and Servant, which we use at work.

Of course, there are other Haskell web frameworks out there. But in the interest of time, I chose to limit myself to these three due to their popularity and my familiarity with them.

Some of the other frameworks, which I did not look into here, include:

Non-trivial request handler

To create a "non-trivial" web request handler to use as an example, I thought back at previous real-world web applications I had worked on and what similar things they needed to do.

While certainly not exhaustive, these are the features I decided my example web request handler would need to showcase:

Read configuration values (using ReaderT)
Logging (using MonadLogger and monad-logger-aeson)
Make database calls (using postgresql-simple)
Make HTTP calls (using req)
Fork threads for asynchronous behavior (using concurrently from UnliftIO.Async)
Throw and handle asynchronous exceptions (using catch from UnliftIO.Exception)
Acquire and release resources (using bracket from UnliftIO.Exception)
Return early with an error response (which depends on the web framework)

I included the Haskell solution I chose for my test, but there are other options. Also, note that these are general web application features and not specific to Haskell.

Custom monad

The one Haskell-specific thing I wanted my example to demonstrate was using a custom monad. Each framework provides a built-in type (ex: Handler), which we can use to define web request handlers. Instead, we'll replace or combine it with our custom monad (ex: App).

I see this monad as a way to provide "dependency injection" to our handlers for logging, database connection pools, HTTP clients, etc. Accessing such dependencies is frequent in "non-trivial" web applications.

There are different flavors of custom monads used in Haskell applications. One is creating a monad transformer stack (ex: stacking ReaderT, LoggingT, DatabaseT, etc.). Another one that is gaining popularity is the so-called ReaderT design pattern. We'll use the latter in our example. We define our custom monad as:

newtype App a = App
  { unApp :: ReaderT AppEnv IO a
  }

AppEnv holds everything we'll need in our web handlers (configuration values, database connection pool, etc.):

data AppEnv = AppEnv
  { appEnvConfig :: Config,
    appEnvLogFunc :: LogFunc,
    appEnvHttpConfig :: HttpConfig,
    appEnvDbPool :: Pool Connection
  }

Fake cart purchase for a booking site

To illustrate a somewhat realistic request handler, imagine we're implementing the cart purchase functionality for an event booking site. We'll create a single endpoint:

POST /cart/:cartId/purchase

We'll assume the cart is already filled by the user and is stored in the database.

Below is what will happen when a request is made to the purchase endpoint. It covers all of the common web application features listed earlier:

We retrieve the cart status by making a database call.
If no cart exists for that cart ID, or if the cart is already purchased or locked, we return an error response.
Before starting the purchase, acquire the cart as a resource by locking it. When the purchase is successful or when an error occurs, release the cart by unlocking it. We implement this in the example by making a database call to update the cart's status, although other ways exist for an actual application. (We use this resource mechanism to prevent, for instance, the double purchase of a cart in the case of multiple requests for the same cart ID.)
Purchasing a cart involves making two requests to external services: one to a booking partner to reserve the seats and another to a payment provider to process the payment. These requests can be done in parallel. So we fork threads and wait for both to complete.
We use HTTP calls to external services to process the booking and payment.
The API URLs for the booking partner and the payment provider are read as configuration values.
In the scenario where either the booking or the payment fails, an asynchronous exception is thrown. We catch it and return a specific error response. (In a real application, we'll also want to do some cleanup here, such as reimburse the payment or cancel the seat reservations.)
If everything goes well, we mark the cart as purchased using a database call and return a successful response.
During all of this, we use logging in key places, such as when we are unable to purchase the cart or before and after making the booking and processing the payment.

This is what the complete handler looks like, using Servant:

postCartPurchaseHandler :: CartId -> App CartPurchaseResponse
postCartPurchaseHandler cartId = do
  cartStatusMaybe <- getCartStatus cartId
  case cartStatusMaybe of
    Nothing -> do
      logWarn $ "Cart does not exist" :# ["cart_id" .= cartId]
      throwIO $ jsonError err404 "Cart does not exist"
    Just CartStatusPurchased -> do
      logWarn $ "Cart already purchased" :# ["cart_id" .= cartId]
      throwIO $ jsonError err409 "Cart already purchased"
    Just CartStatusLocked -> do
      logWarn $ "Cart locked" :# ["cart_id" .= cartId]
      throwIO $ jsonError err409 "Cart locked"
    Just CartStatusOpen -> do
      withCart cartId $ do
        logInfo $ "Cart purchase starting" :# ["cart_id" .= cartId]
        let action :: App (Either Text (BookingId, PaymentId))
            action = Right <$> concurrently (processBooking cartId) (processPayment cartId)
            handleError :: CartException -> App (Either Text (BookingId, PaymentId))
            handleError (CartException msg) = pure $ Left msg
        result <- catch action handleError
        case result of
          Left msg -> do
            logWarn $ ("Cart purchase failed: " <> msg) :# ["cart_id" .= cartId]
            throwIO $ jsonError err500 ("Cart purchase failed: " <> msg)
          Right (bookingId, paymentId) -> do
            markCartAsPurchased cartId
            logInfo $ "Cart purchase successful" :# ["cart_id" .= cartId]
            pure $
              CartPurchaseResponse
                { cartPurchaseResponseCartId = cartId,
                  cartPurchaseResponseBookingId = bookingId,
                  cartPurchaseResponsePaymentId = paymentId
                }

Key takeaways

You can find the source code for the full example on GitHub. It implements the same web request handler and custom monad based on ReaderT IO in all three frameworks: Scotty, Yesod, and Servant.

I won't dive into the implementations' details and the differences between each framework. That could make for another blog post in itself. Instead, I'll highlight what I took away from this experiment.

Scotty:

Scotty's default ActionM (alias for ActionT TL.Text IO) doesn't allow you to do much outside of IO, so most will want to use ActionT right from the start (for example, to use a ReaderT to access configuration and other dependencies).
To use ActionT with a custom monad (let's call it App), you must write the runApp :: App a -> IO a transformation. Although there is an example in Scotty's repo, it may take some work to find, especially for beginners.
Scotty's ActionT is stateful (contains ExceptT and StateT), so we can't define a MonadUnliftIO instance for it. We'll need to use lift in the web handler for all operations with that constraint to run them directly in our custom monad App.
Using lift means we can't call Scotty's functions inside those blocks. For example, we can't send an HTTP response directly from the error handler of catch. (This might not be a bad thing, to help separate HTTP concerns from business logic).
I think I would still recommend Scotty to beginners (over Yesod and Servant). However, I'd suggest having some basic monad transformer knowledge and using ActionT directly. I'd point to an example of such usage (for instance, nicolashery/example-handlers-haskell or eckyputrady/haskell-scotty-realworld-example-app).

Yesod:

Yesod's Handler (alias for HandlerFor AppEnv) is similar to a ReaderT over IO monad, and it already has useful instances for MonadUnliftIO, MonadLogger, etc. There is no real need to define a custom monad stack if we want to use ReaderT IO anyway, so there is no need to call lift or runApp around our business logic functions.
However, access to AppEnv will require using getsYesod instead of asks. We can define instances of the type HasFoo for HandlerData AppEnv AppEnv, to use with a (MonadReader env m, HasFoo env) constraint. But that felt like going into Yesod's internal implementation.
Aside from the web handlers, Yesod's usage of Template Haskell for routes and scaffolding does introduce a bit of indirection and "magic". Although the tradeoff in reduced boilerplate and increased type safety is probably worth it for more experienced Haskell developers, I don't know if I would recommend it for beginners.

Servant:

Servant's documentation explains how to use a custom monad for your web handlers. However, the tutorial uses the simple example of Reader String, and the cookbook suggests defining a ReaderT AppEnv Handler.
Since Handler is stateful (contains ExceptT), we can't define a lawful MonadUnliftIO instance for it. After searching the web (for example here and here), we realize we can use ReaderT AppEnv IO instead of ReaderT AppEnv Handler. We do this by using try and wrapping with an ExceptT and Handler in our "natural transformation" function.
After this initial setup, everything else is relatively straightforward since the web handlers run directly in our custom monad App.
Aside from the web handlers, Servant's usage of the type system to define routes does make for a steep learning curve. Although the tradeoff in type safety is probably worth it for a team of experienced and professional Haskell developers, I'm not sure I would recommend it for beginners.

Wrapping up

I had set out to explore how to implement a non-trivial request handler and integrate with a custom monad App based on ReaderT IO in three popular Haskell web frameworks: Scotty, Yesod, and Servant.

I would say that Servant makes this the easiest since you can write all of your handlers directly in the custom monad App. You then provide a transformation function from App to Servant's Handler. Servant's type-level DSL for defining routes means you get all the request parameters and body as arguments to the handler function. Sending responses is done by returning data from the handler function or throwing an error.

Yesod is a close second. Its Handler type is already equivalent to a ReaderT IO, so you don't need to define a custom App if that's what you want. It also already has useful instances such as MonadLogger, MonadReader, and MonadUnliftIO. This means you generally can use business logic functions directly without using lift or runApp.

Scotty makes it possible to use a custom monad thanks to ActionT. However, as the internet comments from the introduction pointed out, it has some shortcomings. You can't define an instance for MonadUnliftIO, so you must lift any action that uses it. Other instances might be tricky for beginners to write. Also, parsing request parameters and body is slightly more manual than the other two frameworks.

Nevertheless, Scotty is by far the easiest framework to get started with. It doesn't use advanced type system concepts or code generation with Template Haskell. This is why I would still recommend it for people starting out with Haskell.

On parting thoughts, there might be a place for another Haskell web framework in the spirit of Scotty. One that uses basic Haskell concepts but has a few more features out-of-the-box and is easier to integrate with a custom monad. Perhaps such a framework, or something close to it, already exists.

Prototyping and data modeling in the browser using TypeScript

2022-07-28T00:00:00Z

A few months ago, my team was working on a large new feature for our web application. It was tricky to fit into the product, especially without introducing a lot of technical debt. We were having many data modeling conversations on how to best represent the new concepts and functionalities. I noticed we were doing a lot of hand-waving and had trouble understanding what each one of us was picturing in our minds.

I suggested we build a small throwaway prototype that could serve as more concrete material to support the discussion. Using TypeScript, we could explore how to model the new data types. Using React, we could see if the data representation we chose was easy to use in the UI. We would use real data as much as possible to test our assumptions. We would timebox the research, and not worry about re-using any of the code.

I think the experiment was successful. All of a sudden the discussions were a lot more precise and efficient. We could point at a particular data type or UI component, and make a comment or suggest an alternative solution. The time invested in writing this throwaway code was paid off by requiring fewer meetings and making better decisions.

In this article, I'll use a made-up feature as an example. But I'll leverage the same tools we used in our actual prototype. Of course, there are other ways to create prototypes. But if you like this setup, reusing it could save you time to focus on your specific business concerns. The full source code for the example is available on GitHub.

Table of contents

Why prototype in the browser?

As they say, a picture is worth a thousand words. A code snippet as well. As mentioned above, we realized we were doing a lot of hand-waving during meetings trying to explain different implementation ideas for the feature. We needed to get something down on paper, a whiteboard, or in this case, a code editor and a browser screen.

We went from staring at each other's avatars in a Google Meet to screen-sharing side-by-side data types in TypeScript and a basic rendered UI. That was a huge productivity improvement in our meetings.

Trying to get a point across by hand-waving, hidden behind a static avatar...

...versus talking through an idea while pointing at code snippets and a prototype.

The UX designers on the team used Figma a lot, which can also provide supporting material for such conversations. While it is still a very useful tool for exploring ideas and designing, I felt we didn't leverage developers enough at this stage of the project. With code and a prototype, you can show a lot of things that are hard to convey in Figmas and other static mockups.

For example, with code, you can easily work with real data and large amounts of it. Once you write a function that takes the data as input, it is then easy to switch and try different data, or a bigger data set. Realistic and non-trivial data forces you to consider edge cases: long lists, long text values, the absence of values, etc.

On the team, we already leveraged sum types (also known as "tagged union") a lot in our production code. Using code and sum types in the data modeling phase of the prototype pushes you to consider all options or branches, and easily communicate them to other developers.

A prototype in the browser also allows you to show what the URL, routing, and navigation structure could look like, an important part of the technical design for a web application.

Finally, the prototype can be useful for two different audiences: software developers as well as UX designers. The developers will probably focus more on the TypeScript types and the data modeling decisions. The UX designers can review the low-fidelity UI rendered by React. The data model and the UI are two different views of the same underlying concepts, which helps the team build a common vocabulary and work more efficiently together.

Why TypeScript?

When it comes to sketching out data types in code, for illustrative or exploratory purposes, you can use any language that has a good enough type system for your needs. For example, the backend developers on my team have used Haskell to draft technical designs, since that was the language they used in production. Other similar options would be F# for a team that works with .NET, or ReScript for a browser-based language.

I think TypeScript is a good choice for sketching out data types as a team. It is popular and many developers are familiar with its syntax, making it a good lingua franca to communicate types to both frontend and backend developers. It has a powerful type system that was well-suited for the data modeling exercise my team wanted to do. Notably, it can express sum types (or discriminated unions). It also has some more advanced features, allowing you to define types from types to avoid repetition and prototype faster.

TypeScript also has great tooling that generally works out of the box without too much setup or configuration. For instance, VS Code supports TypeScript without installing anything else, complete with type information on hover, autocomplete, etc. It also supports React's JSX syntax natively. For prototyping together as a team and screen sharing, I've found VS Code pleasant to work with. A familiar interface and smooth scrolling make it easier for people to follow along, notably compared to very personalized terminal-based text editors. We also used Live Share which worked quite well, allowing anyone to jump in and fix a type error or sketch out an alternative data type.

VS Code has good built-in tooling for TypeScript: autocomplete, type information on hover, etc.

Finally, TypeScript works well with React and Redux, two other technologies that are well-adapted for this prototyping work. They are both popular, and have well-documented patterns (ex: Redux Toolkit), allowing us to focus on designing the technical solution to our business problem, and not reinventing the wheel. The Redux DevTools Chrome extension is also great for showing co-workers how example data is represented under the chosen data model.

The Redux DevTools extension allows you to explore the data behind the UI

The example prototype

To illustrate this workflow of prototyping in the browser, we're going to pretend that we are building an application that allows the user to create online forms or surveys and collect responses. Something very similar to Google Forms.

Let's imagine that before starting to implement the production-ready version of the feature, we'd like to brainstorm with the team of developers on how to represent, persist, and render the different concepts related to this project. Concepts such as forms, questions, responses, statistics, etc. We'll also want to discuss them with the team of designers, establish a common vocabulary, and map out the different cases the code will need to account for. Building a quick, throwaway prototype during this technical design phase will prove very useful.

With the "online forms creator" example, I tried to pick something less trivial than the popular "to-do list", but small enough to be digestible in this article. The actual feature my team worked on had a lot more complexity and options when it came to the data model, which made the prototyping work even more valuable. It also leveraged sum types, in a similar way that the example in this article will use them to represent different question types. The feature my team worked on had even more branching, with multiple levels of nested sum types, making a strong type system even more helpful.

Of course, there are multiple ways to implement a Google Forms clone, and the way presented here is only one of them. Also, some of the data modeling decisions in this article were made specifically to illustrate a concept. They might have been different decisions in a real business setting.

Below is a screenshot of what the final prototype will look like.

The example prototype we'll be building

Project setup

In this section, we'll set up everything we need before working on the prototype: tooling, dependencies, folder structure, etc.

This is typically the part where one can waste a lot of time. The JavaScript and npm ecosystems have the reputation to provide many choices and configuration options. For a production project maintained over many years, that setup time is easily offset. But for a throwaway prototype, it is even more important to start the actual work quickly.

For this reason, I'll give a detailed step-by-step guide of the specific project setup I used. If you want to adopt this setup, you'll be able to follow along and get started more quickly. Of course, this is only one of many ways to do it, so if you have a favorite project setup you are used to, feel free to use that. Just don't spend too much time on it!

The following instructions will work on any Unix system (macOS, Linux) or WSL on Windows.

We'll install Node using nvm. Copy the curl command from the installation instructions and run it. Then install the latest version of Node with:

nvm install node
nvm use node

For the frontend tooling, I chose Vite mainly because it promises a quick recompile and feedback loop. Webpack is also fine and a popular option.

We'll use the create-vite tool to scaffold the project (create-react-app is also an option if not using Vite). Don't create a project folder yet, the tool will do it for you. Run:

npm create vite@latest

Choose react as a framework and react-ts as a variant.

Navigate into the new project directory and install dependencies:

npm install

Check the Node version you are using and create an .nvmrc file (replace "18" with the version you are using):

node --version
echo "18" > .nvmrc

You'll see an initial project structure similar to this:

tree -L 2 -a -I "node_modules|.git|dist" --dirsfirst --noreport
.
├── src
│   ├── App.css
│   ├── App.tsx
│   ├── favicon.svg
│   ├── index.css
│   ├── logo.svg
│   ├── main.tsx
│   └── vite-env.d.ts
├── .gitignore
├── .nvmrc
├── index.html
├── package-lock.json
├── package.json
├── tsconfig.json
├── tsconfig.node.json
└── vite.config.ts

At this point we can create the first commit:

git init
git add .
git status
git commit -m "Initial commit"

Try running the app:

npm run dev

Install Visual Studio Code, and open the project folder with:

code .

The editor will automatically type-check .ts and .tsx files as you edit them. Let's also add a command-line script to type-check the whole project (which is useful to run before committing, for example). In the scripts section of package.json, add the command "typecheck": "tsc --noEmit". Now run:

npm run typecheck

We'll install the Prettier code formatter. This way we won't be tempted to waste time manually formatting our code and be able to focus on the prototyping work.

npm install --save-dev prettier

Create a .prettierrc.json file with the following content:

{
  "semi": false,
  "singleQuote": true
}

Note: Prettier is meant to be used without any customization. Here, we're just tweaking it to match the preferred style of Vite's scaffolding.

In VS Code, search for the Prettier extension and install it. Open the Settings with Cmd + , (Ctrl + , on Windows), search for "format on save", and enable it for the User (or for the Workspace if you prefer).

Now open a TypeScript file in VS Code such as src/App.tsx. Open the Command Palette with Cmd + Shift + p (Ctrl + Shift + p on Windows), search for "format document", and run it. You may need to select a formatter the first time you do this. You should see no changes to the file if it is already formatted properly.

In the scripts section of package.json, add the command "prettier": "prettier src --write". Now run:

npm run prettier

At the time of writing, Vite does not display the compiler type errors directly in the browser. This would be a nice development feature that could help with our prototyping productivity. We can add it using the vite-plugin-checker:

npm install --save-dev vite-plugin-checker

Add the plugin to vite.config.ts:

import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'
import checker from 'vite-plugin-checker'

// https://vitejs.dev/config/
export default defineConfig({
  plugins: [checker({ typescript: true }), react()],
})

Try it out. Start the development server with npm run dev, then open App.tsx and create a type error. For example:

setCount((count) => count + "1")

You should see the type error reported in your browser.

This concludes our initial project setup. At this point, it is a good idea to create another commit:

git add .
git commit -m "Add tooling"

Adding CSS

Our prototype doesn't have much functionality yet, but let's add some styling now. The goal will be to spend as little time as possible on the way things look. We'll want to focus on data modeling and prototyping different behaviors. This is not a high-fidelity UI mockup.

For this reason, it will be helpful to pick a CSS framework that has plenty of standard components, layout helpers, and utility classes. Picking a framework your team is already familiar with is also good, as there will be no learning curve.

For the prototype my team worked on we used Bootstrap, mainly because of its familiarity. It worked well, but one thing I realized is that the styles didn't give enough of a "low-fidelity" look. Each time I presented my screen, I felt I had to insist to the audience: "ignore the styling, this is not a UI design mockup".

I later came across some hand-drawn style CSS frameworks, such as PaperCSS and DoodleCSS. Although they have fewer components and utility classes, I think they would have been a good fit. Such a rough look would have made it immediately obvious to the audience that this is more of an interactive wireframe than a high-fidelity design. It reminds me of the "fat marker sketches" concept from Basecamp's Shape Up book.

We'll use PaperCSS in this example, but again feel free to pick whatever makes you most productive:

npm install --save papercss

Let's first clean up the scaffolding created by Vite:

Change the <title> in index.html to "Prototype" or something else
Remove the favicon link <link rel="icon" /> in index.html
Remove files we're not going to use: rm src/favicon.svg src/logo.svg src/App.css

Now update src/App.tsx with some basic layout helpers from PaperCSS:

function App() {
  return (
    <div className="container margin-top-large">
      <p>You don't have any forms created yet.</p>
      <p>
        <a href="#" className="paper-btn btn-primary">
          {'Create a new form'}
        </a>
      </p>
    </div>
  )
}

export default App

In src/main.ts, import the PaperCSS styles:

import 'papercss'

The first component we'll add is a Navbar. Almost every app has one. Our prototype doesn't strictly need it, especially if it is focused on a single feature, but it will help put the other elements into context.

Create a new file src/Navbar.tsx containing:

function Navbar() {
  return (
    <nav className="border split-nav">
      <div className="nav-brand">
        <h3>
          <a href="#">Forms</a>
        </h3>
      </div>
      <div className="collapsible">
        <input id="collapsible1" type="checkbox" name="collapsible1" />
        <label htmlFor="collapsible1">
          <div className="bar1" />
          <div className="bar2" />
          <div className="bar3" />
        </label>
        <div className="collapsible-body">
          <ul className="inline">
		  			<li>
              <a href="#">Forms</a>
            </li>
            <li>
              <a href="#">Account</a>
            </li>
            <li>
              <a href="#">Logout</a>
            </li>
          </ul>
        </div>
      </div>
    </nav>
  )
}

export default Navbar

Note: An HTML to JSX tool becomes handy when copying examples from a CSS framework's documentation site and converting them to React components.

Import the Navbar component into src/App.tsx and add it above the container div.

Open the src/index.css file, and remove all initial styles generated by Vite's scaffolding tool.

We'll add a couple of fixes and overrides to the PaperCSS styles. Try to limit these custom CSS rules as much as possible. Again, this prototyping work is not a UI design exercise.

/* PaperCSS fixes
-------------------------------------------------------- */

a:visited {
  color: var(--secondary);
}

nav a:visited {
  color: var(--primary);
}

/* PaperCSS tweaks
-------------------------------------------------------- */

nav ul.inline li a {
  font-size: 1rem;
}

nav.subnav ul.inline {
  margin-top: 0;
}

We'll also add a couple of utility classes that we'll use later:

/* Utility classes
-------------------------------------------------------- */

.text-small {
  font-size: 0.8rem;
}

.inline-block {
  display: inline-block !important;
}

.text-right {
  text-align: right;
}

Start the server with npm run dev, and open the app in your browser to see what we have so far.

Adding some CSS

Now is a good time to commit:

git add .
git commit -m "Add PaperCSS"

Using real data

I can't stress enough the importance of working with real data as much as possible when designing a new feature or application. In my team's case, we used anonymized production data. We had to write a script that re-shaped the raw data to make it fit the new feature's data model. It may seem like a lot of work compared to creating "dummy" data, but in my experience, it is always worth the investment.

Even if you need to make up data, make it as realistic as possible. Don't use words like "test", "foo", or dog breeds (unless you're creating an application about dogs of course). Try to use real business domain terms. For example, in this article's prototype, we'll create an "Order Request" form, a "Customer Feedback" form, etc. These are forms that a company using the feature would realistically create.

By using real data, you'll notice things that you would miss with trivial and fake data. For example, you'll see that the distribution of items per category is often messier than what is shown in high-fidelity designs. It could contain a long tail of categories with just 1 or 2 items, which might not fit cleanly in a horizontal list of filters for instance. You'll see that the length of a text field, such as the name of an item or person, can sometimes be very short or very long. The UI will need to account for that, by wrapping the text or by using an ellipsis with a tooltip.

The ability to work with real data, and large amounts of it, is one of the big differentiators of a prototype in code compared to static designs such as those created in Figma. You can write a single rendering function, then .map() over hundreds of items and see what the result looks like, what patterns arise, and all the cases that need to be handled. As you iterate and get feedback, a change to that rendering function will apply to all of the items in the sample data set. Whether there are a couple of items or a hundred items, it is the same amount of work.

For the online forms prototype in this article, I did not use real data since it is just an example. But if it were real prototyping work, I imagine we would take the time to gather a couple of actual forms and their responses that customers are using as part of their business processes.

Once you have obtained a real data set, how do you use it in the prototype? Unless the data set is very large, a flat-file format checked in the repository gives you some persistence without complicating the prototype's stack by introducing a dependency on PostgreSQL or similar.

If you use a serialized format like JSON or YAML instead of creating the sample data directly in code with TypeScript, then it will force you to think about how the data will be serialized and persisted. This is an important part of a feature's technical design. To give an example, in my team's prototyping work, one approach we first took had functions attached to each record of the chosen data structure. We quickly realized that we wouldn't be able to persist these functions or send them to the frontend, and that we needed to find a way to represent them as data or change our approach.

We'll install vite-plugin-content, which allows us to import directly many text formats such as yaml or csv. Bundlers like Vite or Webpack support json out of the box, but it is not always the easiest to work with. In my team's project, we already had a lot of data in YAML format. It was also slightly easier to write YAML by hand compared to JSON. In this example, we'll use YAML for forms and questions, and CSV for responses.

Install the plugin with:

npm install --save-dev @originjs/vite-plugin-content

Update vite.config.ts to load the plugin:

// ...
import content from '@originjs/vite-plugin-content'

export default defineConfig({
  plugins: [checker({ typescript: true }), content(), react()],
})

Create a src/data/ directory where we'll put all of our .yaml and .csv sample data files. Also create a src/data.ts file that we'll use to load the data and annotate it with proper types.

Let's test the setup by adding a basic form object, with just an id, a name, and a published flag. Create form.ts in the src/ directory, with the following data type declaration:

export type FormId = string

export type Form = {
  id: FormId
  name: string
  published: boolean
}

Now create a few sample data files: data/order-request.yaml, data/work-request.yaml, data/customer-feedback.yaml. They should look like this:

id: aeP73iO5Wz6
name: Order Request
published: true

As a side note, when manually creating new sample data, you often need to generate unique IDs. I recommend making these IDs realistic (i.e. not "123"). An Online UUID Generator is useful if that is the format you are using. In this example, we're using Nano ID with a custom format. After installing it with npm install --save nanoid, I added a script to package.json to make it easier to generate new IDs:

{
  "nanoid": "nanoid --alphabet 0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz --size 11"
}

I can then use the script by running:

npm run nanoid

Back to our sample data files, we can import them directly in data.ts thanks to the Vite plugin, and re-export them with proper type annotations:

import { FormId, Form } from './form'

import customerFeedback from './data/customer-feedback.yaml'
import orderRequest from './data/order-request.yaml'
import timeOffRequest from './data/time-off-request.yaml'

export const formsData: { [key: FormId]: Form } = {
  [customerFeedback.id]: customerFeedback,
  [orderRequest.id]: orderRequest,
  [timeOffRequest.id]: timeOffRequest,
}

If you try it now you should get an error similar to:

Cannot find module './data/customer-feedback.yaml' or its corresponding type declarations.ts(2307)

We can fix that by adding a type declaration for .yaml files. We'll add the declaration, as well as one for .csv files, to the existing vite-env.d.ts file:

// Other file types that TypeScript can't resolve module for

declare module '*.yaml' {
  const data: any
  export default data
}

declare module '*.csv' {
  const data: any
  export default data
}

Let's render the sample data we have so far. Create a new FormList.tsx component:

import { Form } from './form'

function FormList(props: { forms: Array<Form> }) {
  return (
    <>
      {props.forms.map((form) => (
        <p>
          <a href={`#${form.id}`}>{form.name}</a>
        </p>
      ))}
    </>
  )
}

export default FormList

Finally, import and use formsData and FormList inApp.tsx:

// ...
import FormList from './FormList'

import { formsData } from './data'

function App() {
  const forms = Object.values(formsData).sort((a, b) =>
    a.name.localeCompare(b.name)
  )

  return (
    <>
      <Navbar />
      <div className="container margin-top-large">
        <FormList forms={forms} />
      </div>
    </>
  )
}

You should now see a basic list of forms in the browser:

List of forms

Adding routes with React Router

Next, we'll add support for routes and URL handling. You might wonder if this is strictly necessary for a prototype. It depends on what you are prototyping, but I've found that designing routes is an important part of data modeling. It forces you to think about information hierarchy, and the relationship between different data elements, notably when picking parent and child routes. What's more, having shareable and bookmarkable URLs is important for the user experience of any web application, and is often mishandled in Single-Page Apps (SPAs). Finally, URLs can't be easily conveyed in static Figma designs, making the prototype a good place to design them.

We'll use the popular React Router library. It takes some getting used to if you haven't worked with it before, but the following example hopefully covers most use-cases a prototype would need. We'll skip over some details to not repeat React Router's documentation.

As a reminder, the full source code of this example is available on GitHub.

Install the React Router library with:

npm install --save react-router-dom

Add BrowserRouter to main.tsx:

root.render(
  <React.StrictMode>
    <BrowserRouter>
      <App />
    </BrowserRouter>
  </React.StrictMode>
)

Let's start splitting things up into separate React components that can be used for routing. Create a couple of placeholder "page" components:

Forms.tsx (with the contents of what we had so far in App.tsx)
Account.tsx (will give us another page in the top-level navigation)
FormSettings.tsx, FormQuestions.tsx, FormResponses.tsx (will be used as child routes for a specific form)

For now, just add placeholder text in each of these page components. For example:

function FormQuestions() {
  return (
    <>
      <p>Questions</p>
    </>
  )
}

export default FormQuestions

Next, let's add links to these pages in our nav components. Update Navbar.tsx to use <NavLink> instead of <a>:

<li>
  <NavLink to="forms">Forms</NavLink>
</li>
<li>
  <NavLink to="account">Account</NavLink>
</li>

Create a new FormNav.tsx component with links to Settings, Questions, and Responses:

function FormNav() {
  return (
    <>
      <nav className="row flex-center">
        <ul className="inline">
          <li>
            <NavLink to="settings">Settings</NavLink>
          </li>
          <li>
            <NavLink to="questions">Questions</NavLink>
          </li>
          <li>
            <NavLink to="responses">Responses</NavLink>
          </li>
        </ul>
      </nav>
    </>
  )
}

The NavLink component from React Router will add an .active CSS class to links that represent the current route. If the CSS framework you are using doesn't already have a rule for it, make sure to add one to index.css:

/* React Router
-------------------------------------------------------- */

nav a.active {
  color: var(--muted);
  border-bottom-color: var(--muted);
}

Now, let's create a couple of "layout" components that will leverage React Router's Outlet to render child routes with the same layout structure. We'll add these directly to App.tsx since that's the only place they will be used.

Add a component Layout to App.tsx with the same contents we previously had in the App component but replace the contents of <div class="container"> with <Outlet>:

function Layout() {
  return (
    <>
      <Navbar />
      <div className="container margin-top-large margin-bottom-large">
        <Outlet />
      </div>
    </>
  )
}

Also add a FormLayout component to App.tsx using the new FormNav we created:

function FormLayout() {
  return (
    <>
      <FormNav />
      <div className="margin-top-large margin-bottom-large">
        <Outlet />
      </div>
    </>
  )
}

Now that we have all the building blocks, we can create our Routes component hierarchy. We'll use the App component for that:

function App() {
  return (
    <Routes>
      <Route path="/" element={<Layout />}>
        <Route index element={<Navigate to="forms" replace />} />
        <Route path="forms">
          <Route index element={<Forms />} />
          <Route path=":formId" element={<FormLayout />}>
            <Route index element={<Navigate to="settings" replace />} />
            <Route path="settings" element={<FormSettings />} />
            <Route path="questions" element={<FormQuestions />} />
            <Route path="responses" element={<FormResponses />} />
          </Route>
        </Route>
        <Route path="account" element={<Account />} />
      </Route>
    </Routes>
  )
}

A few things to note here:

We're using our "layout" components Layout and FormLayout in the wrapper routes / and /forms/:formId
We use the index attribute to render the forms list for the route /forms
We use the <Navigate to="..." replace /> component to redirect / to /forms, and to redirect /forms/:formId to /forms/:formId/settings

Let's update the remaining <a> elements to <Link> in FormsList.tsx:

<p key={form.id}>
  <Link to={form.id}>{form.name}</Link>
</p>

Note the relative link to={form.id} instead of to={'/forms/' + form.id}, which would potentially allow us to re-use this component elsewhere in the route hierarchy.

For links, as a rule of thumb use <Link> unless you need the .active class automatically added (usually for navbars). In that case, use <NavLink>.

To access URL params, such as the form ID in /forms/formId, we use React Router's useParams() hook. We can do this in a relatively type-safe way by declaring the type Params that we expect and using type casting. Do this in FormNav.tsx to find the form in the data store and display its name:

import { NavLink, useParams } from 'react-router-dom'
import { formsData } from './data'
import { FormId } from './form'

type Params = {
  formId: FormId
}

function FormNav() {
  const params = useParams() as Params
  const form = formsData[params.formId]

  return (
    <>
      <h3 className="row flex-center margin-none">{form.name}</h3>
      <nav className="row flex-center">
        {/* ... */}
      </nav>
    </>
  )
}

You can now run the app and test out the new navigation and routes.

Navigation with React Router

Managing data with Redux

For the prototyping work, Redux works as an in-memory database. It allows us to persist in memory the changes from UI interactions we want to demonstrate. Redux selectors also give us a framework to query data and make it available to React components.

With Redux, we can emulate data persistence without having to introduce a SQL database, fetching data over HTTP, handling loading and failure states, keeping the local in-memory store in sync, etc.

Redux is also popular and has been around for a while. It has developed a set of best practices giving us a well-documented framework to manage data. This means we don't need to reinvent the wheel and can focus on prototyping and the technical design of our feature.

These best practices have been packaged in a library, Redux Toolkit, which we'll be using here. Install it, along with react-redux, by running:

npm install --save @reduxjs/toolkit react-redux

Note: For a good video introduction to Redux Toolkit, see Let’s Learn Modern Redux! (with Mark Erikson).

Before we begin using Redux, let's talk about file organization. You'll see recommendations in the documentation to split up your business logic into different folders, for example src/user/ and src/forms/. However, I recommend keeping everything under the top-level src/ folder for a quick prototype. This way you don't over-think file organization and focus on the important things. And if you really do have a lot of files, maybe what you need is two different prototypes.

Follow the instructions to configure Redux Toolkit for TypeScript. As mentioned above, we won't nest the files under a src/app/ directory. We'll create top-level files, src/store.ts and src/hooks.ts.

Once that is done, follow the steps to provide the store to React in main.ts.

Next, we'll create a new formsSlice.ts by following the instructions to create a slice with TypeScript. We'll set the initial state to be the YAML files that we loaded into formsData earlier. The result should look like this:

import { createSlice } from '@reduxjs/toolkit'
import { RootState } from './store'
import { formsData } from './data'
import { Form, FormId } from './form'

type FormsState = { [key: FormId]: Form }

const initialState: FormsState = formsData

export const formsSlice = createSlice({
  name: 'forms',
  initialState,
  reducers: {},
})

export const formsReducer = formsSlice.reducer

Import formsReducer into store.ts:

// ...
import { formsReducer } from './formsSlice'

export const store = configureStore({
  reducer: {
    forms: formsReducer,
  },
})

Back in formsSlice.ts, let's add a couple of first selectors:

export const selectFormById = (state: RootState, id: FormId): Form => {
  const result = state.forms[id]
  if (!result) {
    throw new Error(`Expected to find form with id ${id}`)
  }
  return result
}

export const selectAllFormsSortByName = (state: RootState): Array<Form> => {
  return Object.values(state.forms).sort((a, b) => a.name.localeCompare(b.name))
}

Note that all selectors accept RootState as the first argument.

Also note that in selectFormById, we throw an error if the lookup returns undefined. This is a prototype, so we want to focus on the happy path and temporarily put aside error-handling. In a production application, we may have the selector return Form | undefined and handle the case of missing data in the UI.

Finally, we'll replace all direct usage of the formsData initial state in React components and use the Redux store instead. Using VS Code, you can open data.ts, right-click onformsData and select "Find All References". Use the useAppSelector hook from src/hooks.ts as well as the selectors we defined in formsSlice.ts.

For example, in Forms.tsx:

// ...
import { selectAllFormsSortByName } from './formsSlice'
import { useAppSelector } from './hooks'

function Forms() {
  const forms = useAppSelector(selectAllFormsSortByName)
  // ...
}

And in FormNav.tsx:

// ...
import { selectFormById } from './formsSlice'
import { useAppSelector } from './hooks'

function FormNav() {
  const params = useParams() as Params
  const form = useAppSelector((state) => selectFormById(state, params.formId))
  // ...
}

If you flip back to the browser, the forms list should still render properly as before. And clicking on a form should still display its name in the top section.

Basic CRUD

In this next part, we'll add some basic CRUD behavior to our prototype. We already have some "Read" functionality from the previous section that displays a list of forms. We'll add "Create" and "Update" for forms, as well as "Update" for the user account. We'll skip "Delete" for this prototype.

For this prototype, the CRUD functionality may not be the most interesting part of the technical design. But it will be an opportunity to get familiar, if you aren't already, with Redux and Redux Toolkit. Other prototyping work might have more sophisticated Add/Remove behavior to showcase, and familiarity with the tools will help get a demo ready faster.

In formsSlice.ts, add a new reducer called formAdded with the following definition. Remember to export it so we can use it later.

import { Form, FormId, generateFormId } from './form'
// ...

export const formsSlice = createSlice({
	// ...
  reducers: {
    formAdded: {
      reducer: (state, action: PayloadAction<{ id: FormId; name: string }>) => {
        const { id, name } = action.payload
        const form: Form = {
          id: id,
          name: name,
          published: false,
        }
        state[id] = form
      },
      prepare: (name: string) => {
        return {
          payload: {
            id: generateFormId(),
            name: name,
          },
        }
      },
    },
  },
})

// ...
export const { formAdded } = formsSlice.actions

There are a couple of things to note here. First, we're adopting the past tense naming convention by using formAdded vs. addForm. This is explained in the Redux style guide's Model Actions as Events, Not Setters section. Although picking good names is arguably less important for prototype code that will be thrown away, adopting a convention means there is one less thing we need to think about.

The second thing to note is that we are using the longer form of the reducer definition, with the reducer and prepare attributes. It allows us to customize the generated action creator. We do so to generate a random ID for the new form in the action creator instead of the reducer, following the rule that reducers must not have side effects. The generateFormId() function here is just a wrapper around nanoid(), see form.ts for more details.

Now let's use this "form added" action. Create a new component FormCreate.tsx:

function FormCreate() {
  // ...

  return (
    <>
      <h3>New form</h3>
      <form onSubmit={handleSubmit}>
        <div className="form-group">
          <label htmlFor="name">Name</label>
          <input
            type="text"
            id="name"
            name="name"
            value={name}
            onChange={handleNameChange}
          />
        </div>
        <input type="submit" className="paper-btn btn-primary" value="Create" />
      </form>
    </>
  )
}

export default FormCreate

We will need to hook it up to the rest of the app. Add a route inApp.tsx, nested under the "forms" route:

<Route path="new" element={<FormCreate />} />

In Forms.tsx, add a create button that navigates to this new route:

<p>
  <Link to="new" className="paper-btn btn-primary">
    Create a new form
  </Link>
</p>
<FormList forms={forms} />

Now back in FormCreate.tsx, let's define the component state and its change handler (name and handleNameChange), as well as the form submit handler (handleSubmit):

const dispatch = useAppDispatch()
const navigate = useNavigate()

const [name, setName] = useState('')
const handleNameChange: React.ChangeEventHandler<HTMLInputElement> = (e) =>
setName(e.target.value)

const handleSubmit: React.FormEventHandler = (e) => {
  e.preventDefault()

  const action = formAdded(name)
  dispatch(action)
  navigate(`/forms/${action.payload.id}`)
}

There are a few things to note here. First, we use the ChangeEventHander and FormEventHandler type annotations on the input change handler and form submit handler. If we didn't, e would have the inferred type any. See also the "Forms and Events" section in this React TypeScript Cheatsheet.

Second, we're dispatching the new form action with the useAppDispatch() hook defined in hooks.ts, and the formAdded action creator defined in formsSlice.ts.

And third, to redirect to the newly created form page, we retrieve the generated form ID from the action returned by formAdded. We use it to construct the route passed to the useNavigate() hook from react-router-dom.

Also notice that there are no validation errors. For example, we could leave the form name empty. This is intended. Remember that this is a prototype, so we want to focus on the happy path.

Create a new form

Redirect to the newly created form

Let's finish this part by adding "Update" functionality to:

For the implementation details, see the source files linked above. It is similar to the FormCreate component, except we don't need to redirect after dispatching the update action.

We also need to add two new action creators:

formSettingsUpdated to formsSlice.ts
accountUpdated to userSlice.ts

The userSlice.ts file is new. To integrate it with the rest of the app, we need to add or update a couple of other files:

Add user.ts with the type definition of a User
Add data/user.yaml with a sample user
Update data.ts to export userData with proper type annotations
Update store.ts to include user in the main reducer

Data modeling

We're getting to the heart of this prototyping exercise. Designing your data model, and how to represent the different concepts related to the feature, is where you'll likely spend most of your time. The reason we went into details in the previous sections was to provide a guide so you could spend less time setting up tools and libraries such as React, Redux, and React Router. But the real value of the prototyping work is in this data modeling phase.

In the case of this article's example, most of the interesting work will be the TypeScript type declarations in forms.ts. Again, this is a simple example so the data modeling will be relatively straightforward. But working this way on more complex problems has proved very useful. The solutions were refined through multiple iterations thanks to this method.

If pressed on time, one could reduce the exercise to only writing TypeScript types and some example data, skipping the prototype UI part with React. Sketching out a data model with a single code snippet is definitely better than having no supporting material to use in meetings.

However, I find that trying to display the data model in the browser helps with the technical design. I've seen APIs send back JSON data that is unnecessarily hard to work with on the frontend (we'll see an example below of missing identifiers to use as React keys for lists). Conversely, the data model sketched out in the browser prototype might not be exactly how we persist and store it in the database. We'll probably use data types and shapes that are more optimized for the backend, with a translation layer in between.

In that sense, it is useful to often flip back and forth between the abstract type definition in form.ts, the sample data in data/, and the React components in .tsx files. Continuously modeling, serializing, and rendering the data will help validate that the data model chosen is easy to work with.

Back to our example, we already have a Form data type. It has an id, a name, and a published flag. We'll add the FormQuestion data type. Each form has a list of questions. Note that the order of questions is relevant, so we use a list (and not a map or a set).

export type Form = {
  id: FormId
  name: string
  published: boolean
  questions: Array<FormQuestion>
}

export type FormQuestion = unknown

There are different types of questions in a form. To represent this, we'll leverage sum types (also known as "tagged unions" or "discriminated unions"). TypeScript doesn't have first-class support for sum types, but you can represent them with a union of object types. All of the object types in the union contain a common field that has a unique and constant value. This allows the compiler to discriminate between each branch. We'll choose to use the tag field for this purpose:

export type FormQuestion =
  | { tag: 'shortText'; question: ShortTextQuestion }
  | { tag: 'longText'; question: LongTextQuestion }
  | { tag: 'singleChoice'; question: SingleChoiceQuestion }
  | { tag: 'multipleChoice'; question: MultipleChoiceQuestion }
  | { tag: 'scale'; question: ScaleQuestion }

export type ShortTextQuestion = unknown
export type LongTextQuestion = unknown
export type SingleChoiceQuestion = unknown
export type MultipleChoiceQuestion = unknown
export type ScaleQuestion = unknown

Note that we could also include the tag attribute directly in the named data type, like so:

export type FormQuestion =
  | ShortTextQuestion
  | LongTextQuestion
  | SingleChoiceQuestion
  | MultipleChoiceQuestion
  | ScaleQuestion

export type ShortTextQuestion = {
  tag: 'shortText'
  // ...
}
export type LongTextQuestion = {
  tag: 'longText'
  // ...
}

// ...

Here we chose instead to define types (ShortTextQuestion, LongTextQuestion, etc.) that can be used independently of the sum type.

Also, keep in mind that each branch of a sum type doesn't have to have the same attributes, as long as it has the tag attribute used to discriminate. For example:

type LoadingState =
  | { tag: 'pending' }
  | { tag: 'failure'; error: AppError }
  | { tag: 'success'; data: Data; timing: number }

Let's now write out the data types corresponding to the different kinds of questions. Each question has a title and an optional description, a flag indicating if answering the question is required or not, as well as some additional attributes that depend on the question type:

export type ShortTextQuestion = {
  title: string
  description: string | null
  required: boolean
}

export type LongTextQuestion = {
  title: string
  description: string | null
  required: boolean
}

export type SingleChoiceQuestion = {
  title: string
  description: string | null
  required: boolean
  choices: Array<{
    id: ChoiceId
    value: string
  }>
}

export type MultipleChoiceQuestion = {
  title: string
  description: string | null
  required: boolean
  choices: Array<{
    id: ChoiceId
    value: string
  }>
}

export type ScaleQuestion = {
  title: string
  description: string | null
  required: boolean
  start: number
  end: number
  startLabel: string
  endLabel: string
}

As you can see, there is a lot of repetition. This can be improved by using generics and introducing a type parameter T. This type parameter will represent the attributes that are specific to each question type. If the question type has no specific attribute, we'll provide null as the concrete type for T.

We define a generic Question<T>, as well as concrete types for each kind of question (ShortText, LongText, SingleChoice, MultipleChoice, and Scale):

export type Question<T> = {
  title: string
  description: string | null
  required: boolean
  definition: T
}

export type ShortText = null

export type LongText = null

export type SingleChoice = Array<string>

export type MultipleChoice = Array<string>

export type Scale = {
  start: number
  end: number
  startLabel: string
  endLabel: string
}

Finally, let's update the sum type FormQuestion to use the new generic Question<T>:

export type FormQuestion =
  | { tag: 'shortText'; question: Question<ShortText> }
  | { tag: 'longText'; question: Question<LongText> }
  | { tag: 'singleChoice'; question: Question<SingleChoice> }
  | { tag: 'multipleChoice'; question: Question<MultipleChoice> }
  | { tag: 'scale'; question: Question<Scale> }

The next step is to try this data model by creating some sample data. We'll update the .yaml files under data/ and use the shape of the data defined above. In a real-world setting, we'll want to use realistic data as much as possible. For instance, we could've asked potential customers to share some real forms and surveys and see how those fit in our data model.

Bellow, we added some questions to data/order-request.yaml:


id: aeP73iO5Wz6
name: Order Request
published: true
questions:
  - tag: singleChoice
    question:
      title: Are you a new or existing customer?
      description:
      required: false
      definition:
        - I am a new customer
        - I am an existing customer
  - tag: shortText
    question:
      title: What is the item you would like to order?
      description: Please enter the product number
      required: true
      definition:
  - tag: multipleChoice
    question:
      title: What color(s) would you like to order?
      description:
      required: false
      definition:
        - Red
        - Blue
        - Green
        - Yellow
  # ...

The last step in the "model, serialize, render" loop is to take this sample data and try to render it on screen.

We'll update the FormQuestions React component and render the list of questions for a form. Not everything will be shown here, so refer to the full source in FormQuestions.tsx for more details.

Use React Router's useParams() hook and the selectFormById selector we created earlier to get the correct form data based on the URL. Then .map() over the list of questions to render them:

import { selectFormById } from './formsSlice'
import { useAppSelector } from './hooks'

type Params = {
  formId: FormId
}

function FormQuestions() {
  const params = useParams() as Params
  const form = useAppSelector((state) => selectFormById(state, params.formId))

  return (
    <>
      {form.questions.map((formQuestion) => (
        <QuestionCard formQuestion={formQuestion} />
      ))}
    </>
  )
}

In the QuestionCard component, we'll leverage TypeScript's exhaustiveness checking to make sure we render every question type. For this, we use a switch statement with a special default branch that assigns any case that we missed to the never type . Let's use this pattern in renderQuestion:

function QuestionCard(props: { formQuestion: FormQuestion }) {
  const { formQuestion } = props

  const renderQuestion = (): JSX.Element => {
    switch (formQuestion.tag) {
      case 'shortText':
        return <QuestionShortText question={formQuestion.question} />
      case 'longText':
        return <QuestionLongText question={formQuestion.question} />
      case 'singleChoice':
        return <QuestionSingleChoice question={formQuestion.question} />
      case 'multipleChoice':
        return <QuestionMultipleChoice question={formQuestion.question} />
      case 'scale':
        return <QuestionScale question={formQuestion.question} />
      default:
        const _exhaustiveCheck: never = formQuestion
        return _exhaustiveCheck
    }
  }

  return (
    <div className="card margin-bottom">
      <div className="card-body">
        <h5 className="card-subtitle">{showQuestionType(formQuestion.tag)}</h5>
        {renderQuestion()}
      </div>
    </div>
  )
}

Try commenting out the 'scale' case, and you'll see that the compiler raises a type error. The use of never allows us to see which cases we have missed.

The compiler telling us that we missed a case of our sum type in this switch statement

To render each question type, we extract common pieces into a re-usable QuestionMetadata<T> component:

function QuestionMetadata<T>(props: { question: Question<T> }) {
  const { question } = props

  const requiredElement = question.required ? (
    <span className="text-danger"> *</span>
  ) : null

  const descriptionElement =
    question.description && question.description !== '' ? (
      <p className="text-small text-muted">{question.description}</p>
    ) : null

  return (
    <>
      <p>
        {question.title}
        {requiredElement}
      </p>
      {descriptionElement}
    </>
  )
}

Now, each question component can focus on rendering what is specific to that question type:

function QuestionShortText(props: { question: Question<ShortText> }) {
  return (
    <>
      <QuestionMetadata question={props.question} />
      <div className="form-group">
        <input className="input-block" type="text" readOnly={true} />
      </div>
    </>
  )
}

// ...

function QuestionSingleChoice(props: { question: Question<SingleChoice> }) {
  return (
    <>
      <QuestionMetadata question={props.question} />
      <fieldset className="form-group">
        {props.question.definition.map((choice) => {
          return (
            <label className="paper-radio">
              <input type="radio" value={choice} disabled={true} />{' '}
              <span className="inline-block">{choice}</span>
            </label>
          )
        })}
      </fieldset>
    </>
  )
}

// ...

When all question types have been implemented as React components, start the development server and view the result in the browser.

It is good practice to periodically check the browser's console for any errors or warnings, notably those produced by React. If you check now, you should see a warning about a missing "key" prop for list items.

React warning about a missing "key" prop for list items

We could try to quickly fix this by using as a key the index of the item in the list. But that is not recommended. If the items in the list are reordered or removed, the indices will change. We need a more stable identifier.

This is an example of a limitation in our data model that we would probably not have noticed if we had not gone through this prototyping exercise and tried to render some sample data on the screen. I've seen this happen in a real setting. The backend did not produce unique IDs for items in a list, and the frontend had to go through all kinds of workarounds to implement the add/remove/edit functionality for that list.

Let's introduce unique IDs for both questions and choices in form.ts:

export type QuestionId = string
export type ChoiceId = string

// ...

export type Question<T> = {
  id: QuestionId
  title: string
  description: string | null
  required: boolean
  definition: T
}

// ...

export type Choice = {
  id: ChoiceId
  value: string
}

Note: Here we are using simple type aliases for QuestionId and ChoiceId. In the production version of the feature, we may want to consider newtypes if the programming language supports them. They will prevent us from mistakenly passing a ChoiceId to a function that expects a QuestionId. TypeScript does have a way to emulate newtypes, with libraries that implement it, but it seemed a little too complex to introduce for a prototype.

We now need to update our sample data to include some random IDs (generated using npm run nanoid):

id: aeP73iO5Wz6
name: Order Request
published: true
questions:
  - tag: singleChoice
    question:
      id: 6funH6yhl1w
      title: Are you a new or existing customer?
      description:
      required: false
      definition:
        - id: 26LQFQmxRk5
          value: I am a new customer
        - id: NqJ5OnW2f7h
          value: I am an existing customer
  # ...

Finally, use question.id and choice.id as a key in the rendering functions of FormQuestions.tsx:

export function FormQuestionsList() {
  // ...

  return (
    <>
      {form.questions.map((formQuestion) => (
        <QuestionCard
          key={formQuestion.question.id}
          formQuestion={formQuestion}
        />
      ))}
    </>
  )
}

// ...

function QuestionSingleChoice(props: { question: Question<SingleChoice> }) {
  return (
    <>
      <QuestionMetadata question={props.question} />
      <fieldset className="form-group">
        {props.question.definition.map((choice) => {
          return (
            <label key={choice.id} className="paper-radio">
              <input type="radio" value={choice.value} disabled={true} />{' '}
              <span className="inline-block">{choice.value}</span>
            </label>
          )
        })}
      </fieldset>
    </>
  )
}

// ...

Again, see form.ts for the full data model, data/ for all of the sample data, and FormQuestions.tsx for the complete rendering logic.

Using HTML forms

In the previous section, we iterated and converged on a data model for questions that we are happy with. Rendering sample questions as low-fidelity React components helped us refine that data model. It also provides an illustration we can use when discussing the feature with other developers and non-developers, such as UX designers or Product Managers.

Remember that this is a prototype. We don't mean to implement the full functionality of the feature. The point we have reached now wouldn't be a bad place to stop. However, we'll push the exercise further and implement more CRUD behavior for questions. This will allow us to cover additional techniques that could be useful in your prototyping work.

Previously, the research was focused on how to represent data. In this section, we'll use the prototype to explore and showcase different application behaviors, notably how to add or edit a question.

From a technical perspective, it will also be an opportunity to learn or review how to handle HTML forms and their state with React. With controlled inputs and state hooks, HTML forms in React can involve some boilerplate. Being familiar and comfortable with these concepts can help you move faster, which is important for prototyping.

In this article, we will stick with "vanilla" React forms to keep it simple. But if forms are central to your prototype and you require many of them, introducing a library such as Formik or React Hook Form could be worth it.

Let's start with the functionality to create a new question. In FormQuestions.tsx, add a new component QuestionCreate:

function QuestionCreate(props: { formId: FormId }) {
  const { formId } = props
  const dispatch = useAppDispatch()
  const initialFormQuestion: FormQuestion = newFormQuestion('shortText')

  const [showQuestionCreate, setShowQuestionCreate] = useState(false)
  const [questionType, setQuestionType] = useState<QuestionType>(
    initialFormQuestion.tag
  )
  const [formQuestion, setFormQuestion] =
    useState<FormQuestion>(initialFormQuestion)

  // ...

  return (
    <div className="card margin-bottom">
      <div className="card-body">
        <h5 className="card-subtitle">New question</h5>
        <form onSubmit={handleSubmit}>
          <div className="form-group">
            <label htmlFor="questionType">Question type</label>
            <select
              id="questionType"
              value={questionType}
              onChange={handleQuestionTypeChanged}
            >
              {allQuestionTypes.map((type) => (
                <option key={type} value={type}>
                  {showQuestionType(type)}
                </option>
              ))}
            </select>
          </div>
          {renderQuestion()}
          <div className="row flex-right">
            <input
              type="button"
              className="paper-btn margin-right"
              onClick={handleCancel}
              value="Cancel"
            />
            <input
              type="submit"
              className="paper-btn btn-primary"
              value="Add"
            />
          </div>
        </form>
      </div>
    </div>
  )
}

The newFormQuestion: (questionType: QuestionType) => FormQuestion helper returns an empty question to use as the form's initial values.

There are three calls to the useState() React hook in this component. With the first piece of state showQuestionCreate: boolean, we want to showcase the behavior where the form toggles on and off by clicking "Add question" or "Cancel". Let's add the necessary handler and early return statement for this:

function QuestionCreate(props: { formId: FormId }) {
  // ...
  const resetForm = () => {
    setQuestionType(initialFormQuestion.tag)
    setFormQuestion(initialFormQuestion)
  }

  const handleCancel: React.MouseEventHandler = (e) => {
    e.preventDefault()
    setShowQuestionCreate(false)
    resetForm()
  }

  if (!showQuestionCreate) {
    return (
      <p className="row flex-right">
        <button
          className="btn-primary"
          onClick={() => setShowQuestionCreate(true)}
        >
          Add question
        </button>
      </p>
    )
  }
  // ...
}

With the second piece of state, questionType: QuestionType, we want to showcase the behavior where selecting a different question type displays different inputs depending on the question's specific attributes. Let's add the necessary event handler and a render function that will switch on the question type:

function QuestionCreate(props: { formId: FormId }) {
  // ...
  const handleQuestionTypeChanged: React.ChangeEventHandler<
    HTMLSelectElement
  > = (e) => {
    const type = e.target.value as QuestionType
    setQuestionType(type)
    setFormQuestion(newFormQuestion(type))
  }

  const renderQuestion = (): JSX.Element => {
    switch (formQuestion.tag) {
      case 'shortText':
        return (
          <QuestionShortTextEdit
            question={formQuestion.question}
            onQuestionChange={(question) =>
              setFormQuestion({ tag: 'shortText', question: question })
            }
          />
        )
      case 'longText':
        return (
          <QuestionLongTextEdit
            question={formQuestion.question}
            onQuestionChange={(question) =>
              setFormQuestion({ tag: 'longText', question: question })
            }
          />
        )
      // ...
      default:
        const _exhaustiveCheck: never = formQuestion
        return _exhaustiveCheck
    }
  }
  // ...
}

Creating a new question of type Short Text

Creating a new question of type Single Choice

The components suffixed with Edit also have their own state, which they bubble back up thanks to the onQuestionChange prop. We also extract QuestionMetadataEdit to re-use across question types for common attributes.

function QuestionShortTextEdit(props: {
  question: Question<ShortText>
  onQuestionChange: (question: Question<ShortText>) => void
}) {
  const { question, onQuestionChange } = props

  return (
    <>
      <QuestionMetadataEdit
        question={question}
        onQuestionChange={onQuestionChange}
      />
    </>
  )
}

function QuestionMetadataEdit<T>(props: {
  question: Question<T>
  onQuestionChange: (question: Question<T>) => void
}) {
  const { question, onQuestionChange } = props

  const handleTitleChange: React.ChangeEventHandler<HTMLInputElement> = (e) =>
    onQuestionChange(
      produce(question, (draft) => {
        draft.title = e.target.value
      })
    )

  // ...

  return (
    <>
      <div className="form-group">
        <label htmlFor="title">Title</label>
        <input
          type="text"
          id="title"
          name="title"
          className="input-block"
          value={question.title}
          onChange={handleTitleChange}
        />
      </div>
      {/* ... */}
    </>
  )
}

The main thing to call out here is that we use the produce function from Immer to update the React component state with immutable data structures. Immer should already be installed with Redux Toolkit.

The last piece of state of QuestionCreate is formQuestion: FormQuestion. It gets updated by our question-specific components suffixed with Edit. We then use this state in the submit handler to dispatch a new questionAdded Redux action:

import { questionAdded } from './formsSlice'
// ...

function QuestionCreate(props: { formId: FormId }) {
  // ...
  const handleSubmit: React.FormEventHandler = (e) => {
    e.preventDefault()
    setShowQuestionCreate(false)
    resetForm()
    dispatch(questionAdded(formId, formQuestion))
  }
  // ...
}

The questionAdded() action creator in formsSlice.ts follows a similar pattern to the one that adds a new form. We keep side effects out of the reducer and use the prepare attribute, in this case to generate a new random question ID:

import { generateQuestionId } from './form'
// ...

export const formsSlice = createSlice({
  name: 'forms',
  initialState,
  reducers: {
    // ...
    questionAdded: {
      reducer: (
        state,
        action: PayloadAction<{ formId: FormId; formQuestion: FormQuestion }>
      ) => {
        const { formId, formQuestion } = action.payload
        state[formId].questions.push(formQuestion)
      },
      prepare: (formId: FormId, formQuestion: FormQuestion) => {
        return {
          payload: {
            formId: formId,
            formQuestion: produce(formQuestion, (draft) => {
              draft.question.id = generateQuestionId()
            }),
          },
        }
      },
    },
  },
})

Editing an existing question is similar to adding a new question. We also re-use the components suffixed with Edit. We won't cover the implementation here in detail, see FormQuestions.tsx for the full source code.

With the edit functionality, we can showcase the behavior of toggling between a "view" and "edit" version of the question, by clicking the "Edit" and "Cancel" buttons respectively.

Prototyping these forms also allows us to demonstrate interesting behavior such as adding and removing choices for single and multiple-choice questions.

While these types of UI interactions could be represented as Figma mockups, it is valuable to have a live prototype that you can interact with. I've used this in the past to suggest alternative and simpler implementations of certain features to UX designers.

Toggling a question between "view" mode...

...and "edit" mode

Generating random data

As stated earlier in the article, prefer using real data when available. You may need some post-processing to get it in the right shape, but the overhead is worth it. If real data is not available, then we have the fallback option of generating random data. This is what we'll use to create sample data for responses in our online form creator example.

With the form and questions sample data, we were focused on data modeling and using sum types to represent different kinds of questions. With the responses sample data, we will focus on the quantity of data. This is useful to prototype things such as pagination, how to handle partial or missing data, and other edge cases. With code, handling ten items or hundreds of items is the same amount of work. This is where prototyping in the browser has a clear advantage over static mockups such as Figma.

We'll use the Faker library to create random data. It includes a lot of common generators such as name, email, phone number, Lorem ipsum, etc. You can also easily create your own generators with functions such as number() and arrayElements(). For instance, we'll use these to create generators for single choice and multiple choice responses.

Create a new file data/generate-responses.ts. We'll run it as a script to produce a .csv file. This file will be checked-in the repository, and then loaded using the vite-content-plugin installed earlier. There are a couple of reasons we do this instead of generating random data at runtime. First, we want to mimic using real data, in which case we would've probably gotten our hands on an export file such as a CSV containing response data for some existing forms. Second, at the time of writing Faker doesn't take a seed as input, so the data will be re-generated to a different set every time we make a change to our prototype, which would be distracting. Finally, Faker is a large library so this allows us to keep the JavaScript bundle size smaller and the development feedback cycle slightly quicker.

The structure of our data/generate-responses.ts script looks like this:

import * as fs from 'fs'
import { faker } from '@faker-js/faker'
import { stringify } from 'csv-stringify/sync'

type ResponseDefinition = {
  question: string
  response: ResponseGenerator
}

type ResponseGenerator = () => string

const generateFormResponses = (
  definitions: Array<ResponseDefinition>,
  count: number
): Array<Array<string>> => {
  // ...
}

const orderRequestResponses: Array<ResponseDefinition> = [
  // ...
]

// response definitions for other forms...

const run = () => {
  fs.writeFileSync(
    './src/data/order-request-responses.csv',
    stringify(generateFormResponses(orderRequestResponses, 154))
  )

  // generate responses for other forms...
}

run()

We use csv-stringify and Node's fs module to serialize and write the data to a file. Since this is a one-off script and performance is not a concern, we can use the sync APIs of both.

Notice that we chose to generate 154 responses for the "Order Request" form. That number is completely arbitrary. We can easily change it to 1,000 or 10,000, and it would not take more code to render it on screen. Unless, of course, it makes us realize that we need to add pagination of some kind.

The random response definitions make use of Faker's generators:

const orderRequestResponses: Array<ResponseDefinition> = [
  {
    question: 'Are you a new or existing customer?',
    response: optional(
      singleChoice(['I am a new customer', 'I am an existing customer'])
    ),
  },
  // ...
  {
    question: 'What color(s) would you like to order?',
    response: optional(multipleChoices(['Red', 'Blue', 'Green', 'Yellow'])),
  },
  // ...
  { question: 'Your name', response: faker.name.findName },
  {
    question: 'Phone number',
    response: () => faker.phone.phoneNumber('###-###-####'),
  },
  { question: 'Email', response: optional(faker.internet.email) },
  {
    question: 'Preferred contact method',
    response: multipleChoices(['Phone', 'Email']),
  },
  {
    question: 'How satisfied are you with our service?',
    response: () => faker.datatype.number({ min: 1, max: 5 }) + '',
  },
  {
    question: 'Questions and comments',
    response: optional(() =>
      faker.lorem.sentences(faker.datatype.number({ min: 3, max: 10 }))
    ),
  },
]

We also create a few helper functions:

const optional = (response: ResponseGenerator): ResponseGenerator => {
  return () => (faker.datatype.boolean() ? response() : '')
}

const singleChoice = (choices: Array<string>): ResponseGenerator => {
  return () => faker.helpers.arrayElement(choices)
}

const multipleChoices = (choices: Array<string>): ResponseGenerator => {
  return () =>
    faker.helpers
      .arrayElements(
        choices,
        faker.datatype.number({ min: 1, max: choices.length })
      )
      .join(', ')
}

See data/generate-responses.ts for the full source.

Note that to be more realistic, some responses should depend on previous responses. For example, the email likely contains a variation of the name. But, in the interest of simplicity and since this is fake data for a prototype, we'll assume that every response is independent. This is another reason it is better to use real data as much as possible. It will contain such patterns and we can make sure our technical design handles them.

We'll run the script without any build step thanks to ts-node. Add an entry to the scripts section in package.json:

"generate-responses": "ts-node --compilerOptions '{\"module\":\"commonjs\"}' src/data/generate-responses.ts"

We can now re-generate the data using:

npm run generate-responses

Next, import the files in data.ts, and re-export them in a map const responsesData: { [key: FormId]: Array<Array<string>> }.

Create a new Redux slice responsesSlice.ts. This one will be "read-only" and not contain any reducers, since we're focused on displaying responses and not creating them.

import { responsesData } from './data'
// ...

type ResponsesState = { [key: FormId]: FormResponses }

export type FormResponses = {
  header: Array<string>
  responses: Array<Array<string>>
}

const initialState: ResponsesState = Object.fromEntries(
  Object.entries(responsesData).map(([key, rows]) => [
    key,
    {
      // First row contains header
      header: rows.length > 0 ? rows[0] : [],
      responses: rows.slice(1),
    },
  ])
)

export const reponsesSlice = createSlice({
  name: 'responses',
  initialState,
  reducers: {},
})

// ...

See the full responsesSlice.ts file for more details. Remember to hook up this new reducer to store.ts.

Next, update the placeholder FormResponses.tsx component to contain a simple HTML table with all of the responses:

// ...
import { selectFormResponses } from './responsesSlice'

type Params = {
  formId: FormId
}

function FormResponses() {
  const params = useParams() as Params
  const { header, responses } = useAppSelector((state) =>
    selectFormResponses(state, params.formId)
  )

  return (
    <>
      <p>
        Responses: <strong>{responses.length}</strong>
      </p>
      <div className="table-container-scroll">
        <table className="table-scroll">
          <thead>
            <tr>
              {header.map((name, index) => (
                <th key={index} title={name}>
                  {index === 0 ? '#' : name}
                </th>
              ))}
            </tr>
          </thead>
          <tbody>
            {responses.map((row, index) => (
              <tr key={index}>
                {row.map((response, index) => (
                  <td title={response} key={index}>
                    {response}
                  </td>
                ))}
              </tr>
            ))}
          </tbody>
        </table>
      </div>
    </>
  )
}

export default FormResponses

Start the development server, navigate to a form, and you should now see the responses displayed on the screen:

Working with realistic data to spot the limits of a design

The first thing we notice is that the horizontal space is insufficient for the responses. We must remember that a form can have any number of questions. In our sample data, it is important to have forms with more than 2 or 3 questions to notice this. I've seen static mockups too often use examples that fit nicely on the screen. But as soon as we implement them with real-world data, we need to revise the design.

In this case, we'll add basic horizontal scrolling to handle the variable number and length of responses. There might be another solution with a nicer user experience, but that is not the objective of the prototype. The horizontal scrolling will be enough to point out this concern to our UX team, when we review this research work together.

In index.css, add the following rules:

/* Tables
-------------------------------------------------------- */

.table-container-scroll {
  overflow-x: scroll;
}

.table-scroll th, .table-scroll td {
  max-width: 200px;
  overflow: hidden;
  white-space: nowrap;
  text-overflow: ellipsis;
}

Save the file, and flip back to the browser. You should see an improved version of the responses table:

The responses table with additional CSS

Notice that every table cell is set to a max width of 200px, and any overflowing content will be truncated with an ellipsis. In the React component above, we use the title HTML attribute to add a simple hover affordance to view the whole content when truncated.

A first enhancement could be to set the max width based on the question type. For instance, LongText would typically be given more room than ShortText.

A second enhancement could be to allow the user to click on a single row. This would open a modal with the response details displayed in a vertical layout.

These UX improvements are all probably out of scope for this prototyping work. But working in the browser and with realistic data has allowed us to notice these concerns. Our prototype can then serve as a basis for discussions and ideas on how to address them.

Charts and data visualization

The table view above gives the user access to raw response data. But with many responses, having some kind of visualization could be useful, especially for quantitative questions. Let's add some charts to explore visualizing response data. This could even be a standalone prototype.

Designing a data visualization screen is another task where leveraging code and real data has clear advantages compared to only using static mockups such as Figma. I've seen many cases where a chart looks great on a static design with clean, made-up data. But as soon as it is implemented in the application and populated with real data, the chart looks empty and underwhelming.

For example, I recently saw a beautiful Figma dashboard with a sunburst chart as the centerpiece. But once it was implemented, we realized the sunburst chart provided very little value. As another example, in my early work with Tidepool's diabetes data visualization app, I put together a quick prototype using D3.js and a real data set. This allowed me to find that bars for insulin boluses and hovering circles for carbs were a simple visualization that worked well with data that could be both scattered out in time or bunched up close together.

For this article, we'll use a chart library that includes common chart components out of the box (bar chart, line chart, etc.). To stick with the style of PaperCSS and to emphasize that this is a sketch and not a high-fidelity design, we'll use hand-drawn style charts. I ended up picking chart.xkcd (GitHub) for this, which mimics the style of charts found in the famous webcomic xkcd. Another one I tried was roughViz.js, which uses the same Rough.js library that Excalidraw uses. But I found the rendering not as nice as chart.xkcd.

If you need more fancy charts you could look at something like nivo, or even drop down to using D3.js. However, remember that this is only a prototype. Reaching for those libraries with more features might only make sense if the sole focus of the prototype is exploring data visualization.

As I mentioned earlier, most of the value of prototyping data visualizations comes from using real data. Unfortunately for this example of online forms and their responses, I don't have access to real data. So I will cheat a little and hand-craft some sample data. I'll try to create edge cases, such as a pie chart with only one category containing data, because this is something the real data would teach us. We also won't compute the statistics feeding the charts from the raw response data generated earlier. Since it was randomly generated, with a large number of responses we would see an even distribution of selected choices, which is probably not representative of what real data would look like.

Part of this prototyping work is also deciding how this feature will be implemented, and creating a technical design for it. For example, we'll say that the statistics over the aggregate response data will be computed on the backend, possibly cached, and sent back to the frontend via a JSON HTTP API.

Let's work on the shape of that data. Create a new file statistics.ts, with the contents:

import { QuestionId } from './form'

export type QuestionStatistics =
  | { tag: 'bar'; statistics: Statistics<Bar> }
  | { tag: 'pie'; statistics: Statistics<Pie> }
  | { tag: 'scale'; statistics: Statistics<Scale> }

export type Statistics<T> = {
  questionId: QuestionId
  title: string
  data: T
}

export type Bar = Array<{ label: string; value: number }>
export type Pie = Array<{ label: string; value: number }>
export type Scale = {
  startLabel: string
  endLabel: string
  values: Array<{ label: string; value: number }>
}

Notice that we're using sum types again to represent different kinds of charts in QuestionStatistics, similarly to FormQuestion earlier. We use the tag attribute for the type of chart, and a type parameter T for the shape of a chart's data.

Also notice that except for the "scale" chart, we don't differentiate between question types. For example, two different question types can both use the bar chart. This will also make it relatively easier to switch a question to another type of chart.

We'll now create a file for each form, for example data/order-request-statistics.ts, to generate the sample response statistics data. Again, in an actual prototype, we'll prefer to use real data.

In data.ts, import the data files and re-export them with a type annotation. This shape Array<QuestionStatistics> is what our JSON API would likely return for a particular form's data.

import { QuestionStatistics } from './statistics'
import orderRequestStatistics from './data/order-request-statistics'
// ...

export const statisticsData: { [key: FormId]: Array<QuestionStatistics> } = {
  [orderRequest.id]: orderRequestStatistics,
  // ...
}

Add the statistics data to responsesSlice.ts:

export type FormResponses = {
  header: Array<string>
  responses: Array<Array<string>>
  statistics: Array<QuestionStatistics>
}

const initialState: ResponsesState = Object.fromEntries(
  Object.entries(responsesData).map(([key, rows]) => [
    key,
    {
      // First row contains header
      header: rows.length > 0 ? rows[0] : [],
      responses: rows.slice(1),
      statistics: statisticsData[key],
    },
  ])
)

Before rendering this data, we'll create a small wrapper around chart.xkcd to use it in React components. See chart.tsx for the implementation. The library doesn't have TypeScript type definitions, so we'll also need to add declare module 'chart.xkcd' to vite-env.d.ts to avoid getting a type error.

In FormResponses.tsx, replace the existing FormResponses component with three components:

FormResponsesLayout uses React Router's Outlet to render the table or the statistics charts
FormResponsesNav allows the user to navigate between the table and the statistics
FormResponsesTable contains the response table implemented earlier

Add a new component FormResponsesStatistics that will render the charts:

import chartXkcd from 'chart.xkcd'
import { BarChart, PieChart } from './chart'
// ...

export function FormResponsesStatistics() {
  const params = useParams() as Params
  const { statistics } = useAppSelector((state) =>
    selectFormResponses(state, params.formId)
  )

  const renderQuestionStatistics = (item: QuestionStatistics): JSX.Element => {
    switch (item.tag) {
      case 'bar':
        return (
          <BarChart
            config={{
              title: item.statistics.title,
              data: {
                labels: item.statistics.data.map((x) => x.label),
                datasets: [
                  {
                    data: item.statistics.data.map((x) => x.value),
                  },
                ],
              },
              options: {
                yTickCount: 4,
              },
            }}
          />
        )
      // ...
      default:
        const _exhaustiveCheck: never = item
        return _exhaustiveCheck
    }
  }

  return (
    <>
      {statistics.map((item) => (
        <div key={item.statistics.questionId} className="chart margin-bottom">
          {renderQuestionStatistics(item)}
        </div>
      ))}
    </>
  )
}

We make use of exhaustiveness checking for our sum type in renderQuestionStatistics, as we did for rendering different question types earlier.

See FormResponses.tsx for the full source.

Finally, update App.tsx with these new routes:

<Route path="responses" element={<FormResponsesLayout />}>
  <Route index element={<Navigate to="table" replace />} />
  <Route path="table" element={<FormResponsesTable />} />
  <Route path="statistics" element={<FormResponsesStatistics />} />
</Route>

Start the development server and navigate to a form's response statistics to see the result.

Visualizing statistics of a form's responses

In this prototyping work, we notice that statistics work well for quantitative questions (scale), or questions with a finite set of possible answers (single or multiple choice). For qualitative questions (short or long text), we decided to display some statistics on the length of the answer (number of words), grouping them into buckets. We used a pie chart, but perhaps a histogram would have been more appropriate. We could've also left out statistics for qualitative questions altogether, as the analysis of the raw answers is more valuable. These are all good discussion points that the prototyping work helps discover.

Data export

The last feature we'll add to our prototype is the ability to serialize and preview the sample data directly in the UI.

We'll serialize the form questions as YAML, and add a screen to show the output in a code block. We'll also serialize the form responses as CSV. The output for those will look like the files we have checked in the data/ directory.

This isn't necessarily a feature that we want to add to the real product, but I found it useful in the prototype for a couple of reasons.

First, if we create sample data for questions using the CRUD functionality of the prototype, we can easily copy-paste the result from the YAML screen into a file in the data/ directory. This can speed up the sample data creation workflow.

Second, we can use the feature to quickly show developers what the data behind the UI looks like, without having to switch to a code editor or the browser developer tools. Even for non-developers, such as the UX team, I've found it useful to show a preview of what the data looks like "under the hood". This helps build a common understanding between developers and the design team.

To serialize the questions as YAML, we'll use js-yaml, which is the same library used by vite-plugin-content.

At the time of writing, this library doesn't have TypeScript type definitions. In the interest of speed, we'll declare the module as any. To do so, add the line declare module 'js-yaml' in vite-env.d.ts.

To serialize the responses as YAML, we'll use csv-stringify, the same library we used in our Node script to generate the random data files. We'll just need to make sure to import the browser version, which contains polyfills for Node dependencies such as streams (see the Browser ESM section in the library documentation).

In FormQuestions.tsx, similarly to what we did when we added the response statistics, split the FormQuestions component into:

FormQuestionsLayout uses React Router's Outlet to render the list of questions or the YAML data export
FormQuestionsNav allows the user to navigate between the list and the YAML
FormQuestionsList contains the questions list implemented earlier

Next, add a new FormQuestionsYaml component to display the serialized questions:

import yaml from 'js-yaml'
import Code from './Code'
// ...

export function FormQuestionsYaml() {
  const params = useParams() as Params
  const form = useAppSelector((state) => selectFormById(state, params.formId))

  return <Code language="yaml" code={yaml.dump(form.questions)} />
}

See FormQuestions.tsx for the full source.

In FormResponses.tsx, we already have a subnav component FormResponsesNav. Add a link to it that navigates to the CSV screen.

Next, add a new FormResponsesCsv component:

import { stringify } from 'csv-stringify/browser/esm/sync'
import Code from './Code'
// ...

export function FormResponsesCsv() {
  const params = useParams() as Params
  const { header, responses } = useAppSelector((state) =>
    selectFormResponses(state, params.formId)
  )

  return <Code language="csv" code={stringify([header].concat(responses))} />
}

Update the routes in App.tsx:

<Route path="questions" element={<FormQuestionsLayout />}>
  <Route index element={<Navigate to="list" replace />} />
  <Route path="list" element={<FormQuestionsList />} />
  <Route path="yaml" element={<FormQuestionsYaml />} />
</Route>
<Route path="responses" element={<FormResponsesLayout />}>
  <Route index element={<Navigate to="table" replace />} />
  <Route path="table" element={<FormResponsesTable />} />
  <Route path="statistics" element={<FormResponsesStatistics />} />
  <Route path="csv" element={<FormResponsesCsv />} />
</Route>

Finally, we'll add some syntax highlighting for the serialized output using Prism.

Create a new Code.tsx component, which we'll use to wrap Prism in React (roughly following this tutorial). We'll make use of the useEffect() hook, as well as Prism.highlightAll():

import { useEffect } from 'react'
import Prism from 'prismjs'
import 'prismjs/components/prism-yaml'
import 'prismjs/components/prism-csv'

export default function Code(props: { code: string; language: string }) {
  const { code, language } = props

  useEffect(() => {
    Prism.highlightAll()
  }, [])

  return (
    <pre>
      <code className={`language-${language}`}>{code}</code>
    </pre>
  )
}

Notice that we need to import the Prism language definitions that we need (in this case YAML and CSV). Doing it this way is fine for a prototype. For a production application, we'll probably want to use something like babel-plugin-prismjs (see the Prism documentation).

We'll also need to add declare module 'prismjs/components/*' to vite-env.d.ts.

In main.tsx, import the CSS file for the default Prism theme (or prism-<theme>.css, replacing <theme> with one of those provided by Prism):

import 'prismjs/themes/prism.css'

We'll also adjust the font size by adding some small overrides to the theme in index.css:

/* Code blocks
-------------------------------------------------------- */

code[class*="language-"], pre[class*="language-"] {
  font-size: 16px;
}

Start the development server, and navigate to a form's questions. Then select the YAML tab to see the serialized data:

Serialize form questions to YAML

Similarly, on the form's responses page, select the CSV tab to see the serialized output:

Serialize form responses to CSV

Bonus: Deploying to Netlify

The actual prototype my team used for the technical design of a feature was only run locally and screen shared over Google Meet. So there was no need to host it anywhere.

But you may want to give coworkers access to the prototype, so they can look at it outside of a meeting. In that case, you need to make it available at a URL they can reach. Also, the source code containing the data model is probably as valuable as the UI. So make sure to share the code as well, using a private GitHub repository for example.

One quick way to share the single-page app would be to run it on your machine and expose it to your local network. Vite has that functionality built-in, using the preview command with the --host option:

npm run build
npm run preview -- --host

Another option, which I used for the example in this article, would be to host it remotely using a service such as Netlify. In that case, keep in mind that the prototype UI will be public to anyone that knows the URL. However, Netlify by default generates random URLs that are hard to guess. This semi-privacy might be enough for your use case. Paid plans also offer site-wide password protection, so that is an option for prototypes that need to be truly private.

Deploying a static site to Netlify is straightforward and well-documented. Here, I'll highlight the specific steps I took for this example.

First, add a public/robots.txt file that tells search engine crawlers not to index the site if ever they come across it:

User-agent: *
Disallow: /

Note: Anything in the public/ directory is picked up by Vite and copied over to the build output.

Next, add a netlify.toml configuration file at the root of the project directory:

[build]
  publish = "dist"
  command = "npm run build"

You don't strictly need to do this. Netlify is good at automatically detecting that a site is using Vite as the build tool. But I prefer to be explicit.

We'll also use the netlify.toml configuration file to add a redirect rule. It will make refreshing the page work when the URL contains a path other than the root path (for example /forms/aeP73iO5Wz6):

[[redirects]]
  from = "/*"
  to = "/index.html"
  status = 200

Commit the changes and push them to the remote repository. We'll be using GitHub in this example.

From the Netlify dashboard, click "Add a new site" and select "Import existing project". Choose GitHub as the Git provider. If the prototype is in a private repository, you may need to authorize access to that repository by going to "Configure the Netlify app on GitHub".

After the build finishes, check the result by navigating to the .netlify.app URL that was generated for this site.

Optionally, customize the URL by changing the site name. In this case, I changed it to example-typescript-prototype.

You can view the full prototype built in this article at example-typescript-prototype.netlify.app.

Wrapping up

Putting together a small prototype can be very useful when designing a new feature. The prototyping work helps guide the technical design and data modeling. It allows the team to validate assumptions with real data and code. The prototype and its source code can be used as supporting material during meetings, resulting in less hand-waving and more efficient discussions.

In this article, I used the made-up example of an online forms creator, similar to Google Forms, to illustrate a prototyping workflow. I gave a detailed walkthrough on how to set up and implement the prototype, using libraries such as React, Redux, React Router, a CSS framework, and the Vite build tool. Of course, this is only one way of doing things. But if you like the approach and follow these steps, it may help you spend less time choosing libraries and setting up tools. This will free up time to focus on the more important prototyping and technical design work.

The full source code for the example is available on GitHub, and the prototype is hosted here.

Moving a blog from Jekyll to Eleventy

2022-03-21T00:00:00Z

Earlier this year I spent some time moving this site from Jekyll over to Eleventy. It was probably a way to procrastinate on writing. If you have also fallen into this classic trap, this post might help you spend less time setting things up and get back to writing. And even if you are starting a blog from scratch and not migrating from Jekyll, there should be some useful things here to help you get started with Eleventy.

Table of contents

Why Eleventy?

Jekyll is still a good option for a static site generator. It is popular and has been around for a while, so you can find results on Google for almost any issue or question. I did read that some people found it slow for larger sites. In my case, I just wanted to try out one of the newer static site generators. I also didn't mind removing the dependency on Ruby, which I don't use much.

I looked at a few options. I wanted to stay with something relatively popular, so I kept to the top of the Jamstack Site Generators list. This means I didn't pick something like Zola, although its zero-dependencies and minimalist aspects were appealing.

Using a generator based on Node.js and JavaScript presented a few benefits. Many frontend tools and libraries require it, so if later I wanted to be more sophisticated and pull in something like Tailwind.css, I would already have the Node.js dependency installed. I'm also comfortable writing JavaScript, so I wouldn't have too much of a hard time extending it if I needed to.

Hugo also looked like a good option, but it would mean I'd have to convert all my Liquid templates to Go templates. And if I got fancier with my frontend CSS or JavaScript, I would probably need to pull in Node.js and npm anyways.

Eleventy seemed closer in spirit to Jekyll compared to the more popular JavaScript options such as Next.js and Gatsby.js. Eleventy is focused on generating static pages, whereas Next and Gatsby seem to do a lot more, such as dynamic content and single-page app functionality.

The fact that Mozilla also moved a site from Jekyll to Eleventy and had come to the same conclusion in terms of similarity between both, was another validation for me.

Finally, Eleventy had just hit 1.0 when I was doing my research, so that was a good sign of future stability.

Why Netlify?

Since I was already changing the static site generator, I also took the opportunity to simplify how the site was remotely built and published. For this, I switched over to using Netlify.

Previously, the site's build artifacts were stored in an S3 bucket, with Cloudflare CDN sitting in front, using Cloudflare's CNAME flattening to serve off of nicolashery.com instead of www.nicolashery.com (a personal choice: it is perfectly fine to serve off of www., and in some cases recommended). I had CircleCI set up to build the site on each push to GitHub, and used the Python awscli to sync the build artifacts to S3.

This worked fine. But Netlify fills the roles of a build environment (replacing CircleCI in my case), a hosting provider (replacing S3), and a CDN (replacing Cloudflare). And if you point your registrar to Netlify so it can manage your DNS, you can also serve off of the bare domain (replacing Cloudflare's DNS with CNAME flattening). Fewer things to manage and think about.

Note that Netlify also fully supports Jekyll. So I could've made this switch without touching the site generator.

Migrating from Jekyll

Porting the site from Jekyll to Eleventy wasn't complicated. Granted, my site is small so that probably made things easier.

After a bit of googling to see what to look out for, I found Xavier Decuyper's post and Mozilla's post to be useful. There is also the official Eleventy base blog starter that is good to use as a reference.

Here are some changes I made to get the source code ready to build with Eleventy:

Remove Ruby's Gemfile and Gemfile.lock, replace them with Node's package.json and package-lock.json (via npm init and npm install)
Add .nvmrc (see nvm)
Update .gitignore
Remove Jekyll's _config.yml, replace it with Eleventy's .eleventy.js and .eleventyignore
Jekyll has a special _posts/ folder, Eleventy doesn't, so I just renamed it to posts/ (no underscore)
Jekyll has support for Sass out of the box, Eleventy doesn't, so I added a separate workflow following Mike Aparicio's Add a Sass workflow recipe (I later removed Sass to use plain CSS, more on that later in this article)
Add a _data/site.json global data file, and hold some site-wide template data previously in Jekyll's _config.yml file (ex: site.title, site.description, etc.)

Configuring Eleventy

Eleventy doesn't make a lot of assumptions on how your site is structured, which gives it great flexibility. But it also means you need to tweak a few things before the site builds and renders properly. Here are some configuration changes I made:

Add README.md to .eleventyignore
By default Eleventy searches for layouts in the _includes/ directory; However, I liked how Jekyll has a special _layouts/ directory, so I updated the Eleventy configuration to have a separate directory for layouts
Copy static files over to the build directory using Eleventy's passthrough file copy configuration option (ex: css/, img/, robots.txt)
Dates can appear off by one day; To fix this with Liquid templates, I updated the template configuration to always display dates in UTC:

eleventyConfig.setLiquidOptions({
  // Display dates in UTC (so they don't risk being off by one day)
  timezoneOffset: 0,
});

In addition to the configuration changes above, I created a posts/posts.json directory data file. I used it to add the following metadata to all posts:

Set "permalink": "{{ page.fileSlug }}/"; This means that a post with a filename of 2022-01-31-blog-with-eleventy.md will be accessible at the URL nicolashery.com/blog-with-eleventy (no date in the URL)
Add "tags": ["posts"]; This allows me to display a list of posts using collections:

{% assign posts_latest_first = collections.posts | reverse %}
{% for post in posts_latest_first %}
<li>
  <a class="posts-link" href="{{ post.url }}">{{ post.data.title }}</a>
  <div class="posts-date">{{ post.date | date: "%d %B, %Y" }}</div>
</li>
{% endfor %}

Syntax highlighting

Syntax highlighting for code blocks is done with Prism. Follow Eleventy's documentation to install the required plugin.

You can browse the list of themes, and download the CSS file for the one you select. For example, save the file to css/prism-one-dark.cssand include the proper tag in your layout:

<link rel="stylesheet" href="/css/prism-one-dark.css">

You might need to make a few tweaks to the file. These were the ones I made in my case:

Remove the theme's font-family attribute for pre and code elements (I manage all fonts in my main.css file)
Remove the margin attribute on the pre (I also manage this in the main.css file with other block spacing rules)

One pitfall I fell into is that I had some plain text code blocks without any language declarations, like so:

```
plain text code block
```

These were not being picked up by Prism. Sol I had to manually update them to add the text language:

```text
plain text code block
```

404 page

I created a 404.md "not found" page, and made sure to add the permalink: 404.html (so it shows up at /404.html and not 404/index.html).

Netlify will automatically use 404.html if present.

RSS feed

The documentation has instructions to set up an RSS feed, with a plugin to install and a sample feed.njk template to use.

For my setup, I moved the metadata from the template to the global data file _data/site.json.

Sitemap

Adding a sitemap is also straightforward. Mike Aparicio has a recipe with a sample sitemap.njk template to use.

Make sure to add eleventyExcludeFromCollections: true to the frontmatter of pages that should not appear in the sitemap, such as the 404.md page.

SEO

Jekyll has an SEO plugin that adds a lot of meta tags to the site. I did find eleventy-plugin-seo, but I got a runtime error when trying to use it. Regardless, I liked the idea of not having a dependency for this, and of having a bit more control on the tags I added. It was also an opportunity to learn a bit more about these tags. So I decided to implement my own solution.

I created an _includes/seo.html that I added in the <head> tag.

I used Eleventy's computed data functionality to centralize and produce all the values used by the template include (instead of doing so directly in the template). For this, I created lib/seo.computed.js that looks a bit like:

module.exports = {
  // Full title for `<title>` tag
  title: (data) => {
    if (data.title) {
      return `${data.title} - ${data.site.title}`;
    }

    return `${data.site.title} - ${data.site.tagline}`;
  },

  // Page title without site title or description appended
  pageTitle: (data) => data.title || data.site.title,

  // ...
};

To add these attributes to the computed data available when generating the site, you need to create _data/eleventyComputed.js with:

const seo = require("../lib/seo.computed");

module.exports = {
  seo: seo,
};

You can then use those value in _includes/seo.html:

<title>{{ seo.title }}</title>

<meta property="og:title" content="{{ seo.pageTitle }}">
<meta name="twitter:title" content="{{ seo.pageTitle }}">

One thing to look out for is how to access different data attributes in the computed data JavaScript file. For example, in my case:

data.title looks for the title attribute in a page's front matter
data.site.title points to the title attribute in my global data file _data/site.json
data.page.url uses Eleventy's supplied page data

There are a lot of SEO tags that exist. I'm not an expert, and I used Jekyll's SEO plugin source code as inspiration. The tags I ended up including were:

Standard tags (title, canonical link, description, author)
Open Graph
Twitter Card
LD JSON

License

It's good to remember to add a license to open-source projects and public content. The only thing I want to call out here is that I opted to provide two licenses:

A Creative Commons license for all of the content of the site
An Unlicense for any custom code that builds the site

I saw this particular setup in Oskar Wickström's blog and used it as inspiration.

Deploying with Netlify

Netlify has become a popular option to build and host static sites and single-page apps. Both the onboarding after creating an account and the documentation are great, so I won't repeat it here. I'll just highlight a few settings that I used.

You can configure the build directly in the Netlify UI, but I chose a file-based configuration. This way if I change the build command in the repository later, I don't have to go back to the Netlify UI. I created a netlify.toml in the root directory of the repository with:

[build]
  publish = "_site"
  command = "npm run build"

We can also add some configuration options for the processing of static assets. I enabled minifying CSS files by adding the following to netlify.toml:

[build.processing]
  skip_processing = false

[build.processing.css]
  bundle = false
  minify = true

There is a setting to enable bundling of CSS or JS files. I chose not to because I only have two source files, main.css and prism-one-dark.css. It also gives me the option to optimize by not including the syntax highlighting CSS on some of the site's pages.

These post-processing options from Netlify are nice because they allow me to keep the build setup simple in the repository itself, and reduce the number of Node.js dependencies. The tradeoff is that it ties me to Netlify a little bit.

UPDATE (2023-07-25): Netlify has since deprecated the asset optimization feature. One solution is to use a CSS minification CLI tool such as Lightning CSS in our production build step.

Make sure your .toml file doesn't contain any syntax errors before pushing. If you don't want to install an editor extension to do so, you can easily check it with an online TOML validator.

To specify the Node version used during the build, note that Netlify will pick up a .nvmrc file if it finds one. Since I use nvm locally, I had that checked in my repository already.

After creating a Netlify account, you will need to grant GitHub access to Netlify via OAuth, so it can pull from your repositories when you push changes. During this step, I chose to only give it access to selected repositories (github.com/nicolashery/nicolashery.com in my case).

After deploying, Netlify will automatically do some additional post-processing to check the site contents (ex: detection of mixed content, and insecure links using http instead of https). This is useful, so make sure to check the deployment logs for any warnings.

Finally, Netlify generates a random site name. Even though we'll set up DNS to use a custom domain, I changed the site name to nicolashery.netlify.app to easily identify it in the Netlify UI.

Custom domain with Netlify DNS

There are two ways you can set up a custom domain to point at your Netlify site:

Use Netlify DNS to manage your domain (and its subdomains)
Or use your existing DNS provider and add a CNAME record to point to the Netlify site

The advantage of choosing Netlify DNS is that it makes it easy to use either the bare domain (nicolashery.com) or a subdomain (www.nicolashery.com). I was using the bare domain before with Cloudflare's CNAME flattening. I also didn't have any special DNS requirements for this domain that pushed me to use another DNS provider. For these reasons, I went with Netlify DNS.

I'm using Gandi.net as the registrar for my domain. Gandi also provides email addresses. The steps to make those work with Netlify DNS were:

Click "Domains" in "Site Settings", and add nicolashery.com as the primary domain
Click "Activate Netlify DNS"
Log in to Gandi and under "Nameservers" use external ones setting them to those given by Netlify
In Gandi, under "Email", copy the mail DNS records given and add them in Netlify DNS
Wait 12-24 hours for propagation

Analytics

Previously the site used Google Analytics. For this new version, I took the opportunity to switch to something that was more:

User privacy-focused
Lightweight

After a bit of research, I found a list of options (in no particular order): Plausible, Fathom, Matomo (formerly Piwik), Simple Analytics.

The ability to self-host was not a requirement for me. In the end, I opted for Plausible as it seemed to be gaining in popularity. There are many articles out there comparing the different options.

After creating an account, setting it up is as simple as adding a script tag to your website.

I didn't want to load this script during local development, because it creates a warning in the console. This isn't a big issue, but it is some noise that might cause me to miss an important console warning. To decide when to load the script, we can check the NETLIFY environment variable. It is automatically set by Netlify during its build step, and absent during local development.

To expose the environment variable to templates, I created a JavaScript data file named_data/env.js:

module.exports = function () {
  return {
    NETLIFY: process.env.NETLIFY,
  };
};

Then in my Liquid template:

{% if env.NETLIFY %}
  {% include "plausible.html" %}
{% endif %}

The last thing useful to set up is the Google Search Console integration. This allows Plausible to display the search terms visitors used when coming in as organic traffic from Google. The documentation walks you through how to set things up.

When setting up the Search Console on Google, I picked a Domain Property (vs. a URL Prefix Property). It is more general and can help you catch unexpected traffic, for instance to a subdomain or to http instead of https. Google will need to verify that you own the site. Pick the Any DNS Provider verification and copy the TXT record that Google gives you. In my case, Netlify handles DNS for the site. So I logged into Netlify, selected my Team, clicked Domains, selected my domain, and added the TXT record. Once that is done, click "Verify" in the Google Search Console. Come back after about 24 hours once the verification to complete.

Back in Plausible, under site Settings, click Search Console. When you connect with Google OAuth, make sure to grant Plausible permission to view your Search Console website data. Choose the sc-domain property in Plausible. Wait 24-36 hours before seeing the data come in.

Note: Once this is done, don't remove the TXT DNS record you added. Leave it in order to stay verified.

Commenting

My previous setup was using the Disqus plugin to add comments to article pages. I wasn't getting a lot of value from it, and in the interest of simplicity, I decided not to add comments in this new version.

I may add a link to a Hacker News, Reddit, or Lobsters submission and let readers know that they can comment over there.

I did a bit of research for Disqus alternatives, so I'll post the results here, even if I didn't use them in the end:

Commento (paid, by design, which means you truly own your data)
Talkyard (does a little more than blog comments, "with features like in StackOverflow, Slack, Discourse, Reddit, Disqus")
Isso (self-hosted, uses SQLite to store data)

CSS

When I set up the first version of this site with Jekyll, I was used Sass as a CSS pre-processor. But since then, CSS has come a long way. Features such as var() and calc() are now well-supported across modern browsers. For this reason, I chose to switch to plain CSS.

The style needs of this site are small, so I kept everything in a single css/main.css file. This forces me to keep things minimal. I organize the file with simple comment headers like so:

/* Variables
-------------------------------------------------------- */

:root {
  --content-width: 640px;
}

/* Reset
-------------------------------------------------------- */

*,
::before,
::after {
  box-sizing: border-box;
  background-repeat: no-repeat;
}

/* etc. */

Since I'm using Netlify's post-processing, I don't need to install a library to minify the CSS. Nor would I need a library to bundle multiple files if I had more than one.

UPDATE (2023-07-25): Netlify has since deprecated the asset optimization feature. One solution is to use a CSS minification CLI tool such as Lightning CSS in our production build step.

Hugo's Paper theme was one source of inspiration for the design of this site. I stumbled across it via Philipp Tanlak's blog. I liked the simplicity and contrast.

Some of the CSS reset rules were taken from sanitize.css, although I didn't use all of it.

For color picking, I use a palette website such as Color Hunt to look for inspiration. I tend to stick to just one or two colors for a small site like this one.

Google web fonts

Google web fonts make it easy to include custom font files to a site. It is good to keep in mind that there is a performance cost to having the user download fonts, but typography is a big part of the style of a blog or other content-centric site.

To help pick a font, you can use the browser developer tools to check the font used by a site you particularly like. Or you can search Google and find lists of popular options.

Once you've picked a font, head over to its page on Google Fonts (Source Sans Pro in this example). Select the weights and styles you need. Typically you'll want Regular 400, Italic Regular 400, and Bold 700. Having both Bold and Italic at the same time is not something we use much so let's leave it out. Keep in mind the more variants you include, the bigger the downloaded payload for the visitor.

After selecting the variants, the right-side panel should open with HTML tags to copy and paste into the <head> section.

Finally, update the site's CSS to set the proper font, along with fallbacks. For example:

html {
  font-family:
    /* Google Font */ "Source Sans Pro",
    system-ui,
    /* macOS 10.11-10.12 */ -apple-system,
    /* Windows 6+ */ "Segoe UI",
    /* Android 4+ */ "Roboto",
    /* Ubuntu 10.10+ */ "Ubuntu",
    /* Gnome 3+ */ "Cantarell",
    /* KDE Plasma 5+ */ "Noto Sans",
    /* fallback */ sans-serif;
}

(Taken from sanitize.css.)

Images with Cloudinary

In the previous setup, I had images (.png, .jpg) and other non-text files (like a .pdf version of my resume) checked in the Git repository. Although that didn't amount to a lot of space, I like the idea of not bloating the Git repository and keeping it for text files only.

Netlify offers a Large Media solution based on Git LFS (for "Large File Storage"). I don't have a lot of experience using Git LFS, but after some research, I saw some negative points mentioned on the web. Netlify also calls out some limitations directly on its documentation page.

I opted for Cloudinary which offers storage, delivery through a CDN, dynamic resizing and transformations, as well as other features which I don't need just now but could be useful to others. It seemed like a good option, even if it means a little more manual work to upload files to this third-party service. It will also be more portable than Netlify's Large Media solution if I ever switch hosting from Netlify to something else.

At the time of writing, Cloudinary offers a free tier for 25k transformations or 25GB bandwidth per month, so it should be plenty for a small blog. After signing up for the free account, it asks you what product you want to use, and I selected "Programmable Media". I kept the auto-generated Cloud Name, which will be used in the URLs for the images.

One thing to note is that Cloudinary's Media Library automatically adds random characters to uploaded file names, to avoid duplicates. This might be useful for some workflows, but it isn't something I wanted in my case. To disable it, go to "Settings > Upload", scroll down to "Upload Presets" and edit the default one used by the Media Library to have Unique filename: false. I then simply drag-and-dropped all images from my system's file explorer to the Media Library.

On the Eleventy side, I added cloudinaryCloudName to _data/site.json. There is nothing secret about the Cloud Name so it is safe to check in a repository.

I then created a shortcode in the file lib/cloudinaryImage.shortcode.js:

module.exports = function (cloudinaryCloudName) {
  return function (path, title, caption, transformations) {
    if (transformations && transformations !== "") {
      transformations = transformations + "/";
    } else {
      transformations = "";
    }
    const url = `https://res.cloudinary.com/${cloudinaryCloudName}/image/upload/${transformations}${path}`;

    return (
      `<figure>` +
      `<img src="${url}" alt="${title || ""}">` +
      `<figcaption>${caption || ""}</figcaption>` +
      `</figure>`
    );
  };
};

It's a closure around the Cloud Name, and you can pass in the value when you register the shortcode in .eleventy.js:

const cloudinaryImageShortcode = require("./lib/cloudinaryImage.shortcode");
const site = require("./_data/site.json");

module.exports = function (eleventyConfig) {
  // ...
  eleventyConfig.addShortcode(
    "cloudinaryImage",
    cloudinaryImageShortcode(site.cloudinaryCloudName)
  );
};

I can then use the shortcode in my Liquid templates like so:

{% cloudinaryImage "user-interface-data-03.png" "UI as data" "The final version of the app with CSS" "c_scale,w_640" %}

Note: I did find juanfernandes/eleventy-plugin-cloudinary, but it is simple enough that I decided not to introduce the dependency. It will also make it easier to adapt to my specific needs in the future.

Conclusion

This wraps up the main steps I took to set up a personal website and blog. There are many options out there to choose from. I picked the static site generator Eleventy, deployed it using Netlify, added analytics with Plausible, and hosted images on Cloudinary.

Feel free to browse the full source code for the site at github.com/nicolashery/nicolashery.com. Keep in mind that there may have been changes since I wrote this article.

Hopefully, some of this can be useful to speed up your own blog setup. Because as much fun as it is to spend time with these tools and technologies, it is more important to spend time writing.

Exploring Isomorphic JavaScript

2015-02-20T00:00:00Z

Isomorphic JavaScript applications (you'll also hear the terms "Server-Side Rendering" or "Progressive JavaScript") is something we hear and read quite a bit about lately. Node.js made it possible, Airbnb blogged about it in 2013, React renders HTML on the server, Ember.js is working on "FastBoot", Andres Suarez from SoundCloud and Michael Ridgway from Yahoo did talks on the subject, etc. I decided to dig in and see what it was all about, what kind of problems you encountered, and possible ways to solve them.

I found that it is still a young idea, and I expect the patterns and available libraries (ex: Fluxible, React Router, Nexus React, Taunus, etc.) to evolve quite a bit in the future. This is one reason why, for a good part of this post, I'll try to stay away from any specific implementation or framework, and focus more on the general concepts and problems to be solved (although of course, you might recognize the influence of certain libraries if you've used them). At the end, I will look at how these ideas translate in a more practical implementation, using Yahoo's Fluxible and React Router.

The code in the examples below is close to "pseudo-code". It stays high-level and the implementation of the functions don't matter as much as what they do. I also assume basic knowledge in Express, React, and Flux, but even if you've never used them, you should be able to follow along.

Simplest example

At the most basic level, an isomorphic app should be able to render both on the server (in Node.js) and in the browser. React does this perfectly well (and other libraries too I'm sure).

// server.js
var server = express();

server.use(function(req, res) {
  var appHtml = React.renderToString(<App />);
  var html = injectIntoHtml({app: appHtml});
  res.send(html);
});

// client.js
React.render(<App />, document.getElementById('app'));

The only "tricky" bit here, is for the client to pick up where the server left off. That is, after the app is loaded in the browser, the first render doesn't "destroy" the DOM generated by the server, but rather "hooks" onto it. This is something that React, with its Virtual DOM implementation, does for you and you don't have to worry about it.

Let's say the app renders a set of data fetched from a remote API (a pretty common scenario). This data is part of the app's "state" (the data can take different values over time, and the app will render differently). Since we need to create this state on the server, it is probably a good idea to share it with the client, so it can pick up where the server left off (instead of re-creating and re-fetching the data, work already done by the server).

// server.js
server.use(function(req, res) {
  var state = {};
  fetchData(function(err, data) {
    state.data = data;
    var exposedState = 'window.__STATE__=' + JSON.stringify(state) + ';';
    var appHtml = React.renderToString(<App data={data} />);
    var html = injectIntoHtml({
      app: appHtml,
      state: exposedState
    });
    res.send(html);
  });
});

// client.js
var state = window.__STATE__;
React.render(<App data={state.data} />, document.getElementById('app'));

We see the need to be able to serialize all of the state on the server, and then parse and instantiate it on the client (you'll hear the terms dehydrate and rehydrate). This allows us to initialize the app on the client in the exact same state it was on the server before being sent back in HTML form.

Also note that on the server the app state is bound to each request, i.e. there is one state per request (since a server serves multiple client requests). On the client, the app state is global, i.e. we only need one state for the whole browser session.

Simple routing

Most apps render according to different possible routes in the URL. Let's assume our top-level <App> React component takes a route object as a property and renders accordingly.

On the server, we can simply get the route by matching the request URL against our "routes declaration" (not shown here):

// server.js
server.use(function(req, res) {
  var route = matchPath(req.url);
  var appHtml = React.renderToString(<App route={route} />);
  var html = injectIntoHtml({app: appHtml});
  res.send(html);
});

On the client, we use the path from the browser's URL location API to match a route (implementation not shown here):

// client.js
function render(route) {
  React.render(<App route={route} />, document.getElementById('app'));
}

// first render
var route = matchPath(getCurrentPath());
render(route);
// re-render on browser location change
addLocationChangeListener(function(path) {
  var route = matchPath(path);
  render(route);
});

Since subsequent route changes will happen on the client, we listen to browser URL location changes (this could be implemented on top of HTML5 History), and re-render the new route.

Shared state and routing

Let's combine the previous two examples. On the server, we match the route and attach it to the app state (this makes sense since the route can take different values, each producing different rendering outputs). We then fetch the required data, render the app, and send the HTML back.

// server.js
server.use(function(req, res) {
  var state = {};
  var route = matchPath(req.url);
  state.route = route;

  fetchData(route, function(err, data) {
    state.data = data;
    var exposedState = 'window.__STATE__=' + JSON.stringify(state) + ';';
    var appHtml = React.renderToString(<App route={route} data={data} />);
    var html = injectIntoHtml({
      app: appHtml,
      state: exposedState
    });
    res.send(html);
  });
});

Note that the fetchData function takes the route as an argument. Indeed, the data needed by the app will probably depend on the route (ex: for /contacts we fetch the list of contacts, for /contact/1/messages we fetch the contact's details and messages).

Note: This means that we declare our data-fetching needs at the route-level (in the "routes declaration"), which might not be the best place to do it in terms of modularity. But we'll see later that some libraries (like React Router) make it possible to declare it at the component-level instead.

On the client, we follow the same logic:

// client.js
var state = window.__STATE__;

function render() {
  React.render(
    <App route={state.route} data={state.data} />,
    document.getElementById('app')
  );
}

// first render
render();
// re-render on browser location change
addLocationChangeListener(function(path) {
  var route = matchPath(path);
  state.route = route;

  fetchData(route, function(err, data) {
    merge(state.data, data);
    render();
  });
});

Note that for the first render we don't need to fetch data (since it was already sent by the server). But for subsequent route changes, we need to fetch the necessary data for that new route.

However, we can improve the logic of that last piece. Indeed, in its current form we wait for the data fetching to finish before actually rendering the route change. While this was necessary on the server, on the client we can render the route change immediately, fetch the data in the background, and re-render when it comes back (maybe showing a loading spinner in the meantime):

// client.js
// ...
addLocationChangeListener(function(path) {
  var route = matchPath(path);
  state.route = route;
  // render immediately with no/old data (can show a loading spinner)
  render();

  // fetch data in the background then re-render
  fetchData(route, function(err, data) {
    merge(state.data, data);
    render();
  });
});

Authentication

Most apps have some sort of authentication logic. They protect access to certain routes and redirect to a sign-in screen when needed. How would that work with isomorphism?

If authentication is persisted from one browser session to the next, as is often the case, then cookies are the shared persistence layer between client and server (both have access to them). If we use an "auth token" to access the remote data API, we can store that token in a cookie (this is not a post on security, so make sure you do your due-diligence regarding setting expiration dates, being able to revoke tokens, using SSL, etc.).

Being authenticated or not is also part the the app's state. We can create the following helper function:

// isAuthenticated.js
function isAuthenticated(state) {
  return Boolean(state.authToken);
}

Now, for each request on the server-side, we first read the value of the cookie, and check if the auth token is valid. If it is not valid and we are trying to access a protected route, we redirect to the sign-in page. In other cases, we proceed with fetching the data for that route and rendering.

// server.js
server.use(function(req, res) {
  var state = {};
  var authToken = getCookie('authToken');
  checkSession(authToken, function(err, isValidToken) {
    if (isValidToken) {
      state.authToken = authToken;
    } else {
      clearCookie('authToken');
      state.authToken = null;
    }
    var route = matchPath(req.url);
    if (isAuthRequiredForRoute(route) && !isAuthenticated(state)) {
      res.redirect(303, signInPath());
      return;
    }
    state.route = route;
    // fetch data, render app, expose state, send html...
  });
});

On the client, the first render already has the authentication check done by the server, and the auth token was already loaded from the cookie into the serialized state. However, for each route change happening on the client we also need to perform the check "is this route protected and are we authenticated", and redirect appropriately. "Redirecting" on the client means replacing the browser's URL location (which will trigger a change event, in our implementation).

// client.js
// grab server state, first render... then:
addLocationChangeListener(function(path) {
  var route = matchPath(path);
  if (isAuthRequiredForRoute(route) && !isAuthenticated(state)) {
    replaceLocation(signInPath());
    return;
  }
  state.route = route;
  render();
  // fetch data, update state, render...
});

Configuration

Using configuration values is also a common pattern for apps. For instance, you might set API_HOST to point to different environments (development, staging, production) of the remote data API.

// config.js
var config = {
  API_HOST: process.env.API_HOST || 'http://localhost:3000/api'
};

// server.js
var config = require('./config');
api.useConfig(config);

On the server, we can get config values from environment variables (or a local text file). However, that's not possible on the client.

So we'd like to share the config values with the client, just like we share the app state (for security purposes, you might want to filter some config values like API secrets before sending it to the client). We'll keep it separate, because unlike the app state, config isn't suppose to change during the app's life cycle. Indeed, we initialize services (for example the api client) with the config values when the server starts. Config is shared across requests, whereas the app state is created on every request. The "serialize, parse & instantiate" mechanism between server and client stays the same.

// server.js
var config = require('./config');
api.useConfig(config);

server.use(function(req, res) {
  var state = {};
  // check session, get route, fetch data... then:
  var exposedConfig = 'window.__CONFIG__=' + JSON.stringify(config) + ';';
  var exposedState = 'window.__STATE__=' + JSON.stringify(state) + ';';
  var appHtml = React.renderToString(<App route={route} data={data} />);
  var html = injectIntoHtml({
    app: appHtml,
    config: exposedConfig,
    state: exposedState
  });
  res.send(html);
});

// client.js
var config = window.__CONFIG__;
var state = window.__STATE__;

api.useConfig(config);
// first render, add location change listener...

Disabling isomorphism

If isomorphism means the app can run both on the server and in the browser, we should be able to "turn off" the server part and still have a working app. This could be useful during development for example, so we can take more advantage of the browser's developer and debugging tools.

Let's use a config value to turn isomorphism on and off:

// config.js
var config = {
  // ...
  DISABLE_ISOMORPHISM: process.env.DISABLE_ISOMORPHISM === 'true' || false
};

On the server, when isomorphism is off, we only need to send the serialized config in an empty HTML page (that contains a link to the JS app bundle), and the client will take care of the rest:

// server.js
server.use(function(req, res) {
  if (config.DISABLE_ISOMORPHISM) {
    var exposedConfig = 'window.__CONFIG__=' + JSON.stringify(config) + ';';
    var html = injectIntoHtml({
      config: exposedConfig
    });
    res.send(html);
  } else {
    var state = {};
    // check session, fetch data, render app,
    // expose state & config, send html...
  }
});

The client now has more things to do for the first render. It needs to perform the tasks that were usually done by the server when isomorphism was turned on: create a new app state, read the auth token from the cookie, check if it's a valid session, match a route from the browser's URL location, fetch the data for that route, and finally render.

// client.js
var config = window.__CONFIG__;
api.useConfig(config);

var state;
// first render
if (config.DISABLE_ISOMORPHISM) {
  state = {};
  var authToken = getCookie('authToken');
  api.checkSession(authToken, function(err, isValidToken) {
    if (isValidToken) {
      state.authToken = authToken;
    } else {
      clearCookie('authToken');
      state.authToken = null;
    }
    var route = matchPath(getCurrentPath());
    if (isAuthRequiredForRoute(route) && !isAuthenticated(state)) {
      replaceLocation(signInPath());
      route = matchPath(signInPath());
    }
    state.route = route;
    // render immediately with no data (can show a loading spinner)
    state.data = {};
    render();
    // fetch data in the background then re-render
    fetchData(route, function(err, data) {
      merge(state.data, data);
      render();
    });
  });
} else {
  state = window.__STATE__;
  render();
}

// add browser location change listener...

With this in place, the app should now function normally with isomorphism turned on or off. The first render will just take a little more time in the browser when isomorphism is off.

Progressive enhancement

In the previous section we saw how to handle turning off the server-side rendering portion. What if we "turned off" the client-side portion? That would probably mean disabling JavaScript in the browser, and "going back" to the good old request/response cycle where every route is rendered on the server.

This is also known as "Progressive Enhancement". There are already plenty of good articles that explain why and when you should do it, so I won't go into that here.

The bare-minimum of Progressive Enhancement is to be able to go from one route to another with JS turned off. This means using actual link tags (<a>) with their href attribute defined (this is also important for accessibility). When JS is enabled, you can intercept the click event on the <a> elements to prevent a page refresh and handle the routing on the client. A simple React component achieving this could look like:

// Link.jsx
var Link = React.createClass({
  render: function() {
    return (
      <a {...this.props} onClick={this.handleClick}>{this.props.children}</a>
    );
  },
  handleClick: function(e) {
    e.preventDefault();
    navigateTo(this.props.href);
  }
});

This should cover most, if not all, of the "read" scenarios for the app. What about "writes", i.e. creating and updating data? For that, we need to make sure to capture user input in a <form>, and create an additional route to handle the form's POST action (we set this route in the form's action attribute). Just like previously, when JS is enabled we intercept the submit event on the <form> element, and handle the routing on the client.

// NewContact.jsx
var NewContact = React.createClass({
  getInitialState: function() {
    return {working: false};
  },

  render: function() {
    <form action="/contacts/create" onSubmit={this.handleSubmit}>
      <input ref="name" name="name" placeholder="New contact" />
      <button type="submit" disabled={this.state.working}>Create</button>
      {this.state.working ? 'Working...' : null}
    </form>
  },

  handleSubmit: function(e) {
    e.preventDefault();
    this.setState({working: true});
    var name = this.refs.name.getDOMNode().value;
    navigateTo('/contact/create?name=' + urlEncode(name));
  }
});

Note that when JS is enabled on the client, we can enhance the experience by showing a "loading" indicator while we process the form.

For this to work properly, we will need to add hooks to the routing logic, specifically before the route change, so we can process the form data and then redirect to a new route when done. It could look like this in the routes declaration:

// routes.js
var routes = {
  'contacts': {
    path: '/contacts'
  },
  'contact-new': {
    path: '/contact/new'
  },
  'contact-create': {
    path: '/contact/create',
    before: function(params, query, done) {
      createContact(query, function() {
        done({redirect: true, path: '/contacts'});
      });
    }
  },
  // ...
};

One thing to note is that routing is now asynchronous. The before() hook takes a done() callback, and the route will only change once it is called (or in our case, we redirect to another route). This means that instead of only having to match a path to a route with var route = matchPath(path), we will have something like executeRouting(path, function(route) { /*...*/ }).

// server.js
server.use(function(req, res) {
  var state = {};
  // check session... then:
  // routing is now async
  executeRouting(req.url, function(route) {
    // check if a routing hook flagged the route as a redirect
    if (route.redirect) {
      res.redirect(303, route.path);
      return;
    }
    if (isAuthRequiredForRoute(route) && !isAuthenticated(state)) {
      res.redirect(303, signInPath());
      return;
    }
    state.route = route;
    // fetch data, render app, expose state & config, send html...
  });
});

So now, when JS is disabled, submitting the "new contact" form will sent a POST request to /contact/create. The executeRouting() call will wait until createContact() is finished (in the route's before() hook), and then redirect to /contacts.

We need to update the client-side with the new routing logic as well:

// client.js
// grab server state, first render... then:
addLocationChangeListener(function(path) {
  executeRouting(path, function(route) {
    if (route.redirect) {
      replaceLocation(route.path);
      return;
    }
    if (isAuthRequiredForRoute(route) && !isAuthenticated(state)) {
      replaceLocation(signInPath());
      return;
    }
    state.route = route;
    render();
    // fetch data, update state, render...
  });
});

If JS is enabled on the client, submitting the "new contact" form will set the state of the <NewContact> component to {working: true}, thus showing a loading indicator. No page refresh happens: the form is processed entirely client-side, as well as the redirect to /contacts when finished, thanks to the same before() routing hook.

Flux

Flux is a common data-flow architecture for React apps that have a non-trivial amount of data and state to deal with, so let's see how it would look like in an isomorphic app. I purposely left it for last, because I wanted to show that all the previous concerns (sharing state, routing, authentication, progressive enhancement, etc.) have nothing to do with Flux (or React for that matter), and are completely valid in other app architectures.

With Flux, the app state is contained in stores, and it gets updated by sending actions through a dispatcher. So just like our simple state object used in previous examples, on the server-side we need to make sure to have one "Flux instance" of stores and dispatcher for each request (so different app states don't collide with each other between requests). We'll imagine we can do this with:

var flux = createFlux();

The instance has its own dispatcher, and we can send actions to it with action creators, like so:

createContact(flux, {name: 'Bob'});

The instance has its own stores, registered to its dispatcher, and we can access these stores like so:

var contact = flux.ContactStore.get('1');

Checking the user session and fetching data both update the app state, so it makes sense to do them through actions. However, for this to work inside a request on the server-side, we need to provide action creators with a done() callback so we can wait for the session-checking and data-fetching to finish before sending the server response. Be careful though not to pass any arguments to this done() callback, in order to keep the "fire and forget" nature of actions when they are used inside components on the client (we don't want components to "wait" for actions to finish and update their internal state, we want them to listen to changes from stores, the "single source or truth").

// actions/loadSession.js
function loadSession(flux, payload, done) {
  var authToken = getCookie('authToken');
  flux.dispatch('LOAD_SESSION_START');
  api.checkSession(authToken, function(err, isValidToken) {
    if (err) {
      flux.dispatch('LOAD_SESSION_FAILURE', err);
      done();
      return;
    }
    if (isValidToken) {
      flux.dispatch('LOAD_SESSION_SUCCESS', authToken);
      state.authToken = null;
    }
    else {
      clearCookie('authToken');
      flux.dispatch('LOAD_SESSION_SUCCESS', null);
    }
    done();
  });
}

// actions/fetchContact.js
function fetchContact(flux, payload, done) {
  flux.dispatch('FETCH_CONTACT_START');
  api.getContact(payload.contactId, function(err, contact) {
    if (err) {
      flux.dispatch('FETCH_CONTACT_FAILURE', err);
      done();
      return;
    }
    flux.dispatch('FETCH_CONTACT_SUCCESS', contact);
    done();
  });
}

Similarly, routing can also be done through an action since it changes app state:

// actions/navigateTo.js
function navigateTo(flux, payload, done) {
  var route = matchPath(payload.path);
  var isAuthenticated = flux.AuthStore.isAuthenticated();
  if (isAuthRequiredForRoute(route) && !isAuthenticated) {
    flux.dispatch('REDIRECT', signInPath());
    done();
    return;
  }
  flux.dispatch('CHANGE_ROUTE', route);
  done();
}

Data fetching depends on the route, so we can define it in the routes declaration, and use action creators to execute it:

// route.js
var routes = {
  'contact-details': {
    path: '/contact/:id',
    fetchData: function(flux, params, query, done) {
      fetchContact(flux, {contactId: params.id}, done);
    }
  },
  // ...
};

We create a general fetchData function, which will get the current route from the app state and fetch any required data defined for that route:

// fetchData.js
function fetchData(flux, done) {
  var route = flux.RouteStore.currentRoute();
  var dataFetcher = route.fetchData;
  if (!dataFetcher) {
    return done();
  }
  dataFetcher(flux, route.params, route.query, done);
}

We still need to be able to serialize the app state (now contained in stores) on the server, and parse and instantiate it on the client. Let's imagine our flux instance has two methods, var serializedState = flux.dehydrate() and flux.rehydrate(serializedState) that accomplish this. On the server, it asks each store to serialize its state, and on the client it re-populates the state of each store.

Bringing it all together, this is what our server and client would look like:

// server.js
server.use(function(req, res) {
  var flux = createFlux();
  loadSession(flux, {}, function() {
    navigateTo(flux, {path: req.url}, function() {
      var redirectPath = flux.RouteStore.redirectPath()
      if (redirectPath) {
        res.redirect(303, redirectPath);
        return;
      }
      fetchData(flux, function() {
        var exposedState = 'window.__STATE__=' + flux.dehydrate() + ';';
        var appHtml = React.renderToString(<App flux={flux} />);
        var html = injectIntoHtml({
          app: appHtml,
          state: exposedState
        });
        res.send(html);
      });
    });
  });
});

// client.js
var state = window.__STATE__;
var flux = createFlux();
flux.rehydrate(state);

function render() {
  React.render(<App flux={flux} />, document.getElementById('app'));
}

// first render
render();

addLocationChangeListener(function(path) {
  navigateTo(flux, {path: path}, function() {
    var redirectPath = flux.RouteStore.redirectPath()
    if (redirectPath) {
      replaceLocation(redirectPath);
      return;
    }
    render();
    fetchData(flux, render);
  });
});

Of course, this is one possible way to combine Flux and isomorphism, and I'm sure there are other completely valid or even better ways.

Practical implementation

Now that we've explored what general concepts and problems to solve arise with Isomorphic JavaScript apps, let's look at a more real-world implementation. This GitHub repository contains an example app built using Yahoo's Fluxible and React Router.

The is one possible choice of libraries, and I'm sure another one would work just as well.

Fluxible is a Flux implementation that was built with isomorphism in mind, and provides a solution to the "one Flux state for each request" problem. It does add some additional abstractions you need to learn (ex: different "contexts" for actions, stores, components), but you get used to it. They also tried to keep it modular (for instance I didn't have to use their router solution, and used React Router instead), and extensible (with "Fluxible plugins"). Actions have "done" callbacks, which means we can use them on the server inside a request handler.

React Router is quite popular, and has a "nested" route definitions. This allows you to declare data fetching logic at the component-level (instead of the route-level), and when the route changes the router automatically "walks the component render tree", so you can aggregate and fetch each component's required data. On the flip-side, the router owns the "route state", which means it will be separate from the rest of the app state (i.e. not in a Flux store). But you could still "sync" it to a store if you wanted to.

With Fluxible, the flux "instance" (i.e. state) that we create on each request is called a context. The action creators, stores, and components get access to only part of the context: this ensures, for example, that only action creators can "dispatch" actions, but also means you have to use an additional abstraction.

We can also use the (undocumented) React.withContext() function (careful not to use it to replace props though). With it, every component has access to Fluxible's component context without having to pass it down as a property.

Update (2015-04-21): React.withContext() is deprecated, but the React context feature is not going away. Fluxible now provides a FluxibleComponent that we use to wrap our top-level component, providing child components with Fluxible's context.

I ended up putting config values in Fluxible plugins, since they also have dehydrate/rehydrate methods. For example, I have an apiPlugin that holds the API_HOST config value, read from an environment variable on the server, and sent to the client via the plugin's dehydrate method.

The routing logic is executed by React Router. Checking authentication and redirecting is done through the willTransitionTo static method (at the time of writing I had to use a fork of React Router that supports passing the context to that method).

The fetchData function takes as an argument the routerState that was passed to React Router's router.run() callback. Using routerState.routes, an array of all handler components that will be rendered for that route, we can grab all the fetchData static methods from components that have one, and run them in parallel.

Since React Router is separate from the Flux implementation, to be able to "dispatch routing actions" (for example, redirect after sign-in), I wrapped the router in a Fluxible routerPlugin. This way I can call context.getRouter().transitionTo() from inside an action creator. It might not be the most elegant way, but it works.

The first render on the client happens inside the router.run() handler, just like subsequent renders. We want to "flag" the first render and treat it differently, so we don't re-fetch all the data we already got from the server-side rendering.

Regarding progressive enhancement, specifically supporting posting forms with JS disabled, we work with the fact that React Router "handlers" are just components. So we need a "fake" component to handle the form's POST route (ex: <form action="/contact/create">), and we use that component's willTransitionTo static method to perform the API call.

If you're going through the example GitHub repository, a good place to start is to look at server.js and client.js.

Conclusion

Throughout my exploration of the subject and this post, I've tried to extract some general concepts related to Isomorphic JavaScript. Hopefully these are common across different implementations, under some form or another. I've also given a small practical example using existing libraries (Fluxible, React Router).

There are of course other things to explore, for example data loading indicators on the client-side, "optimistic" updates and the impact on progressive enhancement, handling API call errors, handling "not found" routes, setting/removing cookies, other data fetching strategies, etc.

My conclusion so far is that Isomorphic JavaScript is still young, and I expect different interesting approaches and libraries to appear in the future. There are also still ongoing debates about the pros & cons, and whether or when should you use it. I do find that it adds some complexity to your app, and increases the "surface area for bugs". But it's definitely something we should pay attention to, and keep an eye on as it evolves.

In some way, it gives a web app a more "original web"-like experience (but with enhanced performance on the client), versus just a "packaged app" that you download. URLs become truly "first-class", and you end up spending more time thinking about them. It also pushes you to embrace the "stateless" nature of URLs. You can see the app somewhat becoming a "function" taking as arguments a URL and some auth information in a cookie, and returning rendered data.

If you haven't already, give "Isomporphic JavaScript" a try, and do share your thoughts and findings!

Describing UI state with data

2015-01-19T00:00:00Z

If you think about it, the majority of (if not all) software programs are just transformations of data. Given input A, the program P produces output B. It's easy to see this pattern in small Unix command-line tools, but you could also apply it to larger systems, and why not a user-interface?

When looking at a mockup of a UI screen, along with some use-cases (ex: user clicks here, this happens), imagine what could be the JSON (or other data format) representation of that screen? What transformations to that JSON would cover the given use-cases?

In the land of front-end web applications, new libraries and patterns really encourage to think that way (ex: React, Flux, Om, DataScript). Even the "good old" MVC architecture gives data a front-row seat, in the M for "Model" (unfortunately in MVC, sometimes part of the app state ends up in the V for "View"). Modeling a UI as data is an important step, and it's worth taking time away from the keyboard to explicitly think it through.

Disclaimer: I don't pretend to invent anything new in this post. The concepts used already exist, and libraries and frameworks have already been written around them. What I'd like to do is simply follow the process of building a small example app, focus on the concepts and not a particular framework or library, and hopefully show along the way some interesting things and advantages related to this approach. The example is pretty simple, so some things might seem trivial or overkill, but the idea is that the same concepts applied to larger apps make things easier to reason about and work with.

Sketching the interface and user experience

When building an app, solving an actual problem and providing a good user experience is what matters most. The user doesn't care how elegant our data model is. So it's a good idea to take a step back, start on paper, sketch out what the app would look like and what use-cases it would need to cover.

The example app we'll be building here is a screen that's part of a larger app, some kind of game. This screen should allow the user to build an army before going into battle, based on units that are available to her as well as the resources she has gathered. Here's what that could look like:

Sketching the user interface

What are the use-cases of this app? Let's separate them into two groups. The first group covers what the user can do:

When I click the "+" button, it adds one unit of that type to my army
When I click the "-" button, it removes one unit of that type from my army
When I click the "Reset" button, it removes all units from my army
When I click on of the "Sort by" buttons ("Cost", "Name"), it sorts the unit selection list and the army summary by the corresponding property

The second group of use-cases describes what the user can see:

I can see a sorted list of units available, and each unit will have a "+" button visible if I can add it, and a "-" button visible if I can remove it
I can see the resources I have remaining
I can see a sorted summary of units I have so far in my army
I can see a graph to help me evaluate the strength of my army against different types of units

Now that we have the visuals and use-cases sketched out, we can dive into code. The full and final version of the code presented next is available on GitHub.

Modeling the app state as data

After the sketch, this is probably the most important (and fun) part, so it's worth spending some time thinking about it (our example app is pretty simple, but larger apps will require more thought). It's also worth noting that there are different ways to model things, and no right or wrong. Whatever model we come up with however, we will make sure to respect the following:

We identify and separate "core state" from "derived state". We only store the core state in the "app state" data object, and we keep it to the bare minimum (i.e. if it can be derived from something else, don't store it).

So what would that look like in our example?

We need a list of the units with their properties that is currently available to the user (maybe fetched from the backend storing the game progress)
We need the resources the user has available at that stage in the game (fetched from the backend too)
For each unit, we'll need a counter (initially set to 0) that we can increment every time the user adds a unit of that type to her army
We need to keep track of which property the list of units is sorted by ("cost" or "name")

As a JavaScript object, here is one way we can model our app state:

{
  resources: {gold: 1500, supply: 10},
  sort: 'cost',
  units: {
    archer: {
      count: 0,
      name: 'Archer',
      description: 'Ranged unit, strong vs light armor',
      cost: 140,
      attack: 5,
      hp: 40,
      bonus: {light: 4}
    },
    // ...
  }
}

Note that this object only contains the "core state". For example, the resources attribute holds the starting resources and we don't keep track of the remaining resources the user actually sees on screen. Indeed, the remaining resources are a function of the starting resources and the count of each unit, so it is "derived state":

f(starting resources, count of each unit) -> remaining resources
f(core state) -> derived state

The same goes for the sorted unit selection list (the core state just holds a map of available units), the sorted army summary, and the data behind the army balance graph. We'll call these "projections", and we'll see more about them later.

We verified that this model of the app state, with the appropriate "projections", covers the second group of use-cases (what the user can see). What about the first group (what the user can do)?

Adding/removing a unit changes its count property
Clicking "Reset" changes every unit's count back to 0
Clicking a "Sort by" button changes the sort attribute

Our data model for the app state seems to work, so we can move on to the next step.

Writing a console version of the app

Let's create some functions that will transform the state object defined previously.

We'll be using the data structures from the library immutable (Map, List). The two main reasons for this are:

Projections from core state to derived state often involve manipulating objects (adding/removing properties, etc.). Instead of having to worry about "defensive cloning" of objects and arrays, we can just use immutable persistent data structures and focus on more important things.
The library has a lot of useful methods for manipulating data structures (map, reduce, merge, getIn, etc.), which is convenient since manipulating data is really the focus here.

Even if you've never worked with immutable before, the API is straightforward and you should be able to follow along without any problem.

We'll first create a State.initial() function that returns the initial app state:

State.initial = function() {
  return Immutable.fromJS({
    resources: {gold: 1500, supply: 10},
    sort: 'cost',
    units: {
      footman: {
        count: 0,
        // ...
      },
      archer: {
        count: 0,
        // ...
      },
      // ...
    }
  });
};

Now we'll create Actions to cover the use-cases where the user interacts with the app. An action is just a function that takes the current version of the app state and returns a new state:

action(current state) -> new state

For example, let's implement Action.select(state, unit) called when the user clicks on the "+" button next to a unit:

Actions.select = function(state, unit) {
  return state.updateIn(['units', unit, 'count'], 0, function(count) {
    return count + 1;
  });
};

We can immediately try it out in the console:

var s = State.initial();
s.getIn(['units', 'archer', 'count']); //-> 0
s = Actions.select(s, 'archer');
s.getIn(['units', 'archer', 'count']); //-> 1

You can find the rest of the actions implemented in Actions.js.

Next we'll create Projections to shape the core application state in a way that's ready to be rendered into a UI component. A projection is simply a function that takes the app state and returns some "derived state" or "view" of the app state:

projection(state) -> view

For example, let's implement Projections.resourcesRemaining(state) that will be used to populate the remaining resources widget in the UI:

Projections.resourcesRemaining = function(state) {
  var goldInitial = state.getIn(['resources', 'gold']);
  var goldSpent = state.get('units').reduce(function(result, unit) {
    return result + unit.get('count') * unit.get('cost');
  }, 0);
  var goldRemaining = goldInitial - goldSpent;
  // Similar logic for `supplyRemaining` not shown here...
  return Immutable.fromJS([
    {id: 'gold', name: 'Gold', quantity: goldRemaining},
    {id: 'supply', name: 'Supply', quantity: supplyRemaining}
  ]);
};

We can also immediately try it out in the console:

var s = State.initial();
Projections.resourcesRemaining(s).toJS();
// [
//   {id: 'gold', name: 'Gold', quantity: 1500},
//   {id: 'supply', name: 'Supply', quantity: 10}
// ]
s = Actions.select(s, 'archer');
s = Actions.select(s, 'footman');
Projections.resourcesRemaining(s).toJS();
// [
//   {id: 'gold', name: 'Gold', quantity: 1240},
//   {id: 'supply', name: 'Supply', quantity: 8}
// ]

You can find the rest of the projections implemented in Projections.js.

With the functions in the modules State, Actions, and Projections, what we have done is basically create a "console version" of the whole app. We can play around with it in the console, and check that it behaves like we expect it to.

It is also very easy to add unit tests, since these are all just pure functions (no side-effects, given input A they always produce output B) operating on plain data:

For an action, given this state does it produce the expected new state?
For a projection, given this state and these parameters, does it produce the expected view?

// test.js
describe('Actions.select', function() {
  it('increments count of selected unit', function() {
    var state = Immutable.fromJS({
      units: {archer: {count: 0}}
    });

    state = Actions.select(state, 'archer');

    expect(state.getIn(['units', 'archer', 'count'])).to.equal(1);
  });
});

describe('Projections.resourcesRemaining', function() {
  it('returns correct quantities of remaining resources', function() {
    var state = Immutable.fromJS({
      resources: {gold: 10, supply: 4},
      units: {
        footman: {cost: 1, count: 1},
        archer: {cost: 3, count: 2}
      }
    });

    var resourcesRemaining = Projections.resourcesRemaining(state);

    expect(resourcesRemaining.getIn([0, 'id'])).to.equal('gold');
    expect(resourcesRemaining.getIn([0, 'quantity'])).to.equal(3);
    expect(resourcesRemaining.getIn([1, 'id'])).to.equal('supply');
    expect(resourcesRemaining.getIn([1, 'quantity'])).to.equal(1);
  });
});

What's more, we don't even need a browser or a DOM for these tests, they can run on the server in Node.

At this point, without even beginning to worry about the browser, HTML, and CSS, we already have all the "business logic" of our app implemented.

Adding the graphical interface

We can finally start rendering our app state and its projections in the browser, and hook up our actions to user interactions. We'll use React for this, not only because it's a popular choice, but also because it fits very well with the pattern "given this piece of data describing my application state, this is how the DOM should look like" (i.e. keeping the DOM in sync with the app state). However, we'll see later that it's fairly easy to switch to another "rendering engine".

We'll also wrap our pure functions in a stateful app object. This is only an implementation detail (the pure functions are the important bit: they hold all the logic, and are the ones we test). The main purpose of this object is to emit an event that will trigger a re-render when the application state changes:

function render() {
  var gold = app.projections.resourcesRemaining()
    .getIn([0, 'quantity']);
  console.log('gold ' + gold);
}
app.addChangeListener(render);

render();
// console log: "gold 1500"
app.actions.select('archer');
// console log: "gold 1360"

Our top-level React component App will listen to these changes and re-render when they happen:

// App.jsx
var App = React.createClass({
  componentDidMount: function() {
    app.addChangeListener(this.handleChange);
  },

  componentWillUnmount: function() {
    app.removeChangeListener(this.handleChange);
  },

  handleChange: function() {
    // Trigger a re-render
    this.forceUpdate();
  },

  render: function() {
    // ...
  }
});

We'll also start with an "HTML-only" version. CSS can be a big time sink, and we want to get as fast as possible to a usable prototype that we can actually click through and test. (We can even have a designer or other developer work on the CSS in parallel.)

Thanks to the notion of "projections", which massage the app state before it's shown on screen, we can keep our UI components thin and simple. All of the "business logic" is contained in the projection's code, the UI component just maps its result to DOM elements. For the interactions, they are simply implemented by hooking up DOM events (clicks, etc.) to actions.

Let's implement the UnitList component which allows the user to view units that are available and add or remove them from the army. The component simple renders the result of Projections.unitList(state):

// Projections.js
Projections.unitList = function(state) {
  // First, create a list from the `units` map
  var result = return state.get('units')
    .map(function(unit, key) {
      return unit.set('id', key);
    })
    .toList();

  // Next, sort by the appropriate attribute
  var sortAttribute = state.get('sort');
  result = result.sortBy(function(unit) {
    return unit.get(sortAttribute);
  });

  return result;
};

// UnitList.jsx (simplified)
var UnitList = React.createClass({
  render: function() {
    return (
      <div>
        {app.projections.unitList().map(this.renderUnit).toArray()}
      </div>
    );
  },

  renderUnit: function(unit) {
    return (
      <p key={unit.get('id')}>
        <strong>{unit.get('name')}</strong>
        {' (' + unit.get('count') + ') '}
        <button
          onClick={app.actions.select.bind(null, unit.get('id'))}
          style={{opacity: this.isSelectable(unit) ? 1 : 0}}>
          {'+'}
        </button>
        {' '}
        <button
          onClick={app.actions.remove.bind(null, unit.get('id'))}
          style={{opacity: this.isRemovable(unit) ? 1 : 0}}>
          {'-'}
        </button>
      </p>
    );
  },

  isSelectable: function(unit) {
    var goldRemaining = app.projections
      .resourcesRemaining().getIn([0, 'quantity']);
    return goldRemaining >= unit.get('cost');
  },

  isRemovable: function(unit) {
    return unit.get('count') > 0;
  }
});

This looks pretty nice. But do you see anything we can make better? Remember how we mentioned that all app logic should be contained in projections and actions, and not in UI components? Well, our UnitList component actually still holds some pretty important logic. Let me demonstrate: I run this, and I start adding a bunch of "Footman" units, which are pretty cheap. Oops! At 10 footman, the "+" button is still visible but my remaining supply is 0! Silly me, I forgot to take supply into account in the isSelectable(unit) method. Better yet, I should move both the "isSelectable" and "isRemovable" logic out of the component and into my projection, and add a test for the bug I just found:

// Projections.js
Projections.unitList = function(state) {
  // First, create a list from the `units` map
  // ...

  // Next, sort by the appropriate attribute
  // ...

  // Finally, add the selectable and removable flags
  var resourcesRemaining = this.resourcesRemaining(state);
  var goldRemaining = resourcesRemaining.getIn([0, 'quantity']);
  var supplyRemaining = resourcesRemaining.getIn([1, 'quantity']);
  result = result.map(function(unit) {
    return unit.merge({
      isSelectable: goldRemaining >= unit.get('cost') &&
                    supplyRemaining >= 1,
      isRemovable: unit.get('count') > 0
    });
  });

  return result;
};

// test.js
describe('Projections.unitList', function() {
  it('sets isSelectable flag to false if not enough supply', function() {
    var state = Immutable.fromJS({
      resources: {gold: 10, supply: 0},
      units: {
        archer: {cost: 3, count: 2}
      }
    });

    var unitList = Projections.unitList(state);

    expect(unitList.getIn([0, 'isSelectable'])).to.be.false;
  });
});

// UnitList.jsx (simplified)
var UnitList = React.createClass({
  render: function() {
    return (
      <div>
        {app.projections.unitList().map(this.renderUnit).toArray()}
      </div>
    );
  },

  renderUnit: function(unit) {
    return (
      <p key={unit.get('id')}>
        <strong>{unit.get('name')}</strong>
        {' (' + unit.get('count') + ') '}
        <button
          onClick={app.actions.select.bind(null, unit.get('id'))}
          style={{opacity: unit.get('isSelectable') ? 1 : 0}}>
          {'+'}
        </button>
        {' '}
        <button
          onClick={app.actions.remove.bind(null, unit.get('id'))}
          style={{opacity: unit.get('isRemovable') ? 1 : 0}}>
          {'-'}
        </button>
      </p>
    );
  }
});

Much better! Now my UnitList component is truly thin, and only maps the data from my app state projection to the DOM.

The same guideline applies to other components. For instance, I could create the ArmySummary component directly from the units map in the app state, sorting the units inside the component's render() method. But what if I want to change the logic and always sort by unit name, or unit count? It might be a better idea to create Projections.armySummary(state), and have the sorting and any other logic happen there, where it's easily testable, and re-usable by future components.

When we're done with the other components, we already have a fully-functional (albeit ugly) app. We can even start giving it to very early adopters (probably internally), to test the interactions, hook it up to other parts of the system (ex: a backend), etc.

A fully-functional 'no-CSS' version of the app

The last step is adding the CSS (pardon my poor designer skills), tweaking the components to output the necessary DOM elements with correct class names, and voilà!

The final version of the app with CSS

You can find a running version of the app here.

Using a different rendering engine

As I mentioned earlier, one advantage of having most (if not all) of the "brains" of our app contained in these functions operating on plain data is that we are not particularly tied to any DOM/UI framework or library. We can actually switch React for a different "rendering engine".

To demonstrate this point, let's re-implement the whole rendering portion of the app using D3.js. "Wait, what? I thought D3.js was for creating cool SVG charts?" Well, that's true, but D3.js handles plain HTML just fine. In fact, "d3" actually stands for "data-driven documents", which fits well with the "data-centric" approach we followed here. D3.js is also different enough from React to make it interesting to re-implement the app with. But it's also similar in the way that it follows the "given this data describe what the DOM should look like" principle, also optimizing re-renders with its "enter, update, exit selections" (React uses a "virtual DOM"). Granted, using D3.js for the whole app is a little abusive and not really something I'd recommended, but we're only trying to illustrate a point.

We'll use a single function render() called every time the app state changes:

// engine.js
var el;

function create(element) {
  el = element;
  app.addChangeListener(render);
}

function destroy() {
  app.removeChangeListener(render);
}

function render() {
  var selection = d3.select(el);
  renderGame(selection);
  renderUnitList(selection.select('.js-UnitList'));
  // ...
}

Let's look at the D3.js equivalent of the UnitList React component seen earlier:

// unitList.js (simplified)
function renderUnitList(selection) {
  var data = app.projections.unitList().toJS();

  // Data join
  var unit = selection.selectAll('.unit')
    .data(data, function(d) { return d.id; })
    .order();

  // Enter
  var enter = unit.enter().append('p').attr('class', 'unit');
  enter.append('strong').text(function(d) { return d.name; });
  enter.append('span').attr('class', 'count');
  enter.append('button')
    .attr('class', 'select')
    .text('+')
    .on('click', function(d) { app.actions.select(d.id); });
  enter.append('span').text(' ');
  enter.append('button')
    .attr('class', 'remove')
    .text('-')
    .on('click', function(d) { app.actions.remove(d.id); });

  // Enter + Update
  unit.select('.count')
    .text(function(d) { return '(' + d.count + ')'; });
  unit.select('.select')
    .style('opacity', function(d) { return d.isSelectable ? 1 : 0; });
  unit.select('.remove')
    .style('opacity', function(d) { return d.isRemovable ? 1 : 0; });

  // Exit
  unit.exit().remove();
}

Notice that, like the React component, this D3.js code is simply a mapping of the app.projections.unitList() data to DOM elements and attributes, and some wiring to hook up app.actions.select(unit) and app.actions.remove(unit) to DOM events. The rendering function stays simple and doesn't contain any other logic.

Going further

This concludes our example. By not using any framework or library (other than immutable of course) for the "data" part of the app that deals with the state logic, I hoped to keep the focus on the concepts and process of thinking about UIs in terms of data. Of course, for a "real" production app, I'd recommend using one of the proven patterns and library out there.

Flux is one of those patterns, and is very similar in its approach. You'll find the concept of "actions" again, which are data objects that will cause the state to update. This state is managed in different "stores", and it's up to you to define which stores contain "core state" and which contain "derived state" (using store dependencies, through a waitFor method). RefluxJS is an interesting Flux implementation with a slightly different take: stores can listen to other stores, so it has a more explicit "core state vs. projections" proposition.

Of course, our example was purposely simple, and additional things would need to be thought of in a real app. For example, where would we put the I/O (talking to a server via HTTP, changing the browser URL, persisting data in the browser's local storage, etc.)? How would we handle the asynchronous nature of web apps (ex: responses coming back from an HTTP request)? Could we sometimes perform some validations in order to prevent the app from getting in a "bad state"? Maybe we would need a "commands" layer between user interactions and the "actions" that change the app state, but that might mean adding a "queue" for the actions to make sure each command handler sees the most up-to-date state before deciding what to do. Also, we recalculate "projections" every time we render, which could be expensive if they require a lot of computation. Maybe we'll need to wrap them in some stateful object, and only recalculate them when the part of the state they use has changed.

Conclusion

To sum up, we built an small app by thinking of the UI as something that can be completely described by a piece of data called the application state. Interacting with the app means taking the current state and returning a new one. We used pure functions to implement all our application logic, and saw some advantages of this approach in testing, reasoning about the application, and not being tied to a particular rendering engine.

As a reminder, the full code for the example app is available on GitHub, as well as a running version of the app.

Using environment variables with Webpack and Divshot

2014-09-26T00:00:00Z

If you haven't heard of Webpack, it's a great way to bundle modules of any type (scripts, styles, images, etc.), into static files ready for deployment. It comes with a dev server for local development, with fast incremental rebuilds.

Divshot is a great service that lets you easily deploy those files right from the command line. Think Heroku, but for static web apps. Divshot's command-line tool provides a server that lets you test your build locally.

A common problem in static web apps is configuration. A Node.js app for example, can read values directly from environment variables (var apiUrl = process.env.API_URL). Of course, that's not possible for the web app, which is delivered and run in the user's browser.

Divshot mimics environment variables for your deployed web apps, by allowing you to set values like:

$ divshot env:add development API_URL=http://development-api.example.com

It will then deliver a file at the /__/env.js URL that looks like:

window.__env = {
  API_URL: 'http://development-api.example.com'
};

And your web app can use this global __env object.

What would be nice would be to use this locally, both for development with the Webpack dev server, and to test a build with the Divshot CLI's server.

Situation

The Divshot server expects a .env.json file in the project directory, and uses it to serve a __/env.js file
The .env.json file used by Divshot server is static, i.e. we can't directly use environment variables set in our terminal
The Webpack dev server can serve any static file from the project directory, but we need to create a __/env.js file ourselves

Proposed solution

Your index.html in the project root directory looks something like:

<!DOCTYPE html>
<html>
  <!-- ... -->

  <body>
    <p>Loading app...</p>
    <script type="text/javascript" src="/__/env.js"></script>
    <script type="text/javascript" src="bundle.js"></script>
  </body>

</html>

Create a .env.js file that looks something like:

module.exports = {
  API_URL: process.env.API_URL || 'http://localhost:8081'
};

Create a scripts/ directory with two files.

The first, scripts/buildenv.js, will create __/env.js for the Webpack dev server and .env.json for the Divshot static server:

require('shelljs/global');
var util = require('util');

var output;
var env = require('../.env.js');

// Used for local development with webpack-dev-server
output = util.inspect(env, {depth: null});
output = 'window.__env = ' + output + ';\n';
output.to('__/env.js');

// Used to test build with divshot server
output = JSON.stringify(env, null, 2);
output = output + '\n';
output.to('.env.json');

The second, scripts/build.js, will build the Webpack app ready for deployment in the dist/ directory:

require('shelljs/global');

console.log('Cleaning output directory "dist/"...');
rm('-rf', 'dist');
mkdir('-p', 'dist');

console.log('Bundling all the things...');
exec('webpack --colors --progress');

console.log('Copying "index.html"...');
cp('index.html', 'dist/index.html');

console.log('Build successfull');

Make sure to update your .gitignore with the necessary things:

.divshot-cache
dist
__
.env.json

We can create npm scripts in our package.json to easily call what we just created:

{
  "scripts": {
    "start": "npm run build-env && webpack-dev-server --devtool eval-source-map --cache --colors --progress",
    "build-env": "node scripts/buildenv",
    "build": "node scripts/build",
    "server": "npm run build-env && divshot server -p 8080",
    "deploy": "divshot push"
  }
}

Now, if we want to work on the app, hitting a particular API, we can do:

$ export API_URL=http://localhost:3001
$ npm start

If we want to deploy, we'll first build the app:

$ npm run build

Then we can test our build, for example hitting a remote development API:

$ export API_URL=http://development-api.example.com
$ npm run server

If all seems to work, we can deploy!

$ npm run deploy

Integrating D3.js visualizations in a React app

2014-09-07T00:00:00Z

I've been working with D3.js and React lately, and in this post I wanted to share a few ways I found in building components and the interface between them. I think they help make both libraries work together quite nicely.

We'll be building a small example to illustrate this, and you can find the full code on GitHub, as well as a running version of the app. I'll also assume that you have some basic knowledge of React and D3.js.

Three simple guidelines

I think that using React and D3.js together makes sense because they share a common philosophy of "give me a set of data, tell me how it should be rendered, and I'll figure out which parts of the DOM to update for you". Indeed, React has its virtual DOM diffs, and D3.js has its update selections, making both quite efficient in the business of keeping the UI in sync with data changes.

I find that making D3 components play nice inside a React app becomes easier when you follow these simple guidelines:

(1) "One Source Of Truth": The D3 visualization should get all of the data it needs to render passed down to it. In this example, we'll see that the single source of truth is in the main <App> React component's state, and it is used by the D3 component (d3Chart) and other React components (for example <Stats>).

(2) "Stateless All The Things": This is related to (1). D3 and React components alike should be as stateless as possible, i.e. they shouldn't hide/encapsulate something that makes them render differently given the same "input". In this example, you'll notice that if you call d3Chart.update() at anytime with the same arguments, you always get the same result on screen.

(3) "Don't Make Too Many Assumptions": This is related to (1) and (2). Components shouldn't make too many assumptions about how they will be used. In this example, we'll see that d3Chart doesn't prescribe when tooltips should show, it only shows whatever it receives in the tooltips array. This allows us to show tooltips on hover, but also to easily create a "show/hide all tooltips" toggle.

Enough "theory", let's write some code.

A first basic chart

We'll call our D3.js chart component d3Chart. Let's define its public interface, which also represents its "lifecycle":

// d3Chart.js

var d3Chart = {};

d3Chart.create = function(el, props, state) {
  var svg = d3.select(el).append('svg')
      .attr('class', 'd3')
      .attr('width', props.width)
      .attr('height', props.height);

  svg.append('g')
      .attr('class', 'd3-points');

  this.update(el, state);
};

d3Chart.update = function(el, state) {
  // Re-compute the scales, and render the data points
  var scales = this._scales(el, state.domain);
  this._drawPoints(el, scales, state.data);
};

d3Chart.destroy = function(el) {
  // Any clean-up would go here
  // in this example there is nothing to do
};

Notice how the D3 component is completely stateless (guideline #2), i.e. it doesn't "hang on" to anything and gets everything it needs to render passed down to it from whatever code is using it. I find that doing this makes it easier to use in different contexts (in our case, we'll use it inside a React component).

The _drawPoints() function is your usual D3.js code with "enter", "update", and "exit" patterns:

// d3Chart.js

d3Chart._drawPoints = function(el, scales, data) {
  var g = d3.select(el).selectAll('.d3-points');

  var point = g.selectAll('.d3-point')
    .data(data, function(d) { return d.id; });

  // ENTER
  point.enter().append('circle')
      .attr('class', 'd3-point');

  // ENTER & UPDATE
  point.attr('cx', function(d) { return scales.x(d.x); })
      .attr('cy', function(d) { return scales.y(d.y); })
      .attr('r', function(d) { return scales.z(d.z); });

  // EXIT
  point.exit()
      .remove();
};

Now let's use this chart in a <Chart> React component:

// Chart.js

var d3Chart = require('./d3Chart');

var Chart = React.createClass({
  propTypes: {
    data: React.PropTypes.array,
    domain: React.PropTypes.object
  },

  componentDidMount: function() {
    var el = this.getDOMNode();
    d3Chart.create(el, {
      width: '100%',
      height: '300px'
    }, this.getChartState());
  },

  componentDidUpdate: function() {
    var el = this.getDOMNode();
    d3Chart.update(el, this.getChartState());
  },

  getChartState: function() {
    return {
      data: this.props.data,
      domain: this.props.domain
    };
  },

  componentWillUnmount: function() {
    var el = this.getDOMNode();
    d3Chart.destroy(el);
  },

  render: function() {
    return (
      <div className="Chart"></div>
    );
  }
});

Notice how we hook into React's lifecycle methods componentDidMount, componentDidUpdate, componentWillUnmount, which fit well with our D3 chart's create, update, destroy functions.

Finally let's create our main <App> React component, and use <Chart> to plot some data:

// App.js

var Chart = require('./Chart');

var sampleData = [
  {id: '5fbmzmtc', x: 7, y: 41, z: 6},
  {id: 's4f8phwm', x: 11, y: 45, z: 9},
  // ...
];

var App = React.createClass({
  getInitialState: function() {
    return {
      data: sampleData,
      domain: {x: [0, 30], y: [0, 100]}
    };
  },

  render: function() {
    return (
      <div className="App">
        <Chart
          data={this.state.data}
          domain={this.state.domain} />
      </div>
    );
  }
});

React.renderComponent(App(), document.body);

And behold! We have a pretty chart with circles:

A basic D3.js chart wrapped in a React component

Adding pagination and statistics widgets

At the top, we have our main <App> React component whose state is the "One Source Of Truth" (guideline #1). At the bottom, we have our d3Chart which is actually quite "dumb": you pass it an object with data and domain properties, and it just renders what it gets. These two things make it rather easy to add features.

First, let's add some pagination controls which will allow us to explore a larger data set than we have room to display. We create a <Pagination> React component that will shift the domain and data displayed when the user clicks "Next" or "Previous":

// Pagination.js

var Pagination = React.createClass({
  propTypes: {
    domain: React.PropTypes.object,
    getData: React.PropTypes.func,
    setAppState: React.PropTypes.func
  },

  render: function() {
    return (
      <p>
        {'Pages: '}
        <a href="#" onClick={this.handlePrevious}>Previous</a>
        <span> - </span>
        <a href="#" onClick={this.handleNext}>Next</a>
      </p>
    );
  },

  handlePrevious: function(e) {
    e.preventDefault();
    this.shiftData(-20);
  },

  handleNext: function(e) {
    e.preventDefault();
    this.shiftData(+20);
  },

  shiftData: function(step) {
    var newDomain = _.cloneDeep(this.props.domain);
    newDomain.x = _.map(newDomain.x, function(x) {
      return x + step;
    });
    var newData = this.props.getData(newDomain);
    this.props.setAppState({
      data: newData,
      domain: newDomain
    });
  }
});

We update our main <App> component to support this new functionality:

// App.js

var Pagination = require('./Pagination');

var App = React.createClass({
  getInitialState: function() {
    var domain = [0, 30];
    return {
      data: this.getData(domain),
      domain: {x: domain, y: [0, 100]},
    };
  },

  _allData: [/* some big dataset, too much to display at once */],

  getData: function(domain) {
    return _.filter(this._allData, this.isInDomain.bind(null, domain));
  },

  isInDomain: function(domain, d) {
    return d.x >= domain[0] && d.x <= domain[1];
  },

  render: function() {
    return (
      <div className="App">
        <Pagination
          domain={this.domain}
          getData={this.getData}
          setAppState={this.setAppState} />
        <Chart
          data={this.state.data}
          domain={this.state.domain} />
      </div>
    );
  },

  setAppState: function(partialState, callback) {
    return this.setState(partialState, callback);
  }
});

That's it, we now have pagination! The <Pagination> component changes our "One Source Of Truth" through the convenience function setAppState(), which triggers a re-render and the new data and domain get passed down to the <Chart> component, causing d3Chart to display the correct "page" of data. If we wanted to remove the functionality, or use a different widget for it, all we need to do is remove or replace <Pagination ... /> in App.render().

Adding the pagination widget

We can also do something else with App.state.data. For example, let's add a <Stats> widget that will show some fancy statistics on the data being displayed:

// Stats.js

var Stats = React.createClass({
  propTypes: {
    data: React.PropTypes.array
  },

  render: function() {
    var data = this.props.data;
    return (
      <div className="Stats">
        {this.renderCount(data)}
        {this.renderAverage(data)}
      </div>
    );
  },

  renderCount: function(data) {
    return (
      <div className="Stats-item">
        {'Count: '}<strong>{data.length}</strong>
      </div>
    );
  },

  renderAverage: function(data) {
    var avg;
    var n = data.length;
    if (!n) {
      avg = '-';
    }
    else {
      var sum = _.reduce(data, function(sum, d) {
        return sum + d.z;
      }, 0);
      avg = Math.round(sum/n * 10)/10;
    }
    return (
      <div className="Stats-item">
        {'Average size: '}<strong>{avg}</strong>
      </div>
    );
  }
});

And back in the main <App> component, all we need to do is drop in our new <Stats> component:

// App.js

var Stats = require('./Stats');

var App = React.createClass({
  // ...

  render: function() {
    return (
      <div className="App">
        <Pagination
          domain={this.domain}
          getData={this.getData}
          setAppState={this.setAppState} />
        <Chart
          data={this.state.data}
          domain={this.state.domain} />
        <Stats data={this.state.data} />
      </div>
    );
  }
});

Thanks to "One Source Of Truth", we know that the statistics shown always correspond to the data displayed on the D3.js chart.

Adding the statistics widget

Adding tooltips

The last thing we'll add to this example are tooltips showing the number value (size) of each circle in the visualization.

We want a tooltip to appear when we hover the mouse over a circle. Since the D3 chart created the element corresponding to the circle, we need some way for d3Chart to "tell" its parents that a mouseover event happened on a circle. There are a couple ways we can do that. Here we'll just use a simple Node.js EventEmitter, that we'll call dispatcher:

// d3Chart.js

var EventEmitter = require('events').EventEmitter;

d3Chart.create = function(el, props, state) {
  // ...

  var dispatcher = new EventEmitter();
  this.update(el, state, dispatcher);

  return dispatcher;
};

d3Chart.update = function(el, state, dispatcher) {
  // ...
  this._drawPoints(el, scales, state.data, dispatcher);
};

d3Chart._drawPoints = function(el, scales, data, dispatcher) {
  // ...

  // ENTER & UPDATE
  point.attr('cx', function(d) { return scales.x(d.x); })
      .attr('cy', function(d) { return scales.y(d.y); })
      .attr('r', function(d) { return scales.z(d.z); });
      .on('mouseover', function(d) {
        dispatcher.emit('point:mouseover', d);
      })
      .on('mouseout', function(d) {
        dispatcher.emit('point:mouseout', d);
      });
  // ...
};

Notice that the d3Chart receives domain and data from "upstream", i.e. its parents <Chart> and <App>. I like to think of this as data flowing down. Using the dispatcher, we're able to send the mouseover and mouseout events and their associated data back "upstream". This is data flowing back up.

Now why go through all this trouble instead of just showing the tooltip right in d3Chart? Remember, we're trying to follow "Don't Make Too Many Assumptions" (guideline #3), and thus we don't want to assume that the code using d3Chart wants to show a tooltip when a circle gets hovered. We're just providing the information "hey, this circle was hovered".

Additionally, if we were to display a tooltip directly, we would introduce state in d3Chart, which goes against "Stateless All The Things" (guideline #2). Indeed, given the same domain and data, the chart could be rendered differently (whether a circle is hovered or not). The outside code would have no way to know what state the chart is rendered in, or any control over it.

So now that we have these mouse events flowing back up, let's do something with them. We add a tooltip object in our "One Source Of Truth":

// App.js

var App = React.createClass({
  getInitialState: function() {
    var domain = [0, 30];
    return {
      data: this.getData(domain),
      domain: {x: domain, y: [0, 100]},
      tooltip: null
    };
  },

  // ...
});

And in <Chart>, we listen to the dispatcher to update the tooltip object when a mouse event happens:

// Chart.js

var Chart = React.createClass({
  propTypes: {
    data: React.PropTypes.array,
    domain: React.PropTypes.object,
    setAppState: React.PropTypes.func
  },

  dispatcher: null,

  componentDidMount: function() {
    var el = this.getDOMNode();
    var dispatcher = d3Chart.create(el, {
      width: '100%',
      height: '300px'
    }, this.getChartState());

    dispatcher.on('point:mouseover', this.showTooltip);
    dispatcher.on('point:mouseout', this.hideTooltip);
    this.dispatcher = dispatcher;
  },

  componentDidUpdate: function(prevProps, prevState) {
    var el = this.getDOMNode();
    d3Chart.update(el, this.getChartState(), this.dispatcher);
  },

  // ...

  showTooltip: function(d) {
    this.props.setAppState({tooltip: d});
  },

  hideTooltip: function() {
    this.props.setAppState({tooltip: null});
  }
});

This is nice, but we're still not showing any tooltips. Let's add a tooltips array that gets passed to d3Chart.update(), as well as a function to draw the tooltips with D3:

// d3Chart.js

d3Chart.update = function(el, state, dispatcher) {
  // ...
  this._drawTooltips(el, scales, state.tooltips);
};

d3Chart._drawTooltips = function(el, scales, tooltips) {
  var g = d3.select(el).selectAll('.d3-tooltips');

  var tooltipRect = g.selectAll('.d3-tooltip-rect')
    .data(tooltips, function(d) { return d.id; });

  // ENTER
  tooltipRect.enter().append('rect')
      .attr('class', 'd3-tooltip-rect')
      .attr('width', TOOLTIP_WIDTH)
      .attr('height', TOOLTIP_HEIGHT);

  // ENTER & UPDATE
  tooltipRect.attr('y', function(d) { return scales.y(d.y) - scales.z(d.z)/2 - TOOLTIP_HEIGHT; })
      .attr('x', function(d) { return scales.x(d.x) - TOOLTIP_WIDTH/2; });

  // EXIT
  tooltipRect.exit()
      .remove();

  var tooltipText = g.selectAll('.d3-tooltip-text')
    .data(tooltips, function(d) { return d.id; });

  // ENTER
  tooltipText.enter().append('text')
      .attr('class', 'd3-tooltip-text')
      .attr('dy', '0.35em')
      .attr('text-anchor', 'middle')
      .text(function(d) { return d.z; });

  // ENTER & UPDATE
  tooltipText.attr('y', function(d) { return scales.y(d.y) - scales.z(d.z)/2 - TOOLTIP_HEIGHT/2; })
      .attr('x', function(d) { return scales.x(d.x); });

  // EXIT
  tooltipText.exit()
      .remove();
};

Notice another instance of "Don't Make Too Many Assumption". The D3 chart asks for a tooltips array (vs. a single tooltip object), because hey, who tells me you won't want to show more than one tooltip at once? (We'll actually see this come up in just a bit.)

Let's construct this tooltips array in <Chart> and pass it down with domain and data to the D3 chart:

// Chart.js

var Chart = React.createClass({
  propTypes: {
    data: React.PropTypes.array,
    domain: React.PropTypes.object,
    tooltip: React.PropTypes.object,
    setAppState: React.PropTypes.func
  },

  // ...

  componentDidUpdate: function(prevProps, prevState) {
    var el = this.getDOMNode();
    d3Chart.update(el, this.getChartState(), this.dispatcher);
  },

  getChartState: function() {
    return {
      data: this.props.data,
      domain: this.props.domain,
      tooltips: [this.props.tooltip]
    };
  },

  // ...
});

And voilà! We have tooltips on hover:

Adding tooltips on hover

Now, let's see why we went through all this trouble of making things stateless and not making too many assumptions. Let's say we want to add a widget with buttons that allow you to "Show all" or "Hide all" tooltips. The way we have everything set up, it will be rather easy!

We add a boolean flag showingAllTooltips to our "One Source Of Truth":

// App.js

var App = React.createClass({
  getInitialState: function() {
    var domain = [0, 30];
    return {
      data: this.getData(domain),
      domain: {x: domain, y: [0, 100]},
      tooltip: null,
      showingAllTooltips: false
    };
  },

  // ...
});

We create a <ShowHideTooltips> React component that will toggle showingAllTooltips between true and false (the code for that widget isn't very interesting, so I won't show it here).

And finally we tweak the way we construct the tooltips array passed to our D3 chart in <Chart>:

// Chart.js

var Chart = React.createClass({
  propTypes: {
    data: React.PropTypes.array,
    domain: React.PropTypes.object,
    tooltip: React.PropTypes.object,
    showingAllTooltips: React.PropTypes.bool,
    setAppState: React.PropTypes.func
  },

  // ...

  getChartState: function() {
    var tooltips = [];
    if (this.props.showingAllTooltips) {
      tooltips = this.props.data;
    }
    else {
      tooltips = [this.props.tooltip];
    }

    return {
      data: this.props.data,
      domain: this.props.domain,
      tooltips: tooltips
    };
  },

  // ...
});

And there we go, with just a few lines of code we added a "show/hide all tooltips" functionality:

Adding the 'show/hide all tooltips' widget

Conclusion

As a reminder, the full source code for this example is available on GitHub, as well as a running demo version. You'll notice that there are a few additional things, like an animation when you page back and forth (implement in a way that follows the 3 guidelines presented earlier). But to keep this post at a reasonable length, I decided not to cover it here.

I also want to point out that React allows you to work directly with SVG. Although I haven't used it much, I'm sure you could re-create most, if not all, of this example in "pure React". But here I assumed a scenario where you might already have some D3 components you want to re-use, and/or you would rather write the data visualization parts of your app in D3.

There are many ways to organize code in React and D3 (these are libraries that provide you with a set of tools, and don't really prescribe how you should use them). What I presented here is just one way, that I personally find easy to work with and reason about. I'm sure there are other great methods out there, so feel free to share your preferred React and D3 integration!

Parse data files using Node.js streams

2013-07-09T00:00:00Z

Streams and pipes are an important part of the Unix philosophy:

Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

-- Doug McIlroy, inventor of Unix pipes and one of the founders of the Unix tradition

This means that we can have small, focused programs that are easily assembled together to tackle more complex problems. Sort of like building awesome things with Lego blocks.

This is also possible in Node.js thanks to the Stream interface, and its .pipe() method.

I won't go over the general aspects of streams and pipes in Unix and Node.js. For that, I strongly recommend reading @substack's excellent Stream Handbook. In this post, I'd like to show how I started using them.

During one of my projects, I had to parse text files containing data from some "legacy" system in order to make it more usable and load it into a database. The parsing had a few different steps to it (translate the file to JSON, add and modify data for each record, etc.), and the files varied in size and could get relatively big.

I thought this would be an ideal candidate for streams. I could split each step into its own "streaming program", assemble them, and pipe the data through the resulting chain. It would also be fast and memory-efficient, even with bigger files. With streams, the whole file isn't loaded into memory, rather the data is "buffered" to each program of the chain and throttled to the speed at which they can process the data.

You can follow along with the code for the example in this article at https://github.com/nicolashery/example-stream-parser.

The data file to parse

In this example, I'm going to use some silly made-up data (its format inspired by some files I've had to parse).

Let's say our objective is to turn a data.csv file containing something like:

Game Export (v1.2)
GameId,1234567
Player,1,Homer Simpson
Player,2,Bart Simpson
Player,3,Marge Simpson
Map,101,Crossroads
Time Range,2013-01-11 02:50:40,2013-01-12 05:34:56
Number of Records,100
Index,Timestamp,Event Type,Player Id,Event Data
1,2013-01-11 02:54:42,ResourcesGathered,3,"resource_type=Wood, quantity=11"
2,2013-01-11 03:00:26,ResourcesGathered,2,"resource_type=Gold, quantity=7"
3,2013-01-11 03:05:42,ResourcesGathered,1,"resource_type=Gold, quantity=2"
4,2013-01-11 03:08:05,UnitTrained,3,"unit_type=Knight, health=270, damage=12-15"
5,2013-01-11 03:24:05,DestroyedEnemy,1,"unit_type=Pig Farm"

Into this more usable format:

{"header":{"Title":"Game Export (v1.2)","GameId":"1234567","Players":[{"id":"1","name":"Homer Simpson"},{"id":"2","name":"Bart Simpson"},{"id":"3","name":"Marge Simpson"}],"Map":{"id":"101","name":"Crossroads"},"Time Range":{"start":"2013-01-11 02:50:40","end":"2013-01-12 05:34:56"},"Number of Records":"100","Columns":["Index","Timestamp","Event Type","Player Id","Event Data"]}}
{"Index":"1","Timestamp":"2013-01-11 02:54:42","Event Type":"ResourcesGathered","Player Id":"3","Event Data":{"resource_type":"Wood","quantity":"11"}}
{"Index":"2","Timestamp":"2013-01-11 03:00:26","Event Type":"ResourcesGathered","Player Id":"2","Event Data":{"resource_type":"Gold","quantity":"7"}}
{"Index":"3","Timestamp":"2013-01-11 03:05:42","Event Type":"ResourcesGathered","Player Id":"1","Event Data":{"resource_type":"Gold","quantity":"2"}}
{"Index":"4","Timestamp":"2013-01-11 03:08:05","Event Type":"UnitTrained","Player Id":"3","Event Data":{"unit_type":"Knight","health":"270","damage":"12-15"}}
{"Index":"5","Timestamp":"2013-01-11 03:24:05","Event Type":"DestroyedEnemy","Player Id":"1","Event Data":{"unit_type":"Pig Farm"}}

A first basic stream

Let's create a first Node.js stream to pipe our data file to. This stream will do nothing really, just pass the data as-is to the next stream in the chain.

For that, we will be using the very handy Transform stream, which is both Writable and Readable (in other words, data comes in and also comes out).

All we need is to create a Transform stream and implement its _transform method. For now we'll just push data without doing anything with it.

// parser.js
var Transform = require('stream').Transform;

var parser = new Transform();
parser._transform = function(data, encoding, done) {
  this.push(data);
  done();
};

Now let's add a bit to pipe data coming in from stdin, through our parser, and then out to stdout:

// Pipe the streams
process.stdin
.pipe(parser)
.pipe(process.stdout);

// Some programs like `head` send an error on stdout
// when they don't want any more data
process.stdout.on('error', process.exit);

Now if we run in the terminal:

$ cat data.csv | node parser
Game Export (v1.2)
GameId,1234567
Player,1,Homer Simpson
# ...

We should see our data being streamed through the parser correctly!

Parse the CSV file

The first thing I'd like to do is split those comma-separated values into objects I can manipulate.

Instead of reinventing the wheel, an npm search brings up csv-streamify which I can use. It will be easy to plug in my own parser logic on top of it, thanks to .pipe(). This is the beauty of streams.

I'll set the objectMode option to true for both csv-streamify and my parser to work with JavaScript objects (instead of strings). The stdout stream expects strings though, so I'll convert back to them with JSONStream:

var Transform = require('stream').Transform
  , csv = require('csv-streamify')
  , JSONStream = require('JSONStream');

var csvToJson = csv({objectMode: true});

var parser = new Transform({objectMode: true});
parser._transform = function(data, encoding, done) {
  this.push(data);
  done();
};

var jsonToStrings = JSONStream.stringify(false);

process.stdin
.pipe(csvToJson)
.pipe(parser)
.pipe(jsonToStrings)
.pipe(process.stdout);

Now if we run our parser we should see the CSV file is being turned into arrays:

$ cat data.csv | node parser
["Game Export (v1.2)"]
["GameId","1234567"]
["Player","1","Homer Simpson"]
# ...

Add some data transforms

Now that we have some basic streams wired up, it's time to add some data manipulation to the parser.

We can, for instance, aggregate the header into one object:

parser.header = null;
parser._rawHeader = [];
parser._transform = function(data, encoding, done) {
  if (!this.header) {
    this._rawHeader.push(data);
    if (data[0] === 'Index') {
      // We reached the last line of the header
      this.header = this._rawHeader;
      this.push({header: this.header});
    }
  }
  // After parsing the header, push data rows
  else {
    this.push({row: data});
  }
  done();
};

Running the program now will output:

$ cat data.csv | node parser
{"header":[["Game Export (v1.2)"],["GameId","1234567"],["Player","1","Homer Simpson"],["Player","2","Bart Simpson"],["Player","3","Marge Simpson"],["Map","101","Crossroads"],["Time Range","2013-01-11 02:50:40","2013-01-12 05:34:56"],["Number of Records","100"],["Index","Timestamp","Event Type","Player Id","Event Data"]]}
{"row":["1","2013-01-11 02:54:42","ResourcesGathered","3","resource_type=Wood, quantity=11"]}
# ...

I won't describe the rest of the transform logic to get to the result mentioned at the beginning of this post, but you can find the full code on GitHub.

More streaming data fun

From here, there are many things we might want to do with our parsed data. Again, thanks to the composable and elegant interface of Node.js streams, this is very easy to do.

We already have a command line tool working (examples/cmd.js), which allows us to use it with other Unix tools, such as cat, head, wc, etc.

We could also pipe the parser into another stream that would manipulate the data further and possibly save it to a database or HTTP API (examples/transform.js).

Another usage could be to include the parser into a server (request and response passed by http.Server are also streams!). This way we could provide a "parser-as-a-service" through HTTP (examples/server.js), like what @maxogden describes in "Gut: Hosted Open Data Filet Knives".

Conclusion

Streams made it really pleasant to build a relatively complex data file parser out of small and simple components. On top of that, I don't have to worry about how big the data files get. The parser will remain fast and memory-efficient thanks to the pipe interface between the streams.

If you weren't familiar with streams before, I hope this example made you curious and want to give them a try.

Or maybe you've already worked with Node.js streams to process data? Feel free to share your experience!

Creating a fast and mobile-friendly website with Jekyll

2013-06-21T00:00:00Z

Jekyll is a static site generator: it transforms plain text files (in Markdown for content, Liquid for templates) into an HTML, CSS, and JavaScript website ready for deployment.

There are other options out there to create a website or blog, like the popular Wordpress, or the more recent Squarespace and Ghost. These are dynamic, which means your content is stored in a database and is filled in templates when a user visits the site. They also have an administration section, which allows you to log in and edit your content. That usually makes the website or blog easier to update, especially for less technical people.

But using a static site generator like Jekyll has a few advantages over these other solutions:

The website is usually very fast: Since it's only static files, there are no round-trips to a database, and web servers have become really good at serving plain files.
It's also very scalable: Even if you have a ton of traffic, it's easier for a static web server to handle than, for instance, a PHP Wordpress application.
It's very easy to deploy: There are many static file web server options out there, you don't need to hook up and configure a database, or install and maintain any application.
You really own your content: It's all plain text files, usually in Markdown, that you can safely keep in your Dropbox or GitHub repository.
It's extensible: Since a static site generator like Jekyll only focuses on turning plain text files into HTML, and provides you with a plugin interface, you can basically do anything: add third-party libraries like Disqus or Google Analytics, create your own logic to insert images into pages, etc.

Building a website "from scratch" using Jekyll might feel a little complicated at first, as there are a lot of moving parts, things to configure, etc. In this post, I'd like to walk you through how I went about it, and hopefully that will help you get started more quickly and serve as a checklist of the things you need to think about for your website. Keep in mind that this is just one way of doing things. You are more than welcome to adapt this workflow to your liking!

This is also a fairly long read, since I wanted to include the end-to-end process from installing the tools to deploying the site. The table of contents below will help you jump to different sections.

If you want, you can follow along with this source code.

Installing Jekyll

We will use Jekyll (Ruby) to generate the HTML files, the Grunt build tool (Node.js) to compile and minify CSS and JavaScript files, and Pygments (Python) for code syntax highlighting in our posts.

This means you will need to have installed:

Ruby with RubyGems
Node.js with NPM
Python with pip

If you don't have all of these installed and are using a Mac, you can check out my Mac Dev Setup guide for help.

Install Jekyll with:

$ gem install jekyll

Jekyll has great documentation to help you learn how to use it.

A basic directory structure looks something like this:

.
├── _includes/
|   ├── footer.html
|   ├── header.html
|   └── posts.html
├── _plugins/
|   ├── asset_url.rb
|   └── image_tag.rb
├── _layouts/
|   ├── default.html
|   ├── page.html
|   └── post.html
├── _posts/
|   ├── 2013-02-11-pictures-of-cats.md
|   └── 2013-01-31-hello-world.md
├── _site/
├── _config.yml
└── index.html

The static website is generated in the _site folder. Any other file or directory than the ones listed above (like CSS and JavaScript files) will be copied over into the _site directory, except if you explicitly exclude them in _config.yml.

We use Git for version control, so let's immediately add _site to our .gitignore file.

Installing Grunt

Jekyll processes our layout and content text files into HTML. We will use Grunt to process our CSS and JavaScript files.

Install the Grunt command line tool with:

$ npm install -g grunt-cli

Let's create directories for our source files, and add them to the exclude list in _config.yml so they are not copied to the site:

...
├── css/
└── js/

If you use preprocessors, like Sass, LESS, or CoffeeScript, you can add corresponding sass/, less/, or coffee/ directories.

We also create directories for our concatenated (debug/) and minified (build/) files, which will be copied to the site, and add them to .gitignore so they are not tracked in version control:

...
├── debug/
└── build/

Finally, we create an assets/ directory for larger files such as images, also excluded from version control in .gitignore:

...
└── assets/

A word of explanation: we will create our stylesheets and scripts in their source directories, using as many files as we like to keep the code maintainable. We then combine those files into one CSS file style.css and one JavaScript file main.js in the debug/ directory, which keeps the number of HTTP requests to a minimum for best performance. To do so, we can use the following Grunt plugins, depending on which tools you use (I like Compass for CSS and Browserify for JavaScript):

For deployment, we will also minify the files into style.min.css and main.min.js in the build/ directory, using:

All these tasks are defined in the Gruntfile.js of your project. You can check out Grunt's documentation and my example Gruntfile.js for help setting it up.

Workflow

I set up some Grunt tasks to help me automate my workflow. During development, when I'm modifying my site layout, writing content, changing CSS, etc., I open up a first terminal and run:

$ grunt debug

This tasks watches for changes in CSS and JS files (and/or Sass, LESS, CoffeeScript), compiles and concatenates source files in the debug/ directory. In a second terminal I run:

$ grunt server

This task is basically an alias for jekyll serve --watch, which runs the development server on http://localhost:4000/ and also watches for changes in Liquid layout or Markdown content files and rebuilds the site.

Before deployment, I run:

$ grunt build

This compiles, concatenates, and minifies styles and scripts, as well as re-generates the whole Jekyll site, and uses the environment variable JEKYLL_ENV by setting it to "production" to tell my Jekyll templates to point to the minified files in the build/ directory instead of debug/.

We will see later, but to deploy the website and publish changes, I use the command:

$ jekyll-s3

The Grunt tasks described above use a combination of the plugins mentioned earlier. I invite you to check out my Gruntfile.js and use it as inspiration for your own workflow.

CSS

Writing plain CSS has its limits, and there are preprocessors like Sass (Ruby) and LESS (Node.js) that help you overcome them. There are also frameworks like Bootstrap (LESS) and Foundation (Sass) that give you a head start with styles for elements like grids, buttons, typography, etc.

Personally, I use Sass with Compass. I also like Inuit.css as a lightweight, extensible framework. I will explain how to set those up, but you can use whichever tool you prefer.

To install Compass and its Inuit framework plugin:

$ gem install compass compass-inuit

I then configure the Grunt plugin grunt-contrib-compass in my Gruntfile.js and add it to my grunt debug and grunt build tasks.

Using preprocessors helps you keep your CSS maintainable and modular. If interested, you can dive into the work around Object-Oriented CSS (OOCSS) or SMACS. To start simple, I find it a good practice to separate styles specific to certain parts your website in different files, and prefix the CSS rules with those "module" names. For example:

sass/
├── ui/
|   ├── _footer.scss
|   ├── _header.scss
|   ├── _page.scss
|   └── _scaffolding.scss
├── _config.scss
└── main.scss

// _header.scss

.header--nav {
  @extend .nav;
  @extend .nav--banner;

  margin-bottom: 0;
}

// ...

Now is a good time to say a few words on building a mobile-friendly website. It has a become best practice to use Responsive Web Design, i.e. building a site that adapts nicely to different device screen sizes. To do so, add the following in your <head> HTML tag:

<meta name="viewport" content="width=device-width, initial-scale=1.0">

You can then use CSS3 media queries to define rules for certain screen sizes. I like the mobile-first approach, so I often design my site for the smallest screen size first, then add rules to complement or override the styles as the screen gets bigger. For example:

body {
  max-width: 700px;
  padding: 12px;
  padding-bottom: 0;
}

@media only screen and (min-width: 481px) {
  body {
    padding-top: 24px;
    padding-right: 24px;
    padding-left: 24px;
  }
}

Finally, include the link to the concatenated stylesheet in the <head> tag of _layouts/default.html:


{% if site.env == 'production' %}
  <link rel="stylesheet" href="/build/style.min.css">
{% else %}
  <link rel="stylesheet" href="/debug/style.css">
{% endif %}

JavaScript

I use the Bower package manager to install front-end JS libraries, like jQuery. It has a pretty extensive registry of these front-end components. Some front-end libraries can also be found on NPM.

Bower will install libraries in the components/ directory. For example:

$ bower install jquery fastclick

I then use Browserify and the Grunt plugin grunt-browserify2 to combine these different libraries into one single JS file. My main JS file will look something like:

// js/main.js

var $ = require('jquery')
  , FastClick = require('fastclick');

// On DOM ready
$(function() {

  // Eliminates the 300ms delay between a physical tap and the firing of a
  // click event on mobile browsers
  new FastClick(document.body);

  // ...

});

To be able to write require('jquery'), you need to use browserify-shim in your Gruntfile.js and provide the path to each component.

In your default.html layout, link to the concatenated JS file just before the closing </body> tag (for best performance):


  <!-- ... -->
  {% if site.env == 'production' %}
    <script src="/build/app.min.js"></script>
  {% else %}
    <script src="/debug/app.js"></script>
  {% endif %}
</body>

Images

Regarding website performance, images are a tricky thing. We took the time to concatenate and minify our CSS and JS files, but a single image can weight more kilobytes than both files combined.

As Dave Rupert's article "Ughck. Images." shows, the solution to "Responsive Images" isn't quite here yet.

In the meantime, I needed something relatively simple that would allow me to include images on a website in a way that keeps it loading fast. I focused on two things:

Lazy load images only once the user scrolls down to them, thanks to the JAIL jQuery plugin
Responsive images that load a different version depending on the screen size and resolution, thanks to a modified version of the picturefill library (small tweak to make it compatible with JAIL)

After installing and setting up the JavaScript libraries, I create a Jekyll plugin called image_tag.rb, that allows me to easily insert images in my Markdown files with a custom Liquid tag:

{% image my-image.png "Image alt text" "Optional image caption" %}

With this in place, pages of the website should feel faster as images aren't loaded until they become visible in the browser viewport.

I chose this solution because it is simple, and my website doesn't use images much, but there are other options out there. For instance, Paul Stamatiou, who also does photograpy, offers a more sophisticated solution in his article "Developing a responsive, Retina-friendly site".

Code blocks

With a blog on programming, I'm going to be using a lot of code examples. It's important that they look good, both on desktop and mobile.

For syntax highlighting, Jekyll has a nice integration with Pygments. To install, run:

$ pip install pygments

And add to your _config.yml file:

pygments: true

I like using fenced code blocks, as found in GitHub-Flavored Markdown, instead of Liquid highlight tags. To do so, I switched my Jekyll Markdown parser to redcarpet:

$ gem install redcarpet

And add to the _config.yml file:

markdown: redcarpet

And now you can include code blocks with:

javascript var msg = 'Hello world!';

For me, this also has the benefit of making the Markdown files of my blog posts compatible with Marked (also check "Strip YAML Front Matter" in Marked's preferences).

As I said, I find it important that the code blocks look good on a small mobile screen as well as the desktop. To achieve this, I first use the following CSS rule:

pre {
  white-space: pre-wrap;
}

This will wrap the code when a line is longer than the screen size, instead of displaying a horizontal scroll bar (I find horizontal scrolling awkward, but that might be just me). Wrapped code isn't great though, and a little difficult to read. To limit this on small screens, diminish the font-size to make as much code as possible fit on one line, using media queries:

pre, pre > code {
  /* Make more code fit on small screens */
  font-size: 14px;
}

@media only screen and (min-width: 481px) {
  pre, pre > code {
    /* Bigger font on bigger screens */
    font-size: 16px;
  }
}

Finally, for the colors of the syntax highlighting, I like the Solarized theme. I put together two CSS stylesheets to use with Pygments and Jekyll, the "Light" and "Dark" versions.

Here I realized the importance of testing a website on an actual mobile device (versus just in a resized desktop browser). Indeed, the Solarized Dark theme was fine on the bright screen of my MacBook Air, but too dark and diffictult to read on my iPhone screen. That's one of the reasons I opted for Solarized Light.

Comments

Many blogs and websites use Disqus to manage their comments. Setting it up with Jekyll is very easy.

First you need to create a Disqus account if you don't have one already, then register your site.

Once that is done, paste the universal code Disqus gives you into a Jekll include, _includes/disqus.html, and replace the shortname with a Liquid output:


var disqus_shortname = '{{ site.disqus.shortname }}';

Then, in your _config.yml, insert your Disqus shortname:

disqus:
    shortname: 'nicolashery'

Now, anywhere on your site you want to add Disqus comments (for instance, in _layouts/post.html), all you have to do is include disqus.html with the Liquid tag:

{% include disqus.html %}

SEO

I'll be honest, I don't know much about Search Engine Optimization. This might be a bit naive, but to me the best way to be on top of search results is to have great content, tell people about it, and if they like it they will link to your content, and that will bring more people and help your ranking.

I did learn however, mostly thanks to Segment.io's article "The Quickest Wins in SEO", that there are a couple basic steps you should take.

First, it is good to include a short meta description in the <head> tag of your page, as Google will display that under the link to your site in the search results:

<meta name="description" content="Short description of my website that will appear in search results.">

Next, add a robots.txt file in your root directory, which tells crawlers (but doesn't force them to) which parts of the site they should index or not. Don't worry about it too much, just use the simplest form of robots.txt:

User-agent: * Allow: /

I'm not sure if this is really SEO-related, but it's also good practice to include an error.html or 404.html file in the root directory, that will be displayed for example if a user mistypes a link to your site. Usually that page displays a short "Not Found" message with a link to other parts of the site (example: GitHub 404).

Finally, include a Sitemap, which is an XML file that helps crawlers find content to index on your site. With Jekyll, all you need to do is copy the sitemap_generator.rb plugin to your _plugins folder. Re-generate the site, and a sitemap.xml file will appear in the _site folder.

RSS feed

An RSS feed is a good way for visitors to keep updated on new blog posts when they add it to their feed reader.

To generate the XML file for your RSS feed, just copy the feed.xml template to your project root directory, and add a link to /feed.xml somewhere on your site.

Analytics

I'm going to explain how to set up Google Analytics on the site, but you can adapt this to other analytics services like Mixpanel, GoSquared, etc. (for a nice list of different services out there, see the Segment.io documentation).

First, register for Google Analytics with your Google account, if you haven't done so already.

Then, add the code snippet provided in the documentation into a Jekyll include, like _includes/google_analytics.html. Replace the tracking code with a Liquid output:


_gaq.push(['_setAccount', '{{ site.google_analytics_id }}']);

You could put your Google Analytics ID in the _config.yml, but it's best not to commit these kind of tokens inside the Git repository. Instead, I set it as an environment variable in the terminal that's running the Jekyll build. In order to do this, I created a simple environment_variables.rb plugin in which I add the line:

site.config['google_analytics_id'] = ENV['GOOGLE_ANALYTICS_ID']

That way I can run:

$ export GOOGLE_ANALYTICS_ID='UA-XXXXX-Y'
$ grunt build

Which will make the environment variable's value available to the Liquid template through site.google_analytics_id.

As explained in Google's documentation, include the snippet just before the closing </head> tag with:


  <!-- ... -->
  {% include google_analytics.html %}
</head>

Deployment

Deploying has a few different pieces that we'll need to select:

A host for the static files (ex: Amazon S3, GitHub Pages)
A registrar for the domain name (ex: Gandi)
Optionally, a separate DNS provider (ex: DNSimple, Amazon Route 53)
Optionally, a CDN (ex: CloudFront)

As mentioned at the beginning of this post, one advantage of a static website is that it can be deployed to just about any host that can serve static files. Here I'll explain how to deploy to Amazon S3, which has a "pay-according-to-traffic" pricing model (it ends up being pretty cheap, unless you really have a ton of traffic). If you don't have many images or other big files, GitHub Pages is also a nice option.

I'm also going to explain how to host your site at the "root" (or "naked" or "apex") domain (i.e. "nicolashery.com" instead of "www.nicolashery.com"), which is a little more tricky. This is my preference, but if you'd like to host at the "www" domain, you should be able to adapt these instructions fairly easily.

Pushing a Jekyll website to Amazon S3 is really easy, thanks to the jekyll-s3 tool. Create an Amazon Web Services (AWS) account if you don't have one already, then install the tool with:

$ gem install jekyll-s3

In the Amazon S3 console, create a bucket (select the "US Standard" region) with a name that matches your domain name, (in my case "nicolashery.com").

Note: Since I'm hosting at the root domain "nicolashery.com", I will also create a second bucket "www.nicolashery.com" that I will leave empty, and configure it to redirect all requests to "nicolashery.com", as explained in the AWS documentation.

Next, create a _jekyll_s3.yml file with your Amazon credentials declared as environment variables:

s3_id: <%= ENV['S3_ID'] %>
s3_secret: <%= ENV['S3_SECRET'] %>
s3_bucket: nicolashery.com
gzip: true

Obtain your Amazon credentials from the AWS console (click on your name in the top right and "Security Credentials"), and set the environment variables:

$ export S3_ID='YOUR_AWS_S3_ACCESS_KEY_ID'
$ export S3_SECRET='YOUR_AWS_S3_SECRET_ACCESS_KEY'

Once that is done, run the S3 bucket configuration tool provided with jekyll-s3, which will prepare it to host a static website (for now, say "no" when it asks you if you want to configure it for the CloudFront CDN):

configure-s3-website --config-file _jekyll_s3.yml

Finally, build your site for production and deploy to Amazon S3 with:

$ grunt build
$ jekyll-s3

You can already visit the live site with the example.com.s3-website-us-east-1.amazonaws.com address provided (replace "example.com" with your domain of course).

To point our domain name to this Amazon S3 bucket, we need to configure its DNS records. If we were hosting at the "www.example.com" domain, all we would need to do is create a CNAME record:

CNAME www.example.com www.example.com.s3-website-us-east-1.amazonaws.com

However, we are hosting at the root domain "example.com", which requires an A record that has to point to an IP address. This is where it gets tricky: the IP address of our Amazon S3 can change.

To work around that, we need to use a separate DNS provider, DNSimple or Amazon Route 53. They both have special ALIAS records (DNSimple ALIAS, Route 53 ALIAS) that allow you to point a root domain ("example.com") to another domain, much like a CNAME.

In the case of DNSimple, we create two records:

ALIAS example.com example.com.s3-website-us-east-1.amazonaws.com CNAME www.example.com www.example.com.s3-website-us-east-1.amazonaws.com

To use a separate DNS provider, you need to configure your registrar (mine is Gandi) to point to its DNS servers.

I won't dive right now into how to configure a CDN like CloudFront. Different CDN, DNS, host combinations could be a whole other article. Feel free to read through the CloudFront documentation, it should be pretty straighforward to set up using jekyll-s3 and Route 53.

Conclusion

Using Jekyll does require a bit of tinkering and setting up, but hopefully this walkthrough will help you get started faster.

I like the fact that creating "from scratch" forces you to learn and understand what goes into making a fast and modern website. I also appreaciate the flexibility it gives you, since you can use plugins and other tools like Grunt to adapt it to what's specific about your project.

Did you find this article helpful? Do you use Jekyll and have a different workflow you'd like to share? Feel free to reach out!

Nicolas Hery

Decoding JSON sum types in Go without panicking

Whether we find them useful or not, sum types exist

My first nil pointer panic in Go was due to lack of sum types

Decoding JSON sum types in Go, take one

How do OpenAPI and Protobuf handle this?

Decoding JSON sum types in Go, take two

Alternative implementations

What Go could have been: V lang?

Examples in other languages

Nesting APIs and ReaderT environments with Servant

Introduction

Server vs. request environment

An example nested API

Nested ReaderT environments

Implementing the request handlers

The Has type class pattern

Wrapping up

Using MonadLogger without LoggingT

What are MonadLogger and LoggingT?

Background

With LoggingT

Without LoggingT

Wrapping up

Comparing request handlers in Scotty, Yesod, and Servant

Context

Non-trivial request handler

Custom monad

Fake cart purchase for a booking site

Key takeaways

Further reading

Wrapping up

Prototyping and data modeling in the browser using TypeScript

Why prototype in the browser?

Why TypeScript?

The example prototype

Project setup

Adding CSS

Using real data

Adding routes with React Router

Managing data with Redux

Basic CRUD

Data modeling

Using HTML forms

Generating random data

Charts and data visualization

Data export

Bonus: Deploying to Netlify

Wrapping up

Moving a blog from Jekyll to Eleventy

Why Eleventy?

Why Netlify?

Migrating from Jekyll

Configuring Eleventy

Syntax highlighting

404 page

RSS feed

Sitemap

SEO

License

Deploying with Netlify

Custom domain with Netlify DNS

Analytics

Commenting

CSS

Google web fonts

Images with Cloudinary

Conclusion

Exploring Isomorphic JavaScript

Simplest example

Sharing state

Simple routing

Shared state and routing

Authentication

Configuration

Disabling isomorphism

Progressive enhancement

Flux

Practical implementation

Conclusion