Getting started with Snowplow Micro

After working with Snowplow for about three years, I finally managed to write my first blog post. Getting started with Snowplow was not easy, partly because I found not mutch content that really explains Snowplow from the ground up. The official Snowplow documentation is solid, but in my opinion it too often assumes technical knowledge that many colleagues in the analytics field simply don’t have if they have primarily worked with packaged analytics solutions like Google Analytics.

What I didn’t realize in my first days with Snowplow is how well Snowplow Micro is suited to understand the data collection with Snowplow holistically. Time to write the article that would have helped me back then.

The Snowplow Pipeline in a nutshell

What is not visible in a packaged analytics solution like Google Analytics is how exactly the data pipeline that processes an event works. Fortunately, this is different with Snowplow. Very simplified, the pipeline consists of four core applications:

Stream Collector: API endpoint that receives the analytics request and forwards it as a message to downstream applications
Schema registry: server which holds event definitions in the form of JSON schemas
Enrich: validates the event against schemas and enriches it with further data via dedicated enrichments e.g. device classification based on the user agent
Loader: loads the event into a data warehouse or file storage e.g. BigQuery or Google Cloud Storage

A Snowplow Analytics event therefore typically runs through the pipeline as follows: Client or server-side Snowplow Tracker/SDK -> Collector -> Enrich -> Loader. More details in the official “How the pipeline works” article from Snowplow.

Enrichment debugging - or how I came across micro

At the beginning of my work with Snowplow enrichments, debugging them was a pain because enrichments were not yet supported in Snowplow Micro. I used Snowplow Mini, a single instance version of Snowplow running on Google Cloud compute engine. The Enrich component of Mini needs to be restarted after every change in the enrichment configuration. That leads to annoying delays in the development and debugging process of enrichments.

Snowplow Micro to the rescue

Micro supports enrichments since April 2023, making it surprisingly similar to a full-fledged Snowplow pipeline, but as the name implies, only for testing and debugging. Besides enriching, Micro can validate events and split them into good and bad event streams. The enriched events are stored in-memory and can be consumed via terminal standard output (a.k.a. stdout) or a simple user interface. The official documentation of Micro is a bit overwhelming because Micro can be configured with plenty of different settings. Let’s focus in this guide on the essentials: debugging web events and enrichments

This guide covers

how to run Micro on your machine
how to send Snowplow events to your local Micro endpoint with the simplest tracker
how to create and use enrichments with Micro
how to connect Micro with a “Schema Server”

Run Micro on your machine

The Micro Docker Image

Only a one-liner is needed to run Micro in its simplest setup. It runs per default on ports 9090, change the host port in case it conflicts with other application using the same port e.g. to 8080:9090.

The --name <<container name>> flag is optional, but I recommend it to be able to restart Micro with the simple command like docker restart <<container name>>. As of writing this guide the current version of the image is 2.1.2, check the changelog on github for the latest version.

docker run --name micro -p 9090:9090 snowplow/snowplow-micro:2.1.2

Run the Docker container

Follow the steps below to run a local instance of Micro:

Install a tool like Ranger Desktop or Docker Desktop to run a Docker container on your machine.
Open ranger or docker desktop which automatically starts the docker daemon
Past the command from above in your terminal and run the container. If it was successful you can see several info logs including [INFO]...- UI available at /micro/ui. That means you can open the Micro user interface via http://localhost:9090/micro/ui. Learn more about the UI in the official documentation.

Sending events to Micro

Events can be send from any Snowplow Tracker to Micro. Let’s use the simple Pixel tracker for the first event:

event_name: page_view
page_title: Testing and debugging | Snowplow Documentation
page_url: https://docs.snowplow.io/docs/testing-debugging/

The URL encoded event properties can be used in a Pixel tracker request like this:

http://localhost:9090/i?e=pv&page=Testing%20and%20debugging%20%7C%20Snowplow%20Documentation&url=https%3A%2F%2Fdocs.snowplow.io%2Fdocs%2Ftesting-debugging%2F&tv=no-js-0.1.0

Past the URL in your browser, hit return and check the UI if the event has successfully reached your local endpoint. You should see one event in your “good” event stream.

Setup enrichments with Micro

Snowplow offers multiple enrichments to modify or enhance events. One of the most powerful enrichments is the JavaScript enrichment, which we will going to set up in Micro in the following.

Example use case:

Let’s use Snowplows documentation site as an example. The page titles on docs.snowplow.io are always ending with the suffix “| Snowplow Documentation”. Lets assume you are picky analyst at Snowplow, who want’s to get rid of the suffix for cleaner reporting. That can be achieved with the enrichment below. It checks if the page title ends with “| Snowplow Documentation”. If thats’s the case, it strips the suffix from the title. This guide is not a deep dive in the JavaScript enrichments, but I may write a dedicated post in the future.

function process(event) {
  let docTitle = event.getPage_title()
  
  if (docTitle != null && docTitle.endsWith(' | Snowplow Documentation')) {
    event.setPage_title(docTitle.split(' | Snowplow Documentation')[0]);
  }

  return [];
}

Prepare the enrichment configuration file:

To use the JavaScript code from a above in the Custom JavaScript enrichment config, we need to encode it to base64 via a visual studio code plugin or e.g. via base64encode.org and paste the encoded string into parameters.script key in the JSON configuration file:

{
  "schema": "iglu:com.snowplowanalytics.snowplow/javascript_script_config/jsonschema/1-0-0",
  "data": {
    "vendor": "com.snowplowanalytics.snowplow",
    "name": "javascript_script_config",
    "enabled": true,
    "parameters": {
      "script": "ZnVuY3Rpb24gcHJvY2VzcyhldmVudCkgewogIGxldCBkb2NUaXRsZSA9IGV2ZW50LmdldFBhZ2VfdGl0bGUoKQogIAogIGlmIChkb2NUaXRsZSAhPSBudWxsICYmIGRvY1RpdGxlLmVuZHNXaXRoKCcgfCBTbm93cGxvdyBEb2N1bWVudGF0aW9uJykpIHsKICAgIGV2ZW50LnNldFBhZ2VfdGl0bGUoZG9jVGl0bGUuc3BsaXQoJyB8IFNub3dwbG93IERvY3VtZW50YXRpb24nKVswXSk7CiAgfQoKICByZXR1cm4gW107Cn0gCg=="
    }
  }
}

The Docker command to use enrichments with Micro

Open a new Terminal tab or window and remove the previously created Micro Docker container with the terminal command docker rm -f micro. The -f (force) flag is necessary in case the container is running.
Create a folder e.g. snowplow_micro_enrichments on your machine to store enrichment configuration files
Create a file named javascript_enrichment.json in the previouly created folder snowplow_micro_enrichments which contains the JSON config from above
An additional line in the docker command is needed to attach a local directory/folder with enrichment configurations to the container, the --mount flag is used for this. So the complete docker command to run Micro with enrichments is:

docker run --name micro -p 9090:9090 \
  --mount type=bind,source=$(pwd)/<YOUR_LOCAL_ENRICHMENT_FOLDER>,destination=/config/enrichments \
  snowplow/snowplow-micro:2.1.2

Test the enrichment

Send the previously created pixel tracker request again and check if it successfully reached the endpoint

http://localhost:9090/i?e=pv&page=Testing%20and%20debugging%20%7C%20Snowplow%20Documentation&url=https%3A%2F%2Fdocs.snowplow.io%2Fdocs%2Ftesting-debugging%2F&tv=no-js-0.1.0

Expand the enriched event object in the UI and check the page_titleproperty. The suffix “| Snowplow Documentation” should be removed
Expand the rawEvent object, to see the raw event property page as it reached the endpoint.

Beyond the UI: Query Micro Event APIs with jq

Micro is offering two event stream APIs (good & bad) and an event counter.

localhost:9090/micro/good: succefully enriched events
localhost:9090/micro/bad: failed events that do not comply with a schema
localhost:9090/micro/all: number of total, good and bad events e.g, {“total”:3,“good”:2,“bad”:1}

Good event stream

The JSON structure of an event in this stream in mainly devided into two parts: event which holds all enriched event information and rawEvent which holds the raw event payload. “Enriched” is not primarily referring to custom enrichments, instead it means that the raw event payload is derialized as a JSON with key/value pairs based on the raw event payload. The amount of available fields with null values might be confusion, but this is how Enrich outputs Snowplows Canonical Event Model.

Enriched:

event.<<standard field name>> standard/atomic fields
event.unstruct_event custom events incl. their event properties
event.contexts context of the incoming event payload
event.derived_contextscontext which are added through enrichments
rawEvent. raw event parameters - details in tracker payload documentation

Querying the API endpoints can be faster than using the UI, especially when you have to check the same event properties over and over again in the development process of enrichments. I prefer to use jq, an excellent open source command-line JSON processor, to query the APIs. Let’s stick to the previous page title enrichment example: what you previously checked via the UI, can be done via the API as follows:

Open a new terminal tab or window and use the following queries

Raw event payload: page title suffix “| Snowplow Documentation” is present:

curl -s localhost:9090/micro/good | jq '.[0].rawEvent.parameters.page'

Enriched event: page title suffix “| Snowplow Documentation” is removed:

curl -s localhost:9090/micro/good | jq '.[0].event.page_title'

The query explained:

the -s (silent) flag removes some clutter in the terminal, so that the API response is easier to read
.[0] selects the first event, .[1] the second etc. jq is very powerful, which is why I cannot explain it in detail in this post.
everything behind .[0] is a JSON path notation

JSON paths can be tricky, especially in such a huge Snowplow event payload. A tool like jsonpathfinder.com helps to unterstand JSON structures and path notation. To select an event, run the query curl -s localhost:9090/micro/good | jq '.[0]' and past the whole reponse in the tool. Use the right side of the tool to navigate through the event payload.

To come back to jq - the command-line JSON processor: A good starting point for jq is ChatGPT. Just provide ChatGPT the JSON event payload and describe what you want to extract or filter.

Bad event stream

The bad event stream has a different JSON structure than the good stream. The event is not fully enriched and contains an error array which describes the validation error. Let’s use the previous pixel tracker event again to create a failed event on purpose: change the event type parameter e from e=pv (page_view) to e=xyz (unknown event type).

http://localhost:9090/i?e=xyz&page=Testing%20and%20debugging%20%7C%20Snowplow%20Documentation&url=https%3A%2F%2Fdocs.snowplow.io%2Fdocs%2Ftesting-debugging%2F&tv=no-js-0.1.0

Ways to check the error message

Micro’s logs in the terminal tab in which you started the docker container
Micro’s UI, drill down deep into the failed event. The actual error message is data -> failure -> messages -> error -> dataReports -> message.
The Bad event stream API:

curl -s localhost:9090/micro/bad | jq '.[0].errors.[1]' | jq 'fromjson' | jq '.data.failure.messages'

The query explained:

Since the erros object in the Snowplow event payload is not so simple, the query is not simple either.

'.[0].errors.[1]' selects the first events and its first error
'fromjson' converts the stringified JSON in the erros array into a JSON
'.data.failure.messages JSON path notation to select the error message

Finally there is another Micro endpoint to reset the event counter or restart the app after a crash:

curl localhost:9090/micro/reset

How to connect Micro with a “Schema Server”

Micro can be connected to local and remote schema servers, which can be super handy for schema development and debugging. I am not going into details what an event schema is, since snowplow describes this in detail in the official documentation.

Option 1: Create a local folder structure

Create the following folder structure on you machine: /snowplow_micro_schemas/schemas/com.mycompany/schema_name/jsonschema/
Create a file named 1-0-0. The file name represents the schema version. Past a valide JSON schema of your choice. Do not use any file suffix like 1-0-0.json
Open a new Terminal tab/window and remove the previously created Micro Docker container with the terminal command docker rm -f micro
Run the following terminal command to attach the local schema folder structure to the container:

docker run --name micro -p 9090:9090 \
  --mount type=bind,source=$(pwd)/snowplow_micro_schemas/schemas,destination=/config/iglu-client-embedded/schemas \
  snowplow/snowplow-micro:2.1.2

Option 2: Connect Micro with a remote Schema Registry

Only two environment variables are needed to point micro to your schema server:

docker run -p 9090:9090 \
  -e MICRO_IGLU_REGISTRY_URL=<YOUR_IGLU_SERVER_API_ENDPOINT> \
  -e MICRO_IGLU_API_KEY=<YOUR_API_KEY> \
  snowplow/snowplow-micro:2.1.2

More details and how you can find your API endpoint and key in the official Micro documentation.

Final thoughts

I haven’t covered everything in detail, but more information would simply be too much for a “getting started” article. I am pretty sure there are a lot of open questions if you haven’t worked with Snowplow before. Nevertheless, I hope that this article has given you a better understanding of Snowplow Micro, and I’ve tried to write it in such a way that it can be understood even without prior technical knowledge. Have fun with Mirco and geek out.

The Snowplow Pipeline in a nutshell¶

Enrichment debugging - or how I came across micro¶

Snowplow Micro to the rescue¶

This guide covers¶

Run Micro on your machine¶

The Micro Docker Image¶

Run the Docker container¶

Sending events to Micro¶

Setup enrichments with Micro¶

The Docker command to use enrichments with Micro¶

Test the enrichment¶

Beyond the UI: Query Micro Event APIs with jq¶

Good event stream¶

Bad event stream¶

How to connect Micro with a “Schema Server”¶

Option 1: Create a local folder structure¶

Option 2: Connect Micro with a remote Schema Registry¶

Final thoughts¶