How to automatically create webhooks on new streams

Introduction

VersionVault Express can be configured to send out notifications when interesting things happen inside projects,. These notifications are called webhooks. A webhook is simply a JSON payload sent to an endpoint over HTTP as a POST request. Any system that can respond to an HTTP POST request can be a webhook endpoint and perform any action it chooses to. That action can even use a REST API to interact with the VersionVault Express instance that sent the webhook. This opens up some interesting possibilities to extend the basic VersionVault Express functionality. In this article we’ll look at one such extension and show how to automatically give newly created streams the ability to request builds.

Projects and streams

VersionVault Express is organized into projects. Developers join projects and create streams where they can store the code they’re working on. When a developer is ready to deliver code, some or all of the contents of their stream is delivered to a parent stream (usually the integration stream) where it can be merged with other code delivered by other developers from other streams.

Both projects and streams can be configured to send webhooks when certain events occur. A project that has been configured to send a “Create stream” webhook will send a notification whenever any new stream is created.

The “Request build” webhook can be added to any stream. It’s a little different because unlike most other webhooks, it triggers when a developer specifically asks for it (there’s a button in the user interface that he or she can press to request a build). Your build system might respond to a “Request build“ webhook by pulling down the code for the specific stream that requested the build and then building the code.

Newly created streams do not come with “Request build“ or any other kinds of webhooks configured. Whoever has the “builder“ role on your project would have to add them. It’s not hard to create webhooks on a stream, but it could become tedious to have to create them every time a new stream is created. Developers can, and often do, create streams at will.

A “builder” may prefer to set up a VersionVault Express project to automatically add the “Request build” (or perhaps another type of) webhook whenever developers create new streams. Here’s how to do it …

Endpoint

The first thing needed is something to catch the incoming webhook, and implement whatever it needs to do. Anything that can act as an HTTP server can be a webhook endpoint. For this article we’ll use Node-Red because it’s simple, visual and effective.

Once you have Node-Red installed and started, create a new flow. A pair of HTTP nodes are all you need to make your flow into a webhook endpoint.  Add an “http in” node and wire it to an “http response” node. You’ll need to configure a URL on your “http in” node. You might choose to make it “/webhook”. Make sure you choose the POST method.

You may also want to add some error handling and debugging to your flow. Wire your “http in” node to a “debug” node, and have it output “msg.payload.action”. Action is the JSON field inside a VersionVault Express webhook payload that tells you what type of webhook this is (“Create stream”, or “Request build” for example).

Now add a “catch” node and wire that to a new debug node set to output “msg.error”, and also wire it to a function node which sets msg.statusCode to “500” and msg.payload to msg.error. Don’t worry about wiring the output of the function node yet .. we’ll get to that shortly. Deploy your flow.

Configuring a project webhook

Log into VersionVault Express and open your project. Make sure you have the “builder” role. If you don’t, ask your project owner to assign you that role. In the navigator pane click the “Settings” link. Create a new webhook. You can call it anything you want, but perhaps “Create a new stream” might be a good idea. The “Payload URL” should be the address of your Node-Red endpoint – something like “https://your.ip.address:1880/webhook”, assuming you called your URL “/webhook”. It’s a good idea to specify a secret. Think of this like a password that VersionVault Express and your webhook endpoint share so that they know to trust one another. If your endpoint uses https and you do not have a valid certificate you’ll want to check the “Disable SSL validation” box, but this is not recommended – use a valid certificate ! Select the “Create stream” event. Click the “Create” button.

Make sure Node-Red is running and that your flow is deployed, then click the Validate button. If you see a “200 OK” status then you’re ready to move on.

Validate the secret

Although it’s not a strict requirement that webhook sender and receivers should share a secret, it’s a good idea. A VersionVault Express webhook that has been configured with a secret will send a custom HTTP header called “x-vv-signature” in its POST request. This header will contain a hash calculated from the JSON payload and the secret. When given a JSON payload, an endpoint that knows the same secret should be able to generate the same hash.

Add a “function” node and a “switch” node to your Node-Red flow. Wire the “http in” node to the “function” node and the “function” node to the “switch node”. Call the “function” node “validate secret” and call the “switch” node “Check statusCode”.

Add this code to your function node and be sure to use the same secret that you used when you created your webhook in VersionVault Express.

NOTE: You’ll need to add the “crypto” package to Node-Red before you can run this code.

Configure your “switch” node to have two outputs. Inspect the “msg.statusCode” property and send values between 200 and 299 to one of them and everything else to the other one.

For testing purposes, you can wire both outputs to the “http response” node, deploy your flow and hit that validate button again in VersionVault Express. If both secrets match, you can expect a “200 OK” response. For fun, you might try changing the secret in the VersionVault Express webhook and trying to validate. You should see a “401 Unauthorized”. Remember to change it back !

Set some flow variables

Your Node-Red flow is going to need some generally useful variables. Some of it may be hardcoded and some of it may come from the incoming webhook. Add a new “function” node and drop it between the first “switch” node’s output and the “http response” node.  Call it “Set flow variables”.

Add this code to the function node, where the “builder” is your builder’s VersionVault Express user name and the “password” is their password.

Add another switch

Add another “switch” node to your flow. This one will look at what kind of webhook has been received, and route it to the appropriate subflow for processing. We’re going to use three webhooks in this article; “Create stream”, “Request build” and “Test”, so give your “switch” node four outputs.

Inspect the “msg.payload.action” property. If the property equals “Create stream”, route to the first output. If it equals “Request build”, route to the second output. If it equals “Test”, route to the third output. And if it’s anything else, route to the fourth output.

Wire the output from the “Set flow variables” node to the input of the new “switch” node.

Add another “function” node. Call it “error 404”. This will be the catch all node for any incoming webhooks that this endpoint isn’t designed to recognize or handle. Use the following code.

Connect the fourth output of the switch node to this new node’s input.

Create a sub flow

It’s good practice to put code that implements a distinct path into subflows. Since you can now route “Create stream” webhooks, you should route to a subflow.

Create a new sub flow, name it “Create stream” and give it one input and one output.

As with the main flow, add a “function” node and call it “Set flow variables”. Add the following code to the function node.

Notice that all the values are either inherited or derived from the variables you already set in the parent flow or from JSON fields in the incoming webhook.

Note: This code assumes that your VersionVault Express sever knows it’s own hostname and port. If your network configuration uses a proxy server, a custom host name/ip address or port forwarding to route to your VersionVault Express server you’ll need to account for that when you set the “host” variable.

Login and get a bearer token

In response to a “Create stream” webhook, you want to log into whichever VersionVault Express system that sent the webhook as a “builder” and use the REST API to create one more webhook on the newly created stream. The first thing to do it log in.

Add a “function” node, an “http request” node, a “json” parser node, a “switch” node (with two outputs) and another two “function” nodes. Wire the first four together, send the output of the “Set flow variables” node (created above) to the input of the first one, and send each output of the “switch” node to one of the last two “function” nodes.

Call the first “function” node “Login payload” and add the following code to build the request payload and URL for the VersionVault Express authenticate API.

Call the “http request” node “Login” and make sure its method is POST.

Call the “json” parser node “Parse” and make sure its action is “Convert between JSON String & Object”. It’s property should be “msg.payload”.

Call the “switch” node “check status” and make sure it’s property is “msg.statusCode”. Send values between 200 and 299 to the first output and everything else to the second output.

The “function” node wired to the “switch” node’s first output should be called “Parse bearer token”. This is where the token that was returned by a successful call to the authenticate REST API will be extracted from the parsed JSON response payload and stored in a variable, so that subsequent REST API calls will be authenticated.

Your flow should look like this.

Create webhook payload – invoke the API

Once your builder is logged into VersionVault Express’s REST API the next step is to create a “Request build” webhook on the newly created stream. The pattern to do this is similar to the login pattern .. make a payload and URL for the REST API, call it and handle the response. So add three more nodes to the flow – another “function” node, another “http request” node and another “switch” node.

Wire the output from the “Parse bearer token” node to the input of the new “function” node and call it “Create webhook payload”. The purpose of this node is to build an HTTP request for the REST API that creates a webhook. As with the login node, you’ll need a request body and a URL, but since this REST API is restricted to project members who have the builder role, you’ll need a header to identify yourself. This header will store the login token from above. Add this code to your node.

Notice that you specify the id of the newly created stream as a parameter in the URL. You will remember that you parsed the stream id from the incoming “Create stream” webhook payload. Also notice that the “eventType” is “REQUEST_BUILD”. If you want to put other types of webhooks on this stream simply add them to the “eventList”.

Call the “http request” node “Create stream webhooks”. Make sure it’s method is set to POST.

As before, call the “switch” node “check status” and make sure it’s property is “msg.statusCode”. Send values between 200 and 299 to the first output and everything else to the second output.

Your flow should look like this.

Log out

Remember to log out to close the REST API session. The process to log out involves calling a REST API. The pattern is the same as before .. build a request with headers and a URL (but no payload this time), then call the API.

Add another “function” node and another “http request” node. There’s no need to check the response to the logout API so no need for another “switch” node.

Call the “function” node “Logout payload” and wire it’s input to the output of the “Create stream webhooks” node. This will attempt to log out regardless of whether the API call to create the new stream webhooks succeeds or fails.  The code for the “function” node should look like this.

Call the “http request” node “Logout”. Make sure it’s method is set to POST.

Your flow should look like this.

Error handling

Having now checked the status of both the login and the create webhook REST APIs this is a good time to insert some error handling. When VersionVault Express sends outgoing webhooks, it records the response payload, checks the response status and if it’s been configured to do so it may go into “retry mode” if it doesn’t get successful responses. Since this flow, and its subflow implement a webhook endpoint, the expectation is that it will return meaningful success and error responses.

Let’s start with the sub flow and come back to the parent flow in a little while.

Add two “function” nodes to the flow.

Call the first one “Set success payload”. This is where the response payload to the incoming “Create stream” webhook will be set. Builders will be able to see these response payloads in VersionVault Express by looking at the webhook history. Add this code.

Wire the first output from the second “check status” node to the input of the “Set success payload” node.

Call the second one “Set error payload”. This is also where the response payload to the incoming “Create stream” webhook will be set, but some sort of description of what went wrong will be set instead of a success message. Add this code.

Wire the second output from both “check status” nodes to the input of the “Set error payload” node.

Wire the output from both nodes to the subflow’s output.

Your flow should look like this.

Deploy your flows.

Complete the webhook endpoint flow

Switch back to your main flow. In the palette on the left, under a category called “subflows” you will see you new “Create stream” sublflow. Drag it onto the flow and wire the first output of the switch node to its input.

Add another “function” node to the flow. Call it “Add statusCode”. This node will prepare the overall response to the incoming webhook. Add the following code.

Wire the outputs from the “Create stream” subflow, the second output from the “Check statuscode” node, the outputs from the “error 404” and the “error 500” nodes and the second and third outputs from the switch node to its input, and wire its output to the input of the “http response” node.

Your flow is complete and it should look like this.

Deploy your flow.

Testing it out

Log into VersionVault Express as a builder or a developer and open your project. Create a new stream. You will see that your new stream has an icon to “Request build”. Look for it on the list streams page if you’re a builder or developer and on the code page if you’re a developer.

If you click the icon to request a build you will see a message appear in the Node-Red debug console. Of course, no build will run because your Node-Red webhook endpoint simply routes the payload back to it HTTP response.

For your next exercise, try creating a “Request build” subflow.