While there are some differences among the flavors of Markdown, one rather nice aspect of authoring content with Markdown is that it can be readily converted to HTML using one of the many markdown processing technologies that are available. One way that can facilitate the creation of a website's HTML files, while still authoring content in Markdown, is to use Node.js to convert Markdown content into an HTML file. The resulting HTML output can then be uploaded to Jamstack website hosting, using static HTML files.

In this post we'll use Node.js and CLI commands to read a Markdown file, convert that file to an HTML string and then write the HTML string to a new file. When we have the file created, we can start a local development server to test the file in a web browser. Before following the steps make sure to have Node.js and npm installed.

Setup Node.js CLI Project

To start, setup the package.json file used with Node.js by running the command npm init in a terminal window open to your project folder. Then follow the prompts shown by the npm init process and a package.json file should have been created. With the package.json file in place we can run additional commands to install the npm packages that are used to convert Markdown to HTML.

npm install

In the same terminal window run the command npm install markdown-it highlight.js fs-extra cross-env rimraf @babel/cli @babel/core @babel/preset-env @babel/preset-typescript --save, followed by the command npm install typescript @types/node @types/markdown-it @types/fs-extra --save-dev.

After running both of these commands a new folder named "node_modules" should be present in your project folder. In the "node_modules" folder the following npm packages are installed:

• markdown-it
• highlight.js

Add Support For ES Modules

For this example these packages are also installed, mostly to support using TypeScript and ES modules in Node.js, which is optional.

• fs-extra
• typescript
• cross-env
• rimraf
• @babel/cli
• @babel/core
• @babel/preset-env
• @babel/preset-typescript
• @types/fs-extra
• @types/markdown-it
• @type/node

The remainder of these steps will include setting up the TypeScript and Babel compilers to use ES Modules in Node.js for the CLI script that will convert Markdown into HTML, and write the HTML string to a file.

To support ES modules there is one more configuration that must be included in the package.json file. This is the "type" property with the value set to "module" as indicated below.

{
  "type": "module"
}

package.json Scripts

Additionally, we need to configure the "scripts" section of the package.json file to include the npm CLI scripts that will be used in the following steps. Since we are modifying the package.json file at this time go ahead and also add the following to the scripts property:

{
  "scripts": {
    "typecheck": "tsc --p .",
    "clean": "rimraf dist",
    "compile": "cross-env-shell babel src -d dist --source-maps --extensions '.ts'",
    "start": "npm run clean && npm run compile && node ./dist/index.js",
    "start-typecheck": "npm run typecheck && npm run start"
  }
}

These scripts are responsible for invoking the TypeScript and Babel compilers, to carry out typechecking and the compilation of TypeScript into JavaScript. These use most of the optional packages that were installed for that process. In a later step we can run these package.json scripts as CLI commands to first compile TypeScript and then run the JavaScript output with Node.js to convert Markdown into HTML.

package.json

With all the required packages installed and ES modules configured, the package.json file in your project should look like this:

{
  "name": "convertmarkdowntohtml",
  "type": "module",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "typecheck": "tsc --p .",
    "clean": "rimraf dist",
    "compile": "cross-env-shell babel src -d dist --source-maps --extensions '.ts'",
    "start": "npm run clean && npm run compile && node ./dist/index.js",
    "start-typecheck": "npm run typecheck && npm run start"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@babel/cli": "^7.14.8",
    "@babel/core": "^7.14.8",
    "@babel/preset-env": "^7.14.9",
    "@babel/preset-typescript": "^7.14.5",
    "cross-env": "^7.0.3",
    "fs-extra": "^10.0.0",
    "highlight.js": "^11.2.0",
    "markdown-it": "^12.2.0",
    "rimraf": "^3.0.2"
  },
  "devDependencies": {
    "@types/fs-extra": "^9.0.12",
    "@types/markdown-it": "^12.0.3",
    "@types/node": "^16.4.10",
    "typescript": "^4.3.5"
  }
}

If you are having trouble with the package install try copying the package.json from above and save that as your package.json file, then run the command npm install to install all of the listed packages.

Configure TypeScript Compiler with tsconfig.json

TypeScript is not required to convert Markdown to HTML, but it is not that much extra configuration to add when compared to the benefits of using TypeScript. Since the npm package for TypeScript was just installed we can add a new file to the project folder named "tsconfig.json" and this will contain the TypeScript compiler configuration settings that are recommended when using TypeScript and Babel in the same project.

{
  "compilerOptions": {
    "allowSyntheticDefaultImports": true,
    "isolatedModules": true,
    "strict": true,
    "module": "esnext",
    "lib": ["ES2019"],
    "noEmit": true,
    "moduleResolution": "node",
    "skipLibCheck": true
  },
  "include": ["src/**/*.ts"],
  "exclude": ["node_modules/**/*", "dist/**/*"]
}

The configuration will use TypeScript for type checking only, and the actual compilation of TypeScript into JavaScript will instead be carried out by the Babel compiler.

Configure Babel Compiler with babel.config.json

Just as the tsconfig.json file was added, we can add another file for the Babel configuration settings. This file is named "babel.config.json" and contains the following:

{
  "presets": [
    [
      "@babel/preset-env",
      { "modules": false, "targets": { "node": "current" } }
    ],
    ["@babel/preset-typescript"]
  ],
  "ignore": ["node_modules"]
}

The Babel compiler does not type check TypeScript code and will attempt to output valid JavaScript regardless of the TypeScript source. This is why the TypeScript compiler is used for type checking, and the benefit of using both is that the Babel compiler has presets available to ensure that the JavaScript generated will target a specific environment, in this case the current version of Node.js, and the "modules" property is set to false, which will preserve ES modules.

Create Markdown File

With our Node.js CLI project setup and package.json scripts already configured, the next part of the process to convert Markdown into HTML will be to create a sample Markdown file with a variety of content that includes the basic syntax shared among most Markdown flavors. To do this create a new folder for your project, named "content" and then inside the "content" folder create a new file named "index.md". When you have the index.md file created you can copy the sample Markdown content below into it.

# H1

## H2

### H3

#### H4

**bold text**

_italicized text_

> blockquote

1. First item
2. Second item
3. Third item

- First item
- Second item
- Third item

`code`

---

```javascript
function() {
  console.log("This is some javascript included in a markdown code block, and it will be converted to valid HTML with code syntax highlighting.");
}
```

<kbd>this is a keyboard input html element</kbd>

```html
<span>this will remain html even after the Markdown is converted to HTML</span>
```

[Dev Extent](https://www.devextent.com)

![Dev Extent](https://www.devextent.com/images/devextent.png)

Create Node.js CLI Script

Now that there is a Markdown file in the project we can add a new folder named "src" and in that folder add a new file named "index.ts". This is the Node.js script responsible for converting the Markdown file into an HTML file, and to start it looks like this:

(async function convertMarkdownToHtml() {
  console.log("Converting Markdown to HTML...");
})();

You can now run the command npm run start-typecheck or npm run start to compile without typechecking and you should see the console log is displayed. This means that the Node.js CLI project is working correctly, first compiling the TypeScript source code and then executing the generated JavaScript output with Node.js, all in one command.

Read Markdown File

After verifying that the Node.js CLI script is working correctly go ahead and add this code:

import fs from "fs-extra";

(async function convertMarkdownToHtml() {
  console.log("Converting Markdown to HTML...");

  // markdown source
  const content = await fs.readFile("./content/index.md", "utf8");
})();

The additional code imports one node module, the fs-extra package, and it provides the "readFile" function to asynchronously read the "index.md" file in the content folder. The contents of the Markdown file are then assigned to the variable named "content". We now have a string of Markdown content that is ready to be converted into HTML, and to do that the markdown-it package will be used.

Configure markdown-it Markdown Parser Options

To configure the markdown parser included in the markdown-it package, create a new folder in the "src" folder named "utils" and then in the "utils" folder create a new TypeScript file named "markdown.ts". In the "markdown.ts" the markdown-it package will be imported and the markdown parser object will be constructed and exported.

import MarkdownIt from "markdown-it";

const markdown: MarkdownIt = MarkdownIt({
  html: true,
});

export { markdown };

There is one configuration option passed into the markdown parser configuration and that is to support HTML tags in the markdown source. This is optional, and not required but it can be helpful to support using HTML for elements that are lacking in Markdown syntax.

Add Code Syntax Highlighting With highlight.js

Besides optionally supporting HTML tags in the Markdown source, the markdown parser included with the markdown-it package can apply syntax highlighting to designated code blocks. Make the following adjustments to the markdown.ts file to include this option:

import hljs from "highlight.js";
import MarkdownIt from "markdown-it";

const markdown: MarkdownIt = MarkdownIt({
  html: true,
  highlight: function (str, lang) {
    if (lang && hljs.getLanguage(lang)) {
      try {
        return (
          '<pre><code class="hljs">' +
          hljs.highlight(str, { language: lang, ignoreIllegals: true }).value +
          "</code></pre>"
        );
      } catch (__) {}
    }
    return (
      '<pre><code class="hljs">' +
      markdown.utils.escapeHtml(str) +
      "</code></pre>"
    );
  },
});

export { markdown };

The highlight.js module is able to dynamically determine the language syntax highlighting based on the "lang" variable valuable that is passed into the "highlight" function the highlight.js module API provides.

Instead of an error when encountering inconsistent syntax, the "ignoreIllegals" parameter configures the highlight.js highlighter to finish highlighting. You may wish to leave this option out, but there is discussion whether the default value of the "ignoreIllegals" options should be changed to true, as is used in this example.

If highlight.js cannot determined the language of the code block it will apply the "escapeHtml" function provided to the markdown string, and also wraps the code block section into a code element nested inside a pre element.

These additions will import the highlight.js module and apply the formatting required to dynamically highlight code blocks based on the language provided. The sample markdown file created in a previous step includes a block of JavaScript code that will have dynamic syntax highlighting applied when converted to HTML.

Convert Markdown to HTML with markdown-it parser

The "markdown.ts" file can now be imported in the "index.ts" file to access the Markdown parser with the previous configuration applied. To import the "markdown.ts" file and use the "render" function provided by the markdown-it module API, make these changes to the "index.ts" file:

import fs from "fs-extra";
import { markdown } from "./utils/markdown.js";

(async function () {
  console.log("Converting Markdown to HTML...");

  // markdown source
  const content = await fs.readFile("./content/index.md", "utf8");

  // converted to HTML
  const rendered = await markdown.render(content);
})();

The Markdown content, converted to HTML, is now assigned to the variable named "rendered". To view the rendered HTML you can output the "rendered" variable to the console and then run the command npm run start-typecheck, once more.

The contents of the "rendered" variable are valid HTML, but they do not represent an entire HTML document. To ensure that the Markdown source is converted into a complete and valid HTML document another variable is added, named "htmlFile", and this wraps the "rendered" variable string value in additional HTML code to create an entire HTML document. The "index.ts" should now look like this:

import fs from "fs-extra";
import { markdown } from "./utils/markdown.js";

(async function () {
  console.log("Converting Markdown to HTML...");

  // markdown source
  const content = await fs.readFile("./content/index.md", "utf8");

  // converted to HTML
  const rendered = await markdown.render(content);

  const htmlFile = `<!DOCTYPE html>
  <html lang="en">
  <head>
  <meta charset="UTF-8" />
  <title>Convert Markdown to HTML with Node.js</title>
  <link rel="stylesheet" href="./default.css">
  </head>
  <body>
  ${rendered}
  </body>
  </html>`;
})();

Note: The "default.css" file referenced in the head of the HTML document will be copied in the following step from the default style sheet theme included with the highlight.js npm package.

Write HTML File

Instead of writing this file in the project folder root, the fs-extra module includes a "mkdirs" function that can programmatically create a folder. Using this function a new folder will be created named "public", and the generated HTML file saved there.

The highlight.js module provides many different style sheet themes to choose from when applying code block syntax highlighting. For this example the "default.css" theme is used, and that file is copied from the highlight.js module, inside the "node_modules" folder into the public folder that is programmatically created for the generated HTML. This way when the style sheet is reference in the "index.html" file, the "default.css" file is available in the same folder.

import fs from "fs-extra";
import { markdown } from "./utils/markdown.js";

(async function () {
  console.log("Converting Markdown to HTML...");

  // markdown source
  const content = await fs.readFile("./content/index.md", "utf8");

  // converted to HTML
  const rendered = await markdown.render(content);

  const htmlFile = `<!DOCTYPE html>
  <html lang="en">
  <head>
  <meta charset="UTF-8" />
  <title>Convert Markdown to HTML with Node.js</title>
  <link rel="stylesheet" href="./default.css">
  </head>
  <body>
  ${rendered}
  </body>
  </html>`;

  await fs.mkdirs("./public");

  await fs.writeFile("./public/index.html", htmlFile, "utf8");

  await fs.copy(
    "./node_modules/highlight.js/styles/default.css",
    "./public/default.css",
    { overwrite: true }
  );

  console.log("HTML generated.");
})();

Run the command npm run start-typecheck once more and a new file "index.html" should be generated inside a new folder named "public" in your project folder, along with the "default.css" file that was copied from the "node_modules" folder.

You can now view the "index.html" file that will contain the Markdown source converted into HTML. The "index.html" file should look similar to this:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <title>Convert Markdown to HTML with Node.js</title>
    <link rel="stylesheet" href="./default.css" />
  </head>
  <body>
    <h1>H1</h1>
    <h2>H2</h2>
    <h3>H3</h3>
    <h4>H4</h4>
    <p><strong>bold text</strong></p>
    <p><em>italicized text</em></p>
    <blockquote>
      <p>blockquote</p>
    </blockquote>
    <ol>
      <li>First item</li>
      <li>Second item</li>
      <li>Third item</li>
    </ol>
    <ul>
      <li>First item</li>
      <li>Second item</li>
      <li>Third item</li>
    </ul>
    <p><code>code</code></p>
    <hr />
    <pre><code class="hljs"><span class="hljs-keyword">function</span>(<span class="hljs-params"></span>) {
  <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(<span class="hljs-string">&quot;This is some javascript included in a markdown code block, and it will be converted to valid HTML with code syntax highlighting.&quot;</span>);
}
</code></pre>
    <p><kbd>this is a keyboard input html element</kbd></p>
    <pre><code class="hljs"><span class="hljs-tag">&lt;<span class="hljs-name">span</span>&gt;</span>this will remain html even after the Markdown is converted to HTML<span class="hljs-tag">&lt;/<span class="hljs-name">span</span>&gt;</span>
</code></pre>
    <p><a href="https://www.devextent.com">Dev Extent</a></p>
    <p>
      <img
        src="https://www.devextent.com/images/devextent.png"
        alt="Dev Extent"
      />
    </p>
  </body>
</html>

You can validate the generated HTML code with the W3C Markup Validation Service, and you can also use the http-server npm package to create a local web server on your computer to view the "index.html" file in a browser.

View HTML File Locally

To test the Markdown converted into HTML, in a browser you can run the command npm install http-server --save-dev to install the http-server npm package. Then add the following to the package.json scripts property:

{
  "scripts": {
    "serve": "http-server"
  }
}

Then you can run the command npm run serve and the generated "index.html" file will be served from the public folder in your project. You should be able to navigate to "localhost:8080" and there you will see the content of the "index.html" with the styles from "default.css" applied to the syntax highlighted code block.

Typically a GraphQL client is used to facilitate the client side query building, and to send HTTP POST requests containing GraphQL queries to the GraphQL server responsible for returning the data. It is not required to use a dedicated GraphQL client, as it is possible to send a GraphQL query as a POST request using the Fetch API, and this is similar to the process used to submit FormData using the Fetch API. To show how to send a POST request containing a GraphQL query with the Fetch API, data from the GraphQL API: https://content.wpgraphql.com/graphql provided by WPGraphQL can be used. After fetching the latest posts from the GraphQL API, by sending a POST request containing the GraphQL query, we can display the data as a list with each item title as a link.

Create HTML File

First, create an HTML file that will link to a JavaScript file containing the code that will send the GraphQL query as a POST request with the Fetch API. After sending the POST request containing the GraphQL query, the result of the query will be displayed as HTML, and before any data is received a no data message is displayed. In the project folder add a new file named "index.html" with the following content:

<!-- index.html -->
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>Post a GraphQL Query with the Fetch API</title>
  </head>
  <body>
    <div id="data-container">
      <p>no data yet!</p>
      <button id="data-button">Get Data</button>
    </div>
    <script src="/script.js"></script>
  </body>
</html>

Add JavaScript File

In the "index.html" file there is a JavaScript file referenced that is named "script.js". We can create that file in the same folder as the index html file. After creating "script.js" in the project folder add the following code:

const dataFetchDisplay = function ({
  eventListenerSelector,
  eventType,
  dataFetcher,
  displayUpdater,
  dataTargetSelector,
}) {
  document
    .querySelector(eventListenerSelector)
    .addEventListener(eventType, async () => {
      displayUpdater(dataTargetSelector, await dataFetcher());
    });
};

The "dataFetchDisplay" function has an options object as the parameter that contains the information needed to send the Fetch API POST request containing a GraphQL query, although we have yet to call this function or define the functions "displayUpdater" and "dataFetcher" that are included in the options parameter and used within the async callback of the event listener that is instantiated, when the "dataFetchDisplay" function is called. Here is how the "dataFetchDisplay" function will be used:

dataFetchDisplay({
  eventListenerSelector: "#data-button",
  eventType: "click",
  dataFetcher: getData,
  displayUpdater: updateDisplay,
  dataTargetSelector: "#data-container",
});

Notice that the "eventListenerSelector" and "dataTargetSelector" parameters correspond to the ID attributes that are present in the index.html file created in the first step. These values can be changed, but the values must match the HTML document ID attributes. Go ahead and add the invocation of the "dataFetchDisplay" function directly below the function definition previously added to script.js.

Fetch API POST Request with GraphQL Query

Now that we have the "dataFetchDisplay" function defined and being called, if we try to run this code it will result in an error because the helper functions to get the data and display it are not yet defined. Directly above the "dataFetchDisplay" function add the following code to define the "getData" function that is referenced in the "dataFetcher" options object parameter key value.

const getData = async function () {
  return (
    await (
      await fetch("https://content.wpgraphql.com/graphql", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          query: "{ posts { nodes { title, link } } }",
        }),
      })
    ).json()
  ).data.posts.nodes;
};

The getData function shown above is where the POST request, sent by the Fetch API, containing the GraphQL query is defined. For this example the GraphQL API is provided by WPGraphQL, and the query will retrieve the link and title information for the ten most recent blog posts. Since we know the format of the data that is returned from the GraphQL query POST request sent with the Fetch API, we can return only the "nodes" in the "getData" function. That way when the "getData" function is used the data is already formatted as an array of post objects.

Display GraphQL Query Data

Now that we have the "getData" function defined and the GraphQL query data is returned after sending a POST request using the Fetch API, we need to display the data once it is returned from the GraphQL API server. To do this the function that is passed in as the "displayUpdater" parameter in the options object will be used. Add this code above the "dataFetchDisplay" function in the "script.js" file:

const updateDisplay = function (selector, data) {
  const list = document.createElement("ul");

  data.forEach(function (item) {
    const listItemLink = document.createElement("a");
    listItemLink.textContent = item.title;
    listItemLink.setAttribute("href", item.link);

    const listItem = document.createElement("li");
    listItem.appendChild(listItemLink);

    list.appendChild(listItem);
  });

  document.querySelector(selector).replaceChildren(list);
};

The "updateDisplay" accepts two parameters: one to indicate the target element to insert the HTML that is generated and the second is the data array. In this example a link item is created for each data object using the title. The list of link elements is then used to replace the html of the target element.

By passing the "getData" and "displayUpdater" functions in as parameters to the "dataFetchDisplay" function, both the query and the way it should be displayed can be changed to suit the usage context. The "dataFetchDisplay" function is generic in that sense as the parameters determine what data to display, and how,based on the specific usage of the function.

Putting all of the code sections together should result in a script.js file that looks like this:

//script.js

const getData = async function () {
  return (
    await (
      await fetch("https://content.wpgraphql.com/graphql", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          query: "{ posts { nodes { title, link } } }",
        }),
      })
    ).json()
  ).data.posts.nodes;
};

const updateDisplay = function (selector, data) {
  const list = document.createElement("ul");

  data.forEach(function (item) {
    const listItemLink = document.createElement("a");
    listItemLink.textContent = item.title;
    listItemLink.setAttribute("href", item.link);

    const listItem = document.createElement("li");
    listItem.appendChild(listItemLink);

    list.appendChild(listItem);
  });

  document.querySelector(selector).replaceChildren(list);
};

const dataFetchDisplay = function ({
  eventListenerSelector,
  eventType,
  dataFetcher,
  displayUpdater,
  dataTargetSelector,
}) {
  document
    .querySelector(eventListenerSelector)
    .addEventListener(eventType, async () => {
      displayUpdater(dataTargetSelector, await dataFetcher());
    });
};

dataFetchDisplay({
  eventListenerSelector: "#data-button",
  eventType: "click",
  dataFetcher: getData,
  displayUpdater: updateDisplay,
  dataTargetSelector: "#data-container",
});

Test GraphQL Post Request Locally

At this point we have the "index.html" and the "script.js" file setup so we can make sure that it is working by testing it locally. To do this we will need to install the http-server npm package. Before proceeding make sure to have Node.js and npm installed as they are required.

npm init package.json

Once installed you can open the project folder in a terminal window and run the npm init command, and follow the prompts that are displayed. This will set up the package.json in the project folder.

npm install http-server

After configuring the package.json file run the command npm install http-server --save-dev. The http-server npm package is now installed as a development dependency.

add npm script

In the "scripts" object of the package.json file configuration add the following script:

{
  "scripts": {
    "dev": "http-server"
  }
}

The dev script can now be run and this will start the local development environment using the http-server npm package. You should now have a "node_modules" folder that was added to the project folder, and the package.json file should look like this:

{
  "name": "post-graphql-query-fetch-api",
  "version": "1.0.0",
  "description": "",
  "main": "script.js",
  "scripts": {
    "dev": "http-server",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "http-server": "^0.12.3"
  }
}

To start the local development environment with http-server run the command npm run dev and navigate to the url that is displayed in the console output. The development url will most likely be "http://localhost:8080", as this is the default setting for the local server configuration.

After running the npm run dev command and navigating "http://localhost:8080" you should see the "no data yet" message in your browser and the "get data" button we created earlier. To send the GraphQL query POST request with the Fetch API click the "get data" button, and the latest ten posts should will display on the page.

In some cases it might be beneficial to include a dedicated GraphQL client in your project, but in others using the Fetch API to send a POST request containing a GraphQL query without a GraphQL client can be sufficient. This can save time if the other more advanced features that come with GraphQL clients are not needed, especially if requests to the GraphQL server are infrequent.

Azure Blob Storage Static Website Hosting

In order to use the static website hosting feature included with Azure Blob Storage you need to create an Azure account if you do not already have one. Once you have created your account and logged in you will need to create the Storage Account resource that will provide the Blob Storage service. In the Azure portal select create a new resource and then search for "storage account". Then click create and follow the setup steps that are displayed to give the storage account a name and region. You can leave any pre-configured settings as the default setting.

Within the Azure portal navigate to your newly created storage account and in the left side navigation find the section labelled "Settings" and then within that section select the "Static website" feature.

Configure Index Document Name and Error Document Path

In the static website settings, enabling the static website feature will automatically generate the primary endpoint based on the storage account name. In the "Index document name" and "Error document path" fields enter the following:

Based on how your static website is configured, or the static site generator that you are using, the "Error document path" might be different than what is shown for this example, however the "Index document name" will most likely remain as "index.html". Make sure to update these settings to correspond to the configuration that you are using. The storage account static website is now available and you can go to the primary endpoint and you will see this:

This is good and it means that the static website is setup and publicly available. Now we can setup GitHub Actions to automated deployments anytime there is a commit pushed to the main branch in our GitHub repository.

Create Remote GitHub Repository and Configure Local Repository Remote

If you haven't already go ahead and create a new repository for your project, and after doing so follow the directions that are displayed to create a new repository from the command line on your local computer. This should result in a new repository that can push to the remote repository created on GitHub and any commits that we make will be applied to the main branch which is named "main". Make sure that you are able to push commits to GitHub and you can see them in your GitHub remote repository interface before proceeding.

Note: The following steps that configure the GitHub Actions workflow require that the branch name is set to "main".

Add .gitignore File

Additionally you will want to add a ".gitignore" file to your project that minimally contains:

node_modules
_output

The "_output" folder is included because this is created by the static site generator build process that is configured in the next step. This folder will always contain generated files so it does not need to be tracked by git.

Setup Static Site Generator

If you already have a repository, that includes a Jamstack site setup with a static site generator, you can proceed without this step. In order to illustrate the git-centric deployment process that is promoted by GitHub Actions we can use, morphic a static site generator built with Node.js and TypeScript. You don't need to use a static site generator in order to use GitHub Actions or the Static Website feature included with Azure Blob Storage, however it can be useful since GitHub Actions can execute commands that will invoke the static site generator build process and upon completion carry out the cloud deployment.

Install Node.js and Configure package.json

Before proceeding make sure to have Node.js and npm installed.

In your project folder run the command npm init and follow the prompts that are displayed to create a package.json file for your project. Then run the command npm install @protolith/morphic. This will install the morphic static site generator and its dependencies.

Add Static Website Content as Markdown Files

In the project folder where the package.json file was created you can then run this series of commands to create some site content.

mkdir content/pages

mkdir templates

cd content/pages

echo '---
title: Home
---
# <%= model.title %>
home page content

<a href="/about/">Go to About Page</a>' > index.md

echo '---
title: Page Not Found
---

The page you are looking for may have been moved or deleted.

[Go to Homepage](/)' > 404.md

echo '---
title: About
---
# <%= model.title %>
This is the about page.

<a href="/">Go to Home Page</a>' > about.md

cd ../../templates

echo '<!DOCTYPE html>
<html>
  <head>
    <title><%= model.title %></title>
  </head>
  <body>
    <%- model.content %>
  </body>
</html>' > index.ejs

cd ..

Note: make sure the files created (both ".md" and ".ejs" files) are encoded as UTF-8, or the static site generation process will not work. In Visual Studio Code you can change the file encoding by selecting the file and then in the bottom right hand corner toolbar the encoding is displayed. Click the encoding display in the bottom right toolbar to save with the UTF-8 encoding if needed.

Build and Serve Static Site Generator Locally

To make sure that morphic is setup correctly and generating the appropriate files you can run the command npx morphic --serve, and a browser window should open with the home page displaying a link to the about page. You can navigate back and forth from the home page and about pages. Also you can verify the 404 page was generated by going to the path "404.html" so the full url when running locally might be "http://localhost:3000/404.html", which corresponds to the error document path setting configured in the Azure Blob Storage Static Website settings.

Github Actions Workflow To Deploy Static Site to Azure Blob Storage

We now have our static site configured and ready to deploy with a GitHub Actions Workflow. To create a new GitHub Actions Workflow, that will build and deploy the static site from our GitHub repository, add a new folder within the current project folder named ".github". In the ".github" folder add another folder named "workflows" and within the "workflows" folder create a new workflow YAML file named "main.yml". The "main.yml" file is where we will create the GitHub Actions workflow that will build and deploy a Jamstack site to the Azure Blob Storage Static website. Here is the complete reference documentation for creating workflows provided by GitHub. We will only use a subset of the available features in our "main.yml" file to build and deploy on a git push command to the main branch in the repository we created. In the "main.yml" file add the following code:

name: MAIN

on:
  push:
    branches: [main]

env:
  BUILD_COMMAND: npx morphic

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0

      - name: Install Dependencies
        run: npm ci

      - name: Build
        run: ${{ env.BUILD_COMMAND }}

      - name: Update Blobs
        uses: azure/cli@v1.0.0
        with:
          inlineScript: |
            az storage blob sync -c '$web' -s _output --connection-string '${{ secrets.AZURE_STORAGE_CONNECTION_STRING }}'

This workflow will be activated anytime there is a git push command for the main branch in the remote git repository. It will setup a GitHub Actions environment that is running on the latest version of the Ubuntu operating system, and then checkout the latest version of the code from the main branch. After checking out the main branch, the workflow invokes the npm ci command, which is similar to the npm install command, and is suitable for continuous integration environments. This is required to install the morphic static site generator npm package, similar to how we ran the static site generator locally in a previous step.

With the package.json dependencies installed the "Build" step in the workflow will run the "BUILD_COMMAND" specified as an environment variable. In our case, with the morphic static site generator, this command is the same one as we used before npx morphic, except this time the --serve flag is omitted since the files will be deployed to Azure Blob Storage Static Website hosting.

After running the Build command the workflow will have access to the site output folder: "_output" in our case, and this is what will be used in the Azure CLI portion of the workflow with name "Update Blobs". This uses the Azure CLI Action to permit the use of Azure CLI commands. In this case we want to use the az storage blob sync command to ensure that "$web" blob container is synced to the the current build "_output" folder that was just created during the Build step of the workflow file.

In order to sync the files to the Azure Blob container named "$web", we need to pass in the "--connection-string" value that acts an access password for the storage account. Instead of creating a security risk by including the blob storage connection string directly in our publicly available workflow we can use GitHub Actions Repository Secrets to store secret variables that only the automated workflow environment can access.

Git Commit Project and Workflow Files

The workflow configuration file is now included in the project, so we can commit the example static files we created and the workflow file using the commands git add . and then git commit -m 'adding website files and workflow'. However, before we push these changes to the remote GitHub repository we need to configure the "AZURE_STORAGE_CONNECTION_STRING" GitHub Actions secret value that is referenced in the workflow file.

Github Actions Secret

In the Azure portal, the storage account connection string can be copied by going to the Storage Account that was created previously. In the left sidebar find the settings section that contains the Access keys configuration.

You can choose the connection string for key1 or key2, but make sure to use the connection string field value and not the Key field value.

Copy the connection string from the Azure portal settings and go to your GitHub repository interface, and in the top navigation there will be a section labelled "Settings". Clicking on "Settings" will bring you to a new page with a left side navigation that contains a section labelled "Secrets". This is where we will use the copied Azure Blob storage connection string to create a new GitHub Actions secret. On the Secrets page click the button to add a "New repository secret".

Then add the secret name that matches the variable name from the "main.yml" workflow file, in this example it is AZURE_STORAGE_CONNECTION_STRING, and paste the Azure Storage connection string into the Value field.

Then click "Add Secret" and our GitHub Actions Workflow can now access the Azure Storage connection string to sync files to Azure Blob Storage. Our GitHub Actions secret is now configured and ready for use.

Git Push to Deploy Static Website with GitHub Actions to Azure Blob Storage Static Website

Now you can run the command git push -u origin main from your local project and you should see the commit made previously has been pushed to the remote repository in the GitHub interface. In the top navigation of the GitHub repository, the "Actions" tab should now show the workflow automation is underway.

When the workflow build and deployment process completes, navigate to the primary endpoint for the Azure Blob Storage static website, that was displayed in the first step.

You should see the home page now displays the content of our site and you can navigate to the About page that was created earlier. If you accidentally navigate to a page that does not exist, the error document path is set to 404.html, and Azure Blob Storage will return the 404.md page content that was added as a page within the static site.

Anytime you want to make changes to your site in the future, you can follow the same process as this example by first committing the changes to the main branch and then pushing the changes to the remote GitHub repository. This will trigger the GitHub Actions workflow and the changes will be automatically updated on the static website.

npm init package.json

You will not need to to do this if you have an existing Node.js project setup but if you do not, then make sure to first install Node.js and npm. Then in a terminal window open new folder for the project and run the command npm init, and follow the prompts that are displayed. The package.json file should have been added to the project folder.

We also need to make one addition to the package.json file after it has been generated so that ES Modules can be used with Node.js. To support ES Modules add a "type" property to the package.json file object with the value set to "module". In the following steps we will configure the TypeScript compiler to output JavaScript using the ES Module format.

npm install

With the package.json generated we can run additional commands to install the npm packages we will use. In the same project folder run the command npm install xml typescript --save, this will install the XML and TypeScript npm packages. After that run another command npm install @types/xml --save-dev. This will install the TypeScript type definitions for the XML npm package. Your package.json should look similar to this:

{
  "type": "module",
  "name": "convertjsontoxml",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "typescript": "^4.2.3",
    "xml": "^1.0.1"
  },
  "devDependencies": {
    "@types/xml": "^1.0.5"
  }
}

Compile TypeScript

Now that we have installed the XML and TypeScript npm packages installed, we can configure TypeScript to compile our code to use with Node.js, by adding an npm package.json script. To do this add the following the the "scripts" property in the package.json file that was created in the first step:

{
  "scripts": {
    "compile": "tsc --allowSyntheticDefaultImports --isolatedModules --moduleResolution node --module esnext index.ts"
  }
}

The compile command will invoke the TypeScript compiler with the CLI flags that will generate the JavaScript output using the ES Module format. This will match the "type" property set to "module" in the package.json configured earlier. You can run this command using npm run compile in the terminal window.

Create Node.js Script

Next we can create a Node.js script, and as referenced in the package.json scripts "compile" command, the name of this file is index.ts. Here we will write the TypeScript code that will use the XML npm package to generate an XML string from a JSON object. In the index.ts file add the following code:

import { promisify } from "util";
import { readFile, writeFile } from "fs";
import xml from "xml";

const readFilePromise = promisify(readFile);
const writeFilePromise = promisify(writeFile);

(async function convertJsonToXml() {})();

This will set up the import statements for the XML npm package and also import the readFile and writeFile functions from the node fs module. Since these functions use callbacks by default, the promisify function is imported from the util module to convert the readFile and writeFile functions into promises. This way we can use async/await syntax.

Read JSON File

In the ConvertJsonToXml function the first thing we can do is read the JSON file containing a sample JSON object that we can convert to an XML string. Create a new file named "data.json" in the same project folder, and add this sample JSON object:

[
  {
    "name": "Next.js",
    "language": "JavaScript",
    "templates": "React",
    "description": "A framework for statically-exported React apps (supports server side rendering)"
  },
  {
    "name": "Gatsby",
    "language": "JavaScript",
    "templates": "React",
    "description": "Build blazing fast, modern apps and websites with React"
  },
  {
    "name": "Nuxt",
    "language": "JavaScript",
    "templates": "Vue",
    "description": "A minimalistic framework for serverless Vue.js applications."
  }
]

In the index.js file, inside of the ConvertJsonToXml function we can add this code to read the JSON file and parse it into a JSON object with the corresponding type signature:

const staticSiteGeneratorData = JSON.parse(
  await readFilePromise("data.json", "utf8")
) as [
  {
    name: string;
    language: string;
    templates: string;
    description: string;
  }
];

Once the json file is read and saved as the "staticSiteGeneratorData" variable we can use the Array.prototype.map() method to shape the data into the format we need, in order to use the XML npm package to convert the JSON object into an XML string. Below the code that is reading the data.json file add this code:

const xmlFormattedStaticSiteGeneratorData = [
  {
    staticSiteGenerators: [
      ...staticSiteGeneratorData.map((item) => {
        return {
          staticSiteGenerator: [
            {
              _attr: {
                language: item.language,
                templates: item.templates,
                description: item.description,
              },
            },
            item.name,
          ],
        };
      }),
    ],
  },
];

The result of the data that is assigned to the "xmlFormattedStaticSiteGeneratorData" variable will look like this:

[
  {
    "staticSiteGenerators": [
      {
        "staticSiteGenerator": [
          {
            "_attr": {
              "language": "JavaScript",
              "templates": "React",
              "description": "A framework for statically-exported React apps (supports server side rendering)"
            }
          },
          "Next.js"
        ]
      },
      {
        "staticSiteGenerator": [
          {
            "_attr": {
              "language": "JavaScript",
              "templates": "React",
              "description": "Build blazing fast, modern apps and websites with React"
            }
          },
          "Gatsby"
        ]
      },
      {
        "staticSiteGenerator": [
          {
            "_attr": {
              "language": "JavaScript",
              "templates": "Vue",
              "description": "A minimalistic framework for serverless Vue.js applications."
            }
          },
          "Nuxt"
        ]
      }
    ]
  }
]

Convert JSON File to an XML String

The JSON data assigned to the "xmlFormattedStaticSiteGeneratorData" variable, is now in the appropriate format to use with the XML npm package. Directly below the code that formats the data, and inside the "convertJsonToXml" function, add the following code:

const staticSiteGeneratorXmlString = xml(xmlFormattedStaticSiteGeneratorData);

The format of the xml string assigned to the "staticSiteGeneratorXmlString" is going to look like this:

<staticSiteGenerators>
    <staticSiteGenerator language="JavaScript" templates="React" description="A framework for statically-exported React apps (supports server side rendering)">Next.js</staticSiteGenerator>
    <staticSiteGenerator language="JavaScript" templates="React" description="Build blazing fast, modern apps and websites with React">Gatsby</staticSiteGenerator>
    <staticSiteGenerator language="JavaScript" templates="Vue" description="A minimalistic framework for serverless Vue.js applications.">Nuxt</staticSiteGenerator>
</staticSiteGenerators>

Write XML File

The XML string assigned to the variable "staticSiteGeneratorDataXmlString" can be written to an XML file with the writeFile module that we imported and promisified at the beginning of the index.ts file. To write the XML string to a file in the same project folder add this code below the XML npm package usage that was included in the prior step:

await writeFilePromise("data.xml", staticSiteGeneratorXmlString, "utf8");

Put all the code together and the index.ts file should look like this:

import { promisify } from "util";
import { readFile, writeFile } from "fs";
import xml from "xml";

const readFilePromise = promisify(readFile);
const writeFilePromise = promisify(writeFile);

(async function convertJsonToXml() {
  const staticSiteGeneratorData = JSON.parse(
    await readFilePromise("data.json", "utf8")
  ) as [
    {
      name: string;
      language: string;
      templates: string;
      description: string;
    }
  ];

  const xmlFormattedStaticSiteGeneratorData = [
    {
      staticSiteGenerators: [
        ...staticSiteGeneratorData.map((item) => {
          return {
            staticSiteGenerator: [
              {
                _attr: {
                  language: item.language,
                  templates: item.templates,
                  description: item.description,
                },
              },
              item.name,
            ],
          };
        }),
      ],
    },
  ];

  const staticSiteGeneratorXmlString = xml(xmlFormattedStaticSiteGeneratorData);

  await writeFilePromise("data.xml", staticSiteGeneratorXmlString, "utf8");
})();

Run Node.js Script with npm package.json Scripts

To test this code out and run the Node.js script we can add another package.json script that will first compile the TypeScript into JavaScript and then run the JavaScript output with Node.js. In the package.json file add a new package.json script named "start" that looks like this:

{ "scripts": { "start": "npm run compile && node index.js" } }

To use the start script run the command npm run start and you should then see the XML file generated and saved to the project folder. The contents of this file should match the format of the XML string shown previously. Anytime you want to change the data or formatting make sure to run the npm run start again to regenerate the data.xml file.

The XML npm package is a convenient way to convert JSON into XML, as long as the JSON data is formatted appropriately, or there is a step involved to properly format the original JSON data source into the format the XML npm package requires. For other usages of the XML npm packages you can read my other posts showing how to generate an XML sitemap and generate an XML RSS feed, like this example, both of these posts are using Node.js and npm.

In this example, TypeScript is used to create a service worker with a network first then cache, caching strategy, to support viewing previously visited pages when offline. When there is no network connection available and a previous page version is not cached, the service worker will display an offline page.

Configure TypeScript and Babel with ES Modules

Before adding the TypeScript code for the service worker there are some prerequisite steps to configure the TypeScript compiler. If you want to include Typescript into your project you can view these posts:

• Setup TypeScript Compilation with npm package.json Scripts
• Import and Export ES Modules in Node.js using TypeScript with Babel Compilation

to find more information about using TypeScript. The configuration will include Babel for TypeScript compilation as shown in the second post above with some minor adjustments, to support browser use rather than Node.js, made to the Babel configuration.

Configure package.json Scripts

Create a package.json file by running npm init and then run the command npm install typescript cross-env @babel/cli @babel/core @babel/preset-env @babel/preset-env @babel/preset-typescript --save. You then need to add three package.json scripts that look like this:

{
  "scripts": {
    "typecheck": "tsc --p .",
    "compile": "cross-env-shell babel $INIT_CWD -d $INIT_CWD --extensions '.ts' --no-comments --source-maps",
    "typecheck-compile": "npm run typecheck && npm run compile"
  }
}

Configure TypeScript for Type Checking Only

The package.json "typecheck" script will invoke the TypeScript compiler and only is responsible for type checking the TypeScript code, and it will not emit any JavaScript. The TypeScript compiler configuration options are specified in a file named "tsconfig.json". You can create this file with the following settings:

{
  "compilerOptions": {
    "allowSyntheticDefaultImports": true,
    "isolatedModules": true,
    "strict": true,
    "module": "esnext",
    "lib": ["es2019", "es6", "dom", "webworker"],
    "noEmit": true,
    "moduleResolution": "node",
    "skipLibCheck": true
  },
  "include": ["*.ts"],
  "exclude": ["node_modules/**/*"]
}

Notice that the "lib" property contains a value for "webworker" among the values. This is important because it indicates to the TypeScript compiler that it should load the type definitions for the Worker APIs.

The package.json "compile" script uses Babel to compile TypeScript and will be responsible for outputting JavaScript. The Babel compiler does not type check the TypeScript source code, so if there are any errors present the Babel compiler may attempt to provide output rather than indicating there was an error. Also the Babel compiler CLI command requires a source folder and output folder to be specified so this is set to the project folder with the "$INIT_CWD" npm CLI variable, for both the source and output folders. This way the JavaScript files will be written to the same folder as the TypeScript files.

Configure Babel with @babel/preset-typescript and @babel/preset-env

We are going to target browsers that are using ES Modules with the preset-env preset, in addition to using the preset-typescript preset. You can configure this by adding a babel.config.json file in the same folder as the package.json with the following settings:

{
  "presets": [
    [
      "@babel/preset-env",
      {
        "modules": false,
        "targets": { "esmodules": true }
      }
    ],
    ["@babel/preset-typescript"]
  ],
  "ignore": ["node_modules"],
  "comments": false,
  "minified": true
}

The presets are applied in the reverse order that they are listed in the Babel.config.json, so the preset-env changes will be applied to the JavaScript output created by preset-typescript.

TypeScript Type Checking with Babel Compilation

To combine both the typecheck and compile scripts you can run the package.json typecheck-compile command: npm run typecheck-compile. This will first use the TypeScript compiler to typecheck the code and then use the Babel compiler to generate JavaScript for the browser. If there are any type checking errors the command will fail and they will be shown in the console.

Register Service Worker

Now that we have TypeScript and Babel configured to use ES Modules, but before creating the actual service worker code, we need to write some code that will register the service worker in the browser. This code can be included in the HTML source directly, or in a script tag. If you are including the code with a script tag make sure to reference the JavaScript output file that is generated as a result of the TypeScript compilation process and not the TypeScript source file. We can name this file "script.ts" and place it in the root of the project.

export default class Main {
  constructor() {
    if ("serviceWorker" in navigator) {
      navigator.serviceWorker
        .register("/service-worker.js", { scope: "/" })
        .then(function () {
          console.log("Service Worker Registered");
        });
    }
  }
}

new Main();

HTML include script type="module"

You can include this script in an HTML file by creating a new file named "index.html", and save it in the root of the project, with this content:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>Service Worker Example</title>
  </head>
  <body>
    This page is loading a service worker!
    <script type="module" src="/script.js"></script>
  </body>
</html>

When the browser loads the HTML page the JavaScript above will attempt to load the service worker. If the browser does not support service workers the code will be skipped.

Besides the index.html file we are also going to need an HTML file for an offline page. In the same folder that the index.html file is saved create a new file name "offline.html" and add this content to show an offline state:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>You are not connected to the internet</title>
  </head>
  <body>
    You are not connected to the internet!
    <script type="module" src="/script.js"></script>
  </body>
</html>

When the service worker code is added in the following step the offline page will be automatically cached for future usage.

Set up Service Worker

Registering the service worker is what invokes the service worker code that is not running on the main thread. Inside of the service worker we will write code to listen for certain types of events that are triggered by the browser. These events are:

• install
• activate
• fetch

We can add the install event first by adding the following code to a file named "service-worker.ts". You can use a different name if you want, but make sure that it matches what is included in the "navigator.serviceWorker.register" function used in the previous step. In the service-worker.js file add the following code:

/// <reference lib="WebWorker" />

// export empty type because of tsc --isolatedModules flag
export type {};
declare const self: ServiceWorkerGlobalScope;

This is the beginning code for using a service worker with TypeScript and Babel compilation. The first line is a triple-slash directive to indicate to the TypeScript compiler to use the Worker API type definitions. This is why it was important to note the inclusion of the Worker API type definitions in the "lib" setting of the tsconfig.json file.

After loading the Worker API type definitions, the next line is and empty type export that prevents an error as a result of the typescript compiler option "isolatedModules". Using "isolatedModules" is recommend when using Babel with TypeScript, but this requires all modules to have either one export or import statement. Since the service worker is a standalone script it cannot import or export modules, and by including the empty type export, the typescript compiler will no longer show the error:

(error TS1208: All files must be modules when the '--isolatedModules' flag is provided)[]

The empty type export will be removed when they TypeScript code is compiled into JavaScript with Babel. This is a workaround until service workers have module support, and only needed when using the "--isolatedModules" flag with TypeScript.

The following declaration allows the type "ServiceWorkerGlobalScope" included in the WebWorker API type definitions, to be specified for the "self" variable that already exists. Without the type declaration the self variable would have the type "Window & typeof globalThis", and the type definitions of that type do not overlap with the "ServiceWorkerGlobalScope" type. This is because the service worker does not have access to the window object or any other global scope, and the self variable used in the service worker code is of type WorkerGlobalScope.self, rather than Window.self.

Similar to the empty type export shown above this is also a workaround that is only needed to inform the TypeScript compiler of the proper types that are being used. There is some discussion regarding the type signatures on the TypeScript Github repository:

• Service Worker Typings
• Unable to access ServiceWorkerGlobalScop via self

Service Worker install Event

The first event listener that is added to the code, listens for the service worker install event to be triggered. When the install event occurs a cache using a combination of the "cacheName" and "version" variables is created, if it does not exist, and the index and offline HTML pages will be automatically added to the cache. Add this code below the "self" variable declaration:

const cacheName = "::yourserviceworker";
const version = "v0.0.1";

self.addEventListener("install", function (event) {
  event.waitUntil(
    caches.open(version + cacheName).then(function (cache) {
      return cache.addAll(["/", "/offline"]);
    })
  );
});

Service Worker activate Event

Below the install event listener code we can add a separate event listener that will be triggered on the activate event. The activate event occurs after the install event, and is used to clean up any old service worker caches.

self.addEventListener("activate", function (event) {
  event.waitUntil(
    caches.keys().then(function (keys) {
      // Remove caches whose name is no longer valid
      return Promise.all(
        keys
          .filter(function (key) {
            return key.indexOf(version) !== 0;
          })
          .map(function (key) {
            return caches.delete(key);
          })
      );
    })
  );
});

On activation if the "version" variable declared at the top of the service worker has been updated, any existing caches that do not match the new variable value will be deleted. This makes sure that the service worker is always using the latest cache values. The version variable acts as a cache bust when making changes to the service worker, after it has been previously deployed, otherwise the cached values will continue to be used until the service worker is no longer registered.

Service Worker fetch Event

After install and activation the service worker will listen for fetch events. In the fetch event listener function we can intercept responses and add them to the cache after the fetch response is complete, or the network can be bypassed and the response can be returned directly from the cache. The fetch event listener can be included below the activate event listener like this:

self.addEventListener("fetch", function (event) {
  const request = event.request;

  // Always fetch non-GET requests from the network
  if (request.method !== "GET") {
    event.respondWith(
      fetch(request).catch(function () {
        return caches.match("/offline");
      }) as Promise<Response>
    );
    return;
  }
});

If the request is not a GET request, for example it could be a POST or PUT request, the service worker will always defer these requests to the network and the response is never cached. In the event that there is no network connection and the network request fails the service worker will return the offline page for any requests that are not GET requests.

Network First then Cache - HTML Caching Strategy

Any HTML request will use a network first then cache, caching strategy. This enables the latest version of the page to be requested from the origin server if there is a network connection. Without a network connection, the service worker will check if a prior version of an HTML page is available in the cache, and if not the offline page will be returned. Below the non-GET requests conditional block add this code to proxy HTML requests:

// For HTML requests, try the network first, fall back to the cache,
// finally the offline page
if (
  request.headers.get("Accept")?.indexOf("text/html") !== -1 &&
  request.url.startsWith(this.origin)
) {
  // The request is text/html, so respond by caching the
  // item or showing the /offline offline
  event.respondWith(
    fetch(request)
      .then(function (response) {
        // Stash a copy of this page in the cache
        const copy = response.clone();
        caches.open(version + cacheName).then(function (cache) {
          cache.put(request, copy);
        });
        return response;
      })
      .catch(function () {
        return caches.match(request).then(function (response) {
          // return the cache response or the /offline page.
          return response || caches.match("/offline");
        });
      }) as Promise<Response>
  );
  return;
}

Cache First then Network - non-HTML Caching Strategy

For non-HTML requests, first the cache is checked and if the resource is in the cache it will be returned. If the non-HTML requests is not in the cache it will be requested from the origin server as long as there is a network connection, and then the response will be added to the cache for future requests. Add this code below the code that handles HTML requests:

// For non-HTML requests, look in the cache first, fall back to the network
if (
  request.headers.get("Accept")?.indexOf("text/plain") === -1 &&
  request.url.startsWith(this.origin)
) {
  event.respondWith(
    caches.match(request).then(function (response) {
      return (
        response ||
        fetch(request)
          .then(function (response) {
            const copy = response.clone();

            if (
              copy.headers.get("Content-Type")?.indexOf("text/plain") === -1
            ) {
              caches.open(version + cacheName).then(function (cache) {
                cache.put(request, copy);
              });
            }

            return response;
          })
          .catch(function () {
            // you can return an image placeholder here with
            if (request.headers.get("Accept")?.indexOf("image") !== -1) {
            }
          })
      );
    }) as Promise<Response>
  );
  return;
}

NOTE: If the request is for a plaintext resource it will be ignored by the service worker with this configuration. Additionally if request Accept header does not include "text/plain", but the response Content-Type header is of "text/plain" then the response will not be stored in the service worker cache. For any other response types like css, js, png, or jpg, the response will be cached by the service worker in this code section.

Here's what the entire service-worker.js file should look like will all the sections put together:

/// <reference lib="WebWorker" />

// export empty type because of tsc --isolatedModules flag
export type {};
declare const self: ServiceWorkerGlobalScope;

const cacheName = "::yourserviceworker";
const version = "v0.0.1";

self.addEventListener("install", function (event) {
  event.waitUntil(
    caches.open(version + cacheName).then(function (cache) {
      return cache.addAll(["/", "/offline"]);
    })
  );
});

self.addEventListener("activate", function (event) {
  event.waitUntil(
    caches.keys().then(function (keys) {
      // Remove caches whose name is no longer valid
      return Promise.all(
        keys
          .filter(function (key) {
            return key.indexOf(version) !== 0;
          })
          .map(function (key) {
            return caches.delete(key);
          })
      );
    })
  );
});

self.addEventListener("fetch", function (event) {
  const request = event.request;

  // Always fetch non-GET requests from the network
  if (request.method !== "GET") {
    event.respondWith(
      fetch(request).catch(function () {
        return caches.match("/offline");
      }) as Promise<Response>
    );
    return;
  }

  // For HTML requests, try the network first, fall back to the cache,
  // finally the offline page
  if (
    request.headers.get("Accept")?.indexOf("text/html") !== -1 &&
    request.url.startsWith(this.origin)
  ) {
    // The request is text/html, so respond by caching the
    // item or showing the /offline offline
    event.respondWith(
      fetch(request)
        .then(function (response) {
          // Stash a copy of this page in the cache
          const copy = response.clone();
          caches.open(version + cacheName).then(function (cache) {
            cache.put(request, copy);
          });
          return response;
        })
        .catch(function () {
          return caches.match(request).then(function (response) {
            // return the cache response or the /offline page.
            return response || caches.match("/offline");
          });
        }) as Promise<Response>
    );
    return;
  }

  // For non-HTML requests, look in the cache first, fall back to the network
  if (
    request.headers.get("Accept")?.indexOf("text/plain") === -1 &&
    request.url.startsWith(this.origin)
  ) {
    event.respondWith(
      caches.match(request).then(function (response) {
        return (
          response ||
          fetch(request)
            .then(function (response) {
              const copy = response.clone();

              if (
                copy.headers.get("Content-Type")?.indexOf("text/plain") === -1
              ) {
                caches.open(version + cacheName).then(function (cache) {
                  cache.put(request, copy);
                });
              }

              return response;
            })
            .catch(function () {
              // you can return an image placeholder here with
              if (request.headers.get("Accept")?.indexOf("image") !== -1) {
              }
            })
        );
      }) as Promise<Response>
    );
    return;
  }
});

Test Service Worker Locally

You can test the service worker by running your project locally with the http-server npm package. First make sure to compile the TypeScript service worker code by using the command npm run build-typecheck. Then, to install the http-server npm package run the command npm i http-server --save-dev. After installing run the command http-server in your project folder where the index.html and offline.html pages are. You should then see your website in the browser by navigating to the default http-server url localhost:8080. With browser dev tools you can inspect your website including managing service worker installation state and cache status.

In Chrome this is in the "Application" tab of chrome DevTools:

In the left hand navigation panel of the Application tab you can also see a section for "Cache Storage", and in the dropdown list should be an entry for the service worker, with the name you chose, listing all of the cached assets. There is three cached responses: the index.html page, the offline.html page, and the script.js file that registers the service worker.

Here's what that looks like in Chrome:

You can test offline mode by selecting the checkbox for "Offline" and deleting the cache item with name "/". Refreshing the page should load the offline.html since the index.html is no longer cached.

Your site can now support offline viewing for pages that are cached with the service worker, or show an offline page when there is no response previously cached and a network connection is unavailable.

npm init package.json

The first thing we need to do is to generate a package.json file so that we can install the xml npm package. If you don't already have a package.json file setup for your project run the command npm init in the project folder and follow the prompts. Once the package.json file is created run the command npm install xml typescript --save. This will install the xml npm package and the TypeScript npm package. Since we are using TypeScript for this example we also need to install the type definitions for the xml package. These can be installed by running the command npm install @types/xml --save-dev.

Configure ES Module format

We are also going to use ECMAScript modules, or ES modules, instead of CommonJS modules, as Node.js now supports ES Module format. In order to use ES Modules the "type" property with value of "module" also needs to be added to the package.json file. Please read my other post for more information regarding how to import and export ES Modules in Node.js With these settings in place the package.json file should look similar to this:

{
  "type": "module",
  "name": "xmlsitemap",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "typescript": "^4.2.3",
    "xml": "^1.0.1"
  },
  "devDependencies": {
    "@types/xml": "^1.0.5"
  }
}

Configure TypeScript with Node.js

After adding the package.json file we can include the configuration steps that are needed to use TypeScript with ES Modules. To do this we can add a "tsconfig.json" file with the following settings:

{
  "compilerOptions": {
    "allowSyntheticDefaultImports": true,
    "isolatedModules": true,
    "strict": true,
    "module": "esnext",
    "lib": ["ES2019"],
    "moduleResolution": "node",
    "skipLibCheck": true
  },
  "include": ["index.ts"],
  "exclude": ["node_modules/**/*"]
}

The "module" setting with value "esnext" is the setting that configures the TypeScript compiler to compile TypeScript into JavaScript using the ES Module format.

Generate XML String with TypeScript

Now we can create the Node.js script that will generate the XML string that will be written to the sitemap file. To do this add a new file to the project named "index.ts". And add the following code:

import xml from "xml";
import { writeFile } from "fs";

async function main() {
  const pages = [
    {
      title: "Sample Page One",
      created: "Dec 22 2020",
      slug: "sample-page-one",
    },
    {
      title: "Sample Page Two",
      created: "Feb 1 2021",
      lastModified: "Feb 2 2021",
      slug: "sample-page-two",
    },
    {
      title: "Sample Page Three",
      created: "Mar 2 2021",
      lastModified: "Mar 5 2021",
      slug: "sample-page-three",
    },
    {
      title: "Sample Page Four",
      created: "Mar 20 2021",
      slug: "sample-page-four",
    },
  ];

  const indexItem = {
    //todo: build index item
  };

  const sitemapItems = pages.reduce(async function (
    items: { url: [{ loc: string }, { lastmod: string }] }[],
    item: {
      title: string;
      lastModified?: string;
      created: string;
      slug: string;
    }
  ) {
    // todo: build page items
    return items;
  }, []);
}

main();

This code is now set up to use sample data that is stored in the "pages" array. In this example we are including data to represent pages that would be included as part of a static site generator build process. Typically with a Jamstack blog this data would be sourced from markdown files, or another common option is requesting data from a headless content management system. For the purposes of this example we are including a short page list directly in the code, but usually this would be dynamically included at build time. After the sample page data there is one object that will contain the data for the sitemap index item, and the other is an array of objects containing the sitemap data for each individual page.

Create Sitemap Index Item

The first item in the sitemap will include optional tags that won't be included in the individual page sitemap items, and that is why it is created separately. Besides including the url location and a last modified time, the index sitemap item includes the change frequency parameter and a priority parameter. These are optional and can be included for each sitemap item, but in this case we are only including it for the root url of the sitemap. Go ahead an add the following inside of the "indexItem" object shown above:

const indexItem = {
  //build index item
  url: [
    {
      loc: "YOUR-DOMAIN-HERE",
    },
    {
      lastmod: new Date(
        Math.max.apply(
          null,
          pages.map((page) => {
            return new Date(
              page.lastModified ?? page.created
            ) as unknown as number;
          })
        )
      )
        .toISOString()
        .split("T")[0],
    },
    { changefreq: "daily" },
    { priority: "1.0" },
  ],
};

Make sure to replace "YOUR-DOMAIN-HERE" with your actual domain. Also note that in order to find the most recent date of all of the pages, the Math.max() function is used, in combination with the function prototype method .apply(), which passes the array of page object dates as parameters to the Math.max function. The first parameter of the .apply method is this, which is not needed so it is set to null.

Additionally since we are using TypeScript the date objects cannot be cast directly from a JavaScript Date object into a number, so they are cast to the unknown type as an intermediary step to prevent the TypeScript compiler from showing type errors. Once the maximum date of all the pages last modified or created dates is determined, it is formatted as an ISO string date format and then trimmed to only include the year, month, and day.

Create Sitemap Page Items

With the index sitemap item created, we can now build the individual page items in the "sitemapItems" array. This process will be similar to creating the index item, but each page item will only include a url location property and a last modified timestamp. To build the sitemap items add the following code to the sitemapItems array creation step:

const sitemapItems = pages.reduce(function (
  items: { url: [{ loc: string }, { lastmod: string }] }[],
  item: {
    title: string;
    lastModified?: string;
    created: string;
    slug: string;
  }
) {
  // build page items
  items.push({
    url: [
      {
        loc: `YOUR-DOMAIN-HERE/${item.slug}`,
      },
      {
        lastmod: new Date(item.lastModified ?? item.created)
          .toISOString()
          .split("T")[0],
      },
    ],
  });
  return items;
}, []);

For each page items url location property make sure to replace the placeholder text with your actual domain.

Build Sitemap Object

Now that both the sitemap index item and the sitemap items for each page are created we can combine them into one object that will be the entire sitemap object. At the end of the main function add in this code:

const sitemapObject = {
  urlset: [
    {
      _attr: {
        xmlns: "http://www.sitemaps.org/schemas/sitemap/0.9",
      },
    },
    indexItem,
    ...sitemapItems,
  ],
};

const sitemap = `<?xml version="1.0" encoding="UTF-8"?>${xml(sitemapObject)}`;

console.log(sitemap);

At this point we can test to make sure that the JSON to xml string conversion is working properly by logging the xml string to the console. To do this we need to add a script command to the package.json file created earlier.

Run Node.js Script with npm Scripts

In order to test the xml sitemap creation process we can add a script to the package.json file named "generate-sitemap". This script will invoke the TypeScript compiler and then run the transpiled JavaScript with Node.js. Here is what the script should look like in the package.json file:

{
  "scripts": {
    "generate-sitemap": "tsc && node index.js"
  }
}

We can run this script with the command npm run generate-sitemap. After running the generate sitemap command, the xml sitemap string should be output to the console. Here is what it will look like:

<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><url><loc>YOUR-DOMAIN-HERE</loc><lastmod>2021-03-20</lastmod><changefreq>daily</changefreq><priority>1.0</priority></url><url><loc>YOUR-DOMAIN-HERE/sample-page-one</loc><lastmod>2020-12-22</lastmod></url><url><loc>YOUR-DOMAIN-HERE/sample-page-two</loc><lastmod>2021-02-02</lastmod></url><url><loc>YOUR-DOMAIN-HERE/sample-page-three</loc><lastmod>2021-03-05</lastmod></url><url><loc>YOUR-DOMAIN-HERE/sample-page-four</loc><lastmod>2021-03-20</lastmod></url></urlset>

Instead of outputting the sitemap as an xml string we can write it to a file using the Node.js write file method in the fs module.

Write XML String to Sitemap File

You can replace the "console.log" statement at the bottom of the index.ts main function with the following code to write the sitemap xml string to a file named "sitemap.xml":

await writeFileAsync("./sitemap.xml", sitemap, "utf8");

You will also need to add one import statements at the top of the index.ts file. This will import the promisify function from the util module. This way we can convert the writeFile module to use promises instead of callbacks, which enables the use of async await syntax.

import { promisify } from "util";
const writeFileAsync = promisify(writeFile);

Here is what the entire index.ts file should look like with all the code included:

import xml from "xml";
import { writeFile } from "fs";
import { promisify } from "util";
const writeFileAsync = promisify(writeFile);

async function main() {
  const pages = [
    {
      title: "Sample Page One",
      created: "Dec 22 2020",
      slug: "sample-page-one",
    },
    {
      title: "Sample Page Two",
      created: "Feb 1 2021",
      lastModified: "Feb 2 2021",
      slug: "sample-page-two",
    },
    {
      title: "Sample Page Three",
      created: "Mar 2 2021",
      lastModified: "Mar 5 2021",
      slug: "sample-page-three",
    },
    {
      title: "Sample Page Four",
      created: "Mar 20 2021",
      slug: "sample-page-four",
    },
  ];

  const indexItem = {
    //build index item
    url: [
      {
        loc: "YOUR-DOMAIN-HERE",
      },
      {
        lastmod: new Date(
          Math.max.apply(
            null,
            pages.map((page) => {
              return new Date(
                page.lastModified ?? page.created
              ) as unknown as number;
            })
          )
        )
          .toISOString()
          .split("T")[0],
      },
      { changefreq: "daily" },
      { priority: "1.0" },
    ],
  };

  const sitemapItems = pages.reduce(function (
    items: { url: [{ loc: string }, { lastmod: string }] }[],
    item: {
      title: string;
      lastModified?: string;
      created: string;
      slug: string;
    }
  ) {
    // build page items
    items.push({
      url: [
        {
          loc: `YOUR-DOMAIN-HERE/${item.slug}`,
        },
        {
          lastmod: new Date(item.lastModified ?? item.created)
            .toISOString()
            .split("T")[0],
        },
      ],
    });
    return items;
  }, []);

  const sitemapObject = {
    urlset: [
      {
        _attr: {
          xmlns: "http://www.sitemaps.org/schemas/sitemap/0.9",
        },
      },
      indexItem,
      ...sitemapItems,
    ],
  };

  const sitemap = `<?xml version="1.0" encoding="UTF-8"?>${xml(sitemapObject)}`;

  await writeFileAsync("./sitemap.xml", sitemap, "utf8");
}

main();

You can then run the npm run generate-sitemap command again, and a new file named "sitemap.xml" should be created in the project folder. The contents of this file should be identical to the sitemap xml string that was logged to the console in the previous step.

Test Sitemap File in a Browser

To test out the sitemap in a browser, in the same project folder that we created the index.ts file run the command npm install http-server --save-dev, and add another script to the package.json file like this:

{
  "scripts": {
    "generate-sitemap": "tsc && node index.js",
    "serve": "http-server"
  }
}

Then to use the http-server npm package run the command npm run serve, and you should see the http-server npm package output the url it is serving. It is most likely the default setting so navigating to "localhost:8080/sitemap.xml" should show the sitemap file that will look similar to this:

Add Sitemap to robots.txt

Now you can include the sitemap generation step in the build process of the static site generator that you might be using for your Jamstack blog. You can also add a line to the robots.txt file to indicate the url of the sitemap file. If you are using a robots.txt file for your site make sure to add the following with your domain included:

Sitemap: https://YOUR-DOMAIN-HERE/sitemap.xml

The SASS team now recommends using Dart Sass in favor of LibSass for new development projects. This means that the sass npm package should be used instead of the node-sass npm package, which is built on top of LibSass, to compile sass with Node.js. The sass npm package is a pure JavaScript implementation of Dart Sass. The Dart Sass JavaScript API strives to be compatible with the existing node-sass API, so that it can be integrated into existing workflows with minimal changes. This post will show how to install the Dart Sass Javascript implementation with npm and use it via the supported JavaScript API and the command line. Before proceeding make sure to have Node.js and npm installed.

npm install sass

After installing Node.js and npm we can create a sample project to demonstrate the functionality of the sass npm package. To do this create a project folder and open it in a terminal window. Then run the command npm init and follow the prompts, which will create a package.json file. Then we can install the sass node module into the project, by running the command npm install sass --save.

We will also be using ES Module format for this example so the package.json requires an additional setting after generating. Add the "type" property to the package.json with the value set to "module", so that Node.js will use ES Modules rather than CommonJS modules. Here is some additional information about how to import and export ES Modules in Node.js, which explains why this setting is necessary.

Your package.json file should now look like this:

{
  "name": "npmsass",
  "type": "module",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "sass": "^1.32.8"
  }
}

SCSS

The sass npm package is now installed, but in order to use it we will need a SCSS file. In the same project folder create a new file named "styles.scss" and place the following code inside:

/* This CSS will print because %message-shared is extended. */
%message-shared {
  border: 1px solid #ccc;
  padding: 10px;
  color: #333;
}

// This CSS won't print because %equal-heights is never extended.
%equal-heights {
  display: flex;
  flex-wrap: wrap;
}

.message {
  @extend %message-shared;
}

.success {
  @extend %message-shared;
  border-color: green;
}

.error {
  @extend %message-shared;
  border-color: red;
}

.warning {
  @extend %message-shared;
  border-color: yellow;
}

The above SCSS code is borrowed from the Sass Basics guide and demonstrates one of the most useful features of Sass which is the @extend at-rule, to share a set of CSS properties among different selectors. Now that we have a SCSS file we can compile it to CSS using the sass npm package.

Compile Sass with Dart Sass JavaScript API

To use the sass npm package JavaScript API, we need to create the index.js file that is set to the "main" property value in the package.json file, created in the first step. This will be the entry point for the Node.js process that will carry out the SASS compilation. In the same project folder create a new file named "index.js", and add the following code:

import sass from "sass";
import { promisify } from "util";
const sassRenderPromise = promisify(sass.render);

async function main() {
  const styleResult = await sassRenderPromise({
    file: `${process.cwd()}/styles.scss`,
  });

  console.log(styleResult.css.toString());
}
main();

This code imports the sass package along with the util.promisify module and converts the sass render function to use promises instead of the default callback implementation. This makes working with the asynchronous API of the sass npm package more manageable because it allows for the use of async/await syntax.

After importing and "promisifying" the sass npm package, the main function contains the code to compile the styles.scss file into CSS. In order to run this code add the following the scripts property in the package.json file:

{
  "start": "node index.js"
}

We can then execute the main function by running the command npm run start, and the css output will be logged to the console.

Instead of logging directly to the console it is much more useful to write the CSS output to a file. The sass npm package does not expose a JavaScript API to write a file directly, however it does support a configuration property to indicate which file the CSS output will be written to. By adding this configuration and using the fs.writeFile module the CSS can be written to an output to a file like this:

import sass from "sass";
import { promisify } from "util";
import { writeFile } from "fs";
const sassRenderPromise = promisify(sass.render);
const writeFilePromise = promisify(writeFile);

async function main() {
  const styleResult = await sassRenderPromise({
    file: `${process.cwd()}/styles.scss`,
    outFile: `${process.cwd()}/styles.css`,
  });

  console.log(styleResult.css.toString());

  await writeFilePromise("styles.css", styleResult.css, "utf8");
}
main();

After running the npm run start command again, you should now see a styles.css file in the same project folder, that contains the compiled CSS output:

/* This CSS will print because %message-shared is extended. */
.warning,
.error,
.success,
.message {
  border: 1px solid #ccc;
  padding: 10px;
  color: #333;
}

.success {
  border-color: green;
}

.error {
  border-color: red;
}

.warning {
  border-color: yellow;
}

SASS Render Configuration Options

The sass npm package supports other render options including:

• sourceMap
• sourceMapContents
• outputStyle

These can be added by modifying the options object passed into the sass render function. When including a source map file, a separate file needs to be written to the project folder containing the sourcemap information. To add these options make the following changes to the index.js:

import sass from "sass";
import { promisify } from "util";
import { writeFile } from "fs";
const sassRenderPromise = promisify(sass.render);
const writeFilePromise = promisify(writeFile);

async function main() {
  const styleResult = await sassRenderPromise({
    file: `${process.cwd()}/styles.scss`,
    outFile: `${process.cwd()}/styles.css`,
    sourceMap: true,
    sourceMapContents: true,
    outputStyle: "compressed",
  });

  console.log(styleResult.css.toString());

  await writeFilePromise("styles.css", styleResult.css, "utf8");

  await writeFilePromise("styles.css.map", styleResult.map, "utf8");
}
main();

Then run the npm run start command again and you should see the "styles.css" and "styles.css.map" files have both been updated.

The styles.css should now output with the blank spaces removed, and it will include a comment at the bottom to indicate the corresponding sourcemap file, which will look like this:

{
  "version": 3,
  "sourceRoot": "",
  "sources": ["styles.scss"],
  "names": [],
  "mappings": "AACA,kCACE,sBACA,aACA,WAaF,SAEE,mBAGF,OAEE,iBAGF,SAEE",
  "file": "styles.css",
  "sourcesContent": [
    "/* This CSS will print because %message-shared is extended. */\r\n%message-shared {\r\n  border: 1px solid #ccc;\r\n  padding: 10px;\r\n  color: #333;\r\n}\r\n\r\n// This CSS won't print because %equal-heights is never extended.\r\n%equal-heights {\r\n  display: flex;\r\n  flex-wrap: wrap;\r\n}\r\n\r\n.message {\r\n  @extend %message-shared;\r\n}\r\n\r\n.success {\r\n  @extend %message-shared;\r\n  border-color: green;\r\n}\r\n\r\n.error {\r\n  @extend %message-shared;\r\n  border-color: red;\r\n}\r\n\r\n.warning {\r\n  @extend %message-shared;\r\n  border-color: yellow;\r\n}\r\n"
  ]
}

The sourcemap will allow for easier debugging and the browser will now load both files. In the debug inspector the browser will show the line in the SCSS source code that corresponds to the CSS output being inspected.

Compile SASS using Dart Sass CLI

It is also possible to use the sass npm package directly from the command line. To do this with the same configuration as the example using the JavaScript API add the following the the package.json scripts property:

{
  "scripts": {
    "compileSass": "sass styles.scss styles.css --style=compressed --embed-sources"
  }
}

This will add a package.json script to run the SASS compiler, by running the command npm run compileSass. To make sure it is working as expected you might want to delete the previously generated styles.css and styles.css.map files, before running the npm run compileSass command.

Using the sass npm package JavaScript API or command line interface, should result in the same output consisting of both the css and css.map files, as both methods rely on the JavaScript implementation of Dart Sass. The main difference is that when using the CLI option the files will automatically be written based on the input and output specified, but when using the JavaScript API we must use the fs.writeFile module to write these files to the project folder.

• opened
• closed
• reopened
• assigned
• unassigned
• review_requested
• review_requested_removed
• labeled
• unlabeled
• synchronize

a POST request can automatically be sent to trigger an integration that is waiting to accept the incoming request. In this example we can set up an Azure Serverless Function using Node.js to accept a GitHub Webhook POST payload. The serverless function will only run when the pull request is from the main branch, the branch associated with the pull request is merged, and the pull request is closed. If all of the following conditions are true, we will also make sure to secure the GitHub webhook by using the @octokit/webhooks npm package to verify the "x-hub-signature-256" request header using an application secret. When the incoming POST request payload is verified to be originating from GitHub's servers any application logic relating to the pull request closing, in the serverless function, can run as expected.

Set up Azure Serverless Function to Accept Webhook Post Request

The first thing we'll need to do is set up an Azure Serverless function so that there is an HTTP endpoint available to accept the incoming webhook POST request that will be sent from GitHub any time an event associated with a pull request occurs. It is not required to use Azure Serverless Functions with GitHub webhooks, so you can exchange this with another technology like a Node.js server using express. All that is required is an HTTP endpoint, using Node.js, that can accept incoming post requests.

Microsoft provides documentation for a quick start to create a function in Azure with TypeScript and Visual Studio code. This steps in this guide will build off that documentation so it is required to set that up before proceeding.

npm install @octokit/webhooks

Once you have the HTTP trigger function setup and you are able to run it locally as indicated in the quick start, we can add the @octokit/webhooks into the package.json that was automatically generated in the functions project. To do this use Visual Studio Code to open a terminal window in the folder where the package.json file was generated for the functions project. Then run the command npm install @octokit/webhooks --save. This will add the @octokit/webhooks npm package into the node_modules folder for the project, so that it can be imported into function code.

import @octokit/webhooks

In the HTTP Trigger function that was created by following quick start guide, this will be called "HTTPExample" if you did not change it, we need to add code to utilize the @octokit/webhooks package that was just installed. You can delete the sample code provided for "HTTPExample" function file named "index.ts". Then go ahead and add the following code into the index.ts file:

import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import { Webhooks } from "@octokit/webhooks";

const httpTrigger: AzureFunction = async function (
  context: Context,
  req: HttpRequest
): Promise<void> {
  context.log("HTTP trigger function processed a request.");

  context.res!.headers["Content-Type"] = "application/json";

  const payload = req.body;

  context.res!.status = 200;
  context.res!.body = { message: "success" };
};

export default httpTrigger;

This is the starting code needed to utilize the @octokit/webhooks npm package verify method. The code to do the verification hasn't been added, only the import statement that is on the second line of code. To use the verification method update the index.ts file to look like this:

import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import { Webhooks } from "@octokit/webhooks";

const httpTrigger: AzureFunction = async function (
  context: Context,
  req: HttpRequest
): Promise<void> {
  context.log("HTTP trigger function processed a request.");

  context.res!.headers["Content-Type"] = "application/json";

  // application/json post request body
  const payload = req.body;

  if (
    payload.action != "closed" ||
    payload.pull_request.base.ref != "main" ||
    !payload.pull_request.merged_at ||
    !new Webhooks({
      secret: process.env["GitHubWebhookSecret"],
    }).verify(payload, req.headers["x-hub-signature-256"])
  ) {
    // this pull request is either:
    //  not closed,
    //  not referencing the main branch,
    //  not merged,
    //  or is not valid
    // so by returning we are not going to process the application logic below
    return;
  }

  // your application logic goes here

  context.res!.status = 200;
  context.res!.body = { message: "success" };
};

export default httpTrigger;

Note: The "GitHubWebhookSecret" is not directly included in the code. Since this is a secret value it is more secure to access this as an environment variable. To add an environment variable within an Azure Functions project, you can view the documentation for how to add an application setting with the Azure portal. This value should be secret and not shared with anyone. In the upcoming steps we will add this to the GitHub repository webhook settings so that the @octokit/webhooks npm package can use this value to verify the request payload. If you are running your function locally you will also need to add the same "GitHubWebhookSecret" setting to the "local.settings.json" file that was automatically generated in the function project folder. Your local.settings.json file can include this value like this:

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "",
    "FUNCTIONS_WORKER_RUNTIME": "node",
    "GitHubWebhookSecret": "<YOUR-SECRET-VALUE>"
  },
  "Host": {
    "LocalHttpPort": 7071,
    "CORS": "*",
    "CORSCredentials": false
  }
}

With this code in place to secure the webhook, we can now be sure that any incoming requests are coming from GitHub's servers and meet the conditional criteria before processing further. Since the POST request from GitHub is sent when any event relating to a pull request occurs, the code above makes sure to only act on the incoming request if the pull request payload data indicates that the pull request is merged, closed, and from the main branch. This way if a pull request is opened or not associated to the main branch the webhook can ignore that request.

Configure GitHub Webhook Settings

Now that we can accept incoming POST requests with webhook payloads from GitHub we need to configure the repository settings to send the request when the pull request event occurs. To do this a GitHub repository will need to be created, if not existing already, and once created navigate to the "Settings" tab. Then in the secondary navigation for the repository settings there will be a nav item labelled "Webhooks". This is where we can configure the url for the webhook and the secret value that is used to verify the incoming request shown in the code above. You can click the button labelled "Add webhook" and GitHub will prompt you to enter your password to continue. Once you have entered your password you will see a screen like this:

To get the value for the Payload URL field, we need to enter the url for the function we created earlier. At this point if you have not deployed your Azure Serverless Function application, you can do this to get the url or, in the next section, follow the steps to set up ngrok to enable testing the locally running functions application.

If you want to deploy to Azure you will find the url in the function app overview settings panel in the Azure portal. This is only the function base url so you will need to append the route of the function that was created. If you kept the default function name the entire Payload URL field value will look something like this:

https://functionapp1.azurewebsites.net/api/httptrigger1

After the Payload URL field, the next field is for the Content type of the request that we are expecting from GitHub, in this case our function is set up to accept:

application/json

so make sure to update this setting. In the following field for the Webhook Secret provide the secret value that was saved as an application setting in the function app settings within the Azure portal. This will also be the same as the "secret" property value that was added to the "local.settings.json" file, shown earlier, within the functions project folder. Next we'll need to update the events that trigger the webhook so select the radio button for "Let me select individual events" then make sure to deselect the checkbox for "Pushes" and only select the checkbox for "Pull requests".

Then select Add webhook at the bottom and this will save the webhook configuration, and GitHub will automatically run a test request to make sure the webhook integration is working as expected.

Use ngrok to Test GitHub Webhook with Azure Functions Locally

If you don't want to deploy your Azure Severless Functions project to the Azure cloud while testing, you can use ngrok to test the GitHub webhook integration while running the function project locally. To do this download ngrok, and follow the installation instructions. Once it is setup, you can run the command ngrok http 7071 and ngrok will provide a publicly available url that will forward the port your functions app is running on. If you changed the default port for the functions project to something different than localhost:7071 make sure to run the ngrok command with the port you are using. With ngrok running you should get a url that looks like this:

http://92832de0.ngrok.io

With that url, go back to the GitHub webhook settings page in the repository settings and update the Payload URL field to match, ensuring that you have appended the entire url for the function so it would look like this:

http://92832de0.ngrok.io/api/httptrigger1

Then run your function app locally inside of Visual Studio Code, and save the webhook settings in GitHub. This will send another test request and you will be able to see the request, from GitHub's servers, being processed in the console output logs of your function app running locally.

Moving back to the "index.ts" file in your function project you can now add the code you need to integrate with the GitHub Webhook and it will use the @octokit/webhooks npm package to verify the incoming request was signed with the secret value you provided. Anytime an event occurs matching the webhook settings criteria, GitHub will send a POST request, and the webhook integration will occur automatically and securely. For an example of what can be done with GitHub webhooks, checkout out how to build a serverless comment system for a jamstack blog. Where you can find detailed instructions about how to setup a GitHub Webhook integration to provide a moderated commenting system for a blog without a database or servers.

npm install uuid

We can create a sample node.js script to test out the functionality of the uuid npm package, but first make sure that Node.js and npm are installed. Then run the command npm init in a test project folder terminal window, and follow the prompts that are shown. This will create a package.json file, which will be used to install the uuid package. Creating a package.json file is not needed, if you are adding uuid to an existing project.

After creating the package.json file run the command npm install uuid --save, and the uuid npm package will be added to the node_modules folder inside the project folder. If you are using TypeScript you can also run the command npm install @types/uuid to install the type definitions for the uuid npm package. See how to compile TypeScript with npm for more information if you want to use the uuid npm package with TypeScript.

Generate uuid with Node.js

With the uuid npm package installed we can now import it into a Node.js script and use the functions provided. Create a new file named "index.js" and include the uuid npm package like this:

import { v4 as uuidv4 } from "uuid";

function main() {
  const uniqueId = uuidv4();
  console.log(uniqueId);
}

main();

What this does is import the RFC4122 version 4 uuid export using ECMAScript modules syntax. If you aren't using ES Modules, you can by following the steps to import and export ES Modules in Node.js, or you can use commonJS modules with a different import syntax that looks like this:

const { v4: uuidv4 } = require("uuid");

For the remainder of this example we will use the ES Module syntax.

To test out the Node.js script, we can add a package.json script to run it. In the "scripts" property of the package.json file add the following:

{
  "scripts": {
    "start": "node index.js"
  }
}

After adding the additional script run the command npm run start, and you should see a uuid output to the console that looks similar to this:

d160b410-e6a8-4cbb-92c2-068112187503

You can re-run the command and a new uuid will be generated each time.

Validate uuid

The uuid npm package can also validate uuid strings to test whether they are a valid uuid. To do this add some additional code to the index.js file we just created.

import { v4 as uuidv4 } from "uuid";
import { validate as uuidValidate } from "uuid";

function main() {
  const uniqueId = uuidv4();
  console.log(uniqueId);

  const isValid = uuidValidate(uniqueId);
  console.log(isValid);
}

main();

Then run the npm run start command again and you will see the result of the uuidValidate method is output as true. If the value passed into the uuidValidate function is not a valid uuid the output will be false.

Detect uuid RFC version

After validation, if we have a valid uuid, the uuid npm package can also detect the version of a uuid string. Building off the index.js sample code add another import to access the uuid version function and test the uniqueId that is generated like this:

import { v4 as uuidv4 } from "uuid";
import { validate as uuidValidate } from "uuid";
import { version as uuidVersion } from "uuid";

function main() {
  const uniqueId = uuidv4();
  console.log(uniqueId);

  const isValid = uuidValidate(uniqueId);
  console.log(isValid);

  const version = uuidVersion(uniqueId);
  console.log(version);
}

main();

Now when we run the index.js script we can see the version of the uniqueId that is generated is output as "4", matching the uuidv4 version we imported initially. If the uuidVersion function is not passed a valid uuid an error will be thrown, so it can be helpful to wrap this function in a try/catch block to capture any errors.

import { v4 as uuidv4 } from "uuid";
import { validate as uuidValidate } from "uuid";
import { version as uuidVersion } from "uuid";

function main() {
  const uniqueId = uuidv4();
  console.log(uniqueId);

  const isValid = uuidValidate(uniqueId);
  console.log(isValid);

  try {
    const version = uuidVersion(uniqueId);
    console.log(version);
  } catch (error) {
    console.log(error);
  }
}

main();

This way any resulting error can be output to the console, or handled in a way that is best for the current usage.

Generate NIL uuid

If you need to generate a NIL uuid, or a uuid that is entirely zeros, the uuid npm package provides the NIL_UUID as an export. Adding it to the index.js sample script looks like this:

import { v4 as uuidv4 } from "uuid";
import { validate as uuidValidate } from "uuid";
import { version as uuidVersion } from "uuid";
import { NIL as NIL_UUID } from "uuid";

function main() {
  const uniqueId = uuidv4();
  console.log(uniqueId);

  const isValid = uuidValidate(uniqueId);
  console.log(isValid);

  try {
    const version = uuidVersion(uniqueId);
    console.log(version);
  } catch (error) {
    console.log(error);
  }

  console.log(NIL_UUID);
}

main();

I haven't come across a use case where a NIL uuid is needed, but it is provided in the uuid npm package so I'm sure there are plenty.

The uuid npm package is a helpful utility to generate a unique id with Node.js that saves a lot of the potential headache from generating one from scratch, while also ensuring the value are unique, secure, and matching a specific RFC version.

Using TypeScript in combination with ES Modules brings many added benefits. To use TypeScript with ES Modules, the TypeScript compiler configuration in tsconfig.json can be updated to process code in ES Module format. Additionally, Babel can be used for TypeScript compilation, and the TypeScript compiler will be used for type checking, as Babel can not type check TypeScript code. Once the TypeScript code is being compiled by Babel into JavaScript, retaining the ES Module format, the ES Modules can be exported, imported, and run with Node.js.

package.json Type Module

The first configuration change we can make, to use ES Modules in Node.js is configuring the package.json file to include the type module property value. To do this add the following code to the package.json file in your Node.js project:

{
  "type": "module"
}

If you are starting a new project you can run the command npm init in a terminal window, follow the prompts that follow, and a package.json file will be generated in the current project folder. Although, before doing so make sure to have Node.js and npm installed. Once the package.json file is added to your project then add the extra configuration shown above as the npm init command does not generate a package.json file with this ES Module setting pre-configured.

npm install

We will also be using some additional npm packages to carry out the compilation and type checking processes.

• cross-env
• @babel/cli
• @babel/core
• @babel/preset-env
• @babel/preset-typescript
• rimraf
• typescript

Before proceeding run the command npm install cross-env @babel/cli @babel/core @babel/preset-env @babel/preset-typescript rimraf typescript --save. This will install the npm packages in the project "node_modules" folder and create a package-lock.json file. The npm packages are now available for usage in the project. Since we are using TypeScript, we can also run the command npm install @types/node --save-dev which will install the Node.js type definitions as a devDependency.

Configure TypeScript compiler to use ES Module format

Using ES Modules does not require the use of TypeScript, however the overhead of including TypeScript is minimal and including it provides many benefits such as static typing, which can enable code editors or an IDE to offer more predictive assistance. You may have heard referred to as intellisense or intelligent code completion. In the same folder as the package.json add a new file named "tsconfig.json" containing this configuration:

{
  "compilerOptions": {
    "allowSyntheticDefaultImports": true,
    "isolatedModules": true,
    "strict": true,
    "module": "esnext",
    "lib": ["ES2019"],
    "noEmit": true,
    "moduleResolution": "node",
    "skipLibCheck": true
  },
  "include": ["src/**/*.ts"],
  "exclude": ["node_modules/**/*", "dist/**/*"]
}

More info on tsconfig settings can be found in the TSConfig reference provided by Microsoft. The most important compiler option included is setting the "module" property to "esnext". This informs the TypeScript compiler to recognize source code in the ES Module format as well as retain the format when generating JavaScript code.

Since Babel will be configured to do the compilation of TypeScript into JavaScript the "noEmit" property is set to true, and what this does is allow for the use of the TypeScript compiler only to indicate when there are type checking errors. When configured this way the tsc compile command will not generate any JavaScript code, but it will output any errors that would occur during compilation to the console. It is also recommended, when using TypeScript with the Babel compiler, to set the "allowSyntheticDefaultImports" and "isolatedModules" to true as this ensures that the TypeScript compiler will process source code similar to how the Babel compiler does. This way the type checking and compilation configurations are in sync, even though separate steps are responsible for each.

Configure Babel to compile TypeScript into ES Modules

With TypeScript configured, we can add the Babel configuration that enables TypeScript compilation with the Babel compiler. To do this create a new file in the same folder as the tsconfig.json file named ".babelrc.json" and add this configuration:

{
  "presets": [
    ["@babel/preset-env", { "modules": false, "targets": { "node": true } }],
    ["@babel/preset-typescript"]
  ],
  "ignore": ["node_modules"],
  "comments": false,
  "minified": true
}

This will configure Babel to use the preset-typescript and preset-env when generating JavaScript code. The presets are executed in a bottom to top order, meaning that first Babel will compile the TypeScript into JavaScript and then on the resulting JavaScript code the preset-env configuration will be applied. This is where Babel is configured to use ES Modules as the "modules" setting is set to false, which is somewhat confusing because ES Modules are being used. It is necessary to set this to false otherwise Babel will use the default CommonJS module format for Node.js. Additionally the compilation target is set to Node so that Babel can apply code transforms that ensure the code will be able to run in the LTS version of Node.js.

In this example there are two extra babel settings included that instruct the Babel compiler to remove any comments in the source code and minify the JavaScript output. These can be removed if not desired for your use case, however this is beneficial for using in production to minimize code size.

Export ES Module

Now we can add some sample TypeScript code to test out the configuration changes.

In the same project folder create a new folders named "src", so that the file structure matches the "include" pattern in the tsconfig.json file. Then in the "src" folder create a new file named "helpers.ts" and place the following code in it:

function log(value: string) {
  console.log(value);
}

export { log };

This code is only logging the value that is passed in to the console, and is not really representative of actual code that would be used, but it allows for the demonstration of using ES Modules with TypeScript and Babel. The export of the "log" function is the key item to notice about this code, as this is all that is needed to export an ES Module. Now we can create another file to import the "log" helper function module.

Import ES Module

In the same "src" folder create a new file named "index.ts" this will be the main entry point for our ES Module code. Once that file is created add in this TypeScript code to import the helper function that was created in the previous step.

import { log } from "./helpers.js";

function main() {
  log("testing es modules");
}

main();

Similar to the helpers.ts file the index.ts files is mainly for demonstrating ES Module import syntax. It imports the helper function and then the main function is called to execute the "log" function. Although it is important to note that the file imported must end with a ".js" file extension rather than a ".ts" file extension. This is because when the code is eventually compiled the ES Module code will be a JavaScript file. Make sure that anytime a module is imported from a separate file the path is relative to the current file and the extension is set to ".js", otherwise both the TypeScript compiler and Babel compiler will not be able to resolve the file location.

Run ES Modules in Node.js

At this point the source code is configured to run with ES Modules, so we can now look at how to compile the code and run it with Node.js. To do this we'll need to add six additional scripts to the "scripts" property in the package.json file.

In the package.json "scripts" property add the following:

{
  "clean": "rimraf dist",
  "compile": "cross-env-shell babel src -d dist --source-maps --extensions '.ts'",
  "build": "npm run clean && npm run compile",
  "typecheck": "tsc --p .",
  "build-typecheck": "npm run typecheck && npm run build",
  "start": "npm run build-typecheck && node ./dist/index.js"
}

The "clean" script will ensure that prior to the compilation, the output directory "dist" will be deleted. This way the latest code will copied into an empty folder.

The "compile" script is where the cross-env package is used to run the babel compilation command. This babel compilation command specifies that the source files will be located in the "src" folder and when compilation is complete the JavaScript output will be copied to a folder named "dist". The flags that are passed in indicate that source maps should be generated for debugging purposes and also the "--extensions" flag is required so that Babel will look for files ending with the ".ts" extension.

To use the "clean" and "compile" script sequentially they are combined in a new script named "build", which can be run using the command npm run build. This will remove the old files from the "dist" folder and compile the TypeScript source code with Babel, however no typechecking errors will be indicated and Babel may fail to compile the code if there are errors present.

To resolve this an additional script "typecheck" is included that will pass the TypeScript source code through the TypeScript compiler, and if there are errors present, they will be output to the console. Since the tsconfig.json settings include the "noEmit" property the typecheck command won't output any JavaScript code.

The command that will be most commonly used is the "build-typecheck" command, which can be used by running npm run build-typecheck. This will sequentially run the "typecheck" command and then if there are no errors present as a result of the TypeScript compilation with the TypeScript compiler, the "build" command will be executed, invoking the Babel compiler and generating JavaScript code that can be run by Node.js in ES Module format.

Since the JavaScript code is being output to a folder named "dist" the "main" property in the package.json should be changed to:

{
  "main": "./dist/index.js"
}

To run the compiled JavaScript code, execute the command npm run start and this will carry out the type checking and compilation steps as well as run the index.js file with Node.js. If everything is setup and working as expected you should see the value included in the "main" function - "testing es modules" output to the console. Now you can use this configuration to create node modules that are statically typed and run in Node.js using the ES Module format.

Setup Azure Storage Emulator

In order to save on development costs, instead of creating cloud resources, we can install the Azure Storage Emulator for development and testing. If you aren't using windows, Azurite is an open source Azure storage API compatible server, and it is recommended by Microsoft to use. Otherwise, after installing, Windows users can search in the start menu for "azure storage emulator" and press enter to start the emulator. This should open a cmd window that will indicate the emulator is running, and some helpful commands. The cmd window can be closed and the emulator will continue to run.

Install Azure Storage Explorer

Next we'll need to download Azure Storage Explorer to interact with the emulated storage environment. This application is available for Windows, Mac, and Linux machines. After installing go ahead and start the Azure Storage Explorer, and in the left hand column navigator find the dropdown section labelled "Local & Attached" and then within that section find the secondary dropdown "Storage Accounts" and within the tertiary dropdown "(Emulator - Default Ports)" is where the resources, that we have not yet created, will be displayed. Here you can see three additional dropdown sections:

• Blob containers
• Queues
• Tables

Our focus will be on the "Tables" section, which should be empty since no tables have been programmatically created yet.

Setup Azure Functions for Local Development

Now that we have the storage emulator and storage explorer configured we can download the Azure Functions extension for Visual Studio Code. If you don't have Visual Studio Code you can download it, and then follow the instructions to configure the local project. You don't need to follow the naming convention indicated in the documentation, but what is important is that there is a package.json created in the functions project. The package.json file is created automatically and allows us to include the npm package provided by Microsoft to interact with Azure Storage.

npm Install azure-storage

In the same folder as the package.json that was created, run the command npm install azure-storage --save and then run the command npm install @types/node --save-dev to install the type definitions for Node.js. This will install the azure-storage npm package to the local functions project so that we can import it in our code.

Congratulations, you made it through the setup configuration!

Http Trigger Serverless Function

Now we can write the code to use Azure Table Storage with Azure Serverless Typescript Functions. To begin find the file "index.ts" in the HTTP trigger function that was created earlier (if using the default it will be called HttpTrigger1). Right now there is sample code in that function that can be deleted, and the code below can be added.

The function should now look like this:

import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import * as azureStorage from "azure-storage";

const httpTrigger: AzureFunction = async function (
  context: Context,
  req: HttpRequest
): Promise<void> {
  context.log("HTTP trigger function processed a request.");

  // set content type for all responses
  context.res!.headers["Content-Type"] = "application/json";

  if (req.method == "POST") {
  } else if (req.method == "GET") {
  } else if (req.method == "PUT") {
  } else if (req.method == "DELETE") {
  } else {
    // request method does not match
    context.res!.status = 500;
  }
};

export default httpTrigger;

Programmatically Create Table If Not Exists

Before we can retrieve data from Azure Storage we need to insert data by using an HTTP POST request, additionally a table must be created to store the data. To ensure there is a table to store data we can programmatically create the entity table if it does not exist with the azure storage npm package, at the time of the POST request. In order to connect to the storage emulator a connection string is required, which can be stored as an environment variable to be passed into the Node.js serverless functions process. To do this add the default local connection string to the file "local.settings.json" that is in the same folder as the HttpTrigger1 function. Additionally we want to add a "Host" configuration to permit CORS requests and set the default port that the functions will run on.

The local.settings.json file should now look like this:

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "",
    "FUNCTIONS_WORKER_RUNTIME": "node",
    "TableStorageConnection": "UseDevelopmentStorage=true"
  },
  "Host": {
    "LocalHttpPort": 7071,
    "CORS": "*",
    "CORSCredentials": false
  }
}

Now we can use the "TableStorageConnection" environment variable to create a table.

import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import * as azureStorage from "azure-storage";

const httpTrigger: AzureFunction = async function (
  context: Context,
  req: HttpRequest
): Promise<void> {
  context.log("HTTP trigger function processed a request.");

  const tableService = azureStorage.createTableService(
    process.env["TableStorageConnection"]
  );

  const createTableIfNotExists = (tableName: string) =>
    new Promise((resolve, reject) => {
      tableService.createTableIfNotExists(tableName, (error, result) => {
        if (error) {
          reject(error);
        } else {
          resolve(result);
        }
      });
    });

  // set content type for all responses
  context.res!.headers["Content-Type"] = "application/json";

  if (req.method == "POST") {
    try {
      await createTableIfNotExists("TestTable");
    } catch (error) {
      console.log(error);
      context.res!.status = 400;
      context.res!.body = {
        message: "An error occurred.",
      };
    }
  } else if (req.method == "GET") {
  } else if (req.method == "PUT") {
  } else if (req.method == "DELETE") {
  } else {
    // request method does not match
    context.res!.status = 500;
  }
};

export default httpTrigger;

Once that table service is initialized the "tableService.createTableIfNotExists" function can be used. This function by default, uses a callback function to obtain the result. Instead of using the callback, the function is wrapped in a Promise which can asynchronously resolve the callback function or return an error. Following that the promise is awaited inside a conditional branch that will only execute if the incoming request is a POST request.

The function can now create a new table if it doesn't exist named "TestTable" on any incoming POST request. To test this out run the function (in Visual Studio Code press F5), and then download Postman to mock requests. Copy the url provided in the terminal window where the function is running, if you kept the default configuration this url will be "http://localhost:7071/api/HttpTrigger1", and change the request method in Postman from GET to POST and send the request. In the response body displayed in Postman all that will show is the number "1", however if we use the Azure Storage Explorer to view the emulator tables we can see that the "TestTable" was successfully created. You may need to select "refresh all" in the storage explorer to see the new table.

Insert Azure Table Storage Entity

Now that the table will be programmatically created if it doesn't exist, we can add a request body to the POST request that is being sent in Postman. This data will be parsed with the querystring module included with Node.js and then a storage entity can be generated from the incoming data. Once the storage entity is generated it can then be saved to the storage table.

To facilitate the saving of the table data we can use the uuid npm package, to install run the command npm install uuid --save and then install the typescript type definitions with the command npm install @types/uuid --save-dev.

Add the following import statements to the index.ts file:

import * as querystring from "querystring";
import { v4 as uuidv4 } from "uuid";

Then add the following inside the POST method conditional branch:

//parses www-form-urlencoded request body
const body = querystring.parse(req.body) as {
  firstName: string;
  lastName: string;
  age: string;
};

if (!(body && body.firstName && body.lastName && !isNaN(Number(body.age)))) {
  context.res!.status = 400;
  context.res!.body = {
    message: "The data is invalid.",
  };
  return;
}

// inform table storage of row types
const entityGenerator = azureStorage.TableUtilities.entityGenerator;

// storing data within the same storage partition
// partition key and row key combo must be unique but also type string
const entityData = {
  PartitionKey: entityGenerator.String("TestPartition"),
  RowKey: entityGenerator.String(uuidv4()),
  firstName: entityGenerator.String(body.firstName),
  lastName: entityGenerator.String(body.lastName),
  age: entityGenerator.Int32(Number(body.age)),
};

try {
  const tableName = "TestTable";

  await createTableIfNotExists(tableName);

  const entity = await insertEntity(tableName, entityData);

  context.res!.status = 200;
  context.res!.body = {
    message: "Data is saved.",
    data: entity,
  };
} catch (error) {
  console.log(error);

  context.res!.status = 400;
  context.res!.body = {
    message: "An error occurred.",
  };
}

Note: Azure Table Storage requires both the partition key and the row key value to be present on storage entities and it also enforces that the type of these columns is a string. The "RowKey" property is utilizing the uuid package that was installed to guarantee that the partition key and row key combination is unique regardless of the other entity data. It's also worth noting that the entity generator isn't required and Azure Table Storage will default to a type of string if the entity row type is not specified.

You will notice that there is no function declared yet with the name "insertEntity". We can add that helper function below the "createTableIfNotExists" function.

const insertEntity = (tableName: string, entity: {}) =>
  new Promise((resolve, reject) => {
    tableService.insertEntity(tableName, entity, (error, result) => {
      if (error) {
        reject(error);
      } else {
        resolve(result);
      }
    });
  });

After adding the code to save the table storage entity run the serverless functions again with visual studio code, and submit a post request containing sample data with Postman.

Here is what the request should look like in Postman:

Checking with the Azure Storage Explorer, inside of the "TestTable" there should be one entity, and now we can add code to retrieve this data entity using the partition key and row key values that are saved.

Retrieve Azure Storage Entity

In order to retrieve the Azure Storage entity we will need to add a second helper function to the index.ts file inside the HttpTrigger1 serverless function. This helper function will allow us to retrieve storage entities using the partition key and the row key. Below the "insertEntity" function declaration add the following code:

const retrieveEntity = (
  tableName: string,
  partitionKey: string,
  rowKey: string
) =>
  new Promise((resolve, reject) => {
    tableService.retrieveEntity(
      tableName,
      partitionKey,
      rowKey,
      (error, result) => {
        if (error) {
          reject(error);
        } else {
          resolve(result);
        }
      }
    );
  });

Then the "retrieveEntity" helper function can be called in the conditional branch that will execute on incoming GET requests, however we will need a way to pass the row key value to the function from the incoming request data. To do this we can customize the http endpoint using the functions.json file that is in the HttpTrigger1 function (the same folder as index.ts). In that file add a new key to the first object in the "bindings" array.

The functions.json file should look similar to this :

{
  "bindings": [
    {
      "authLevel": "anonymous",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req",
      "methods": ["get", "post"],
      "route": "HttpTrigger1/{rowKey:guid?}"
    },
    {
      "type": "http",
      "direction": "out",
      "name": "res"
    }
  ],
  "scriptFile": "../dist/HttpTrigger1/index.js"
}

In the route parameter that is being added the pattern to match the row key in the request is specified. The row key will be of type GUID and is optional since post requests will not have a row key to specify. We can now use the retrieve entity function in combination with the request parameter to query Azure Table Storage for the entity data.

Add this code into the GET request method branch to retrieve and return the data:

try {
  const entity = await retrieveEntity(
    "TestTable",
    "TestPartition",
    context.bindingData.rowKey
  );

  context.res!.status = 200;
  context.res!.body = {
    message: "Data retrieved.",
    data: entity,
  };
} catch (error) {
  console.log(error);
  context.res!.status = 400;
  context.res!.body = {
    message: "An error occurred",
  };
}

Then in Postman change the request method to GET and copy the entity row key from table storage so that the url in Postman looks similar to

http://localhost:7071/api/HttpTrigger1/99baf118-fb0b-495e-b839-432264ff6aaa

The row key will be different in your case since it is automatically generated for each entity, so make sure to change that to the entity row key saved to your local table storage. In the response data from postman you should see the following data returned:

{
  "message": "Data retrieved.",
  "data": {
    "PartitionKey": {
      "$": "Edm.String",
      "_": "TestPartition"
    },
    "RowKey": {
      "$": "Edm.String",
      "_": "99baf118-fb0b-495e-b839-432264ff6aaa"
    },
    "Timestamp": {
      "$": "Edm.DateTime",
      "_": "2021-01-30T20:51:49.323Z"
    },
    "firstName": {
      "_": "test first"
    },
    "lastName": {
      "_": "test last"
    },
    "age": {
      "_": 99
    },
    ".metadata": {
      "metadata": "http://127.0.0.1:10002/devstoreaccount1/$metadata#TestTable/@Element",
      "etag": "W/\"datetime'2021-01-30T20%3A51%3A49.323Z'\""
    }
  }
}

In the data property of the response, each of the table storage columns is returned as an object containing two properties, one indicating the table storage data type and the other is the value of the property. There is also an additional metadata field included in the response that provides extra info about the response from Azure Table Storage, or in this case the Azure Storage Emulator.

Azure table storage entities can now be inserted and retrieved, but it is also useful to be able to update an entity that has been previously saved. To do this we can add the PUT request method to the "methods" property of the first object in the "bindings" array located in the functions.json file. Update the "methods" property to look like this:

{
  "methods": ["get", "post", "put", "delete"]
}

The code for the delete method is going to be added later, so that string value has also been added to the array at this time.

Update Azure Storage Entity

After permitting PUT requests in the functions.json add this code to the PUT method conditional branch:

//parses www-form-urlencoded request body
const body = querystring.parse(req.body) as {
  rowKey: string;
  firstName: string;
  lastName: string;
  age: string;
};

// inform table storage of row types
const entityGenerator = azureStorage.TableUtilities.entityGenerator;

// use request body data to maintain row key for entity
const entityData = {
  PartitionKey: entityGenerator.String("TestPartition"),
  RowKey: entityGenerator.String(body.rowKey),
  firstName: entityGenerator.String(body.firstName),
  lastName: entityGenerator.String(body.lastName),
  age: entityGenerator.Int32(Number(body.age)),
};

try {
  const entity = await updateEntity("TestTable", entityData);

  context.res!.status = 200;
  context.res!.body = {
    message: "Data is updated.",
    data: entity,
  };
} catch (error) {
  console.log(error);
  context.res!.status = 400;
  context.res!.body = {
    message: "An error occurred",
  };
}

A third helper function is also needed, shown as "updateEntity" so it can be added below the "retrieveEntity" helper function, above the request method conditional branches:

const updateEntity = (tableName: string, entity: {}) =>
  new Promise((resolve, reject) => {
    tableService.replaceEntity(tableName, entity, (error, result) => {
      if (error) {
        reject(error);
      } else {
        resolve(result);
      }
    });
  });

The "updateEntity" function takes two parameters one being the table name and the other is the updated entity. The partition key and the row key of the entity must match an existing partition/row key combination, or table storage will return an error. If desired there is a function provided by the azure-storage npm package named "insertOrReplaceEntity" which, as the name indicates can either update existing entities or create a new one if one does not exist. In this example the entity already exists so only the "replaceEntity" function is needed.

The PUT request method branch to update an existing entity is almost the same as the POST method branch to insert a new storage entity. The url is the same for both, and the main difference is that the "rowKey" is included in the request body so that the appropriate entity can have it's data updated. You can try it out by changing one of the fields in the request body to a different value and then check in the storage explorer to confirm the entity that was previously inserted and retrieved has the matching table column value updated.

Delete Azure Storage Entity

Much like the GET request method branch the DELETE request method does not contain a request body, instead the row key will be passed in the request as a parameter, and like the examples above we can add a fourth helper function to carry out the deletion.

const deleteEntity = (tableName: string, entity: {}) =>
  new Promise((resolve, reject) => {
    tableService.deleteEntity(tableName, entity, (error, result) => {
      if (error) {
        reject(error);
      } else {
        resolve(result);
      }
    });
  });

Then use the "deleteEntity" function in the DELETE request method branch by adding this code:

try {
  // inform table storage of row types
  const entityGenerator = azureStorage.TableUtilities.entityGenerator;

  // use request body data to maintain row key for entity
  const entityData = {
    PartitionKey: entityGenerator.String("TestPartition"),
    RowKey: entityGenerator.String(context.bindingData.rowKey),
  };

  const statusMessage = await deleteEntity("TestTable", entityData);

  context.res!.status = 200;
  context.res!.body = {
    message: "Data deleted.",
    data: statusMessage,
  };
} catch (error) {
  console.log(error);
  context.res!.status = 400;
  context.res!.body = {
    message: "An error occurred",
  };
}

To test this copy the row key value from the storage explorer for the entity previously saved and use the same url from the GET request method example in Postman, but change the request method to DELETE. Then execute the delete request with Postman and in the response section a success message will be displayed:

{
  "message": "Data deleted.",
  "data": {
    "isSuccessful": true,
    "statusCode": 204,
    "body": "",
    "headers": {
      "cache-control": "no-cache",
      "content-length": "0",
      "server": "Windows-Azure-Table/1.0 Microsoft-HTTPAPI/2.0",
      "x-ms-request-id": "3c378130-7a6d-4652-9022-d02320d29c05",
      "x-ms-version": "2018-03-28",
      "x-content-type-options": "nosniff",
      "date": "Sun, 31 Jan 2021 21:23:06 GMT"
    }
  }
}

The response status from Azure Table Storage is 204 No Content, since there is no longer an entity saved in the table. We can verify the entity was deleted by refreshing the table in the storage explorer. The response items shown in the "statusMessage" variable, is the response from Azure Table Storage, and it is being included in the response back from the serverless function to show the consumer of the serverless function API that the delete request to Azure Storage was successful. If the delete request failed the status message would indicate that by setting the "isSuccessful" property value to false.

Here is the complete function file with all code include:

import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import * as azureStorage from "azure-storage";
import * as querystring from "querystring";
import { v4 as uuidv4 } from "uuid";

const httpTrigger: AzureFunction = async function (
  context: Context,
  req: HttpRequest
): Promise<void> {
  context.log("HTTP trigger function processed a request.");

  const tableService = azureStorage.createTableService(
    process.env["TableStorageConnection"]
  );

  const createTableIfNotExists = (tableName: string) =>
    new Promise((resolve, reject) => {
      tableService.createTableIfNotExists(tableName, (error, result) => {
        if (error) {
          reject(error);
        } else {
          resolve(result);
        }
      });
    });

  const insertEntity = (tableName: string, entity: {}) =>
    new Promise((resolve, reject) => {
      tableService.insertEntity(tableName, entity, (error, result) => {
        if (error) {
          reject(error);
        } else {
          resolve(result);
        }
      });
    });

  const retrieveEntity = (
    tableName: string,
    partitionKey: string,
    rowKey: string
  ) =>
    new Promise((resolve, reject) => {
      tableService.retrieveEntity(
        tableName,
        partitionKey,
        rowKey,
        (error, result) => {
          if (error) {
            reject(error);
          } else {
            resolve(result);
          }
        }
      );
    });

  const updateEntity = (tableName: string, entity: {}) =>
    new Promise((resolve, reject) => {
      tableService.replaceEntity(tableName, entity, (error, result) => {
        if (error) {
          reject(error);
        } else {
          resolve(result);
        }
      });
    });

  const deleteEntity = (tableName: string, entity: {}) =>
    new Promise((resolve, reject) => {
      tableService.deleteEntity(tableName, entity, (error, result) => {
        if (error) {
          reject(error);
        } else {
          resolve(result);
        }
      });
    });

  // set content type for all responses
  context.res!.headers["Content-Type"] = "application/json";

  if (req.method == "POST") {
    //parses www-form-urlencoded request body
    const body = querystring.parse(req.body) as {
      firstName: string;
      lastName: string;
      age: string;
    };

    if (
      !(body && body.firstName && body.lastName && !isNaN(Number(body.age)))
    ) {
      context.res!.status = 400;
      context.res!.body = {
        message: "The data is invalid.",
      };
      return;
    }

    // inform table storage of row types
    const entityGenerator = azureStorage.TableUtilities.entityGenerator;

    // storing data within the same storage partition
    // partition key and row key combo must be unique but also type string
    const entityData = {
      PartitionKey: entityGenerator.String("TestPartition"),
      RowKey: entityGenerator.String(uuidv4()),
      firstName: entityGenerator.String(body.firstName),
      lastName: entityGenerator.String(body.lastName),
      age: entityGenerator.Int32(Number(body.age)),
    };

    try {
      const tableName = "TestTable";

      await createTableIfNotExists(tableName);

      await insertEntity(tableName, entityData);

      context.res!.status = 200;
      context.res!.body = {
        message: "Data is saved.",
        data: entityData,
      };
    } catch (error) {
      console.log(error);
      context.res!.status = 400;
      context.res!.body = {
        message: "An error occurred.",
      };
    }
  } else if (req.method == "GET") {
    try {
      const entity = await retrieveEntity(
        "TestTable",
        "TestPartition",
        context.bindingData.rowKey
      );
      context.res!.status = 200;
      context.res!.body = {
        message: "Data retrieved.",
        data: entity,
      };
    } catch (error) {
      console.log(error);
      context.res!.status = 400;
      context.res!.body = {
        message: "An error occurred",
      };
    }
  } else if (req.method == "PUT") {
    //parses www-form-urlencoded request body
    const body = querystring.parse(req.body) as {
      rowKey: string;
      firstName: string;
      lastName: string;
      age: string;
    };

    // inform table storage of row types
    const entityGenerator = azureStorage.TableUtilities.entityGenerator;

    // use request body data to maintain row key for entity
    const entityData = {
      PartitionKey: entityGenerator.String("TestPartition"),
      RowKey: entityGenerator.String(body.rowKey),
      firstName: entityGenerator.String(body.firstName),
      lastName: entityGenerator.String(body.lastName),
      age: entityGenerator.Int32(Number(body.age)),
    };

    try {
      const entity = await updateEntity("TestTable", entityData);
      context.res!.status = 200;
      context.res!.body = {
        message: "Data is updated.",
        data: entity,
      };
    } catch (error) {
      console.log(error);
      context.res!.status = 400;
      context.res!.body = {
        message: "An error occurred",
      };
    }
  } else if (req.method == "DELETE") {
    try {
      // inform table storage of row types
      const entityGenerator = azureStorage.TableUtilities.entityGenerator;

      // use request body data to maintain row key for entity
      const entityData = {
        PartitionKey: entityGenerator.String("TestPartition"),
        RowKey: entityGenerator.String(context.bindingData.rowKey),
      };

      const statusMessage = await deleteEntity("TestTable", entityData);

      context.res!.status = 200;
      context.res!.body = {
        message: "Data deleted.",
        data: statusMessage,
      };
    } catch (error) {
      console.log(error);
      context.res!.status = 400;
      context.res!.body = {
        message: "An error occurred",
      };
    }
  } else {
    // method does not match any
    context.res!.status = 500;
  }
};

export default httpTrigger;

Azure serverless functions are a scalable and cost efficient method to create a RESTful API interface to interact with Azure Table Storage. The code above covers the actions needed to insert, retrieve, update and delete Azure Storage entities, while using TypeScript and the azure-storage and uuid npm packages to execute the methods that correspond to the Azure Table Storage API. Azure Storage entities can be accessed from a consumer of the serverless functions REST API, like a web application, and the Azure Storage credentials and connection string remain secure.

Setup Node.js and npm package.json

Make sure to have node.js and npm installed before following these steps. Then run the command npm init and follow the prompts to create a package.json file. Once the package.json file is created add the setting:

{
  "type": "module"
}

This will permit the use of ECMAScript modules in the code, specifically it will allow for the use of es module imports from npm packages. After that we need to install TypeScript, so run the command npm install typescript --save and then run the command npm install @types/node --save-dev. At this point also go ahead and add a new script property called "start", that will initiate the TypeScript compiler and run the JavaScript output with Node.js.

The package.json file should look similar to this:

{
  "type": "module",
  "name": "splitarrayintochunks",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "tsc && node index.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "@types/node": "^14.14.22"
  },
  "dependencies": {
    "typescript": "^4.1.3"
  }
}

Setup TypeScript

After configuring Node.js, add a tsconfig.json file to same folder as the package.json file. This lets us use TypeScript, which we just installed, instead of JavaScript and as a result we get the advantage of generic types among other features. Copy this config into the tsconfig.json file:

{
  "compilerOptions": {
    "allowSyntheticDefaultImports": true,
    "isolatedModules": true,
    "strict": true,
    "module": "esnext",
    "lib": ["ES2019"],
    "moduleResolution": "node",
    "skipLibCheck": true
  },
  "include": ["*.ts"],
  "exclude": ["node_modules/**/*"]
}

Now the output of the TypeScript compilation, indicated in the tsconfig "module" field, will be created as ECMAScript modules, which matches the type field added to the package.json configuration.

Node.js EMFILE Error When Reading Files

The configuration steps are now complete and we can add some code that will demonstrate the EMFILE error that can be prevented by batch processing the array in smaller chunks. This sample code, that results in an error, can be added to index.ts.

import fs from "fs";
import util from "util";
const readFile = util.promisify(fs.readFile);

(async function main() {
  //an array containing ten thousand undefined items
  const originalArray = Array.from(Array(10000));

  try {
    // awaiting all ten thousand promises simultaneously
    await Promise.all(
      originalArray.map(async () => {
        const file = await readFile("./data.json", "utf8");
        console.log(file);
      })
    );
  } catch (error) {
    console.log(error);
  }
})();

At this point also create a sample JSON file referenced in the code above named "data.json". All that you need to add to this file is "{}" which will be interpreted as an empty JSON object. With the data file created run the command npm run start and as expected you should see an error in the console:

[Error: EMFILE: too many open files, open '/../../data.json'] {
  errno: -4066,
  code: 'EMFILE',
  syscall: 'open',
  path: '/../../data.json'
}

What is occurring is that we are trying to asynchronously read the data.json file ten thousand times all at once, and the error is informing us that there are too many file descriptors for the system that the code is being run on. The access to the data.json file is happening too frequently for the system to keep track of and as a result the process crashes.

Rather than trying all ten thousand file read attempts at once, the array can be split into chunks and the read requests can be processed in batches, ensuring the number total number of file descriptors is within a suitable limit for the system that Node.js is operating on. To do this we can create a generic TypeScript function that will split any type of array into chunks of the original array type.

TypeScript Generic Reducer to Split Array Into Chunks

In the index.ts file, and above the main function that is immediately invoked we can create another function named "chunkItems". This will utilize TypeScript generics to create an array containing groups of smaller arrays, that match the type of the original array.

const chunkItems = <T>(items: T[]) =>
  items.reduce((chunks: T[][], item: T, index) => {
    const chunk = Math.floor(index / 512);
    chunks[chunk] = ([] as T[]).concat(chunks[chunk] || [], item);
    return chunks;
  }, []);

The reduce() method is used to create an array containing chunks of smaller arrays, and for this example the chunk size is set to be a limit of 512 items per chunk. This way the maximum number of file descriptors that can be allocated at once, is below the default limit of most systems. Now we can use the generic "chunkItems" function to create a batched process by wrapping the existing file read code in a for...of loop, so that each of the Promise.all() results can be awaited asynchronously.

Putting all the code together in the index.ts file looks like this:

import fs from "fs";
import util from "util";
const readFile = util.promisify(fs.readFile);

const chunkItems = <T>(items: T[]) =>
  items.reduce((chunks: T[][], item: T, index) => {
    const chunk = Math.floor(index / 512);
    chunks[chunk] = ([] as T[]).concat(chunks[chunk] || [], item);
    return chunks;
  }, []);

(async function main() {
  const originalArray = Array.from(Array(10000));
  const chunks = chunkItems(originalArray);
  try {
    for (const chunk of chunks)
      await Promise.all(
        chunk.map(async (item, index) => {
          const file = await readFile("./data.json", "utf8");
          console.log("-----start item------");
          console.log("current array chunk:" + chunks.indexOf(chunk));
          console.log("file contents: " + file);
          console.log("current item index: " + index);
          console.log("-----end item-------");
        })
      );
  } catch (error) {
    console.log(error);
  }
})();

Run the npm run start command again, and the EMFILE error will not occur. The output of the above code will display rather quickly, but it will show the index of each chunk currently being processed synchronously and the contents of the sample data.json file. Watching closely (or by stopping the output after it has ran for sometime), you can see that the chunk index goes in numerical order, but the intentionally limited number of file reads is still happening asynchronously and the current item indexes are not in numerical order. By splitting the array into smaller chunks the system is not overloaded and Node.js is able to process the files asynchronously.

HTML with relative links

<p>
  This is the sample content that contains a
  <a href="/relative-link">relative link</a>, that will be converted into an
  absolute link.
</p>

<p>Post content can also include images like this one:</p>
<img src="/sample-image" />
<p>These will get transformed too.</p>

This isn't a full HTML document, only a fragment that represents a sample of what may be contained in a blog post that has been converted from markdown into HTML with a node.js static site generator. Now that the sample HTML file is created and saved as "sample-post.html" we can read it and process the relative links.

Cheerio npm package

To use the cheerio npm package we need to create a node script, and for this we can optionally use TypeScript. For more info about using TypeScript with Node.js, read about how to compile TypeScript with npm. If you aren't using TypeScript you can omit the type declarations from the following code.

What is important is that the package.json file is configured for the project (if not use the npm init command), and then run the command npm install cheerio fs-extra typescript --save followed by the command npm install @types/cheerio @types/fs-extra @types/node --save-dev to install the cheerio npm package with the corresponding type declaration files.

The script code will use es modules to import these npm package libraries, so at the top of the generated package.json file add the following line:

{
  "type": "module"
}

Your package.json file should look similar to this:

{
  "type": "module",
  "name": "relativeurltoabsoluteurl",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "cheerio": "^1.0.0-rc.5",
    "fs-extra": "^9.0.1",
    "typescript": "^4.1.3"
  },
  "devDependencies": {
    "@types/cheerio": "^0.22.23",
    "@types/fs-extra": "^9.0.6",
    "@types/node": "^14.14.21"
  }
}

You can also copy the above json and save as package.json, then run the command npm install to install all of the dependencies listed.

Transform relative URL to absolute URL

Then create a new file named script.ts and place the following code inside of it:

import cheerio from "cheerio";
import fs from "fs";

(async function convertRelativeToAbsolute() {
  const postContent = await fs.readFile("./sample-post.html");

  const $ = cheerio.load(postContent as string, {
    decodeEntities: false,
  });

  $("a[href^='/'], img[src^='/']").each(function (this: cheerio.Element) {
    const $this = $(this);
    if ($this.attr("href")) {
      $this.attr("href", `YOUR-DOMAIN-HERE/${$this.attr("href")}`);
    }
    if ($this.attr("src")) {
      $this.attr("src", `YOUR-DOMAIN-HERE/${$this.attr("src")}`);
    }
  });

  await fs.writeFile($("body").html() as string);
})();

Make sure to replace "YOUR-DOMAIN-HERE" with actual domain you want to convert the relative links to use.

The code inside the "convertRelativeToAbsolute" function, first reads the sample post file containing the HTML file with relative links. Then it uses the cheerio package to load the file and parse it to find all of the anchor tags and images tags that are referencing relative URLs. The selectors used scope either anchor tags or image tags to those that begin with a forward slash, which can most likely be safely assumed to be a relative link. Depending on whether the element is an anchor link or an image, either the href attribute or the src attribute will be prepended with the site domain, to make it an absolute link. When all of the link and image attributes are processed the sample html file is written back to the original file location.

Compile TypeScript and Run Node script

Now we can add a script to the package.json file that will compile the TypeScript script file and run the "convertRelativeToAbsolute" function. In the package.json file add this line to the scripts property:

{
  "scripts": {
    "convertRelativeToAbsolute": "tsc --allowSyntheticDefaultImports --moduleResolution node --module esnext script.ts && node script.js"
  }
}

This will first run the TypeScript compiler, with the flag options specified to indicate the use of es modules with node.js, to convert script.ts into JavaScript output. Then the script.js file is run using node. We can run the "convertRelativeToAbsolute" package.json script by running the command npm run convertRelativeToAbsolute. After it completes you should be able to see the sample-post.html file has been updated to use absolute links in the sample content included earlier.

Now the sample-post.html file content can be shared and referenced from any source, while ensuring that any of the internal links will load as expected. In a more typical scenario the cheerio parsing can be included as a plugin or middleware in a static site generator's build process, rather than working with HTML files directly. This would enable the output of the build process to apply the relative to absolute link conversion to all of the, possibly syndicated, content for the site.

This is one example of how the cheerio npm package is helpful for DOM parsing and manipulation outside of the browser, especially in the context of a static, pre-rendered site that is built using Jamstack technologies.

This example will show how a publicly available HTML form, can be submitted using the Fetch API, with TypeScript, to first asynchronously retrieve a valid token, and then submit that token in a second request to save the form information. For the server-side components, Azure Functions will be used, however these techniques can be applied to other server-side technologies, including a typical server.

HTML Form

We can create a form containing any fields we would like to submit. Let's create a sample contact form with some standard information to collect. There is one extra field at the bottom of the form that is hidden to act as a decoy field for bots to incorrectly submit. This can be ignored for now, but it will be validated in the serverless function handling submissions of the contact form.

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>Contact Form</title>
  </head>
  <body>
    <form
      id="contactForm"
      action="YOUR-DOMAIN/api"
      method="post"
      data-type="contact"
    >
      <div>
        <label for="firstName">first name</label>
        <input
          required
          type="text"
          id="firstName"
          name="firstName"
          autocomplete="given-name"
        />
      </div>
      <div>
        <label for="lastName">last name</label>
        <input
          required
          type="text"
          id="lastName"
          name="lastName"
          autocomplete="family-name"
        />
      </div>
      <div>
        <label for="email">email</label>
        <input
          required
          type="email"
          id="email"
          name="email"
          autocomplete="email"
        />
      </div>
      <div>
        <label for="website">website</label>
        <input type="text" id="website" name="website" autocomplete="url" />
      </div>
      <div>
        <label for="message">message</label>
        <textarea required rows="5" id="message" name="message"></textarea>
      </div>
      <button type="submit">Submit</button>
      <div style="position: absolute; left: -5000px" aria-hidden="true">
        <input
          id="password"
          type="text"
          name="password"
          tabindex="-1"
          value=""
          autocomplete="off"
        />
      </div>
    </form>
    <div id="form-submit-msg"></div>
    <script src="form.js"></script>
  </body>
</html>

Make sure to replace "YOUR-DOMAIN" in the form action attribute to the domain you are using. For Azure functions local development the form action could be http://localhost:7071/api. We want the form action to end with "/api", rather than include the full url, so that the form "data-type" attribute can be appended to the url later with JavaScript. This way if anyone is attempting to scrape this form they would not get the full url without inspecting the JavaScript code executing the AJAX request.

The bottom of the HTML document includes a reference to a script named "form.js" and this is where the JavaScript code to submit the form will be included. We can create that file now with TypeScript.

TypeScript Form Submit

For this example we'll use TypeScript, which will transpile to the script referenced in the HTML form (script.js). More info on how to use TypeScript with HTML forms can be found in this article showing how to submit a FormData object using the ES6 Fetch Web API. With TypeScript properly configured, we can create the form.ts file and add some of the needed code:

window.addEventListener("load", async function () {
  new FormHandler();
});

Now we can create the FormHandler class that is instantiated when the HTML document is loaded, by adding it directly below the window event listener.

class FormHandler {
  constructor() {
    this.formSubmitListener();
  }

  private formSubmitListener() {
    document.body.addEventListener("submit", async function (event) {
      event.preventDefault();
    });
  }
}

The private method "formSubmitListener" is invoked during the constructor of the FormHandler class, and includes the registration of an additional event listener that will be activated on the HTML form submit event. Currently this only prevents the default event from occurring, so we can add additional code to get the data from the form.

// inform user form is submitting
const submitButton = document.querySelector(
  "button[type=submit]"
) as HTMLInputElement;

submitButton.disabled = true;

const statusMsgElement = document.getElementById("form-submit-msg");

statusMsgElement!.innerText = "Submitting reply... Please wait.";

// gather form element data
const form = event.target as HTMLFormElement;

const formData = new FormData(form);

The first bit of code added, will select the submit button of the form and disable it during the submission so that the form cannot be submitted multiple times. Then the "form-submit-msg" element will show a message that indicates to the viewer the form is processing. After alerting the user, the form is gathered from the event target passed as an argument of the submit event listener. The "event.target" value is cast to an HTMLFormElement so that TypeScript will permit the access of the "target" property. Then a FormData object is instantiated with the form element. Next we can send the formData variable using the Fetch API.

Get csrf Token and Post FormData with Fetch API

Before accessing the result of the form submission, two extra helper functions are created to handle and log any errors that may occur during the Fetch API post request. Once the helper functions are created the Fetch request is stored in the "result" variable.

const errorHandler = async (response: Response) => {
  if (!response.ok) {
    const err = await response.json().then((err) => err);

    throw Error(
      JSON.stringify({
        status: response.status,
        statusText: response.statusText,
        error: err,
      })
    );
  }

  return response;
};

const errorLogger = (error: Error) => {
  // overwrite message to inform user
  error.message = "An error occurred. Please try again.";
  return error;
};

// submit formData with error handling and logging
const result = await fetch(
  `${form.action}/formToken/${new Date(new Date().toUTCString()).getTime()}/${
    form.dataset.type
  }`
)
  .then(errorHandler)
  .then((response: Response) => response.json())
  .then((data) => {
    // anti-forgery
    formData.append("_csrf", data.token);
    return data.type;
  })
  .then(
    async (type) =>
      // casting to any here to satisfy tsc
      // sending body as x-www-form-url-encoded
      // formData convert to array for edge browser support
      await fetch(`${form.action}/${type}`, {
        method: form.method,
        body: new URLSearchParams([...(formData as any)]),
      })
  )
  .then(errorHandler)
  .then((response: Response) => response.json())
  .then((json) => json)
  .catch(errorLogger);

statusMsgElement!.innerText = result.message;
submitButton.disabled = false;

Since we need a CSRF token and the HTML form isn't rendered server-side (it is pre-rendered as is the case with a site built with the Jamstack) there is actually two fetch requests sent. The first one is a GET request to an endpoint that will provide the token, and then that token is appended to the formData object created earlier. The url pattern for this endpoint includes the "data-type" attribute from the form and a current timestamp. The timestamp is an extra step of validation that will occur in the serverless function created later. Additionally the formToken endpoint sends back the form data type sent to it, so that it can be passed to the second request.

After getting a valid token the next request is a POST request to the form "data-type" endpoint, and the body of the request includes the updated formData object with the "_csrf" token appended. This request is responsible for saving the data if it is sent with a valid CSRF token, and the form data is valid.

The last bit of code below the result is showing a message to the user after the Fetch request completes, showing whether the submission was successful, or an error occurred and they should try again. Additionally the submit button is no longer disabled so the form can be submitted again.

The entire form.ts file should look like this:

window.addEventListener("load", async function () {
  new FormHandler();
});

class FormHandler {
  constructor() {
    this.formSubmitListener();
  }

  private formSubmitListener() {
    document.body.addEventListener("submit", async function (event) {
      event.preventDefault();

      // inform user form is submitting
      const submitButton = document.querySelector(
        "button[type=submit]"
      ) as HTMLInputElement;

      submitButton.disabled = true;

      const statusMsgElement = document.getElementById("form-submit-msg");

      statusMsgElement!.innerText = "Submitting reply... Please wait.";

      // gather form element data
      const form = event.target as HTMLFormElement;

      const formData = new FormData(form);

      const errorHandler = async (response: Response) => {
        if (!response.ok) {
          const err = await response.json().then((err) => err);

          throw Error(
            JSON.stringify({
              status: response.status,
              statusText: response.statusText,
              error: err,
            })
          );
        }

        return response;
      };

      const errorLogger = (error: Error) => {
        // overwrite message to inform user
        error.message = "An error occurred. Please try again.";
        return error;
      };

      // submit formData with error handling and logging
      const result = await fetch(
        `${form.action}/formToken/${new Date(
          new Date().toUTCString()
        ).getTime()}/${form.dataset.type}`
      )
        .then(errorHandler)
        .then((response: Response) => response.json())
        .then((data) => {
          // anti-forgery
          formData.append("_csrf", data.token);
          return data.type;
        })
        .then(
          async (type) =>
            // casting to any here to satisfy tsc
            // sending body as x-www-form-url-encoded
            // formData convert to array for edge browser support
            await fetch(`${form.action}/${type}`, {
              method: form.method,
              body: new URLSearchParams([...(formData as any)]),
            })
        )
        .then(errorHandler)
        .then((response: Response) => response.json())
        .then((json) => json)
        .catch(errorLogger);

      statusMsgElement!.innerText = result.message;
      submitButton.disabled = false;
    });
  }
}

CSRF Token Serverless Function

The client-side code is now set up, so we can look at creating the Azure TypeScript Serverless Functions that will provide a server-side environment to generate the CSRF token and then validate the token to save the form submission data. Here is the quickstart documentation for creating an Azure TypeScript function with Visual Studio code. Once that is setup, we are going to create two functions. The first is the formToken endpoint.

In your functions package.json make sure to include the csrf npm package by running the command npm install csrf --save

Here is the functions.json file associated with the index.ts formToken code that follows:

{
  "bindings": [
    {
      "authLevel": "anonymous",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req",
      "methods": ["get"],
      "route": "formToken/{timeStamp:long}/{formType:alpha}"
    },
    {
      "type": "http",
      "direction": "out",
      "name": "res"
    }
  ],
  "scriptFile": "../dist/formToken/index.js"
}

This function only accepts GET requests, and requires two route parameters, timeStamp and formType. These are included in the client side script we created earlier.

Here is the formToken function code:

import { AzureFunction, Context } from "@azure/functions";
import * as csrf from "csrf";

const httpTrigger: AzureFunction = async function (
  context: Context
): Promise<void> {
  context.log("HTTP trigger function processed a request.");

  context.res!.headers["Content-Type"] = "application/json";

  const utcTime = new Date().toUTCString();

  const submitTime = new Date(
    new Date(context.bindingData.timeStamp).toUTCString()
  ).getTime();

  // add some skew
  const futureDateLimit = new Date(utcTime).getTime() + 1000 * 60 * 5;

  const pastDateLimit = new Date(utcTime).getTime() - 1000 * 60 * 5;

  if (submitTime > futureDateLimit || submitTime < pastDateLimit) {
    // don't create token but also don't return error
    context.res!.status = 200;
    context.res!.body = { message: "success" };
  } else {
    const tokens = new csrf();

    const token = tokens.create(process.env["csrfSecret"]);

    context.res!.status = 200;
    context.res!.body = { token: token, type: context.bindingData.formType };
  }
};

export default httpTrigger;

This function first gathers the current time, and then the time submitted as the timeStamp route parameter. Then a past and future date limit are calculated based on the current time. If the submitted timeStamp is not within the date limit range then the request is ignored and a fake success message is sent back. This is to deter any bots from attempting to make anymore additional requests.

If the timestamp is valid, a new token is generated using the csrf npm package tokens.create() function. In order to prevent the secret from being accessed publicly or accidentally stored in a git repository, a process environment variable is referenced to obtain the "csrfSecret" value. This is the documentation on how to add an application setting in the Azure portal. With the generated token, the function returns the response object, including the token and the "formType" route parameter that was sent with the request.

In this example the same secret is used for all tokens that are generated. This may be useful as all tokens can be invalidated by changing the secret, and given the short length of the token date limit range this can work well. However, it may be advantageous to use the csrf npm package token.secret() function to dynamically create a new secret for each token that is generated. You could then store both the token and the secret in a database, or Azure Table Storage, and use the token to look up the stored secret, to later verify the token on the subsequent request.

Contact Form Serverless Function

The second serverless function is going to accept the contact form data with the csrf token appended. Additionally, it will verify the hidden decoy password form field, and the csrf token. If both validations pass then the data can be saved.

Here is the functions.json for the contact serverless function:

{
  "bindings": [
    {
      "authLevel": "anonymous",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req",
      "methods": ["post"]
    },
    {
      "type": "http",
      "direction": "out",
      "name": "res"
    }
  ],
  "scriptFile": "../dist/contact/index.js"
}

Note that the contact function is limited to only accept post requests.

Below is the index.ts function code:

import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import * as querystring from "querystring";
import * as csrf from "csrf";

const httpTrigger: AzureFunction = async function (
  context: Context
): Promise<void> {
  context.log("HTTP trigger function processed a request.");

  context.res!.headers["Content-Type"] = "application/json";

  //sent as x-www-form-url-encoded
  const body = querystring.parse(req.body);

  // check hidden form field
  const verifiedHiddenFormField =
    body && (body.password === undefined || body.password.length);

  // verify token with secret
  const verifiedToken = new csrf().verify(
    process.env["csrfSecret"],
    body._csrf
  );

  if (!verifiedHiddenFormField || !verifiedToken) {
    // failed verification
    context.res!.status = 200;
    context.res!.body = { message: "success" };
    return;
  }

  if (
    !(body && body.firstName && body.lastName && body.email && body.message)
  ) {
    context.res!.status = 400;
    context.res!.body = {
      message: "Contact form is invalid. Please correct errors and try again.",
    };
    return;
  }

  //todo: save the comment form data!

  context.res!.status = 200;
  context.res!.body = {
    message: "Thank you for contacting me! I will reply to you shortly.",
  };
};

export default httpTrigger;

The contact function first parses the request body using the querystring parse method, which will create an object from the form data that was sent. Then the decoy password field is verified to exist, but also not have a value present. The csrf token appended to the form data is then verified using the process.env "csrfSecret" value. If both of these verifications are passing, then the function execution can proceed. Otherwise, like the formToken function, an empty success message is returned to deter further, possibly malicious requests.

After verification, the contact form info is checked to make sure all the fields have a value. If they don't an error message is returned and displayed to the viewer with the client side errorHandler and errorLogger functions created earlier.

At this point, with both verifications passing and valid form data, the data can be saved to the preferred data store. This could be a sql database or a nosql data store like azure storage. Once the save is complete the function will return a success message and the client side code will display that to the viewer.

EJS

Begin by creating a new EJS file named index.ejs. This file will be the template used to generate index.html. If the model is passed into the template it will render the content as a paragraph.

<!-- Sample Page -->

<h1>Sample Page</h1>

<% if (model) { %>
  <%= model.content %>
<% } %>

package.json

If you don't already have a package.json created you can create one by running the command npm init and following the prompts.

You will need your package.json to include these packages:

{
  "name": "package-name-goes-here",
  "version": "0.0.0",
  "devDependencies": {
    "@types/ejs": "^2.6.2",
    "@types/node": "^11.9.4",
    "ejs": "^2.6.1",
    "typescript": "^3.3.3333"
  }
}

You can also copy the devDependencies section and run the command npm install instead of installing one at a time.

Node.js

Create a new TypeScript file named render.ts. Then add the following code to import the modules that we will use.

//imports
import util = require("util");
import fs = require("fs");
import ejs = require("ejs");
//promisify
const mkdir = util.promisify(fs.mkdir);
const readFile = util.promisify(fs.readFile);
const writeFile = util.promisify(fs.writeFile);

The first import is the util module so that we can use the promisify function. Then import the fs module for file system access. Before using three of the functions from the fs module we can promisify them allowing for the use of async/await instead of nested callbacks. The last is for EJS, and since the render file function returns a promise by default we do not need to use promisify.

Below the import statements add an async function named render. This is where the HTML output will be generated and written to a file named index.html. It needs to be marked as an async function so that the keyword await can be used. Then make sure to call the function so the code that is about to be added will execute.

async function render() {
  try {
  } catch (error) {
    console.log(error);
  }
}
render();

Before rendering our EJS file we will need a folder to put the output. So add the following to our render function:

await mkdir("dist", { recursive: true });

This will create a new directory named dist where the html output will be saved. By passing the recursive property we can ensure parent folders are created even if none are necessary. After creating the dist folder we can use EJS to render the index.ejs template to HTML. The resulting HTML string is then written to a file named index.html in the dist folder.

At this point your index.ts file should look like this:

//imports
import util = require("util");
import fs = require("fs");
import ejs = require("ejs");
//promisify
const mkdir = util.promisify(fs.mkdir);
const readFile = util.promisify(fs.readFile);
const writeFile = util.promisify(fs.writeFile);
async function render() {
  try {
    //create output directory
    await mkdir("dist", { recursive: true });

    //render ejs template to html string
    const html = await ejs
      .renderFile("index.ejs", { model: false })
      .then((output) => output);
    //create file and write html
    await writeFile("dist/index.html", html, "utf8");
  } catch (error) {
    console.log(error);
  }
}
render();

In order to run this script we need to add a tsconfig.json file to configure the TypeScript compiler. This will compile the TypeScript into JavaScript so that it can be used by node.js. Add the tsconfig file to the same folder as the render.js script.

{
  "compilerOptions": {
    "module": "commonjs",
    "moduleResolution": "node",
    "rootDir": "./",
    "outDir": "./dist",
    "sourceMap": true
  },
  "include": ["render.js"]
}

We also need to add a script to the package.json file created earlier. This script will compile render.ts and then run it using node. Your package.json should look like this:

{
  "name": "package-name-goes-here",
  "version": "0.0.0",
  "scripts": {
    "render": "tsc && node dist/render.js"
  },
  "devDependencies": {
    "@types/ejs": "^2.6.2",
    "@types/node": "^11.9.4",
    "ejs": "^2.6.1",
    "typescript": "^3.3.3333"
  }
}

EJS render HTML

The render script can be run in a terminal window by typing the command npm run render. Make sure to run this command from the directory where your package.json is located. After running the render script you should now see a folder named dist containing a file named index.html.

The contents of index.html should look like this:

Sample Page

Notice that the conditional block containing the model content, in the index.ejs template, is not included in the html output. This is because in the render script the model was passed in as false. Now we'll create an object to pass in as the model with some sample content to the sample page.

In the render.ts file previously created, after the import statements, create an object and add a property to it called content with the value set to a sample of content.

const pageModel = {
  content: "This is some sample content. Located on the sample page.",
};

Then pass this object in to the ejs.renderFile function instead of false. The render.ts file should look like this:

//imports
import util = require("util");
import fs = require("fs");
import ejs = require("ejs");
//promisify
const mkdir = util.promisify(fs.mkdir);
const readFile = util.promisify(fs.readFile);
const writeFile = util.promisify(fs.writeFile);

const pageModel = {
  content: "<p>This is some sample content. Located on the sample page.</p>",
};
async function render() {
  try {
    //create output directory
    await mkdir("dist", { recursive: true });

    //render ejs template to html string
    //pass pageModel in to render content
    const html = await ejs
      .renderFile("index.ejs", { model: pageModel })
      .then((output) => output);
    //create file and write html
    await writeFile("dist/index.html", html, "utf8");
  } catch (error) {
    console.log(error);
  }
}
render();

With the model object passed into the template we should now see the conditional block rendered in the index.html output file. Run the command npm run render once more.

The index.html file in the dist folder should now look like this:

<h1>Sample Page</h1>
<p>This is some sample content. Located on the sample page.</p>

The index.ejs template can now render dynamic HTML content according to the model object configured in the render.ts file and by running npm run render after each change to generate an updated index.html file.

HTML Form

First we need to create an html file, let's call it index.html, with a form element to capture the input values we are going to submit using JavaScript.

<!-- index.html -->
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>Example Form</title>
  </head>
  <body>
    <form id="myForm" action="myFormAction" method="post">
      <!-- this input is hidden with a value already set -->
      <input type="hidden" id="userId" name="userId" value="3" />
      <label for="firstName">first name</label>
      <input type="text" id="firstName" name="firstName" />
      <button type="submit">Submit</button>
    </form>
    <script src="form.js"></script>
  </body>
</html>

Take note that the value for the form action attribute is a placeholder. In real usage this would be replaced with url that you would like to submit the form to. One of the inputs is type=hidden to show that we can submit hidden elements using this technique. Additionally there is one input to capture first name and a button to submit the form using the HTTP post method.

Typescript Form Submit

Next we'll need to write form.ts so the TypeScript compiler can generate the JavaScript file, form.js, referenced in index.html. The code in form.ts will handle the form submit by making an AJAX request. If you haven't already now is a good time to read my other post Compile Typescript with npm. There you will find instructions on how to install and configure TypeScript to accommodate the usage below.

The code below assumes the endpoint we are submitting the form to, in the sample HTML action="myFormAction", will be returning a response that has the Content-Type header set to application/json.

To begin an event listener is created to listen for all form submit events. Notice that the callback function is marked as an async function. Using the async modifier allows for the use of the await keyword when executing the asynchronous Fetch request.

// form.ts

// listen for any form submit event
document.body.addEventListener("submit", async function (event) {});

Inside the callback the first line of code prevents the default action from occurring. Without preventing the default, the browser would attempt to navigate to the URL of the form action attribute when the form is submitted.

// form.ts

// listen for any form submit event
document.body.addEventListener("submit", async function (event) {
  event.preventDefault();
});

Next a variable for the form element is created and cast to an HTMLFormElement to allow access to the action and method properties.

// form.ts

// listen for any form submit event
document.body.addEventListener("submit", async function (event) {
  event.preventDefault();

  const form = event.target as HTMLFormElement;
});

Fetch API Post Form Data

Then the result variable is created and it is used to store the response sent following the Fetch request. The Fetch request returns a promise and must be awaited so that the result can be obtained. The URL passed into the Fetch method is set to the action of the form, and the options contains keys for method and body values. The form method, like the action, is available from HTMLFormElement.

// form.ts

// listen for any form submit event
document.body.addEventListener("submit", async function (event) {
  event.preventDefault();
  const form = event.target as HTMLFormElement;

  // casting to any here to satisfy tsc
  // sending body as x-www-form-url-encoded
  const result = await fetch(form.action, {
    method: form.method,
    body: new URLSearchParams([...(new FormData(form) as any)]),
  })
    .then((response: Response) => response.json())
    .then((json) => json)
    .catch((error) => console.log(error));
});

Notice the use of spread syntax to transform the FormData into an array of key-value pairs. This may seem redundant, but the Edge browser cannot iterate over FormData objects. By transforming the object into an array, Edge is able to successfully construct the URLSearchParams object.

Interestingly the current TypeScript definition for URLSearchParams does not permit a FormData object to be passed into the constructor, however this is valid JavaScript. To satisfy the TypeScript compiler the FormData object is cast to any. This allows a URLSearchParams object to be constructed from the FormData object which itself is constructed from the HTMLFormElement. Since the body of the Fetch request is of the type URLSearchParams (hint: it looks like a ?query=string) the resulting Content-Type of the request body will be x-www-form-url-encoded. This allows for the server to parse it as it would a normal form submission.

Now when the form in the index.html file is submitted the submit event listener will override the default browser behavior and submit the form using an AJAX request. Even without an actual endpoint to receive the request you should still be able to verify the code is working because the resulting error "Failed to fetch" will be logged to the console.

Create GitHub Git Repository

The first GitHub repo that we need to create will be public and is where our comments will ultimately end up. GitHub provides documentation for creating a repo. After creating the public repository, a private repository is also needed and is going to be used so that comments can be moderated through the creation of pull requests. The private repository also allows for any comment information, like emails, to be filtered out before merging into the public repository.

HTML Comment Form

With the git repositories set up we can now create a standard HTML form that will submit comments to our serverless function (not yet set up) endpoint.

<!-- form.html -->
<form id="commentForm" action="FUNCTION_ENDPOINT" method="post">
  <input id="postId" type="hidden" name="postId" value="POST_ID" />
  <div>
    <label for="comment">comment</label>
    <textarea required rows="5" id="comment" name="comment"></textarea>
  </div>
  <div>
    <label for="authorName">name</label>
    <input
      required
      type="text"
      id="authorName"
      name="authorName"
      autocomplete="name"
    />
  </div>
  <div>
    <label for="authorEmail">email</label>
    <input
      required
      type="email"
      id="authorEmail"
      name="authorEmail"
      autocomplete="email"
    />
  </div>
  <button type="submit">Submit</button>
</form>

In most cases a static site generator would be outputting this form from template files, but the important part is that the form action shown as "FUNCTION_ENDPOINT" will be replaced with the actual url that will be provided by the serverless function in the following section. There also needs to be a way to maintain the relationship between the comment submitted and the blog post it should reference. In this case a hidden field is added with a value of "POST_ID" to maintain this data during the form submit. This can be changed to anything that suits the build process in use, so that comments can be stored with this as a key to indicate which post they belong to.

Azure Serverless Function

Now that the client side HTML form is in place, we need an endpoint to submit the form to. Azure Javascript functions will be used to provide an endpoint configured to accept HTTP POST requests containing comment data, in the request body, that will be committed by our serverless function to the private git repository. Microsoft provides documentation to set up a TypeScript function with Visual Studio Code. Please make sure to reference their documentation before proceeding. Below is the starting code that we will build out TypeScript function with:

// comment.ts
import { AzureFunction, Context, HttpRequest } from "@azure/functions";
const httpTrigger: AzureFunction = async function (
  context: Context,
  req: HttpRequest
): Promise<void> {
  context.log("HTTP trigger function processed a request.");
  context.res!.headers["Content-Type"] = "application/json";
  context.res!.status = 200;
  context.res!.body = { message: "Success!" };
};
export default httpTrigger;

At this point all the function does is set the Content-Type response header and return an HTTP 200 OK success status response code with a success message. Next we will npm install the npm packages needed for the functions code.

npm install

We are going to want to use the following npm packages within the code of the serverless function we are creating:

• uuid
• simple-git
• rimraf
• sendgrid/mail
• octokit/rest

To install these packages, all at the same time, and their corresponding types to use with Typescript, run the command: npm install @sendgrid/mail @octokit/rest rimraf simple-git uuid @types/node @types/rimraf --save-dev.

Then add these import states to the comment.ts file:

import * as querystring from "querystring";
import util = require("util");
import uuidv4 = require("uuid/v4");
import * as SendGrid from "@sendgrid/mail";
import * as simpleGit from "simple-git/promise";
import { formHelpers } from "../common/formHelpers";
import { Octokit } from "@octokit/rest";
import fs = require("fs");
import rimrafstd = require("rimraf");
import { tmpdir } from "os";
const rimraf = util.promisify(rimrafstd);
const mkdir = util.promisify(fs.mkdir);
const writeFile = util.promisify(fs.writeFile);
const readFile = util.promisify(fs.readFile);
SendGrid.setApiKey(process.env["SendGridApiKey"] as string);

The last import statement uses and environment variable to securely access a SendGrid API key. In order to send out notification emails (this will be set up in a later section), create a SendGrid account and configure an API Key. Azure Serverless Functions support adding additional application settings where the API key can be saved as an environment variable. By using an environment variable we prevent the need to store the SendGrid API key directly in the serverless function source code.

Validate POST request body

Next add some basic validation to ensure that the comment form is submitted appropriately.

const body = querystring.parse(req.body);

if (
  !(body && body.comment && body.postId && body.authorEmail && body.authorName)
) {
  context.res!.status = 400;
  context.res!.body = {
    message: "Comment invalid. Please correct errors and try again.",
  };
  return;
}

After parsing the request body using the querystring module the validation code checks to make sure the form fields are filled out with data. In a production environment these checks would need to be much more strict, to ensure there is no CSRF attacks being attempted.

Initialize Git Repository with Simple Git

Next we will begin the process of creating a temporary repository in the serverless functions default directory for temporary files using the os module , adding a new branch, and committing the newly submitted comment so that, in a later step, a pull request for the new branch can be created programmatically.

//Initialize Git Repository with Simple Git

// generate unique folder name for git repository
const tempRepo = uuidv4();

// create empty directory to store comment file
await mkdir(`${tmpdir}/${tempRepo}/comments`, {
  recursive: true,
});

// initialize simple-git
const git = simpleGit(`${tmpdir}/${tempRepo}`);

// initialize git repository in tempRepo
await git.init();

// set up git config
await Promise.all([
  git.addConfig("user.name", "GITHUB_USERNAME"),
  git.addConfig("user.email", "GITHUB_EMAIL"),
]);

// add the private remote
await git.addRemote(
  "private",
  `https://GITHUB_USERNAME:${process.env["GitHubUserPassword"]}@https://github.com/GITHUB_USERNAME/PRIVATE_REPOSITORY`
);

Since this code resides within a serverless function there is no state that is saved in between requests. This requires creating a unique folder and initializing a new git repository every time the serverless function is activated. Once the git repo is initialized in a temp folder the user name and email are configured. These currently set to "GITHUB_USERNAME" and "GITHUB_EMAIL" should be updated to match your account information.

Once the git config is set, a remote is added to reference the private repository that was created earlier. For convenience the remote is named "private", although this can be changed to something more suitable in your case. GitHub requires authentication for private repositories, so the GitHub account password is accessed as an environment variable, similar to the SendGrid API key set up previously. When adding the password application setting it is also a good idea to use a GitHub personal access token (PAT) instead of your main GitHub account password. The GitHub PAT can be included the same way a regular password would be.

Checkout Git Branch with Simple Git

//Checkout git branch with Simple Git

// generate unique id for comment
const commentId = uuidv4();

// create branch
try {
  // fetch main branch to base of off
  await git.fetch("private", "main");

  // use postId to see if comments already are saved for this post
  await git.checkout("private/main", ["--", `comments/${body.postId}.json`]);

  // create new branch named with commentID based off main branch
  await git.checkoutBranch(`${commentId}`, "private/main");
} catch (error) {
  // no previous comments are saved for this post
  await git.checkout("private/main");
  await git.checkoutLocalBranch(`${commentId}`);
}

Each comment needs a unique identifier, and the uuid npm package is used to generate a GUID that we save the the commentId variable. The code that follows is contained in a try catch block, because in the case of a brand new comment there will not be a file corresponding to the post that contains the comments previously submitted. In this case the checkout of the JSON file with the name of the postId from the parsed request body will throw an error because git will indicate that this file does not exist.

In either case of appending a comment to an existing list or committing the first one, the end result of the try catch block will be a new branch checked out with the name of the commentId that was just generated. Be sure to note the difference between checkoutBranch and checkoutLocalBranch in the Simple Git git checkout documentation.

Write JSON File

// Write JSON File with updated Comment data

// create comment object to store as JSON in git repository
const comment = {
  id: commentId,
  timestamp: new Date(new Date().toUTCString()).getTime(),
  authorEmail: body.authorEmail,
  authorName: body.authorName,
  bodyText: body.comment,
};

// list of all comments
let comments = [];

// retrieve existing comments
try {
  comments = JSON.parse(
    await readFile(`${tmpdir}/${tempRepo}/comments/${body.postId}.json`, "utf8")
  );
} catch (error) {
  //no previous comments
}

// add newly submitted comment
comments.push(comment);

// update or create new comments file with new comment included
await writeFile(
  `${tmpdir}/${tempRepo}/comments/${body.postId}.json`,
  JSON.stringify(comments, null, 2),
  "utf8"
);

Now that the temporary git repository is configured and we have checked out a branch with the latest comments (if any exist), we can update the JSON file containing the comments to include the new one. First, an object is created that represents the new comment data. Then in the following try catch block we attempt to read and parse into JSON, the existing file with the name of the postId included in the request body, corresponding to the blog post commented on. In the event that this file does not exists there will be an error that is caught and the execution of the code can proceed. In this case when the file cannot be read, because it does not exist, it means that we have no comments saved previously similar to the try catch block used previously during the branch checkout.

Once the list of all comments is hydrated, or if it remains an empty array, the new comment can be added to it. Then the entire list of comments is written back to the same file corresponding the the postId, and the changes to this file are ready to be committed and pushed to the private git repository.

Git Commit and Push to Private Repository

// stage file modifications, commit and push

await git.add(`${tmpdir}/${tempRepo}/comments/${body.postId}.json`);

await git.commit(`adding comment ${commentId}`);

await git.push("private", `${commentId}`);

// delete temporary repository
await rimraf(`${tmpdir}/${tempRepo}/`);

Here we are adding the modifications from the file we just wrote to, with the name of the postId, to the branch currently checked out with the name of the commentId, and then that branch is pushed to the private remote origin. Once the push is complete, the temporary directory we previously created is no longer needed, and the rimraf npm package is used to recursively delete the entire directory and its contents.

Send Notification Emails and Create Pull Request with Octokit

The last bit of code needed for the comment.ts function, will construct two emails, one to you and one to the reader that submitted the comment. It will also use the GitHub Octokit REST API client to create a pull request for the branch that was pushed with the new comment committed. This way the comment can be moderated before displaying publicly. To prevent the comment from being published the pull request can be declined and the branch with the comment can be deleted all within the GitHub interface.

//send notifications and create pull request

const userEmail = {
  to: body.authorEmail,
  from: "YOUR_NAME@YOUR_WEBSITE",
  subject: "comment submitted",
  text: "Your comment will be visible when approved.",
};

const adminEmail = {
  to: "ADMIN_EMAIL",
  from: "ADMIN_EMAIL",
  subject: "comment submitted",
  html: `<div>from: ${body.authorName}</div>
         <div>email: ${body.authorEmail}</div>
         <div>comment: ${body.comment}</div>`,
};

await Promise.all([
  SendGrid.send(userEmail),
  SendGrid.send(adminEmail),
  new Octokit({
    auth: process.env["GitHubUserPassword"],
  }).pulls.create({
    owner: "GITHUB_USERNAME",
    repo: "PRIVATE_REPOSITORY",
    title: `${commentId}`,
    head: `${commentId}`,
    base: "main",
  }),
]);

Both SendGrid.send() and Octokit.pulls.create() are asynchronous and return a promise. To take advantage of this we use Promise.all() to carry out all three actions: sending two emails and the HTTP Request to the GitHub REST API simultaneously. Using the await keyword ensures that all three promises are resolved before continuing.

When we put all these code sections together the result should look like this:

// comment.ts

import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import * as querystring from "querystring";
import util = require("util");
import uuidv4 = require("uuid/v4");
import * as SendGrid from "@sendgrid/mail";
import * as simpleGit from "simple-git/promise";
import { formHelpers } from "../common/formHelpers";
import { Octokit } from "@octokit/rest";
import fs = require("fs");
import rimrafstd = require("rimraf");
import { tmpdir } from "os";
const rimraf = util.promisify(rimrafstd);
const mkdir = util.promisify(fs.mkdir);
const writeFile = util.promisify(fs.writeFile);
const readFile = util.promisify(fs.readFile);
SendGrid.setApiKey(process.env["SendGridApiKey"] as string);

const httpTrigger: AzureFunction = async function (
  context: Context,
  req: HttpRequest
): Promise<void> {
  context.log("HTTP trigger function processed a request.");

  context.res!.headers["Content-Type"] = "application/json";

  const body = querystring.parse(req.body);

  if (
    !(
      body &&
      body.comment &&
      body.postGuid &&
      body.authorEmail &&
      body.authorName
    )
  ) {
    context.res!.status = 400;
    context.res!.body = {
      message: "Comment invalid. Please correct errors and try again.",
    };
    return;
  }

  //Initialize Git Repository with Simple Git

  // generate unique folder name for git repository
  const tempRepo = uuidv4();

  // create empty directory to store comment file
  await mkdir(`${tmpdir}/${tempRepo}/comments`, {
    recursive: true,
  });

  // initialize simple-git
  const git = simpleGit(`${tmpdir}/${tempRepo}`);

  // initialize git repository in tempRepo
  await git.init();

  // set up git config
  await Promise.all([
    git.addConfig("user.name", "GITHUB_USERNAME"),
    git.addConfig("user.email", "GITHUB_EMAIL"),
  ]);

  // add the private remote
  await git.addRemote(
    "private",
    `https://GITHUB_USERNAME:${process.env["GitHubUserPassword"]}@https://github.com/GITHUB_USERNAME/PRIVATE_REPOSITORY`
  );

  //Checkout git branch with Simple Git

  // generate unique id for comment
  const commentId = uuidv4();

  // create branch
  try {
    // fetch main branch to base of off
    await git.fetch("private", "main");

    // use postID to see if comments already are saved for this post
    await git.checkout("private/main", ["--", `comments/${body.postId}.json`]);

    // create new branch named with commentID based off main branch
    await git.checkoutBranch(`${commentId}`, "private/main");
  } catch (error) {
    // no previous comments are saved for this post
    await git.checkout("private/main");
    await git.checkoutLocalBranch(`${commentId}`);
  }

  // Write JSON File with updated Comment data

  // create comment object to store as JSON in git repository
  const comment = {
    id: commentId,
    timestamp: new Date(new Date().toUTCString()).getTime(),
    authorEmail: body.authorEmail,
    authorName: body.authorName,
    bodyText: body.comment,
  };

  // list of all comments
  let comments = [];

  // retrieve existing comments
  try {
    comments = JSON.parse(
      await readFile(
        `${tmpdir}/${tempRepo}/comments/${body.postId}.json`,
        "utf8"
      )
    );
  } catch (error) {
    //no previous comments
  }

  // add newly submitted comment
  comments.push(comment);

  // update or create new comments file with new comment included
  await writeFile(
    `${tmpdir}/${tempRepo}/comments/${body.postId}.json`,
    JSON.stringify(comments, null, 2),
    "utf8"
  );

  // stage file modifications, commit and push

  await git.add(`${tmpdir}/${tempRepo}/comments/${body.postId}.json`);

  await git.commit(`adding comment ${commentId}`);

  await git.push("private", `${commentId}`);

  // delete temporary repository
  await rimraf(`${tmpdir}/${tempRepo}/`);

  //send notifications and create pull request

  const userEmail = {
    to: body.authorEmail,
    from: "YOUR_NAME@YOUR_WEBSITE",
    subject: "comment submitted",
    text: "Your comment will be visible when approved.",
  };

  const adminEmail = {
    to: "ADMIN_EMAIL",
    from: "ADMIN_EMAIL",
    subject: "comment submitted",
    html: `<div>from: ${body.authorName}</div>
           <div>email: ${body.authorEmail}</div>
           <div>comment: ${body.comment}</div>`,
  };

  await Promise.all([
    SendGrid.send(userEmail),
    SendGrid.send(adminEmail),
    new Octokit({
      auth: process.env["GitHubUserPassword"],
    }).pulls.create({
      owner: "GITHUB_USERNAME",
      repo: "PRIVATE_REPOSITORY",
      title: `${commentId}`,
      head: `${commentId}`,
      base: "main",
    }),
  ]);

  context.res!.status = 200;
  context.res!.body = {
    message: "Success!",
  };
};

export default httpTrigger;

At this point, we have one of the two serverless functions completed! Next we will need a way to moderate comments that are submitted to the comment.ts function shown above. To do this another serverless function will be used, which we will name "comment-merge.ts". The goal of this function will be to integrate moderated comments into the public repo that was created initially, and to filter out any sensitive data that should not be publicly displayed.

GitHub Webhook

Before beginning the code of the comment-merge.ts function a GitHub webhook needs to be created that will send a POST request on pull request events. In the private repository settings on GitHub add a webhook that points to the serverless function url, and select only the pull request event rather than the default of activating for all of the event types. This will enable the comment-merge.ts function to be activated anytime we accept one of the pull requests created as a result of a new comment submission.

Now that the GitHub webhook is configured to listen for pull request events occurring in the private repository we can set up the second serverless function to act on these events. There is one additional npm package that will be needed for this function, and it can be installed by running the command npm install glob @types/glob --save-dev. This will install the glob npm package and the corresponding types.

The same beginning code from the first function can be used for the merge function, so we can skip ahead a bit and look at the imports that will be needed.

// comment-merge.ts
import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import util = require("util");
import * as querystring from "querystring";
import * as simpleGit from "simple-git/promise";
import fs = require("fs");
import { tmpdir } from "os";
import uuidv4 = require("uuid/v4");
import globstd = require("glob");
import rimrafstd = require("rimraf");
const rimraf = util.promisify(rimrafstd);
const glob = util.promisify(globstd);
const mkdir = util.promisify(fs.mkdir);
const writeFile = util.promisify(fs.writeFile);
const readFile = util.promisify(fs.readFile);

These should look similar to the first function, with the glob package also being imported.

Validate GitHub Webhook Post Request

Now we can add code that will parse the request body that is sent from the GitHub webhook. The webhook is sent with the data needed as the value of the payload property. Like the request body of our initial comment function the querystring package is used to parse the payload and then JSON.parse is used to create an object representing the data.

// validate github webhook payload

//request content type is configured in GitHub webhook settings
const payload = req.body;

if (
  payload.action != "closed" ||
  payload.pull_request.base.ref != "main" ||
  !payload.pull_request.merged_at
) {
  return;
}

Since this webhook activates on any event regarding a pull request, whether that be opening or closing, we need to make sure that this code only runs when the pull request is closed. Secondly, the pull request branch needs to match the main branch so that pull requests from other branches are ignored. Lastly, the merged_at value is checked to make sure this pull request has been merged before closing. If the pull request is closed and not merged (the comment is spam) we can ignore the following post request sent by GitHub.

In addition to checking the payload properties shown above, it is a good idea to secure the webhook to make sure the serverless function is only activating when a request is sent from GitHub. This can prevent unwanted requests from being processed, and is a good idea to include when running this code in a production environment.

Add Public and Private GitHub Remotes

// create temp repo and add remotes

const tempRepo = uuidv4();

await mkdir(`${tmpdir}/${tempRepo}/comments`, {
  recursive: true,
});

const git = simpleGit(`${tmpdir}/${tempRepo}`);

await git.init();

await Promise.all([
  git.addConfig("user.name", "GITHUB_USERNAME"),
  git.addConfig("user.email", "GITHUB_EMAIL"),
]);

await Promise.all([
  git.addRemote(
    "private",
    `https://GITHUB_USERNAME:${process.env["GitHubUserPassword"]}@https://github.com/GITHUB_USERNAME/PRIVATE_REPOSITORY`
  ),
  git.addRemote(
    "public",
    `https://GITHUB_USERNAME:${process.env["GitHubUserPassword"]}@https://github.com/GITHUB_USERNAME/PUBLIC_REPOSITORY`
  ),
]);

This code is nearly the same as the temporary git repo creation and initialization that was needed for the first function. The main difference is that two remotes are being added this time, one being the private repository where the comment is stored, and the second is the public repository where moderated comments will be merged into.

Make sure to include the username and password in the remote url for both the private and public remotes, even though for public GitHub repositories this is usually not necessary. This is a result of the Azure serverless function configuration requiring authentication in order to work as expected. If it is not included, when trying to push to the public repository after merging the comment, the git push will fail silently and the function will timeout.

Git Checkout and Fetch

After configuring the remotes some additional git commands are required to checkout the correct branches and fetch the latest file modifications.

// fetch public and integrate with latest modifications from private repo

await git.fetch("public", "main");

await git.checkout("main", ["--", "comments/"]);

await git.checkoutBranch("main", "main");

await git.fetch("private", "main");

await git.checkout("main", ["--", "comments/"]);

This code first fetches the public remote so that the folder containing previously posted comments can be checked out. With the comment data from the main branch of the public repository now included in the temporary repository, the same fetch and checkout commands are used to integrate the private remote where the main branch includes comments that have passed moderation and their corresponding pull request has been merged.

Filter Out Private Data

Now that the temporary git repository has the newest comment, there may be information that should not be made public, like user emails. Before we commit and push the new comment to the public repository we can filter the comment data to remove any information that should not be public. This is also the point where the glob npm package will be utilized.

// filter private data from comments

// retrieve comment file paths
const paths = await glob(`comments/**/*.json`, {
  cwd: `${tmpdir}/${tempRepo}/`,
});

// wait for all paths to process asynchronously
await Promise.all(
  paths.map(async (path) => {
    let pathData = [];

    //read JSON file with comment info
    pathData = JSON.parse(
      await readFile(`${tmpdir}/${tempRepo}/${path}`, "utf8")
    );

    // filter out private info
    const publicData = pathData.map((item) => {
      const { authorEmail, ...store } = item;
      return store;
    });

    // write file back to original with private data removed
    await writeFile(
      `${tmpdir}/${tempRepo}/${path}`,
      JSON.stringify(publicData, null, 2),
      "utf8"
    );
  })
);

This code gets all the paths for the files where comments are stored. Then each path is processed and the file in the temporary folder is read and JSON.parse is used to create an object that we can remove any private data from before publishing. In this case the authorEmail key/value pair is being removed from the comment object, using destructuring assignment syntax, and any remaining properties are kept in place. The filtered data is then written back to the file matching the path using JSON.stringify to retain the original formatting.

Git Commit and Push to Public Repository

// add filtered comment file modifications, commit, and push

await git.add(`${tmpdir}/${tempRepo}/comments/*.json`);

await git.commit("approving comment");

await git.push("public", "main");

await rimraf(`${tmpdir}/${tempRepo}/`);

The last part of the comment merge function includes adding the modifications made the to the comment files to include the new comment with private data filtered out, and committing those changes to the main branch. Once the changes are committed the branch is pushed to the public repository and the comment can now be displayed.

In the case where a static site generator is being used for the blog, this push can trigger a new build and the comment can be included by the build process. The last thing to do, as done in the first function, is to delete the temporary git repository folder since it is no longer needed for the duration of this request.

The comment-merge.ts with all code added should look like this:

// comment-merge.ts
import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import util = require("util");
import * as querystring from "querystring";
import * as simpleGit from "simple-git/promise";
import fs = require("fs");
import { tmpdir } from "os";
import uuidv4 = require("uuid/v4");
import globstd = require("glob");
import rimrafstd = require("rimraf");
const rimraf = util.promisify(rimrafstd);
const glob = util.promisify(globstd);
const mkdir = util.promisify(fs.mkdir);
const writeFile = util.promisify(fs.writeFile);
const readFile = util.promisify(fs.readFile);

const httpTrigger: AzureFunction = async function (
  context: Context,
  req: HttpRequest
): Promise<void> {
  context.log("HTTP trigger function processed a request.");

  context.res!.headers["Content-Type"] = "application/json";

  //request content type is configured in GitHub webhook settings
  const payload = req.body;

  if (
    payload.action != "closed" ||
    payload.pull_request.base.ref != "main" ||
    !payload.pull_request.merged_at
  ) {
    return;
  }

  // create temp repo and add remotes

  const tempRepo = uuidv4();

  await mkdir(`${tmpdir}/${tempRepo}/comments`, {
    recursive: true,
  });

  const git = simpleGit(`${tmpdir}/${tempRepo}`);

  await git.init();

  await Promise.all([
    git.addConfig("user.name", "GITHUB_USERNAME"),
    git.addConfig("user.email", "GITHUB_EMAIL"),
  ]);

  await Promise.all([
    git.addRemote(
      "private",
      `https://GITHUB_USERNAME:${process.env["GitHubUserPassword"]}@https://github.com/GITHUB_USERNAME/PRIVATE_REPOSITORY`
    ),
    git.addRemote(
      "public",
      `https://GITHUB_USERNAME:${process.env["GitHubUserPassword"]}@https://github.com/GITHUB_USERNAME/PUBLIC_REPOSITORY`
    ),
  ]);

  // fetch public and integrate with latest modifications from private repo

  await git.fetch("public", "main");

  await git.checkout("main", ["--", "comments/"]);

  await git.checkoutBranch("main", "main");

  await git.fetch("private", "main");

  await git.checkout("main", ["--", "comments/"]);

  // filter private data from comments

  // retrieve comment file paths
  const paths = await glob(`comments/**/*.json`, {
    cwd: `${tmpdir}/${tempRepo}/`,
  });

  // wait for all paths to process asynchronously
  await Promise.all(
    paths.map(async (path) => {
      let pathData = [];

      //read JSON file with comment info
      pathData = JSON.parse(
        await readFile(`${tmpdir}/${tempRepo}/${path}`, "utf8")
      );

      // filter out private info
      const publicData = pathData.map((item) => {
        const { authorEmail, ...store } = item;
        return store;
      });

      // write file back to original with private data removed
      await writeFile(
        `${tmpdir}/${tempRepo}/${path}`,
        JSON.stringify(publicData, null, 2),
        "utf8"
      );
    })
  );

  // add filtered comment file modifications, commit, and push

  await git.add(`${tmpdir}/${tempRepo}/comments/*.json`);

  await git.commit("approving comment");

  await git.push("public", "main");

  await rimraf(`${tmpdir}/${tempRepo}/`);

  context.res!.status = 200;
  context.res!.body = { message: "success" };
};

export default httpTrigger;

A blog built with the Jamstack can now integrate comments in a way that is very cost effective and maintain a git-centric approach. The comments that readers submit can be moderated, filtered, and are stored right along side the blog content. This way the corresponding JSON files that are created can be integrated into an existing build process and dynamically pre-rendered with the content, eliminating the need to make client side requests to fetch data that would harm the user experience or affect page load time.

Azure serverless functions provide a cost effective way to have on demand cloud compute, without the need to have a server running all of the time, only to be used occasionally. One possible drawback of this approach is that sometimes, due to cold start delays of the the serverless function, when the user submits a comment it can be somewhat slow to process. This is a result of the comment.ts function, while asynchronous, initializing and checking out a git repository, sending two emails and utilizing the GitHub REST API to programmatically create a pull request. It may reduce processing times to remove the email notification component if not needed for your use case.

There are many different ways to compile SCSS, one of the two syntaxes supported by SASS. In this post we will explore the utilization of the node-sass npm package. We'll also look at how we can use the clean-css npm package to minify and optimize the generated output after compiling SCSS into CSS. Both of these techniques are similar to how Bootstrap handles the compilation and minification of its SCSS files. Please make sure you have Node.js and npm installed first.

SCSS

First an SCSS file is needed, and it can be placed in the root of the project folder. To illustrate the preprocessing of our SCSS file into CSS let's add some style rules that are intentionally utilizing the SCSS syntax. We'll look to the Sass Basics guide for some code snippets.

// some variables
$font-stack: Helvetica, sans-serif;
$primary-color: #333;

body {
  font: 100% $font-stack;
  color: $primary-color;
}

// some nesting
nav {
  ul {
    margin: 0;
    padding: 0;
    list-style: none;
  }

  li {
    display: inline-block;
  }

  a {
    display: block;
    padding: 6px 12px;
    text-decoration: none;
  }
}

// a mixin
@mixin transform($property) {
  -webkit-transform: $property;
  -ms-transform: $property;
  transform: $property;
}

.box {
  @include transform(rotate(30deg));
}

Now that we have an SCSS file to process, the next step involves configuring the package.json to install the necessary dependencies and provide a way to compile SCSS with Node.js by adding custom scripts.

package.json

Using the scripts section of an npm package.json file we can execute a series of commands to carry out the compilation of SCSS and optimize the resulting CSS output. A package.json file is required, and can be created running the command npm init in the project folder and following the prompts, or copying below.

{
  "name": "npmcompilesass",
  "scripts": {}
}

Next we'll need to add two packages into the devDependencies of our project. To do so run the following command npm install node-sass clean-css-cli --save-dev. What will occur is that the node-sass and clean-css npm packages will be installed to the devDependencies of the project. You should also see a node modules folder appear in the root of the project, and there should also be a package-lock.json file that was created.

Your package.json file should look like this:

{
  "name": "npmcompilesass",
  "scripts": {},
  "devDependencies": {
    "clean-css-cli": "^4.3.0",
    "node-sass": "^4.12.0"
  }
}

If for some reason your file looks different, you can copy the above and run the command npm install. This will reinstall both packages.

Compile Sass to CSS using node-sass

With the dependencies available we can add a script to compile the SCSS file created earlier with the node-sass npm package.

{
  "name": "npmcompilesass",
  "scripts": {
    "compile-styles": "node-sass --output-style expanded --source-map true --source-map-contents true --precision 6 styles.scss dist/styles.css"
  },
  "devDependencies": {
    "clean-css-cli": "^4.3.0",
    "node-sass": "^4.12.0"
  }
}

Unfortunately, multi-line npm scripts are not supported so the script is quite long, and there are quite a few parameters passed. Luckily the node-sass command line documentation can provided detailed info on all of the possible parameters that are supported.

In this case parameters are used to indicate source maps should be generated (for debugging purposes), the amount of decimal precision is capped at 6, and the scss source file to process is styles.scss, which will be processed and output to a file named styles.css in a new folder named dist, located in the root of the project. The name of the dist folder can be changed if needed, and it will be created when the script runs if it does not already exist.

At this point we can run the compile styles script by running the command npm run compile-styles. However, this is only running node-sass and isn't minifying the css output, so we need to add another script to carry out the css optimization.

Minify CSS with clean-css

Like the node-sass package, we installed the clean-css package in the first step. To utilize it we'll add an additional script to the package.json file.

{
  "name": "npmcompilesass",
  "scripts": {
    "compile-styles": "node-sass --output-style expanded --source-map true --source-map-contents true --precision 6 styles.scss dist/styles.css",
    "css-minify": "cleancss --level 1 --format breaksWith=lf --source-map --source-map-inline-sources --output dist/styles.min.css dist/styles.css"
  },
  "devDependencies": {
    "clean-css-cli": "^4.3.0",
    "node-sass": "^4.12.0"
  }
}

Similar to the compile-styles script, the css-minify script is also quite long and contains many parameters. More info on all the parameters can be found at the clean-css-cli GitHub repo. The parameters being passed in indicate to run clean-css with a certain level of optimization, line break formatting, and to include source maps with the optimized output. The file to optimize is the styles.css file located in the dist folder that was generated by the compile-styles command, and the optimized output will be written to styles.min.css in the same folder. Now that all the required scripts have been added to the package.json file we can first compile, and then minify the scss source, by running the command npm run compile-styles followed by the command npm run css-minify. Then looking in the dist folder that was created there should be four files:

• styles.css
• styles.css.map
• styles.min.css
• styles.min.css.map

The two files we are most interested in are styles.css and styles.min.css. These are the browser compatible style sheets that can now be linked in any HTML file.

CSS

To make sure everything worked correctly your styles.css file should look like this:

body {
  font: 100% Helvetica, sans-serif;
  color: #333;
}

nav ul {
  margin: 0;
  padding: 0;
  list-style: none;
}

nav li {
  display: inline-block;
}

nav a {
  display: block;
  padding: 6px 12px;
  text-decoration: none;
}

.box {
  -webkit-transform: rotate(30deg);
  -ms-transform: rotate(30deg);
  transform: rotate(30deg);
}

/*# sourceMappingURL=styles.css.map */

You can also verify the styles.min.css file because it should have identical content with all of the whitespace removed. Also take note that a comment is included at the bottom for the source map file. This can be left as is and allows for seeing the style rules in the original SCSS file from the browser's dev tools.

Run npm Scripts Sequentially

With the output being generated correctly, there is one additional step we can do to simplify the SCSS processing into one command. Looking back to the scripts section of the package.json file, let's add one more script to combine the other two.

{
  "name": "npmcompilesass",
  "scripts": {
    "compile-styles": "node-sass --output-style expanded --source-map true --source-map-contents true --precision 6 styles.scss dist/styles.css",
    "css-minify": "cleancss --level 1 --format breaksWith=lf --source-map --source-map-inline-sources --output dist/styles.min.css dist/styles.css",
    "process-styles": "npm run compile-styles && npm run css-minify"
  },
  "devDependencies": {
    "clean-css-cli": "^4.3.0",
    "node-sass": "^4.12.0"
  }
}

Now by running the command npm run process-styles, the compile-styles and css-minify scripts will run in series, and it is no longer necessary to execute both scripts separately. The process-styles script is responsible for both compiling the SCSS into css output and minifying it for optimal use.

TypeScript

In a new folder, create a file named script.ts. Then, add some sample code so we can test whether the JavaScript output is being generated properly.

const msg: string = "Hello World!";
console.log(msg);

TypeScript Compiler

In the same folder, create a new file named tsconfig.json. Here is the TypeScript official documentation for configuring tsconfig.json.

Your tsconfig.json file should look like this:

{
  "compilerOptions": {
    "outDir": "output"
  },
  "include": ["./*"],
  "exclude": ["node_modules"]
}

This configuration tells the TypeScript compiler to look for source files in the root of your project, where your tsconfig.json is located. For any TypeScript files it finds there, it will output the compiled JavaScript to a new folder named output.

package.json

In the same folder create a package.json file. Here is the npm official documentation on creating a package.json file.

Then, add the name and version properties required. You will also need to add a property called scripts. This property contains the script instructions that we will use to compile the TypeScript we created. In this case, our compilation script is named compile-typescript, and it runs the command tsc. This is the default TypeScript command, and it will utilize the tsconfig.json we created.

Your package.json file should look like this:

{
  "name": "package-name-goes-here",
  "version": "0.0.0",
  "scripts": {
    "compile-typescript": "tsc"
  }
}

Now that package.json is created and the TypeScript compilation step is listed, we must save TypeScript as a dev dependency. This will give the npm task access.

npm Install TypeScript

To install TypeScript for this project in a terminal window, run the command: npm install typescript --save-dev

After installing TypeScript, your package.json should look like this:

{
  "name": "package-name-goes-here",
  "version": "0.0.0",
  "scripts": {
    "compile-typescript": "tsc"
  },
  "devDependencies": {
    "typescript": "^3.5.3"
  }
}

JavaScript

In a terminal window, navigate to the source code folder you created. Then, run the following command: npm run compile-typescript

Now, you should now see a new folder created named output, that contains one file named script.js. Notice how the output has defaulted to ES5 JavaScript, which is compatible with all major browsers.

Your script.js file should look like this:

var msg = "Hello World";
console.log("msg");

Run Node.js Script

The script.js created as a result of running the "compile-typescript" command can now be run with Node.js. To do this another package.json script is added, which is named "start". The "start" script will run the node cli command which the path of the script.ts file is passed.

{
  "name": "package-name-goes-here",
  "version": "0.0.0",
  "scripts": {
    "compile-typescript": "tsc",
    "start": "node ./output/script.js"
  },
  "devDependencies": {
    "typescript": "^3.5.3"
  }
}

Run the start command by entering npm run start in a terminal window, and you should see the output "Hello World!" printed to the console.

Run npm Scripts Sequentially

To save time the "compile-typescript" and "start" commands can be combined into one command by modifying the start command to include this functionality.

{
  "name": "package-name-goes-here",
  "version": "0.0.0",
  "scripts": {
    "compile-typescript": "tsc",
    "start": "npm run compile-typescript && node ./output/script.js"
  },
  "devDependencies": {
    "typescript": "^3.5.3"
  }
}

Now running the command npm run start will first run the "compile-typescript" command and then use node to run the script.js file that is output.

Before following the steps below make sure to have Node.js and npm installed.

HTML

To demonstrate the features provided by the html-minifier package we will start out with a sample html file. We can name this file index.html, and save it to a folder called "src". The name of the file and containing folder will be needed in the following steps. For this example, the sample file contains different types of elements to highlight the effect of minification, especially in regard to how white space is maintained when using preformatted elements.

<h1>This is our sample html content</h1>

<p>Here is some paragraph text.</p>

<pre>This text is preformatted.

There is more than one line in this text block.

    <code>console.log("code block inside preformatted block.");</code>
</pre>

<div>some more text at the bottom</div>

Note: A more common scenario than starting with a sample file might be applying the minification step to the output of a build process. If you are interested in seeing an example of how to generate HTML output, here is some info on how to render EJS files with Node.js. The steps in that article can be extended to create a static site generator, and the html-minifier package can be included and used as part of the build process.

package.json

Next we will want to set up the package.json file so that we can npm install the html-minifier package. If one is not already created, running the command npm init and following the prompts will create one. Once the package.json file is in place we can run the command npm install html-minifier --save-dev to install the html-minifier npm package.

Your package.json file should look similar to this:

{
  "name": "your-package-name-here",
  "version": "1.0.0",
  "devDependencies": {
    "html-minifier": "^4.0.0"
  }
}

There may be some additional properties created if using the npm init command, and the default values can be left in place. If you did not use the npm init command you can copy the contents above and run the command npm install, which will install all the required dependencies.

Now that the html-minifier package is installed we need a way to utilize it from the command line. To do so, an npm scripts property can be added to the package.json file just created. We will need to add one script that will pass option arguments to the html-minifier package command line interface, and we can name this script "html-minify".

Here is what the package.json file should look like with the script added:

{
  "name": "your-package-name-here",
  "version": "1.0.0",
  "scripts": {
    "html-minify": "html-minifier --input-dir src --output-dir dist --file-ext html --remove-comments --collapse-whitespace --minify-js true --minify-css true"
  },
  "devDependencies": {
    "html-minifier": "^4.0.0"
  }
}

Let's look at each of the options being passed in to the html-minifier cli, and see what each is specifying.

html-minifier options

The first option --input-dir is specifying the folder that our source html file is located. In this case the folder name is "src", which was created during the initial step. Following that, --output-dir is specifying the output directory where the minified html file will be added. In this case it is set to "dist", although this can be changed to any folder name.

The --file-ext option is set to html (in this example it is not needed), however if the input directory contains file types other than "html", errors may occur as a result of the attempted minification of those files. In the html-minifier github repository there is open issue to support multiple file extensions. A possible workaround for the time being is to add multiple package.json scripts, with each one running a separate command for each of the individual file types that will be minified. Additionally there are many other minifier packages available on npm and one of those may be better suited for file types other than html.

The next two options: --remove-comments and --collapse-whitespace do exactly as they are named, and there is no value to pass to them. If comments need to be retained or white space non-collapsed, these options can be deleted and html-minifier will not alter these properties of the original file.

Depending on whether set to true or false (or not included as the default value is false), the last two options, --minify-js and --minify-css will minify the corresponding source of their type, only if included as inline style or script tags in the html being minified. It may also be good to know that the html-minifier options information states that clean-css and uglify-js are used for the minification when these options are included.

To get a full list of all the options supported, it can be helpful to globally install the html-minifier package by running the command npm install html-minifier -g (this may require administrator access). With the package installed globally, running the command html-minifier --help will list all of the command line options, their value if applicable, and a short help text description.

Minify HTML

Now that the html-minify script is added and the options are configured, to use it run the command npm run html-minify. As a result a new folder called "dist" should have been created where the src folder is located. Within that folder should be the minified version of the index.html file initially created.

Here is what the minified html file should look like:

<h1>This is our sample html content</h1><p>Here is some paragraph text.</p><pre>This text is preformatted.

There is more than one line in this text block.

    <code>console.log("code block inside preformatted block.");</code>
</pre><div>some more text at the bottom</div>

Notice that the whitespace within the preformatted element is maintained. This is important as those sections need to keep their whitespace as originally formatted, so the html-minifier does not change the desired formatting. For other elements not intended to maintain whitespace they can be reduced to a single line, and the comment at the top has been removed as well. There is no inline Javascript of CSS in this example, but you can add some in and see the effect.

Using the html-minifier package can be helpful to reduce file size and increase performance for users, especially when using a slower internet connection. Even with the small file used for this example, the response size has decreased from 303 bytes to 275 bytes. This is a small amount, but the savings can add up in a typical scenario where file sizes are much larger.

There is also a web based html minifier made by the same package author.

Before getting started it may be helpful to checkout how to render EJS files with Node.js. The code below assumes a static build process and that the source is tracked in a git repository, as well as having Node.js and npm installed.

Setup TypeScript

If you're interested in more information about setting up Typescript, checkout this post that shows how to compile TypeScript with npm. There you can see how to create a package.json file and add a tsconfig.json to configure the TypeScript compiler.

Since we are using Node.js with TypeScript there are some modifications needed to the tsconfig.json file from the previous post.

Here is what your tsconfig.json file should look like, in order for the code that follows to work correctly.

{
  "compilerOptions": {
    "outDir": "./output",
    "module": "commonjs",
    "target": "ES6"
  },
  "include": ["/*"],
  "exclude": []
}

This configuration instructs the TypeScript compiler to use commonjs modules and output code that targets the ES6 specification, which is needed since an async function will be needed to utilize the npm package we'll use to gather git file metadata.

npm install Simple Git

Next, the Simple Git npm package is needed so that it can be used to access the git metadata. Run the command npm install simple-git --save-dev in the terminal, and that will install the Simple Git package to the node_modules folder.

At this point the package.json file should look similar to this (the package versions might be slightly different):

{
  "name": "package-name-goes-here",
  "version": "0.0.0",
  "scripts": {
    "compile-typescript": "tsc"
  },
  "devDependencies": {
    "simple-git": "^1.129.0",
    "typescript": "^3.7.5"
  }
}

Note: Since we are using TypeScript for this example, usually a type definition package is also required to be "npm installed" just like the actual package. In this case the Simple Git package includes the @types type declarations, so downloading a separate package is not needed.

Use Simple Git with TypeScript

With TypeScript and the npm package.json configured we can now create a TypeScript file, let's call it index.ts. This will contain the code that will access the git metadata of our post file. To get started the Simple Git npm package will be imported, and an async function will be needed to utilize the Simple Git package, immediately following the async build function is called so the result can be output.

// index.ts
import * as simpleGit from "simple-git/promise";

async function build() {
  const git = simpleGit();
}

build();

Since we are using TypeScript the import declaration might look slightly different than expected. This approach is consistent with the Simple Git documentation. Additionally, make sure to import the promise and async compatible version simple-git/promise. Inside of the build function Simple Git is initialized, and the functions provided by the Simple Git API are ready for use. Simple Git may not provide all of the git functionality available from the command line, but it works well for more common usages. We can add some code that will retrieve the created date of a file (based on the first commit) and the last modified date (based on latest commit).

// index.ts
import * as simpleGit from "simple-git/promise";

async function build() {
  const git = simpleGit();

  //list commits
  // git log accepts an options object - from ts definition
  /*
    format?: T;
    file?: string;
    from?: string;
    multiLine?: boolean;
    symmetric?: boolean;
    to?: string;
  */
  const log = await git.log({ file: `sample-post-page.html` });

  // get first commit date of file
  const createdDate = new Date(log.all.slice(-1)[0].date);

  // get latest modified date of file
  const modifiedDate = new Date(log.latest.date);

  // output formatted time stamps
  console.log(createdDate.toLocaleDateString());
  console.log(modifiedDate.toLocaleDateString());
}

build();

Instead of just outputting this git metadata to the console it can be assigned to a variable and then included in rendered content output for the reader to view.

The Simple Git API provides a lot of other examples for the functionality it provides. In this example the focus was on how we can gather the created and last modified dates of a file representing post content that is included in a static build process like one might find in use for a site built with the Jamstack.

SendGrid API Key

If you are having trouble creating an API key, please view the API Keys documentation provided by SendGrid. With the API key obtained, we can begin writing code that will utilize the free SendGrid service. You should not "hardcode" your API key into your application code. A more secure way of granting the application access to your account API key is to store it as an environment variable.

Azure Serverless Function

In order to send the email we can use a serverless function, for this example we will use a JavaScript Azure Function. The serverless function will accept an HTTP post request and trigger the sending of an email to the address provided in the form submission. In order to do this we will need an HTML form. Before using the following code, it's a good idea to check out this other post on Submitting Form Data with the Fetch API.

Once the client side code is setup to post a form with the email address and email message, we can set up the serverless function to handle sending the email with the information from the form submission.

import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import * as querystring from "querystring";

const httpTrigger: AzureFunction = async function (
  context: Context,
  req: HttpRequest
): Promise<void> {
  context.log("HTTP trigger function processed a request.");

  // form data submitted as query string in request body
  const body = querystring.parse(req.body);
};

export default httpTrigger;

This is the base code needed for the Azure serverless function. Below we will look to see how the data contained in the request body will be used to generate the email.

Send Email with SendGrid and Node.js

Note: This code is setup as if the request body contains two keys, one for emailAddress and one for emailMessage. Additionally, the SendGrid API key obtained earlier is accessed here from an environment variable. See the Azure documentation to add an application setting. Application settings are accessed as environment variables in the serverless function code.

import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import * as querystring from "querystring";
import * as SendGrid from "@sendgrid/mail";
SendGrid.setApiKey(process.env["SendGridApiKey"] as string);

const httpTrigger: AzureFunction = async function (
  context: Context,
  req: HttpRequest
): Promise<void> {
  context.log("HTTP trigger function processed a request.");

  // form data submitted as query string in request body
  const body = querystring.parse(req.body);

  // check to make sure form was submitted with field data entered
  if (body && body.emailAddress && body.emailMessage) {
    // create an email options object
    const email = {
      to: process.env["SendGridApiKey"],
      from: "noreply@yourdomain.com",
      subject: "Hello! This email was sent with Node.js",
      html: `<div>This email is from: ${body.emailAddress}</div>
      <div>message: ${body.emailMessage}</div>`,
    };

    try {
      await SendGrid.send(email);

      context.res!.status = 200;
      context.res!.body = {
        message: "Email successful! Check email for the message.",
      };
    } catch (error) {
      context.res!.status = 400;
      context.res!.body = {
        message: "An error occurred.",
      };
    }
  } else {
    context.res!.status = 400;
    context.res!.body = {
      message: "Form submission is invalid. Please try again.",
    };
  }
};

export default httpTrigger;

Prior to the function code the SendGrid Mail Service package is imported. Immediately following the setApiKey method is called and the environment variable, stored as an application setting, is passed in. The SendGrid package is now initialized and ready to be used in the code that sends the email. The api key is typecast as a string here, because in this example TypeScript has been selected as the language. If you are not using TypeScript this typecast should be removed as it is not needed.

The serverless function code first checks to make sure the form was submit with the field data entered. If the form submission was valid and an email options object is created with the form data. ES6 template literals are used here to, instead of standard string concatenation, build the email message that is saved as the email object html key. It is called html because SendGrid permits the sending of html emails. The result of using ES6 template literals are a concise and readable block of code can be easily adjusted in the future if needed.

The email object is then passed to the SendGrid send method provided by the SendGrid Mail Service npm package. Notice, that since this is an async method it must be awaited before allowing the code execution to proceed. The send method call is also wrapped in a try catch block. This way if the email service fails the serverless function will return a server error notifying the client.

Using Sendgrid makes it even easier to manage and prevents potential issues with spam filters. This approach can be useful if we are building a site with the Jamstack, since a server is not required. Additionally, if the email usage limits are within the free plan of SendGrid, the cost savings can be quite substantial. It is also worth noting that when using Azure Serverless Functions, we can use the same Azure account to create and link to a Sendgrid account, which includes tens of thousands of free emails per month. To find it search for SendGrid in the Azure Portal dashboard, and follow the setup instructions from there.

Before proceeding make sure to have node.js and npm installed.

Run npm init

There are some npm packages that will be used to create the rss feed, so first run the command npm init, which will create a package.json file that we can add dependencies to. After creating the package.json these are the npm packages that we will add:

• fs-extra
• xml
• cheerio
• typescript

To install these run the command npm install fs-extra cheerio xml typescript --save, and since we are using TypeScript for this example we need the corresponding type definitions. To install the type definitions run the command: npm install @types/xml @types/cheerio @types/fs-extra --save-dev.

There is one extra field that needs to be added to the package.json file and that is the type field. This permits the use of ECMAScript modules, rather than CommonJS modules.

Your package.json should look similar to this:

{
  "type": "module",
  "name": "xmlrssfeed",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "cheerio": "^1.0.0-rc.5",
    "fs-extra": "^9.0.1",
    "typescript": "^4.1.3",
    "xml": "^1.0.1"
  },
  "devDependencies": {
    "@types/cheerio": "^0.22.23",
    "@types/fs-extra": "^9.0.6",
    "@types/xml": "^1.0.5"
  }
}

Configure tsconfig.json

Typescript is used in this example so tsconfig.json file is also required. You can read more about the tsconfig.json settings in the TypeScript documentation. For our case, create a file named tsconfig.json and copy the code below into it.

{
  "compilerOptions": {
    "allowSyntheticDefaultImports": true,
    "isolatedModules": true,
    "strict": true,
    "module": "esnext",
    "lib": ["ES2019"],
    "moduleResolution": "node",
    "skipLibCheck": true
  },
  "include": ["*.ts"],
  "exclude": ["node_modules/**/*"]
}

The module field is set to "esnext" to match the addition of the "type" field in the package.json. This setting instructs the TypeScript compiler to generate es modules, and allows us to use import in the TypeScript code.

npm package.json script

After configuring TypeScript, we need a way to transpile and then execute the generated JavaScript with node.js. To do this, an npm package.json script can be added to carry out both steps. In the package.json file, add a new scripts property "createRssFeed", so that it looks like this:

{
  "type": "module",
  "name": "xmlrssfeed",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "createRssFeed": "tsc && node index.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "cheerio": "^1.0.0-rc.5",
    "fs-extra": "^9.0.1",
    "typescript": "^4.1.3",
    "xml": "^1.0.1"
  },
  "devDependencies": {
    "@types/cheerio": "^0.22.23",
    "@types/fs-extra": "^9.0.6",
    "@types/xml": "^1.0.5"
  }
}

The createRssFeed script will sequentially compile the TypeScript source file (index.ts) and then use node to execute the JavaScript output. If you try running the command npm run createRssFeed you will get an error, because the index.ts doesn't exist yet. Let's create that now.

Node.js Script

In the same folder as the package.json file create a new file named index.ts, and add the code below to make sure the setup is working.

import fs from "fs-extra";
import xml from "xml";
import cheerio from "cheerio";

(async function createRssFeed() {
  console.log("creating feed");
})();

Then run the createRssFeed command npm run createRssFeed and the output should print to the console the text "creating feed".

Generate RSS Feed

With the setup working we can now begin to use the npm packages that we imported. The xml package accepts a feed object as it's configuration so we can add that to the createRssFeed function. The feedObject will be processed into an xml string and then the fs-extra package will be used to write the output to a file named feed.rss.

import fs from "fs-extra";
import xml from "xml";
import cheerio from "cheerio";

(async function createRssFeed() {
  console.log("creating feed");
  const feedObject = {
    rss: [
      {
        _attr: {
          version: "2.0",
          "xmlns:atom": "http://www.w3.org/2005/Atom",
        },
      },
      {
        channel: [
          {
            "atom:link": {
              _attr: {
                href: "YOUR-WEBSITE/feed.rss",
                rel: "self",
                type: "application/rss+xml",
              },
            },
          },
          {
            title: "YOUR-WEBSITE-TITLE",
          },
          {
            link: "YOUR-WEBSITE/",
          },
          { description: "YOUR-WEBSITE-DESCRIPTION" },
          { language: "en-US" },
          // todo: add the feed items here
        ],
      },
    ],
  };

  const feed = '<?xml version="1.0" encoding="UTF-8"?>' + xml(feedObject);

  await fs.writeFile("/feed.rss", feed, "utf8");
})();

Make sure to replace "YOUR-WEBSITE", "YOUR-WEBSITE-TITLE", and "YOUR-WEBSITE-DESCRIPTION" with the actual values from the website you are generating the RSS feed for.

At this point the createRssFeed npm package.json script should generate a new file named feed.rss in the project folder, although it will be an empty feed. So in the feed object we can replace the todo comment with code that will use some sample post data to generate the feed.

In this case we'll create an array of objects for our sample post data, but a more likely scenario is that they would be dynamically sourced from a content store, like markdown files or a content management system.

Add the sample posts below directly above the feedObject variable.

const posts = [
  {
    title: "Post One",
    date: "1/1/2020",
    slug: "post-one",
    content: "This is some content for post one.",
  },
  {
    title: "Post Two",
    date: "1/2/2020",
    slug: "post-two",
    content: "This is some content for post two.",
  },
  {
    title: "Post Three",
    date: "1/3/2020",
    slug: "post-three",
    content: "This is some content for post three.",
  },
  {
    title: "Post Four",
    date: "1/4/2020",
    slug: "post-four",
    content: "This is some content for post four.",
  },
];

Now that we some posts to include, replace the todo with this function call:

...(buildFeed(posts));

This will take the result of the buildFeed function (we will write this next), which will be an array and spread the results into the feedObject.

Now the index.ts file should look like this:

import fs from "fs-extra";
import xml from "xml";
import cheerio from "cheerio";

(async function createRssFeed() {
  console.log("creating feed");
  const posts = [
    {
      title: "Post One",
      date: "1/1/2020",
      slug: "post-one",
      content: "<p>This is some content for post one.</p>",
    },
    {
      title: "Post Two",
      date: "1/2/2020",
      slug: "post-two",
      content: "<p>This is some content for post two.</p>",
    },
    {
      title: "Post Three",
      date: "1/3/2020",
      slug: "post-three",
      content:
        "<p>This is some content for post three. This is a relative <a href='/relative-link/'>link</a></p>",
    },
    {
      title: "Post Four",
      date: "1/4/2020",
      slug: "post-four",
      content: "<p>This is some content for post four.</p>",
    },
  ];

  const feedObject = {
    rss: [
      {
        _attr: {
          version: "2.0",
          "xmlns:atom": "http://www.w3.org/2005/Atom",
        },
      },
      {
        channel: [
          {
            "atom:link": {
              _attr: {
                href: "YOUR-WEBSITE/feed.rss",
                rel: "self",
                type: "application/rss+xml",
              },
            },
          },
          {
            title: "YOUR-WEBSITE-TITLE",
          },
          {
            link: "YOUR-WEBSITE/",
          },
          { description: "YOUR-WEBSITE-DESCRIPTION" },
          { language: "en-US" },
          ...buildFeed(posts),
        ],
      },
    ],
  };

  const feed = '<?xml version="1.0" encoding="UTF-8"?>' + xml(feedObject);

  await fs.writeFile("./feed.rss", feed);
})();

The feedObject now includes the buildFeed function, which can be added below the createRssFeed function. As the name suggests this is where the feed items will be created and sorted by most recent date. Additionally the cheerio npm package will be used here.

function buildFeed(
  posts: { title: string; date: string; slug: string; content: string }[]
) {
  const sortedPosts = posts.sort(function (first, second) {
    return new Date(second.date).getTime() - new Date(first.date).getTime();
  });

  const feedItems = [];

  feedItems.push(
    ...sortedPosts.map(function (post) {
      const feedItem = {
        item: [
          { title: post.title },
          {
            pubDate: new Date(post.date as string).toUTCString(),
          },
          {
            guid: [
              { _attr: { isPermaLink: true } },
              `YOUR-WEBSITE/${post.slug}/`,
            ],
          },
          {
            description: {
              _cdata: post.content,
            },
          },
        ],
      };
      return feedItem;
    })
  );

  return feedItems;
}

This code can now generate the RSS feed by re-running the command npm run createRssFeed, however any relative links in the post content will not link to the correct website, since RSS feeds require absolute links. We can convert them to absolute links using the cheerio npm package.

Convert relative links to absolute links

Directly above the feed object add the following code:

const $ = cheerio.load(post.content as string, {
  decodeEntities: false,
});

// replace relative links with absolute
$("a[href^='/'], img[src^='/']").each(function (this: cheerio.Element) {
  const $this = $(this);
  if ($this.attr("href")) {
    $this.attr("href", `YOUR-WEBSITE/${$this.attr("href")}`);
  }
  if ($this.attr("src")) {
    $this.attr("src", `YOUR-WEBSITE/${$this.attr("src")}`);
  }
});

const postContent = $("body").html() as string;

Here is some more info on this technique to convert relative urls to absolute urls. Make sure to also replace the description property of the feedItem with the postContent variable. The buildFeed function should now look like this:

function buildFeed(
  posts: { title: string; date: string; slug: string; content: string }[]
) {
  const sortedPosts = posts.sort(function (first, second) {
    return new Date(second.date).getTime() - new Date(first.date).getTime();
  });

  const feedItems = [];

  feedItems.push(
    ...sortedPosts.map(function (post) {
      const $ = cheerio.load(post.content as string, {
        decodeEntities: false,
      });

      // replace relative links with absolute
      $("a[href^='/'], img[src^='/']").each(function (this: cheerio.Element) {
        const $this = $(this);
        if ($this.attr("href")) {
          $this.attr("href", `YOUR-WEBSITE/${$this.attr("href")}`);
        }
        if ($this.attr("src")) {
          $this.attr("src", `YOUR-WEBSITE/${$this.attr("src")}`);
        }
      });

      const postContent = $("body").html() as string;

      const feedItem = {
        item: [
          { title: post.title },
          {
            pubDate: new Date(post.date as string).toUTCString(),
          },
          {
            guid: [
              { _attr: { isPermaLink: true } },
              `YOUR-WEBSITE/${post.slug}/`,
            ],
          },
          {
            description: {
              _cdata: postContent,
            },
          },
        ],
      };

      return feedItem;
    })
  );

  return feedItems;
}

The buildFeed function, first sorts all the posts by most recent date and then maps over the sorted posts to assign post data properties to the corresponding xml fields in the RSS feed. For each of the posts the content is modified, by using the cheerio npm package, to convert all the relative links to absolute links. That way when the RSS feed is shared the in-article links will link back to the correct website. As in the sections above make sure to replace "YOUR-WEBSITE" with the actual domain of your website. Additionally the date is formatted to RFC 822 format, in order to match the RSS specification.

Re-run the command npm run createRssFeed, and the feed.rss file that is generated should reflect the changes we made. You can verify that this file is a valid rss feed by checking it with the w3c Feed Validation Service.

To permit auto discovery of the RSS feed make sure to include the following html in the head tag of your website.

<link
  rel="alternate"
  type="application/rss+xml"
  title="RSS 2.0"
  href="/feed.rss"
/>