Categories

Hugo

How to use Hugo to generate the exact format needed to submit the file to the _bulk endpoint of Elasticsearch.
Last updated
July 7, 2023

Hugo is a static site generator written in Go. It is conceptually similar to Jekyll, albeit with far more speed and flexibility. Hugo also supports generating output formats other than HTML, which allows users to pipe content directly into an Elasticsearch cluster.

In this guide, we are going to use this feature to tell Hugo to generate the exact format needed to submit the file to the _bulk endpoint of Elasticsearch.

First Steps

In order to make use of this documentation, you will need Hugo installed and configured on your system

  1. Make Sure You Have Hugo Installed. This guide assumes you already have Hugo installed and and configured on your system. Visit the Hugo Documentation to get started.
  2. Spin Up a Bonsai Elasticsearch Cluster. This guide will use a Bonsai cluster as the Elasticsearch backend.
  3. Create an Index on the Cluster. In this example, we’re going to push data into an index called <span class="inline-code"><pre><code>hugo</code></pre></span>. This index needs to be created before any data can be stored on it. The index can be created either through the Interactive Console, or with a tool like <span class="inline-code"><pre><code>curl</code></pre></span>:

Use the URL for your cluster. A Bonsai URL looks something like this:

<div class="code-snippet w-richtext"><pre><code fs-codehighlight-element="code" class="hljs language-javascript">curl -XPUT https://user123:pass456@my-awesome-cluster-1234.us-east-1.bonsai.io/hugo</code></pre></div>

Configure Hugo to Output to Bonsai Elasticsearch

Hugo’s configuration settings live in a file called <span class="inline-code"><pre><code>config.toml</code></pre></span> by default. This file may also have a <span class="inline-code"><pre><code>.json</code></pre></span>. or <span class="inline-code"><pre><code>.yaml</code></pre></span>/<span class="inline-code"><pre><code>yml</code></pre></span>yml extension. Add the following snippet based on your config file format:

TOML:

<div class="code-snippet-container"><a fs-copyclip-element="click-1" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-1" class="hljs language-javascript">[outputs]
home = ["HTML", "RSS", "Bonsai"]

[outputFormats.Bonsai]
baseName = "bonsai"
isPlainText = true
mediaType = "application/json"
notAlternative = true

[params.bonsai]
vars = ["title", "summary", "date", "publishdate", "expirydate", "permalink"]
params = ["categories", "tags"]</code></pre></div></div>

JSON:

<div class="code-snippet-container"><a fs-copyclip-element="click-2" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-2" class="hljs language-javascript">

{
 "outputs": {
   "home": [
     "HTML",
     "RSS",
     "Bonsai"
   ]  
},
 "outputFormats": {
   "Bonsai": {
     "baseName": "bonsai",
     "isPlainText": true,
     "mediaType": "application/json",
     "notAlternative": true
   }
 },
 "params": {
   "bonsai": {
     "vars": [
       "title",
       "summary",
       "date",
       "publishdate",
       "expirydate",
       "permalink"
     ],
     "params": [
       "categories",
       "tags"
     ]
   }
 }
}</code></pre></div></div>


YAML:

<div class="code-snippet-container"><a fs-copyclip-element="click-3" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-3" class="hljs language-javascript"> outputs:
 home:
   - HTML
   - RSS
   - Bonsai
outputFormats:
 Bonsai:
   baseName: bonsai
   isPlainText: true
   mediaType: application/json
   notAlternative: true
params:
 bonsai:
   vars:
     - title
     - summary
     - date
     - publishdate
     - expirydate
     - permalink
   params:
     - categories
     - tags</code></pre></div></div>

This snippet defines a new output called “Bonsai”, and specifies some associated variables.

Creating the JSON template

Hugo needs to have a template for rendering data in a way that Elasticsearch will understand. To do this, we will define a JSON template that conforms to the Elasticsearch Bulk API.

Create a template called <span class="inline-code"><pre><code>layouts/_default/list.bonsai.json</code></pre></span> and give it the following content:

<div class="code-snippet-container"><a fs-copyclip-element="click-7" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-7" class="hljs language-javascript">{{/* Generates a valid Elasticsearch _bulk index payload */}}
{{- $section := $.Site.GetPage "section" .Section }}
{{- range .Site.AllPages -}}
 {{- if or (and (.IsDescendant $section) (and (not .Draft) (not .Params.private))) $section.IsHome -}}
   {{/* action / metadata */}}
   {{ (dict "index" (dict "_index" "hugo" "_type" "doc"  "_id" .UniqueID)) | jsonify }}
   {{ (dict "objectID" .UniqueID "date" .Date.UTC.Unix "description" .Description "dir" .Dir "expirydate" .ExpiryDate.UTC.Unix "fuzzywordcount" .FuzzyWordCount "keywords" .Keywords "kind" .Kind "lang" .Lang "lastmod" .Lastmod.UTC.Unix "permalink" .Permalink "publishdate" .PublishDate "readingtime" .ReadingTime "relpermalink" .RelPermalink "summary" .Summary "title" .Title "type" .Type "url" .URL "weight" .Weight "wordcount" .WordCount "section" .Section "tags" .Params.Tags "categories" .Params.Categories "authors" .Params.Authors) | jsonify }}
 {{- end -}}
{{- end }}</code></pre></div></div>

When the site is generated, this will result in creating a file called public/bonsai.json, which will have the content stored in a way that can be pushed directly into Elasticsearch using the Bulk API.

Push the Data Into Elasticsearch

To get the site’s data into Elasticsearch, render it by running <span class="inline-code"><pre><code>hugo</code></pre></span> on the command line. Then send it to your Bonsai cluster with <span class="inline-code"><pre><code>curl</code></pre></span>:

<div class="code-snippet-container"><a fs-copyclip-element="click-4" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-4" class="hljs language-javascript">curl -H "Content-Type: application/x-ndjson" -XPOST "https://user123:pass456@my-awesome-cluster-1234.us-east-1.bonsai.io/_bulk" --data-binary @public/bonsai.json</code></pre></div></div>

You should now be able to see your data in the Elasticsearch cluster:

<div class="code-snippet-container"><a fs-copyclip-element="click-5" href="#" class="btn w-button code-copy-button" title="Copy"><img class="copy-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/6483934eeefb356710a1d2e9_icon-copy.svg" loading="lazy" alt=""><img class="copied-image" src="https://global-uploads.webflow.com/63c81e4decde60c281417feb/64839e207c2860eb9e6aa572_icon-copied.svg" loading="lazy" alt=""></a><div class="code-snippet"><pre><code fs-codehighlight-element="code" fs-copyclip-element="copy-this-5" class="hljs language-javascript">$ curl -XGET "https://user123:pass456@my-awesome-cluster-1234.us-east-1.bonsai.io/_search"{"took":1,"timed_out":false,"_shards":{"total":2,"successful":2,"failed":0},"hits":{"total":1,"max_score":1.0,"hits":[{"_index":"hugo","_type":"doc","_id":...</code></pre></div></div>

View code snippet
Close code snippet