Go in Big Data

Sharing experience in Big Data, Golang and DevOps with examples.

Unit Testing Golang Code Calling Elasticsearch

There is an amazing Elasticsearch client for Go and in Working With Elasticsearch I demonstrated with example how to index and search for documents using it. However, if you want to ensure your code will continue working correctly and will not be inadvertently broken during refactoring, you have to cover the code with tests.

In this post I'll show how to unit test Go code interoperating with Elasticsearch. Keep in mind, however, that the same approach is applicable to unit testing almost any code calling external RESTful API.

A service calling Elasticsearch

Suppose you have a logging service which can

Read more »

How To Correctly Serialize JSON String In Golang

json is one of the most wildly used Go packages. It is simple and, what is more important, very intuitive. So what could be easier than marshalling a string with JSON and unmarshalling it to struct? If you believe (as I did) that the issue is trivial and json.Marshal does the job, read on.

What's wrong with json.Marshal?

It's easier to demonstrate on example. Let's write a simple program which serializes JSON string to bytes and deserializes the bytes into matching struct:

package main

import (  

type Person struct {  
    FirstName string `json:"firstName"`
    LastName  string `json:
Read more »

Collecting Logs In Elasticsearch With Filebeat and Logstash

You are lucky if you've never been involved into confrontation between devops and developers in your career on any side. In this post I'll show a solution to an issue which is often under dispute - access to application logs in production.

The issue at hand

Imagine you are a devops responsible for running company applications in production. Applications are supported by developers who obviously don't have access to production environment and, therefore, to production logs.

Imagine that each server runs multiple applications, and applications store logs in /var/log/apps. A server with two running applications will have log

Read more »

Working With Elasticsearch in Go

Elasticsearch is one of the most popular highly scalable full-text search engines on the market. It is based on Lucene engine and allows you to store, search, and analyze big volumes of data quickly and in near real time. It has rich REST API and clients for most popular programming languages including Go. In this post I'll demonstrate on example how to index and search for application logs using Go.

Installing Elasticsearch

If you haven't set it up already, installation of Elasticsearch is as easy as downloading it (from here) and running the executable file. I installed it on localhost

Read more »

Example Of Using Templates In Golang

Almost every programming language has a library implementing templating. In epoch of server side MVC dominance, templating was so important that it could determine language success or failure. Nowadays, however, when single page applications get momentum, templates are used only occasionally.

The niche where templates are still absolutely indispensable is email content generation. In this post I will demonstrate on an example how to use Golang template package to generate email content based on a template defined in a file. I'll keep example as simple as possible, but beefy enough to show most commonly used features of the package.


Read more »