Oscar Forner

Unit test with HTTP client in Go

Thu, 04 Jun 2026 00:00:00 +0000

Introduction

In this post we will explore an example of how to test code with unit test that uses an HTTP client.

Code with HTTP Client

The following code is a typical example of a function that uses an HTTP client to do some requests.

package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"log"
	"net/http"
	"strings"
)

var (
	ErrFailedToCreateRequest    = errors.New("failed to create request")
	ErrFailedToPerformRequest   = errors.New("failed to perform request")
	ErrFailedToReadBody         = errors.New("failed to read body")
	ErrStatusCodeInvalid        = errors.New("invalid status code")
	ErrBodyDoesNotContainWololo = errors.New("body does not contain wololo")
)

func GetValuesAndProcessThem(ctx context.Context, client *http.Client) ([]byte, error) {
	req, err := http.NewRequestWithContext(ctx, "GET", "http://wololo:1234/", nil)
	if err != nil {
		return nil, ErrFailedToCreateRequest
	}

	res, err := client.Do(req)
	if err != nil {
		return nil, ErrFailedToPerformRequest
	}
	defer res.Body.Close()

	if res.StatusCode != 200 {
		return nil, ErrStatusCodeInvalid
	}

	// Do not read the whole body at once like here. This is just to keep it short.
	body, err := io.ReadAll(res.Body)
	if err != nil {
		return nil, ErrFailedToReadBody
	}

	if !strings.Contains(string(body), "wololo") {
		return nil, ErrBodyDoesNotContainWololo
	}

	return body, nil
}

Testing the code

As you probably know, the problem with testing this kind of code is having to use the package httptest for unit test. Or even spining a whole web server up for integration test.

Both options are valid, but sometimes you do not want to overkill it with either way mentioned before. In those cases, we can mock the RoundTripper for the transport attribute of the http.Client.

Generate mock for RoundTripper

You can use Uber’s mockgen to generate a mock for the RoundTripper with the following statement.

//go:generate go tool mockgen -destination mocks/http.go -package mock net/http RoundTripper

And you can construct the http.Client like the following example:

ctrl := gomock.NewController(t)
transport := mock.NewMockRoundTripper(ctrl)
client := &http.Client{Transport: transport}

Example of the test

package main_test

import (
	"bytes"
	"context"
	"errors"
	"io"
	"net/http"
	"testing"

	"github.com/stretchr/testify/require"
	"go.uber.org/mock/gomock"

	main "2026_go_http_unit_test"
	mock "2026_go_http_unit_test/mocks"
)

//go:generate go tool mockgen -destination mocks/http.go -package mock net/http RoundTripper
//go:generate go tool mockgen -destination mocks/io.go -package mock io Reader

func Test_getValuesAndProcessThem(t *testing.T) {
	tests := []struct {
		name      string
		ctx       context.Context
		mockSetup func(transport *mock.MockRoundTripper, reader *mock.MockReader)
		want      []byte
		wantErr   error
	}{
		{
			name: "success",
			ctx:  context.Background(),
			mockSetup: func(transport *mock.MockRoundTripper, _ *mock.MockReader) {
				transport.EXPECT().RoundTrip(gomock.Any()).Return(&http.Response{
					StatusCode: 200,
					Body:       io.NopCloser(bytes.NewBufferString("<h1>wololo</h1>")),
				}, nil)
			},
			want:    []byte("<h1>wololo</h1>"),
			wantErr: nil,
		},
		{
			name: "http request fails",
			ctx:  context.Background(),
			mockSetup: func(transport *mock.MockRoundTripper, _ *mock.MockReader) {
				transport.EXPECT().RoundTrip(gomock.Any()).Return(nil, errors.New("http request failed"))
			},
			want:    nil,
			wantErr: main.ErrFailedToPerformRequest,
		},
		{
			name: "http status code not 200",
			ctx:  context.Background(),
			mockSetup: func(transport *mock.MockRoundTripper, _ *mock.MockReader) {
				transport.EXPECT().RoundTrip(gomock.Any()).Return(&http.Response{
					StatusCode: 404,
					Body:       io.NopCloser(bytes.NewBufferString("<h1>Not found</h1>")),
				}, nil)
			},
			want:    nil,
			wantErr: main.ErrStatusCodeInvalid,
		},
		{
			name: "fails to read body",
			ctx:  context.Background(),
			mockSetup: func(transport *mock.MockRoundTripper, reader *mock.MockReader) {
				reader.EXPECT().Read(gomock.Any()).Return(0, errors.New("read body failed"))
				transport.EXPECT().RoundTrip(gomock.Any()).Return(&http.Response{
					StatusCode: 200,
					Body:       io.NopCloser(reader),
				}, nil)
			},
			want:    nil,
			wantErr: main.ErrFailedToReadBody,
		},
		{
			name: "body does not contain wololo",
			ctx:  context.Background(),
			mockSetup: func(transport *mock.MockRoundTripper, _ *mock.MockReader) {
				transport.EXPECT().RoundTrip(gomock.Any()).Return(&http.Response{
					StatusCode: 200,
					Body:       io.NopCloser(bytes.NewBufferString("<h1>try again</h1>")),
				}, nil)
			},
			want:    nil,
			wantErr: main.ErrBodyDoesNotContainWololo,
		},
		{
			name:      "ctx fails",
			ctx:       nil,
			mockSetup: func(transport *mock.MockRoundTripper, reader *mock.MockReader) {},
			want:      nil,
			wantErr:   main.ErrFailedToCreateRequest,
		},
	}
	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			t.Parallel()

			ctrl := gomock.NewController(t)
			defer ctrl.Finish()

			transport := mock.NewMockRoundTripper(ctrl)
			reader := mock.NewMockReader(ctrl)
			tt.mockSetup(transport, reader)

			got, err := main.GetValuesAndProcessThem(tt.ctx, &http.Client{Transport: transport})

			if tt.wantErr != nil {
				require.ErrorIs(t, err, tt.wantErr)
				require.Nil(t, got)
			} else {
				require.NoError(t, err)
				require.Equal(t, tt.want, got)
			}
		})
	}
}

Execution

$ go generate ./...
go: downloading go.uber.org/mock v0.6.0
go: downloading golang.org/x/tools v0.36.0
go: downloading golang.org/x/mod v0.27.0
go: downloading golang.org/x/sync v0.16.0

$ go test ./...
go: downloading github.com/stretchr/testify v1.9.0
ok  	2026_go_http_unit_test	0.004s
?   	2026_go_http_unit_test/mocks	[no test files]

Go embed for migrations

Tue, 10 Oct 2023 00:00:00 +0000

Introduction

In this post we will explore an example of a Go binary that combines features such as Go Embed and Go-Migrate to have a self-contained binary that runs the appropriate migrations before starting any actual work.

What is embed?

Go Embed was introduced in Go 1.16 as a core library that allows binaries to contain external files embedded and they are accessible from inside the binary.

There are multiple ways to benefit from this amazing feature, from packing your whole webapp with all its assets, to having binaries containing their DB migrations to always have the correct DB schema to interact with.

Note: I do not recommend to serve all webapp assets from the binary for webapps with high traffic load. Using Nginx, or a CDN would be a better way to do it.

Go-Migrate to run migrations

Go-Migrate is a library to handle all aspects of DB migrations in Go. It is both a CLI tool, and a library.

Go-Migrate CLI tool

Install

$ go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest
go: downloading github.com/golang-migrate/migrate v3.5.4+incompatible
go: downloading github.com/lib/pq v1.10.0

Create new migration

$ migrate create -ext sql -dir migrations -seq create_example_table
/home/maitesin/dev/blog/2023_go_embed_for_migrations/migrations/000001_create_example_table.up.sql
/home/maitesin/dev/blog/2023_go_embed_for_migrations/migrations/000001_create_example_table.down.sql

Go-Migrate with Go Embed

The following source code shows how Go-Migrate can use the embedded migrations in the binary.

package main

import (
	"database/sql"
	"embed"
	"fmt"

	"github.com/golang-migrate/migrate/v4"
	_ "github.com/golang-migrate/migrate/v4/database/postgres"
	"github.com/golang-migrate/migrate/v4/source/iofs"
	_ "github.com/lib/pq"
)

const dbURL = "postgres://postgres:postgres@localhost:54321/examples?sslmode=disable"

//go:embed migrations/*.sql
var migrationsFS embed.FS

func main() {
	dbConn, err := sql.Open("postgres", dbURL)
	if err != nil {
		fmt.Println(err)
		return
	}
	defer dbConn.Close()

	d, err := iofs.New(migrationsFS, "migrations")
	if err != nil {
		fmt.Println(err)
		return
	}

	migrations, err := migrate.NewWithSourceInstance("iofs", d, dbURL)
	if err != nil {
		fmt.Println(err)
		return
	}

	err = migrations.Up()
	if err != nil && err.Error() != "no change" {
		fmt.Println(err)
		return
	}

	// Here goes your awesome code to get rich :D
}

Extra setup

Obviously you will need a DB - PostgreSQL in this example - to connect to in order to run the migrations. And the migration files to be placed in the correct location for the go binary to find them.

For the example used in this post we have the following filesystem hierarchy:

.
├── docker-compose.yml
├── go.mod
├── go.sum
├── main.go
└── migrations
    ├── 000001_create_examples_table.down.sql
    └── 000001_create_examples_table.up.sql

Docker for Postgres

When I am developing, I tend to work with a DB locally running inside Docker. So, for this example I have used the same approach. The following is the dockerfile that sets the PostgreSQL instance up.

Please note that there is no entry point set up for it in order to do a migration. Since the migration will be done completely from the Go binary.

version: "3"

services:
  db:
    image: postgres:${POSTGRES_VERSION:-15}
    environment:
      POSTGRES_USER: ${DB_USERNAME:-postgres}
      POSTGRES_PASSWORD: ${DB_PASSWORD:-postgres}
      POSTGRES_DB: ${DB_NAME:-examples}
    ports:
      - "${DB_HOST:-127.0.0.1}:${DB_PORT:-54321}:5432"
    command: ["postgres", "-c", "log_statement=all"]

Migration files

The migration file to create the examples table is:

CREATE TABLE IF NOT EXISTS examples(
   id serial PRIMARY KEY,
   name VARCHAR (50) UNIQUE NOT NULL
);

The rollback file to revert the creation of the examples table is:

DROP TABLE IF EXISTS examples;

Execution

If we put all of the above steps together, we can get an execution like the one shown in the following gif:

Conclusion

As shown in this post, we can use the embed library and go-migrate tool to build binaries that can run and verify DB migrations.

This is useful in multiple scenarios, such as building binaries for sidecar pods for Kubernetes deployments, or microservices that are self-contained with the DB migrations they require to be run.

Image manipulation in the terminal

Sat, 03 Apr 2021 13:50:46 +0200

Introduction

Lately I have been in need of cropping some images, the easy thing to do would have been to install Gimp and do it there, but that is not fun.

I end up using ImageMagick to do some basic image manipulations. Since it took me few searches to find what I needed I decided to write a post in my blog so whenever I need it again, I know where I can find it easily.

Crop Images

The most important part of the crop command is where the measures and offsets are provided. In the example below the output image will be of 600px by 500px with 10 offset from the X-axis and 20 offset from the Y-axis.

convert input_file.png -crop 600x500+10+20 output_file.png

Add Transparency

If the image has a uniform background it can be replaced with an alpha-channel (transparency) with the following command. The mention of the color white in the command makes reference to the current background color that you want to replace with the transparency.

convert input_file.png -transparent white output_file.png

The End?

I will keep updating this post with any other image manipulation commands that I find useful.

No Naked Boolean Parameters

Sat, 18 Apr 2020 13:50:46 +0200

Introduction

We all have used naked boolean parameters at some point during our lives, specially when we were starting to learn to program. Other times we add them when refactoring code, or we are just hacking somthing together. However, naked boolean parameters are a code smell and signs of a bad code.

First, they hurt the readability of the code and since we spend the vast majority of our time reading code, that is a big reason to avoid naked boolean parameters. Okay… some IDEs add nice information like the name of the parameters passed, but you do not have that when you are doing code reviews :D

Second, naked boolean parameters can be confusing when several of them are present in the same parameter list. Specially, if they are all together one after the other.

func openRemoteFile(url string, allowRetries, allowRedirects bool) {
	if allowRetries {
		fmt.Println("Redirects enabled")
	}
	if allowRedirects {
		fmt.Println("Retries enabled")
	}
}

When calling that method you will see something like the following

openRemoteFile("sftp://myserver.com/wololo.txt", true, false)

As you can see, the first thing you will have to do upon reading this code is to go to the definition of the function, just to know which parameter is being passed as true and which one is being passed as false.

Wrapper functions

One way to improve the readability of the function is to create wrapper functions that set specific values for the boolean parameters. This is useful if these wrapper functions are the public way to call the non-shared function that contains the naked boolean parameters.

func OpenRemoteFile(url string) {
    openRemoteFile(url, false, false)
}

func OpenRemoteFileWithRetries(url string) {
    openRemoteFile(url, true, false)
}

func OpenRemoteFileWithRedirects(url string) {
    openRemoteFile(url, false, true)
}

func OpenRemoteFileWithRetriesAndRedirects(url string) {
    openRemoteFile(url, true, true)
}

As you can imagine, the amount of these wrapper functions grows exponentially based on the number of boolean parameters. This option is only viable if the number of boolean parameters is low and the possible viable combinations of them are small.

Named booleans

Another way to improve the readability of the function is to use named booleans.

const (
	allowRetries = true
	disallowRetries = false
	allowRedirects = true
	disallowRedirects = false
)

Now, the call to openRemoteFile will look something like this.

openRemoteFile("sftp://myserver.com/wololo.txt", allowRetries, disallowRedirects)

The main problem with this solution is that it only fixes the readability issue, but the problem of mixing the boolean parameters order still in place.

This is a totally valid code, but it does not do what you would expect by reading the parameters.

openRemoteFile("sftp://myserver.com/wololo.txt", allowRedirects, disallowRetries)

Bitmask

Bitmask is only an option if all the possible combinations of the boolean parameters are viable. Being viable means that all the combinations make sense.

type RemoteMark uint8

const (
    Redirects RemoteMark = 1 << iota
    Retries
)

func openRemoteFile(url string, mask RemoteMark) {
	if mask & Redirects == Redirects {
		fmt.Println("Redirects enabled")
	}
	if mask & Retries == Retries {
		fmt.Println("Retries enabled")
	}
}

If the openRemoteFile function signature is changed to match the one above, then it can be called like

openRemoteFile("sftp://myserver.com/wololo.txt", Redirects | Retries)

This solution is both more readable and less error prone. It avoids the usage of multiple boolean parameters, however, this approach case can only be used if all the combinations of values of the boolean parameters are viable.

Typed booleans

Last, but not least, typed booleans. You can think of typed booleans as an improvement over named booleans. This is because typed booleans not only solve the readability problem, like names booleans, but they solve the problem of mixing the order of the boolean parameters as well.

type RedirectEnum bool
type RetryEnum bool

const (
	DisallowRedirects RedirectEnum = false
	AllowRedirects

	DisallowRetries RetryEnum = false
	AllowRetries
)

func openRemoteFile(url string, redirects RedirectEnum, retries RetryEnum) {
	if redirects == AllowRedirects {
		fmt.Println("Redirects enabled")
	}
	if retries == AllowRetries {
		fmt.Println("Retries enabled")
	}
}

If the openRemoteFile function signature is changed to match the one above, then it can be called like

openRemoteFile("sftp://myserver.com/wololo.txt", AllowRedirects, DisallowRetries)

The snippet above shows code that is correct and readable. In addition, if you mix the order like in the following example

openRemoteFile("sftp://myserver.com/wololo.txt", DisallowRetries, AllowRedirects)

When compiling it, an error similar to the following would be displayed

./prog.go:29:17: cannot use DisallowRetries (type RetryEnum) as type RedirectEnum in argument to openRemoteFile
./prog.go:29:17: cannot use AllowRedirects (type RedirectEnum) as type RetryEnum in argument to openRemoteFile

Conclusion

Writing readable code is one of the most important parts of software development, as we spend way more time reading code than writing it. Therefore, if you make the effort to use the patterns described above the readability of your code base will increase.

I strongly recommend the usage of typed booleans and bitmasks when possible. They do not only improve the readability of the code, but they also improve the correctness of the code.

Sorting algorithms

Mon, 10 Feb 2020 13:50:46 +0200

Introduction

Before we start, I want to define a couple of terms:

In place: the algorithm does not require extra space to be executed. In other words, if you have to sort and array of n elements, the algorithm will use only that array of n elements.
Stable: if two elements of the array are the same, they will keep the same relative order in the array after being sorted. For example, if you want to sort the array [3, 1, 4, 3, 2], the result would be [1, 2, 3, 3, 4].

Arguably these are the four most important sorting algorithms, Quicksort, Merge Sort, Heapsort and Insertion Sort.

Algorithm	Best Time	Average Time	Worst Time	Space	Stable
Quicksort	O(n log n)	O(n log n)	O(n^2)	O(1)	No
Merge Sort	O(n log n)	O(n log n)	O(n log n)	O(n)	Yes
Heapsort	O(n log n)	O(n log n)	O(n log n)	O(1)	No
Insertion Sort	O(n)	O(n^2)	O(n^2)	O(1)	Yes

Quicksort

Quicksort is the fastest one of the lot, as its constants are smaller than the others. The best case scenario O(n log n) happens when the data to be sorted is not already sorted or all their values are the same. Another of its strong points is that it is in place, however, it is not stable.

How does it works?

Given an input array a pivot element needs to be selected (a simple approach is to select the first element in the array). Then, you have to swap elements in the array to leave all smaller elements than the pivot in the left side of the pivot, and the elements equal or greater than the pivot in the right hand side of it. This will leave the pivot in its sorted place in the array. Now repeat the same process with the two arrays (left and right) from the pivot until the length of the remaining arrays is less than 2.

Optimizations

The are two main problems with the quicksort algorithm that can make it run at the worst performance. In both cases, the array is always split at the beginning of it, having to do the same operation over the whole array over and over instead of having it split in about half:

Sorted array: one way to solve this problem is to randomize the input array before starting to apply the algorithm.
All values in the array are the same: one way to solve this problem is to use two pointers for the pivot. This way when comparing the elements in the array, on top of doing all the work done before, it keeps track of all the equal values to the pointer and then places them at right place. Only the remaining values have to be sorted.

When to use it?

Since quicksort is the fastest of the sorting algorithms it’s the most commonly used. Though, you need to be aware of it not being stable.

Merge Sort

Merge sort has the most predictable performance of the lot, since its best and worst case scenario is the same O(n log n). This is due to it not being affected by the input state or values. In addition, it is a stable algorithm, but it requires an extra n space to run, where n is the length of the array.

How does it works?

Given an input array it gets split in half over and over until the length of the remaining arrays is less than 2. At that point, all these arrays are sorted, now we just need to merge these arrays, but keeping them sorted.

Optimizations

Since merge sort is a really consistent algorithm, the main optimizations have been focused on lowering the constants of its performance and making it work in place. The most notable optimizations are that an in place merge sort and merge sort is highly parallelizable, this means that it can be run in multiple cores, or even distributed among several machines.

When to use it?

Since merge sort has the most predictable execution time, it is used when the worst case scenario cannot be allowed to be greater than O(n log n). Moreover, the merge sort algorithm is used when sorting tables, to take advantage of it being stable.

Because of the consistent performance of the merge sort, being stable and having an optimization to be in place, some standard libraries from programming languages use it as its default sorting algorithm.

Heapsort

Heapsort has an equivalent predictable performance as merge sort, but heapsort is not a stable algorithm. The heapsort algorithm leverages the usage and performance of the priority queue implementation of a max-heap.

How does it works?

Given an input array, first of all an algorithm known as heapify, with cost O(n), must be applied to it. This leaves the array ready to be used as a max-heap. At this point, the only thing left to do is extracting the n elements in the heap. This has a cost of O(log n) per element extracted, giving it a total cost of O(n log n).

Optimizations

Heapsort does not have any optimizations that stand up or that give a clear improvement against the basic implementation.

When to use it?

Heapsort is mostly used when the data to be sorted is contantly changing. For example, if you are not sorting a static array, but a list of elements that are going to change its values, get removed or added to the list. Your best bet is to use priority queue and it will keep the elements sorted.

Insertion Sort

Insertion sort has the best case performance, but it also has the worst average and worst case performance. However, insertion sort is a stable and in place algorithm.

How does it works?

Insertion sort leaves the already sorted elements to the left of the array and the unsorted elements to the right. Hence, it first sorts the first element of the array, then it sorts the second element of the array, and so on and so forth until every element of the arrray has been sorted.

Optimizations

Insertion sort does not have any optimizations that stand up or that give a clear improvement against the basic implementation.

When to use it?

Since insertion sort is the only algorithm of the lot that is iterative in nature and not recursive, it has a good performance for small arrays. This means that even if the average case is worse than the one for quicksort or merge sort, in practice it can be faster than them.

Conclusion

The more you know about the data and use case you have to sort, the better choice of sorting algorithm you will be able to make.

Finally, I want to mention that there are hybrid approaches. For example, when sorting an array, you can first check its length and if it’s a small array you can call insertion sort straight away, if not you can call either quicksort or merge sort and work recursively until the length of the array is small enough for insertion sort to be faster.

Python's for loop explained

Sat, 04 Jan 2020 13:50:46 +0200

Introduction

Have you ever thought about how the for loop is implemented in Python? I always thought it would iterate over all elements in the sequence, like it does, but that first it would query the sequence to know its length and then request that many elements. Turns out, it does not work like that.

>>> for item in range(5):
... 	print(item)
...
0
1
2
3
4

So, how does it work? Basically, the for loop just keeps requesting more elements from the sequence until it raises a StopIteration or an IndexError exception. The Collection protocol that behaves that way is the Iterator protocol. Therefore, anything that implements the Iterator protocol could be used in the for loop.

Iterator protocol

The only requirements for a class to implment the Iterator protocol is the following:

__next__() method that returns the following element in the sequence. It will raise the StopIteration exception when all elements in the sequence have been visited.
__iter__() method that returns the iterator itself.

Alternatively, the Iterator protocol can fallback to be implemented by just using the __getitem__() method.

Default implementation

The following example of a default implementation of the Iterator protocol will show how the for loop behaves when the __iter__ and __next__ methods are available.

class MyIterator:
    def __init__(self, data):
        self._index = 0
        self._data = data
    
    def __iter__(self):
        print('Called __iter__ method')
        return self
    
    def __next__(self):
        print(f'Called __next__ method {self._index} times')
        if self._index >= len(self._data):
            raise StopIteration()
        
        result = self._data[self._index]
        self._index += 1
        return result

>>> for _ in MyIterator([1, 2, 3, 4, 5]):
...     pass
...
Called __iter__ method
Called __next__ method 0 times
Called __next__ method 1 times
Called __next__ method 2 times
Called __next__ method 3 times
Called __next__ method 4 times
Called __next__ method 5 times

Note how the first thing the for loop does is to call the __iter__ method, and then it keeps calling the __next__ method until it raises the StopIteration exception.

Alternative implementation

The following example of an alternative implementation of the Iterator protocol will show how the for loop behaves when the __iter__ and __next__ methods are not available, but the __getitem__ method is.

class AlternativeIterator:
    def __init__(self, data):
        self._data = data
    
    def __getitem__(self, index):
        print(f'Called __getitem__ method with index {index}')
        return self._data[index]

>>> for _ in AlternativeIterator([1, 2, 3, 4, 5]):
...     pass
...
Called __getitem__ method with index 0
Called __getitem__ method with index 1
Called __getitem__ method with index 2
Called __getitem__ method with index 3
Called __getitem__ method with index 4
Called __getitem__ method with index 5

Note how the for loop just keeps calling the __getitem__ method with an ever increasing index until the IndexError exception is raised.

Closing notes

As you have seen, Python’s approach to the for loop is to keep requesting elements until the iterator runs out of data and raises an exception. This is a very interesting approach and proves that Python exceptions are an integral part of the language and not a last minute addition to the language.

Software development using BASH

Sun, 24 Nov 2019 13:50:46 +0200

Introduction

As for any other language I develop with, I like to make use of tools and utilities to help me spot problems as soon as possible.

For that reason I like to have three things in all my projects:

Formatter
Linter (Static Analysis)
Unit Test Framework

When developing in Bash there is no difference in this regard. Some people would say that Bash is just for scripting. I mostly agree with that statement, however, sometimes you have to write a lot of Bash and for these cases I want to be able to develop it in the same way I do for other languages.

As always, all the code used in this post is available in this repo.

Formatter

A formatter’s duty is to keep the code style the same across a whole project. For Bash I like to use shfmt as the formatter. It is a Go project and it can be installed running

go get -u mvdan.cc/sh/cmd/shfmt

The most basic usage for shfmt is to format a file. By default it targets bash, but to target just POSIX compatible shells the flag -p can be used.

An example file before formatting:

#!/bin/bash

echo "Welcome to this example file"
    echo "This is badly formatted"




  VAR=$( pwd )

echo "This script is being ran from ${VAR}"



for item       in   $(seq 1 10)
do
  echo "Number ${item}"
done

The previous file formatted:

$ shfmt mess.sh 
#!/bin/bash

echo "Welcome to this example file"
echo "This is badly formatted"

VAR=$(pwd)

echo "This script is being ran from ${VAR}"

for item in $(seq 1 10); do
	echo "Number ${item}"
done

Two interesting options I use quite often with shfmt are -bn and -ci. They are for binary operations like && and | may start a line and switch cases will be indented, respectively.

An example script with binary operations and a switch case:

#!/bin/bash

first_thing_to_be_run \
&& second_thing_to_be_run \
&& third_thing_to_be_run \
&& forth_and_last_thing_to_be_run

number=$RANDOM
case ${number} in
5)
echo "Lucky number"
;;
7)
echo "Lucky number"
;;
*)
echo "It may not be a lucky number"
;;
esac

shfmt run with default arguments:

$ shfmt binary_ops_and_switch_cases.sh
#!/bin/bash

first_thing_to_be_run &&
	second_thing_to_be_run &&
	third_thing_to_be_run &&
	forth_and_last_thing_to_be_run

number=$RANDOM
case ${number} in
5)
	echo "Lucky number"
	;;
7)
	echo "Lucky number"
	;;
*)
	echo "It may not be a lucky number"
	;;
esac

shfmt run with -bn and -ci flags:

$ shfmt -bn -ci binary_ops_and_switch_cases.sh
#!/bin/bash

first_thing_to_be_run \
	&& second_thing_to_be_run \
	&& third_thing_to_be_run \
	&& forth_and_last_thing_to_be_run

number=$RANDOM
case ${number} in
	5)
		echo "Lucky number"
		;;
	7)
		echo "Lucky number"
		;;
	*)
		echo "It may not be a lucky number"
		;;
esac

Linter (Static Analysis)

Any good linter’s job is to point out all the errors that can found in your code. My favourite Bash linter is ShellCheck. It can be used from their website https://www.shellcheck.net/ or installing their command line application in your system.

Can you spot all the errors in the following snippet?

#!/bin/sh

for file in $(ls *.py); do
	grep -qi wololo* $file \
		&& echo -e 'The file $file contains words that start with wololo'
done

I am sure you have found several errors, but let’s have a look at the output of ShellCheck

$ shellcheck errors.sh 

In errors.sh line 3:
for file in $(ls *.py); do
            ^-- SC2045: Iterating over ls output is fragile. Use globs.
                 ^-- SC2035: Use ./*.py so names with dashes won't become options.


In errors.sh line 4:
	grep -qi wololo* $file \
                 ^-- SC2062: Quote the grep pattern so the shell won't interpret it.
                 ^-- SC2022: Note that unlike globs, o* here matches 'ooo' but not 'oscar'.
                         ^-- SC2086: Double quote to prevent globbing and word splitting.


In errors.sh line 5:
		&& echo -e 'The file $file contains words that start with wololo'
                        ^-- SC2039: In POSIX sh, echo flags are not supported.
                           ^-- SC2016: Expressions don't expand in single quotes, use double quotes for that.

Finding errors like these by just looking at the code is hard. So, let a computer do it for you.

Unit Test Framework

Last but not least, a unit test framework for Bash. My choice is Bats.

Given the following functions.sh file:

sum() {
  if [ ${#} -ne 2 ]; then
    echo "Bad number of parameters"
    echo "Usage: sum <number 1> <number 2>"
  else
    num1=${1}; shift
    num2=${1}; shift
    echo "${num1}+${num2}" | bc
  fi
}

Test file test_functions.sh for the previous file:

#!/usr/bin/env bats

source functions.sh

@test "The addition of 4 and 5 results in 9" {
  run sum 4 5
  [ "$status" -eq 0 ]
  [ "${#lines[@]}" -eq 1 ]
  [ "${lines[0]}" = "9" ]
}

@test "The addition of -5 and 9 results in 4" {
  run sum -5 9
  [ "$status" -eq 0 ]
  [ "${#lines[@]}" -eq 1 ]
  [ "${lines[0]}" = "4" ]
}

@test "Bad number of parameters (1)" {
  run sum 10
  [ "$status" -eq 0 ]
  [ "${#lines[@]}" -eq 2 ]
  [ "${lines[0]}" = "Bad number of parameters" ]
  [ "${lines[1]}" = "Usage: sum <number 1> <number 2>" ]
}

@test "Bad number of parameters (3)" {
  run sum 10 11 12
  [ "$status" -eq 0 ]
  [ "${#lines[@]}" -eq 2 ]
  [ "${lines[0]}" = "Bad number of parameters" ]
  [ "${lines[1]}" = "Usage: sum <number 1> <number 2>" ]
}

When that battery of test is run it outputs:

$ ./test_functions.sh 
 ✓ The addition of 4 and 5 results in 9
 ✓ The addition of -5 and 9 results in 4
 ✓ Bad number of parameters (1)
 ✓ Bad number of parameters (3)

4 tests, 0 failures

Conclusion

Bash is a great scripting language, but when you have to write more than a simple script to perform a task it is worth spending the time in using a formatter, a linter and unit test framework. It may be even useful to use some documentation generation tool as well.

Interactive curriculum in Rust

Mon, 23 Jul 2018 13:50:46 +0200

Introduction

As we all know the process of looking for a job is hard. And you have to stand out from the crowd. Therefore, when I saw s0ulshake’s CV I wanted to do something similar, but with other technologies (you know I do not know JavaScript) and add the ability to move forward and backward.

I decided to code it in Rust because I have been looking for an excuse to start using it and this was as good as any other excuse :D. The way to distribute the binary is using Docker.

How to run my CV application

In order to run my CV application you just have to run

sudo docker run -it maitesin/resume

and you will get something similar to:

For more information about the project please visit https://oscarforner.com/projects/resume.

My two cents about Go

Sat, 09 Sep 2017 13:50:46 +0200

Introduction

There is plenty of meterials such as talks, tutorials and posts talking about Go, but I want to give my opinion about what I think are the strenghts and weaknesses of Go.

Full disclosure: I do not use Go at work. I use Go in my personal projects. Therefore, take my opinions with a grain of salt.

Pros

Go is one of the best programming languages I have seen in my whole life. I think that because of the following characteristics.

Standard library

If you ever asked me “Which programming language has the best and more complete standard library?” before trying Go I would have answered Python. Nowadays, I would answer Go has the best and more complete standard library.

To back up my statement that Go has the best and more complete standard library:

It has a library for encoders and a library for decoders. They contain encoders and decoders for formats such as json, xml, csv, base64, hex, etc.
It has a compress library that contains the following formats bzip2, flate, gzip, lzw and zlib.
It has a crypto library with algorithms such as aes, dsa, md5, rc4, rsa, sha1, sha256, sha512, etc.
It has a database library that contains a generic interface for SQL. It does not contain specific drivers, but it defines a common interface for all of them. Therefore, changing the backend will be pretty simple.
It has a go library that contains all the required dependencies in order to format, parse and build any valid Go source code. It contains modules such as ast, build, doc, format, parser, token, types, etc.
It has an image library that can handle formats such as gif, jpeg and png.
It has a math library that has a big module to handle arbitrary-precision arithmetic.
It has a net library that contains modules such as http, mail, smtp and url.
It has a library for run-time reflexion called reflect.
It has a testing library to perform the unit testing.

Obviously that is no the only content of the standard library, but these were the packages that I found to be the more interesting.

Unit test and benchmarking

I really love the idea of the compiler of the language having the ability to run the unit testing and the benchmarking in such a simple way.

To run all the unit test you just have to do the following:

go test

You can also run the unit test from a specific file with:

go test file.go

To run all the unit test that match a provided pattern you can do the following:

go test -run "Pattern*"

To perform the benchmarking of the unit test you have to do the following:

go test -bench=.

Finally, you can limit the amount of time the benchmark is running:

go test -bench=. -benchtime=20s

Channels

This is one of the big features of Go. If you ever used a Unix system, this feature will remind you of pipes. This is because a channel is used to send data between a reader and a writer (usually Goroutines).

There is plenty of good documentation about channels in Go by Example and in A Tour of Go. So, there is no need for me to write examples.

Cons

Not everything is perfect in Go. There are some perks that once they are fixed will make Go an even better programming language.

Build system

One of the main flaws of Go is the lack of a standard build system. I can understand that older languages such as C or C++ do not have a standard build system. Newer languages usually have a build system, not always from the beginning, such as Python or the more recent Rust.

Getting a standard build system for Go would be a great advantage. Maybe for Go 2.0.

Different versions of the compiler

This issue is not my biggest concern, but it is something that could be problematic for some people. The main problem is having to change the environment variables $GOROOT and $GOPATH. In Python, I can choose to use Python2 or Python3 calling different executables. The same for C and C++, I can call different versions of the compiler because they can coexist in the filesystem.

Other

In this section I talk about some features of Go that are both a Pro and a Con.

Runtime

Since I come from a systems programming background, I find hard to like a programming language that has a runtime. Unless such programming language brings more to the table to compensate for the runtime, I will try to avoid using it.

Go is one of the few languages that actually brings cool features in its runtime like detection of concurrency problems such as deadlocks.

Generics

The lack of generics in the Go language may make some people doubt the usefulness of the language, however, I do not mind not having generics. It could be because I have not yet created a big project using Go, so I have not needed them.

Maybe in Go 2.0 they will introduce some feature to address this perk. Nevertheless, I am not confident using this argument against Go. I have used C for long time and I never felt the need for generics. Therefore, I can live with a language that does not have them.

Goroutines

This is another of the big features of Go and some people will be surprised of it not being a Pro in this post. Don’t get me wrong, I think Goroutines are awesome, but you need the runtime to make them work.

That is because Goroutines do not use the thread system provided by the operating system. It uses its own implementation of thread. And all that dependency, locality, lifetime, etc. of the thread is handled by the runtime.

Conclusion

I think Go is already a great language, even with the small Cons I talk about in this post. Once the people in charge of the language fix this perks, Go will become an even better language. Go is a mature language with an awesome standard library and great tools. I will not be surprised if Go starts replacing Python in some niches even before Go gets a standard build system.

Boost ASIO basics

Sat, 21 Jan 2017 13:50:46 +0200

Introduction

Boost ASIO library is the defacto standard for network and low-level I/O programming. It has a great documentation available online, but there are a lot of methods and classes in the library. Therefore, if it is your first attempt to use it, it can be a bit challenging.

Since there is a lot of ground to cover in the Boost ASIO library, I will only cover the work scheduler and the synchronous methods in this post. The asynchronous methods and timers will be covered in following posts.

During this post I use CMake to configure and build the project and as dependency manager I will use conan.

As always, all the code used in this post is available in this repo.

The videos are made with asciinema, that means you can copy from the video.

IO Service

The IO Service is the mainstay of the Boost ASIO. Basically, it is in charge of scheduling the work to be done. In this section, I will use the IO Service to demonstrate how it performs the scheduling. However, it is not recommended to use it for this purpose, as Sean Parent shows in this talk.

The whole idea of this section is to help you understand how the Boost ASIO works underneath.

One worker

The code for this simple example is to just show how the IO Service can perform some work.

#include <iostream>
#include <boost/asio.hpp>

int main()
{
  boost::asio::io_service service;

  service.post([](){std::cout << "1" << std::endl;});
  service.post([](){std::cout << "2" << std::endl;});
  service.post([](){std::cout << "3" << std::endl;});
  service.post([](){std::cout << "4" << std::endl;});
  service.post([](){std::cout << "5" << std::endl;});
  service.post([](){std::cout << "6" << std::endl;});

  service.run();

  return 0;
}

The code is quite straightforward, it provides 6 tasks to the IO Service and then it executes them.

Execution

The output of the previous code is what everybody can expect.

Two workers

The code for the two workers is a bit more complicated. The idea is the same, but with two threads executing tasks instead of one.

#include <iostream>
#include <vector>
#include <thread>
#include <algorithm>
#include <boost/asio.hpp>

int main()
{
  std::vector<std::thread> worker;
  boost::asio::io_service service;

  service.post([](){std::cout << "1" << std::endl;});
  service.post([](){std::cout << "2" << std::endl;});
  service.post([](){std::cout << "3" << std::endl;});
  service.post([](){std::cout << "4" << std::endl;});
  service.post([](){std::cout << "5" << std::endl;});
  service.post([](){std::cout << "6" << std::endl;});

  for (int i = 0; i < 2; i++) {
    worker.push_back(std::thread([&](){ service.run(); }));
  }

  std::for_each(std::begin(worker), std::end(worker), [](std::thread &t){t.join();});

  return 0;
}

The complexity of the previous code is the usage of threads, but it is not overwhelming.

Execution

The output of the two workers code can be the same as from the one worker section. However, it is not guaranteed that it would be the same. This can be seen in the following execution:

Synchronous

In order to use Boost ASIO for network it is not required to know how the IO Service works. Actually, the explicit usage of IO Service as it is done in the previous section is not recommended.

In this section, the usage of synchronous calls is shown in order to create a client and a server programs.

Client

The code for the client sends a message - provided as a parameter - to the server. After that the client reads the answer from the server and prints it in the standard output.

#include <iostream>
#include <string>
#include <boost/asio.hpp>

using boost::asio::ip::tcp;

int main(int argc, const char *argv[])
{
  // Check provided parameters
  if (argc != 4) {
    std::cerr << "Error. Bad number of parameters" << std::endl;
    std::cerr << "Usage: " << argv[0] << " <hostname> <port|service> <message>" << std::endl;
    exit(-1);
  }

  // Error to not throw exception
  boost::system::error_code not_throw;

  // Resolve hostname and port
  boost::asio::io_service service;
  tcp::resolver resolver(service);
  tcp::resolver::query query(argv[1], argv[2]);
  tcp::resolver::iterator endpoint = resolver.resolve(query, not_throw);
  if (not_throw) {
    std::cerr << "Error resolving host(" << not_throw.value() << "): "<< not_throw.message() << std::endl;
    return 1;
  }

  // Socket and connection
  tcp::socket socket(service);
  boost::asio::connect(socket, endpoint, not_throw);
  if (not_throw) {
    std::cerr << "Error connecting(" << not_throw.value() << "): "<< not_throw.message() << std::endl;
    return 1;
  }

  // Send a message to the end point
  std::string message = {argv[3]};
  socket.write_some(boost::asio::buffer(message), not_throw);
  if (not_throw) {
    std::cerr << "Error sending(" << not_throw.value() << "): "<< not_throw.message() << std::endl;
    return 1;
  }

  // Read a message from the end point
  boost::asio::streambuf response;
  boost::system::error_code error = boost::asio::error::eof;
  do {
    boost::asio::read(socket, response, error);
  } while(error && error != boost::asio::error::eof);

  // Close socket
  socket.close();

  // Show the received message
  std::istream received(&response);
  std::cout << received.rdbuf() << std::endl;

  return 0;
}

The first interesting part of the code is the resolution of the hostname and the port. Pay attention to the fact that the “port” is a string. It is because it can be a number or a service name (i.e. “https” as port would connect to port 443).

  // Resolve hostname and port
  boost::asio::io_service service;
  tcp::resolver resolver(service);
  tcp::resolver::query query(argv[1], argv[2]);
  tcp::resolver::iterator endpoint = resolver.resolve(query, not_throw);
  if (not_throw) {
    std::cerr << "Error resolving host(" << not_throw.value() << "): "<< not_throw.message() << std::endl;
    return 1;
  }

The next interesting part of the code is the send of the message to the end point. The interesting bit is how the data is send. In this case, the message is stored in a std::string, but it is required to be transformed into a boost::asio::buffer.

  // Send a message to the end point
  std::string message = {argv[3]};
  socket.write_some(boost::asio::buffer(message), not_throw);
  if (not_throw) {
    std::cerr << "Error sending(" << not_throw.value() << "): "<< not_throw.message() << std::endl;
    return 1;
  }

The last interesting part of the code is the read a message from the end point. In this case, a boost::asio::streambuf is used to retrieve the data from the server. However, it could have been done in other ways as it is shown in the server case. The errors are another interesting part of the code. The “success” error (when the first part of the conditional is false) will happen when the buffer is full. However, the boost::asio::error::eof will occur when the end point closes the connection.

  // Read a message from the end point
  boost::asio::streambuf response;
  boost::system::error_code error = boost::asio::error::eof;
  do {
    boost::asio::read(socket, response, error);
  } while(error && error != boost::asio::error::eof);

Server

The code for the server receives a message, then it returns the message backwards to the sender.

#include <iostream>
#include <string>
#include <algorithm>
#include <boost/asio.hpp>

using boost::asio::ip::tcp;

int main(int argc, const char *argv[])
{
  // Check provided parameters
  if (argc != 2) {
    std::cerr << "Error. Bad number of parameters" << std::endl;
    std::cerr << "Usage: " << argv[0] << " <port>" << std::endl;
    exit(-1);
  }
  // Get the port from the parameters
  uint16_t port = std::stoul(argv[1]);

  // Error to not throw exception
  boost::system::error_code not_throw;

  // Socket and acceptor
  boost::asio::io_service service;
  tcp::acceptor acceptor(service, tcp::endpoint(tcp::v4(), port));
  tcp::socket socket(service);
  acceptor.accept(socket, not_throw);
  if (not_throw) {
    std::cerr << "Error when binding the port in the socket" << std::endl;
    return 1;
  }

  // Read a message
  size_t size;
  char buf[512];
  std::ostringstream oss;
  boost::system::error_code error = boost::asio::error::eof;
  do {
    size = socket.read_some(boost::asio::buffer(buf), error);
    oss << std::string(buf, sizr);
  } while(error && error != boost::asio::error::eof);

  // Reverse the received message
  std::string original = oss.str();
  std::reverse(std::begin(original), std::end(original));

  // Write the reversed message
  socket.write_some(boost::asio::buffer(original), not_throw);
  if (not_throw) {
    std::cerr << "Error sending(" << not_throw.value() << "): " << not_throw.message() << std::endl;
    return 1;
  }

  // Close socket
  socket.close();

  return 0;
}

The first interesting part of the code is the socket and acceptor. In this case the port is a number. Another good part is how the usage of IPv4 only has been specified.

  // Socket and acceptor
  boost::asio::io_service service;
  tcp::acceptor acceptor(service, tcp::endpoint(tcp::v4(), port));
  tcp::socket socket(service);
  acceptor.accept(socket, not_throw);
  if (not_throw) {
    std::cerr << "Error when binding the port in the socket" << std::endl;
    return 1;
  }

The last interesting part of the code is the read a message. In this case, it is using a plain char array to read parts of the message and store it in a std::ostringstream. Pay attention to how it keeps track of the amount of data that has been read and stores that much information only.

  // Read a message
  size_t size;
  char buf[512];
  std::ostringstream oss;
  boost::system::error_code error = boost::asio::error::eof;
  do {
    size = socket.read_some(boost::asio::buffer(buf), error);
    oss << std::string(buf, size);
} while(error && error != boost::asio::error::eof);

Execution

Conclusion

The simplicity of the Boost ASIO is matched only by its power. It is quite simple to create a synchronous client/server application. These are the basics of Boost ASIO and it should allow you to understand more complex usage of it.

Thread, Future, and Promise

Sun, 11 Sep 2016 13:50:46 +0200

Introduction

In the C++11 standard several concurrency related classes were added. I will talk about thread, future and promise. Yes, I know there are others more useful than these three. However, I think that it is really important to know well the bases to be able to use the more complex ones properly.

As usual, I am working from an Arch Linux computer. Therefore, I can install Clang and the tools from the repository (clang). For other distributions you can find the information in the documentation.

As always, all the code used in this post is available in this repo.

The videos are made with asciinema, that means you can copy from the video.

Thread

Thread is the basic element of concurrency in the C++11 standard and they are intended to map one-to-one the operating system’s threads.

Simple

In this example we use a thread to perform the task done by the function accum_up_to.

#include <iostream>
#include <functional>
#include <thread>

void accum_up_to(int value, int & result) {
  int accum = 0;
  for (int i = 0; i <= value; ++i) {
    accum += i;
  }
  result = accum;
}

int main()
{
  int result;
  std::thread t1(accum_up_to, 50, std::ref(result));

  t1.join();

  std::cout << "Calculated a value of: " << result << std::endl;

  return 0;
}

Some parts to note in the code are the following:

The function that is executing the thread has a return type of void. Therefore, to get a result from that function it is required to provide a parameter to hold it.
Before being able to use the result stored in the parameters passed we have to make sure the thread has finished. To do so we use the join method of the thread to synchronize it.

Execution

Two threads

In this example we use two threads that perform the same task done by the function accum_up_to.

#include <iostream>
#include <functional>
#include <thread>

void accum_up_to(int value, int & result) {
  int accum = 0;
  for (int i = 0; i <= value; ++i) {
    accum += i;
  }
  result = accum;
}

int main()
{
  int result1, result2;
  std::thread t1(accum_up_to, 50, std::ref(result1));
  std::thread t2(accum_up_to, 40, std::ref(result2));

  t1.join();
  t2.join();

  std::cout << "Calculated a value of: " << result1 << std::endl;
  std::cout << "Calculated a value of: " << result2 << std::endl;

  return 0;
}

As you can see above once the code is able to hold a thread, it is quite easy to make it hold two. However, now we have two result variables and two calls to the method join. As you can see it is not really easy to keep track once you have several threads.

Execution

Future

Future provides a way to access the result of asynchronous operations. You can imagine we use this to avoid the tedious part of getting the results from the threads.

Simple

In this example we use a future to perform the task done by the function accum_up_to.

#include <iostream>
#include <future>

int accum_up_to(int value) {
  int accum = 0;
  for (int i = 0; i <= value; ++i) {
    accum += i;
  }
  return accum;
}


int main()
{
  std::future<int> f = std::async(std::launch::async, accum_up_to, 50);

  std::cout << "Calculated a value of: " << f.get() << std::endl;

  return 0;
}

Some changes compared to the thread version:

The function returns the value directly and it can be accesed by the method get. Moreover, the method get synchronizes the future.
Future is gotten from the async function. This is really similar to how the threads are created.

Execution

Two futures

In this example we use two futures that perform the same task done by the function accum_up_to.

#include <iostream>
#include <future>

int accum_up_to(int value) {
  int accum = 0;
  for (int i = 0; i <= value; ++i) {
    accum += i;
  }
  return accum;
}


int main()
{
  std::future<int> f1 = std::async(std::launch::async, accum_up_to, 50);
  std::future<int> f2 = std::async(std::launch::async, accum_up_to, 40);

  std::cout << "Calculated a value of: " << f1.get() << std::endl;
  std::cout << "Calculated a value of: " << f2.get() << std::endl;

  return 0;
}

This time the execution of two tasks in separate threads is way cleaner.

Execution

Promise

Promise is where a task can deposit its result to be retrieved through a future. In other words, promise is a way to have a single point of synchronization between two threads without being the end of one of them.

Simple

In this example we use a promise to send some information (from the user) into a future to perform the task done by the function accum_up_to.

#include <iostream>
#include <functional>
#include <thread>
#include <future>

void accum_up_to(std::future<int>& f) {
  std::cout << "Started accum_up_to method" << std::endl;
  int accum = 0;
  int value = f.get();
  for (int i = 0; i <= value; ++i) {
    accum += i;
  }
  std::cout << "Calculated a value of: " << accum << std::endl;
}


int main()
{
  int limit;
  std::promise<int> p;
  std::future<int> f = p.get_future();
  std::thread t(accum_up_to, std::ref(f));

  std::cout << "Introduce the limit:" << std::endl;
  std::cin >> limit;

  p.set_value(limit);
  t.join();

  return 0;
}

Some key points from the example:

The promise provides you a future and that future is used by the accum_up_to function.
Once the user gives a number, the promise sends it to the future, as it is waiting for the number where the get method is called.
Finally, the method join from the thread is called in order to wait for it to finish the execution of the function accum_up_to.

Execution

In this execution something called data race is shown. That can be seen because the execution of the same program can generate different outputs (even if the result value is always the same).

Conclusion

These are really the basics of concurrency that are provided in the C++11 standard. I know there are other important topics from concurrency such as mutex or condition variable and I will talk about them in the future because they deserve their own post. Futhermore, I know I have not talked about data races, dead locks and all the other problems you face when doing concurrency, again if people are interested in this topic I can write about it too.

Clang sanitizers

Wed, 13 Jul 2016 13:50:46 +0200

Introduction

Clang is a compiler front end for the C, C++, Objective-C and Objective-C++ programming languages. It uses LLVM as its back end. In this post I talk about some of the sanitizers available in Clang (some are avilable in GCC as well). They help you detect problems at run time (dynamic analysis).

As always, all the code used in this post is available in this repo.

The videos are made with asciinema, that means you can copy from the video.

Clang sanitizers

The Clang sanitizers available are:

AddressSanitizer: detects memory errors. http://clang.llvm.org/docs/AddressSanitizer.html
ThreadSanitizer: detects data races. http://clang.llvm.org/docs/ThreadSanitizer.html
MemorySanitizer: detects uninitialized reads. http://clang.llvm.org/docs/MemorySanitizer.html
UndefinedBehaviorSanitizer: detects undefined behavior. http://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html
DataFlowSanitizer: is a generalised dynamic data flow analysis. Unlike other sanitizers this one is not designed to detect a specific class of bugs on its own, it provides a generic dynamic data flow analysis framework to be used by clients to help detect application-specific issues within their own code. http://clang.llvm.org/docs/DataFlowSanitizer.html
LeakSanitizer: detects memory leaks. It can be combined with AddressSanitizer. http://clang.llvm.org/docs/LeakSanitizer.html

In this post I show AddressSanitizer, ThreadSanitizer, MemorySanitizer and UndefinedBehaviorSanitizer. I do not talk about DataFlowSanitizer because it is a work in progress and it is really specific of the application where you need it. Moreover, I do not talk about LeakSanitizzer because it is a subset of checks from AddressSanitizer that can be run independently from it.

AddressSanitizer

Usage of freed memory

The code below is trying to use a region of memory that has been freed already.

int main()
{
  int *array = new int[100];
  delete [] array;
  return array[1];
}

We can compile it with:

clang++-3.8 -fsanitize=address -g -o free free.cpp

When you run the previously generated executable you will get something similar to the following:

Buffer overflow

The code below is trying to access a memory region that is not part of the allocated one.

int main()
{
  int * array = new int[50];
  return array[100];
}

We can compile it with:

clang++-3.8 -fsanitize=address -g -o overflow overflow.cpp

When you run the previously generated executable you will get something similar to the following:

Memory leak

The code below is allocating memory twice but it only frees the memory once.

int main()
{
  int * array = new int[5];
  array = new int[10];
  delete [] array;
  return 0;
}

We can compile it with:

clang++-3.8 -fsanitize=address -g -o leak leak.cpp

When you run the previously generated executable you will get something similar to the following:

Double free

The code below is allocating memory once but it is freeing it twice.

int main()
{
  int * array = new int[5];
  delete [] array;
  delete [] array;
  return 0;
}

We can compile it with:

clang++-3.8 -fsanitize=address -g -o double_free double_free.cpp

When you run the previously generated executable you will get something similar to the following:

ThreadSanitizer

Data race

The code below has a data race due to having two threads modifying the same global variable.

#include <iostream>
#include <pthread.h>

int GLOBAL;

void * SetGlobalTo2(void * x) {
  GLOBAL = 2;
  return x;
}

void * SetGlobalTo3(void * x) {
  GLOBAL = 3;
  return x;
}

int main()
{
  pthread_t thread1, thread2;
  pthread_create(&thread1, NULL, SetGlobalTo2, NULL);
  pthread_create(&thread2, NULL, SetGlobalTo3, NULL);
  pthread_join(thread1, NULL);
  pthread_join(thread2, NULL);
  std::cout << GLOBAL << std::endl;
  return 0;
}

We can compile it with:

clang++-3.8 -fsanitize=thread -g -lpthread -o race race.cpp

When you run the previously generated executable you will get something similar to the following:

MemorySanitizer

Uninitialized values

The code below is reading the value stored in an array that has not been initialized.

int main()
{
  int *array = new int[50];
  return array[42];
}

We can compile it with:

clang++-3.8 -fsanitize=memory -g -o memory memory.cpp

When you run the previously generated executable you will get something similar to the following:

UndefinedBehaviorSanitizer

Function not returning a value

The code below contains a function that must return an integer, but it does not.

int must_return_value()
{
  int result = 0;
  result += 1;
}

int main()
{
  int value = must_return_value();
  return value;
}

We can compile it with:

clang++-3.8 -fsanitize=undefined -g -o function function.cpp

When you run the previously generated executable you will get something similar to the following:

Conclusion

Using these tools to compile your code to run your tests (integration test, smoke test, system test, etc) helps you catch plenty of problems. The downside is that you cannot use two sanitizers at the same time (except AddressSanitizer and LeakSanitizer). Therefore, you need multiple binaries to test your code with all these sanitizers, but the payoff is worth it.

Lambda comparison C++11/14/17

Sat, 14 May 2016 13:50:46 +0200

Introduction

In this post I talk about what has been added in the C++ standard regarding lambda expressions since they were introduced in C++11.

All the code and configuration files used in this post are available in this repo in GitHub.

What is a lambda expression?

A lambda expression is a simplified notation for defining and using an anonymous function object. Instead of defining a named class with an operator(), later making an object of that class and finally invoking it.¹

I do not explain all the options for capturing or specifying return types. There is plenty of material regarding these topics. I focus on what has been introduced in C++14 and what will be introduced in C++17.

Basics of the lambda expression

The following is the smallest lambda expression with its three parts:

[]: capture.
(): parameters.
{}: body.

[](){}

The following lambda expression increments by one the parameter.

#include <algorithm>
#include <iostream>
#include <vector>

int main() {
  std::vector<int> v = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};
  auto my_lambda = [](int x) { return x + 1; };
  std::transform(v.begin(), v.end(), v.begin(), my_lambda);
  for (auto &value : v) {
    std::cout << value << std::endl;
  }
  return 0;
}

The result of the execution of the previous code is:

The following lambda expression shows the difference between capturing by value = and by reference &:

#include <iostream>

int main() {
  int i = 1;
  auto copy = [=](){ std::cout << i << std::endl; };
  auto ref = [&](){ std::cout << i << std::endl; };
  copy();
  ref();
  ++i;
  copy();
  ref();
  return 0;
}

The result of the execution of the previous code is:

What has been added in C++14

In C++14 two new features were added to lambda expressions:

Initialization captures: A capture with an initializer acts as if it declares and explicitly captures a variable declared with type auto, whose declarative region is the body of the lambda expression.²
Generic lambda expressions: Until C++14 parameters of a lambda expression should be of a specific type. Now, lambda expressions accept auto as a valid parameter type.

Example of the initialization captures:

#include <iostream>

int main() {
  int x = 1;
  auto my_lambda = [&z = x]() { z += 1; };
  my_lambda();
  std::cout << x << std::endl;
  return 0;
}

The result of the execution of the previous code is:

Example of a generic lambda expression:

#include <iostream>

int main() {
  auto my_lambda = [](auto &a, auto &b) { return a < b; };
  float af = 1.5, bf = 2.0;
  int ai = 3, bi = 1;
  std::cout << "Float: " << my_lambda(af, bf) << std::endl;
  std::cout << "Integer: " << my_lambda(ai, bi) << std::endl;
  return 0;
}

The result of the execution of the previous code is:

Float: 1
Integer: 0

What will be added in C++17

The current plan is to add two new features for lambda expressions in C++17:

Capture ∗this: This will allow the lambda expression to capture the enclosing object by copy. This will make possible to use safely the lambda expression even after the enclosing object has been destroyed.
constexpr lambda expressions: This will allow to call lambda expressions and use their result to generate constexpr objects at compile time.

Sadly neither GCC or Clang in any stable version supports them. Therefore, there will not be any execution of code, but there is some code that should work once the features are implemented. The information used to do the code of this section has been found in the C++ reference website and in the paper for constexpr lambda expressions.

The following is an example of the capture ∗this:

struct my_struct {
  int x;
  void value();
};

void my_struct::value() {
  [=, *this](){};
}

The following is an example of the constexpr lambda expression:

#include <iostream>

int main() {
  constexpr auto multi = [](int a, int b){ return a * b; };
  static_assert(multi(3,7) == 21, "3x7 == 21");
  static_assert(multi(4,5) == 15, "4x5 != 15");
  return 0;
}

Note: Once GCC or Clang support these features I will try the code above and I will ammend it if necessary.

Definition extracted from the book The C++ Programming Language (Fourth Edition). ↩︎
Definition extracted from the website cppreference.com. ↩︎

ARM C Compiler (ACC) II

Wed, 20 Apr 2016 13:50:46 +0200

Introduction

I am working on a pet project to create a C compiler for the ARM architecture. You can find more information about this topic in my previous post ARM C Compiler (ACC) - Basic Compiler I.

The source code of this project can be found in GitHub.

What is the current state of the project?

Currently, I have implemented a very basic compiler, this version is available in GitHub as v0.2. Basic compiler able to handle a single file with a main function (without parameters) and if/else statements with simple boolean expressions and return statements of a positive integer.

What has been added or modified since the last post?

The difference between both versions can be found in the following link diff v0.1 v0.2.

Unit tests

In the previous version only unit tests were available for the Lexer, but in this new version some unit tests for the Lexer have been added and unit tests for the Grammar have been created. In total, there are 27 (26 new) unit tests for Lexer and 12 unit tests for Grammar.

Lexer and Tokens

There are new Tokens for if, else and boolean operators. Lexer now supports all these new tokens and it has a new feature (a stack) that allows the Grammar to give a Token back to Lexer. That is useful when you are reading a boolean expression, but you do not know if it will be a binary or unary expression. Then in case you try to check if it is a binary expression and it is not, you give back that token to the Lexer and try with the unary expression. Moreover, the Lexer now receives a buffer with the content of the file loaded instead of a file. This is to make it easier to create unit tests.

void init_lexer(lexer * l, char * code)
{
	l->f = code;
	l->stack = NULL;
	l->count = 0;
}

void destroy_lexer(lexer * l)
{
	stack_base * tmp = NULL;
	stack_base * s = l->stack;
	if (s != NULL)
	{
		tmp = s;
		s = s->next;
		free_stack_base(tmp);
	}
}

void free_stack_base(stack_base * s)
{
	free(s);
}

void push_back(lexer * l, token_base * t)
{
	stack_base * s = (stack_base *) malloc(sizeof(stack_base));
	s->token = t;
	s->next = l->stack;
	l->stack = s;
}

char get_char(lexer * l)
{
	return l->f[l->count++];
}

void push_back_chars(lexer * l, size_t amount)
{
	if (l->count >= amount)
	{
		l->count -= amount;
	}
	else
	{
		fprintf(stderr, "Error moving back in the buffer\n");
		exit(EXIT_FAILURE);
	}
}

void push_back_one_char(lexer * l)
{
	push_back_chars(l, 1);
}

struct token_base * next(lexer * l)
{
	char buffer[512];
	char * buff_copy = NULL;
	int pos = 0;
	size_t state = 0;
	char tmp;
	void * result = NULL;
	stack_base * stack = NULL;

	if (l->stack != NULL)
	{
		stack_base * stack = l->stack;
		l->stack = l->stack->next;
		result = stack->token;
		stack->token = NULL;
		free_stack_base(stack);
		return result;
	}
...

Grammar and AST nodes

The most important features added to the AST is the addition of node_if, node_boolean_operator and enum boolean_operator_type. In addition to this, now the base of the AST holds a pointer to the next AST node. That is to hold the whole information contained in the body of a function or an if statement.

// Boolean operators
enum boolean_operator_type {
	B_EQUALEQUAL,
	B_NOTEQUAL,
	B_LTEQUAL,
	B_GTEQUAL,
	B_OROR,
	B_ANDAND,
	B_LT,
	B_GT,
	B_NOT
};

typedef struct node_if
{
	ast_base base;
	ast_base * expression;
	ast_base * i_body;
	ast_base * e_body;
} node_if;

typedef struct node_boolean_operator
{
	ast_base base;
	ast_base * first;
	ast_base * second;
	enum boolean_operator_type oper;
} node_boolean_operator;

/*
 * Init functions for the AST nodes
 */
void init_ast_base(ast_base * base, enum ast_type type, ast_base * next);
void init_ast_base_single(ast_base * base, enum ast_type type);
void init_node_id(node_id * node, char * value);
void init_node_int(node_int * node, int value);
void init_node_function(node_function * node, char * name,
			ast_base * entry_point);
void init_node_return(node_return * node, ast_base * value);
void init_node_if(node_if * node, ast_base * expression, ast_base * i_body,
		  ast_base * e_body);
void init_node_boolean_operator(node_boolean_operator * node,
				enum boolean_operator_type type,
ast_base * first, ast_base * second);

Regarding the Grammar, the method read_function_body has been refactored into the method read_body to be able to re-use it to read the body of the if and else statements. Another interesting piece of code is the method read_boolean_expression, it allows to build a valid AST for a complex boolean expression such as 1 <= 2 && 4 == 4.


/*
 * Read functions to build AST parts
 */
ast_base * read_function_ast_node(grammar * g);
ast_base * read_body(grammar * g);
ast_base * read_return_expression(grammar * g);
ast_base * read_if_statement(grammar * g);
ast_base * read_boolean_expression(grammar * g);
ast_base * read_single_boolean_expression(grammar * g, ast_base * r,
					  int * op_found,
					  enum boolean_operator_type op);
ast_base * read_boolean_binary_expression(grammar * g);
ast_base * read_boolean_unary_expression(grammar * g);

ast_base * read_boolean_expression(grammar * g)
{
	ast_base * root = NULL;
	token_base * token = NULL;
	enum boolean_operator_type op;
	int op_found = 0;

	token = next(g->l);
	while (token->type != T_CPAR)
	{
		switch (token->type)
		{
			case T_INT_VALUE:
				push_back(g->l, token);
				root = read_single_boolean_expression(g, root, &op_found, op);
				break;
			case T_BOOLEAN_OP:
				op = get_boolean_op_value((token_boolean_op *)token);
				op_found = 1;
				break;
			case T_OPAR:
				root = read_single_boolean_expression(g, root, &op_found, op);
				break;
			default:
				fprintf(stderr, "Error reading boolean expression\n");
				exit(EXIT_FAILURE);
		}
		token = next(g->l);
	}
	free_token_cpar((token_cpar *)token);
	return root;
}

Assembly Generator

The Generator of the ARM assembly has new methods to handle all the new AST structures and behaviours. It has some limitations regarding boolean expressions in which I have to do further research. Some pieces of the Generator are the following:

void __generate_code_for_if(generator * g, node_if * ast);
void __generate_code_for_if_expression(generator * g, ast_base * ast,
				       unsigned long long int if_num);
void __generate_code_for_binary_boolean_expression(generator * g,
					node_boolean_operator * op,
					unsigned long long int if_num);
void __generate_code_for_unary_boolean_expression(generator * g,
					node_boolean_operator * op,
					unsigned long long int if_num);
void __generate_code_for_body(generator * g, ast_base * body);

void __generate_code_for_if(generator * g, node_if * ast)
{
    unsigned long long int if_num = g->if_num;
    g->if_num++;
    // Expression
    __generate_code_for_if_expression(g, ast->expression, if_num);
    // If body
    __generate_code_for_body(g, ast->i_body);
	fprintf(g->f, "if_else_%llu:\n", if_num);
    if (ast->e_body != NULL)
    {
        // Else body
        __generate_code_for_body(g, ast->e_body);
    }
}

Example of the current functionality

The code to be compiled into ARM assembly is:

int main()
{
    if (1 < 2)
    {
	    if (4 > 10)
	    {
		return 1;
	    }
	    else {
		return 2;
	    }
    }
    return 0;
}

Compile the example with our compiler (ACC):

./bin/acc example.c -o example.s

The assembly generated is:

	.text
	.global main
main:
	mov r0, #1
	mov r1, #2
	cmp r0, r1
	bge if_else_0
	mov r0, #4
	mov r1, #10
	cmp r0, r1
	ble if_else_1
	mov r0, #1
	bx lr
if_else_1:
	mov r0, #2
	bx lr
if_else_0:
	mov r0, #0
	bx lr

Use GCC to translate that assembly into a executable binary:

gcc example.s -o example

Execute and check the result:

$ ./example
$ echo $?
2

Future

The next step will be to actually be able to generate any possible boolean expression. Currently, the AST can recognise complex boolean expressions, but the Generator is not able to handle them. I have to study and research more about this topic in the ARM architecture. After that, I plan on adding variables (integers).

Linux Kernel Rickroll Module

Sat, 19 Mar 2016 13:50:46 +0200

Introduction

NOTE: The code used to replace the user’s path with the one provided is BAD, never change user’s pointer content unless he/she is expecting that to happen. Don’t do that at home kids

I decided to explain the basics of a Linux Kernel Module with humor. I am not saying this is a good idea for April’s fool, but it is quite close ;)

This module shares some ideas with the post about LD_PRELOAD, but this time it is not to defend ourselves. The module will replace the open syscall for our own where it will detect if we are opening an mp3 or a jpg file. This idea was taken from this talk of Julia Evans.

As always, all the code used in this post is available in this repo.

Skeleton of a Linux Kernel Module

The following code is the skeleton of a Linux Kernel Module

#include<linux/module.h>
#include<linux/init.h>

MODULE_AUTHOR("Oscar Forner Martinez");
MODULE_LICENSE("GPL v2");

static int __init my_init(void)
{
  pr_info("This will be shown when the module is loaded\n");
  return 0;
}

static void __exit my_exit(void)
{
  pr_info("This will be shown when the module is removed\n");
}

module_init(my_init);
module_exit(my_exit);

I think the code above is quite self-explanatory.

What does this Module do?

As said before, this module is aimed to replace the current open syscall by ours, that will detect if the file we are trying to open is an mp3 or a jpg file and it will substitute the files by the ones provided when the module is loaded.

Code explained

This could seem a bit overwhelming, but let’s go through it.

#include <linux/module.h>
#include <linux/moduleparam.h>
#include <linux/init.h>
#include <linux/kallsyms.h>
#include <asm/unistd.h>
#include <linux/uaccess.h>

MODULE_AUTHOR("Oscar Forner Martinez");
MODULE_LICENSE("GPL v2");
MODULE_DESCRIPTION("This module's aim is to replace the open syscall to our own open syscall. Our own open syscall will replace the opening of all *.mp3 files by the file provided as a parameter (song). Moreover it will replace all *.jpg files by the file provided as a parameter (pict).");

static char *song = NULL;
static char *pict = NULL;

module_param(song, charp, 0000);
MODULE_PARM_DESC(song, "Path to the song to open always.");

module_param(pict, charp, 0000);
MODULE_PARM_DESC(pict, "Path to the picture to open always.");

static void **sys_call_table = NULL;

static asmlinkage long (*old_open) (const char __user *filename, int flags, umode_t mode);

static asmlinkage long rick_open(const char __user *filename, int flags, umode_t mode)
{
        size_t len = strlen(filename);
        const char *ext = filename + len - 4;
        if (strncmp(ext, ".mp3", 4) == 0 && song != NULL)
        {
                copy_to_user((void *)filename, song, len);
        }
        if (strncmp(ext, ".jpg", 4) == 0 && pict != NULL)
        {
                copy_to_user((void *)filename, pict, len);
        }
        pr_info("Let's play!\n");
        return old_open(filename, flags, mode);
}

static int __init set_rick(void)
{
        if (song == NULL && pict == NULL)
        {
                pr_info("Rickroll module not loaded. You need to provide a song and/or a picture as a parameter\n");
                return -1;
        }
        sys_call_table = (void **)kallsyms_lookup_name("sys_call_table");
        pr_info("Found sys_call_table at %p\n", sys_call_table);
        old_open = sys_call_table[__NR_open];
        sys_call_table[__NR_open] = rick_open;
        pr_info("Old open: %p; Rick open: %p\n", old_open, rick_open);
        return 0;
}

static void __exit unset_rick(void)
{
        pr_info("Party is over :(\n");
        sys_call_table[__NR_open] = old_open;
}

module_init(set_rick);
module_exit(unset_rick);

Parameters provided to the module are declared and registered as follows:

static char *song = NULL;
static char *pict = NULL;

module_param(song, charp, 0000);
MODULE_PARM_DESC(song, "Path to the song to open always.");

module_param(pict, charp, 0000);
MODULE_PARM_DESC(pict, "Path to the picture to open always.");

Pointer to the old syscall and declaration of the new one. Moreover, we can see the method copy_to_user, this is to copy information from the Kernel Space to User Space.

static asmlinkage long (*old_open) (const char __user *filename, int flags, umode_t mode);

static asmlinkage long rick_open(const char __user *filename, int flags, umode_t mode)
{
        size_t len = strlen(filename);
        const char *ext = filename + len - 4;
        if (strncmp(ext, ".mp3", 4) == 0 && song != NULL)
        {
                copy_to_user((void *)filename, song, len);
        }
        if (strncmp(ext, ".jpg", 4) == 0 && pict != NULL)
        {
                copy_to_user((void *)filename, pict, len);
        }
        pr_info("Let's play!\n");
        return old_open(filename, flags, mode);
}

The most outstanding part of the following piece of code is the use of kallsyms_lookup_name function. We will use it to locate the address of the sys_call_table.

static int __init set_rick(void)
{
        if (song == NULL && pict == NULL)
        {
                pr_info("Rickroll module not loaded. You need to provide a song and/or a picture as a parameter\n");
                return -1;
        }
        sys_call_table = (void **)kallsyms_lookup_name("sys_call_table");
        pr_info("Found sys_call_table at %p\n", sys_call_table);
        old_open = sys_call_table[__NR_open];
        sys_call_table[__NR_open] = rick_open;
        pr_info("Old open: %p; Rick open: %p\n", old_open, rick_open);
        return 0;
}

static void __exit unset_rick(void)
{
        pr_info("Party is over :(\n");
        sys_call_table[__NR_open] = old_open;
}

Note: the sys_call_table is in read-only mode by default. To be able to write on it, the kernel running must have been compiled with the flag CONFIG_DEBUG_RODATA not set.

Example of how it works

Conclusion

This is a great and funny example of the power of the Linux Kernel Modules. I will write about more advanced examples in the future. Moreover, I will keep on trying to get a patch accepted in the main tree :D

Prefix Tree comparison

Fri, 26 Feb 2016 13:50:46 +0200

Introduction

In this post I talk about three data structures I implemented to compare their performance in different scenarios. The three data structures are Trie, Ternary Search Tree and Radix Tree.

All the code for the data structures as well as the tested scenarios are available in this repo in GitHub.

What is a Prefix Tree?

It is an ordered tree data structure that is used to store a dynamic set of elements where the key is a string. All the data structures that I implemented have the following methods:

Operation	Description
clear	Removes all the content of the data structure
find	Returns the value stored in the data structure for a specific key provided
insert	Adds a new pair of key and value to the data structure
size	Returns the amount of elements stored in the data structure
show	Prints the content of the data structure in the dot format
erase	Removes the value stored in the data structure for a specific key provided
contains	Returns true if there is a value associated to the key provided
keys	Returns a std::vectorstd::string with all the keys in the data structure that have the provided prefix. If no prefix is provided, it returns a std::vectorstd::string with all the keys in the data structure.
lcp	Returns the longest common prefix of all keys stored in the data structure.

Trie

Unlike a binary search tree, none of the nodes in the tree stores the key associated with that node. Instead, its position in the tree defines the key with which it is associated. All the descendants of a node have a common prefix of the string associated with that node, and the root is associated with the empty string. Values are not necessarily associated with every node. Rather, values tend only to be associated with leaves, and with some inner nodes that correspond to keys of interest.

Example of a Trie that contains the keys: “He”, “Hello”, “Kthulu”, “No”, “Wololo”, “World”, “Worst”, “he”:

Ternary Search Tree

In a Ternary Search Tree nodes are arranged in a manner similar to a binary search tree, but with up to three children rather than the binary tree’s limit of two. Like other prefix trees, a ternary search tree can be used as an associative map structure with the ability for incremental string search. However, ternary search trees are more space efficient compared to standard prefix trees, at the cost of speed.

Example of a Ternary Search Tree that contains the keys: “He”, “Hello”, “Kthulu”, “No”, “Wololo”, “World”, “Worst”, “he”:

Radix Tree

Radix Tree is a space-optimized Trie in which each node that is the only child is merged with its parent. The result is that the number of children of every internal node is at least the radix r of the radix trie, where r is a positive integer and a power x of 2, having x >= 1. Unlike in regular tries, edges can be labeled with sequences of elements as well as single elements. This makes radix trees much more efficient for small sets (especially if the strings are long) and for sets of strings that share long prefixes.

Example of a Radix Tree that contains the keys: “He”, “Hello”, “Kthulu”, “No”, “Wololo”, “World”, “Worst”, “he”:

Comparison

In this section I show a comparison of the performance for the three data structures in different scenarios. Note: sometimes the Trie runs out of memory. Therefore, there is no data on these cases.

Insert

Search

Found

Not found

Erase

Found

Not found

Get keys

Get keys prefix

Longest common path

Conclusion

The Radix Tree is in average the best of the three data structures. Nevertheless, I found out a paper where it explains how to implement a Balanced Ternary Search Tree, because one of the weakness of the Ternary Search Tree is that its shape depends of the order of the insertions. I have plenty of work to do regarding this topic but I wanted to share with you some of the preliminar results. Anyway, I am working on writing a paper explaining with more detail this experiment and the way I conducted the expertiment.

FFF a mocking framework for C

Fri, 19 Feb 2016 13:50:46 +0200

Introduction

This post is a continuation from a previous post called Unity; unit test for C, but in this post we are going to use FFF.

FFF is one of the available mocking frameworks for C. In this example I will use CMake to configure the project and build it.

All the code and configuration files used in this post are available in this repo in GitHub.

Why do we need to mock functions?

To answer that question I will introduce the signature of the two methods I will use during this post.

The first function is called modulo and it will return 0 if the given number is even, 1 if it is an odd number:

#ifndef MODULO_H
#define MODULO_H
int modulo(int value);
#endif //MODULO_H

The second function is called both_even and it will return 0 if both numbers are even, otherwise:

#ifndef BOTH_H
#define BOTH_H
int both_even(int a, int b);
#endif //BOTH_H

Finally, the implementation of the both_even is the following:

#include "both.h"
#include "modulo.h"

int both_even(int a, int b)
{
	int a_m, b_m;
	a_m = modulo(a);
	b_m = modulo(b);
	return a_m == 0 && b_m == 0;
}

In this case we will use the modulo function to calculate both_even, but we do not have the implementation of modulo. This is intended because we do not want to use the actual code of the modulo function. We want to test only the code in both_even. Therefore, the modulo function will be mocked.

Unit test with FFF

The following code is an example of 6 unit tests written using the FFF framework:

#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <string.h>
#include "both.h"
#include "modulo.h"
#include "fff.h"
DEFINE_FFF_GLOBALS
#define TEST_F(SUITE, NAME) void NAME()
#define RUN_TEST(SUITE, TESTNAME) printf(" Running %s.%s: \n", #SUITE, #TESTNAME); setup(); TESTNAME(); printf(" SUCCESS\n");

FAKE_VALUE_FUNC1(int, modulo, int);

void setup()
{
	RESET_FAKE(modulo);
}

TEST_F(ModuloTest, both_even_numbers)
{
	int a;
	int b;
	int result;

	// Given
	a = 2;
	b = 4;
	modulo_fake.return_val_seq_len = 2;
	modulo_fake.return_val_seq = (int *) malloc(2*sizeof(int));
	modulo_fake.return_val_seq[0] = 0;
	modulo_fake.return_val_seq[1] = 0;

	// When
	result = both_even(a, b);

	// Then
	assert(result != 0);
}

TEST_F(ModuloTest, first_even_and_second_odd)
{
	int a;
	int b;
	int result;

	// Given
	a = 2;
	b = 3;
	modulo_fake.return_val_seq_len = 2;
	modulo_fake.return_val_seq = (int *) malloc(2*sizeof(int));
	modulo_fake.return_val_seq[0] = 0;
	modulo_fake.return_val_seq[1] = 1;

	// When
	result = both_even(a, b);

	// Then
	assert(result != 1);
}

TEST_F(ModuloTest, first_odd_and_second_even)
{
	int a;
	int b;
	int result;

	// Given
	a = 3;
	b = 2;
	modulo_fake.return_val_seq_len = 2;
	modulo_fake.return_val_seq = (int *) malloc(2*sizeof(int));
	modulo_fake.return_val_seq[0] = 1;
	modulo_fake.return_val_seq[1] = 0;

	// When
	result = both_even(a, b);

	// Then
	assert(result != 1);
}


TEST_F(ModuloTest, both_odd_numbers)
{
	int a;
	int b;
	int result;

	// Given
	a = 3;
	b = 5;
	modulo_fake.return_val_seq_len = 2;
	modulo_fake.return_val_seq = (int *) malloc(2*sizeof(int));
	modulo_fake.return_val_seq[0] = 1;
	modulo_fake.return_val_seq[1] = 1;

	// When
	result = both_even(a, b);

	// Then
	assert(result != 1);
}

TEST_F(ModuloTest, one_even_and_other_zero)
{
	int a;
	int b;
	int result;

	// Given
	a = 2;
	b = 0;
	modulo_fake.return_val_seq_len = 2;
	modulo_fake.return_val_seq = (int *) malloc(2*sizeof(int));
	modulo_fake.return_val_seq[0] = 0;
	modulo_fake.return_val_seq[1] = 0;

	// When
	result = both_even(a, b);

	// Then
	assert(result != 0);
}

TEST_F(ModuloTest, one_odd_and_other_zero)
{
	int a;
	int b;
	int result;

	// Given
	a = 0;
	b = 5;
	modulo_fake.return_val_seq_len = 2;
	modulo_fake.return_val_seq = (int *) malloc(2*sizeof(int));
	modulo_fake.return_val_seq[0] = 0;
	modulo_fake.return_val_seq[1] = 1;

	// When
	result = both_even(a, b);

	// Then
	assert(result != 1);
}

int main(void)
{
	RUN_TEST(ModuloTest, both_odd_numbers);
	RUN_TEST(ModuloTest, both_even_numbers);
	RUN_TEST(ModuloTest, first_even_and_second_odd);
	RUN_TEST(ModuloTest, first_odd_and_second_even);
	RUN_TEST(ModuloTest, one_odd_and_other_zero);
	RUN_TEST(ModuloTest, one_even_and_other_zero);

	return 0;
}

On the one hand, the beginning of the code is the initialization of the FFF framework and a couple of definitions that will come in handy:

DEFINE_FFF_GLOBALS
#define TEST_F(SUITE, NAME) void NAME()
#define RUN_TEST(SUITE, TESTNAME) printf(" Running %s.%s: \n", #SUITE, #TESTNAME); setup(); TESTNAME(); printf(" SUCCESS\n");

However, the interesting part is:

FAKE_VALUE_FUNC1(int, modulo, int);

The macro FAKE_VALUE_FUNC1 receives three parameters, first the type of the return, second the name of the function to mock and third the type of the parameter. You can find all the information about this in the documentation.

On the other hand, each test looks as follows:

TEST_F(ModuloTest, both_even_numbers)
{
	int a;
	int b;
	int result;

	// Given
	a = 2;
	b = 4;
	modulo_fake.return_val_seq_len = 2;
	modulo_fake.return_val_seq = (int *) malloc(2*sizeof(int));
	modulo_fake.return_val_seq[0] = 0;
	modulo_fake.return_val_seq[1] = 0;

	// When
	result = both_even(a, b);

	// Then
	assert(result != 0);
}

And the interesting piece is under the Given part. Note that we are configuring the modulo_fake struct that contains the information that the modulo function will return. Actually, we are saying that it will be called twice and which will be the return values.

The execution of the test will be the following:

$ ./run_test 
 Running ModuloTest.both_odd_numbers: 
 SUCCESS
 Running ModuloTest.both_even_numbers: 
 SUCCESS
 Running ModuloTest.first_even_and_second_odd: 
 SUCCESS
 Running ModuloTest.first_odd_and_second_even: 
 SUCCESS
 Running ModuloTest.one_odd_and_other_zero: 
 SUCCESS
 Running ModuloTest.one_even_and_other_zero: 
SUCCESS

Conclusion

FFF, or any mocking framework, is an important tool for developers because it allows to create unit tests for interoperability of different functions. Furthermore, you can create unit tests that are focused on a single function without worring if the functions to which it depends are working properly.

ARM C Compiler (ACC) I

Fri, 12 Feb 2016 13:50:46 +0200

Introduction

Why do you want to create your own compiler? To answer this question I have to give you some background. For Christmas I got a BeagleBone Black, perfect to learn ARM assembly. After a few weeks of doing the usual stuff I decided I wanted a bigger project to improve my knowledge. However, the idea to start the development of a compiler comes from http://www.sigbus.info/how-i-wrote-a-self-hosting-c-compiler-in-40-days.html.

The only aim of this project is to improve my knowledge of the ARM assembly, the C language and Compilers. The project can be found in GitHub.

What is the target of the project?

My target is to have a self-hosting compiler of C. For this reason I want to keep it simple. Moreover, I will only develop the features I need to accomplish the target of the project. As you can imagine the choosen language is C (ANSI C to be specific).

I decided not to use Flex, Bison, Lex or Yacc because I want to work in all stages of the compiler. Yes, it sounds crazy and maybe I will regret it in the future, but at least I will try it.

What is the current state of the project?

Currently, I have implemented a very basic compiler, this version is available in GitHub as v0.1. It just generates code for a file containing the main function without parameters and the body only contains a return statement of a positive integer.

Structure of the project

The project contains three folders:

inc: holds the external libraries needed. Right now, it has the files of Unity for unit testing.
unittest: has the unit tests for the project. It has just one, but once I find a good framework to do mocks in C, it will have more unit tests.
src: contains the actual source code of the project.

This compiler is made of a Lexer that creates Tokens. These Tokens are used by the Grammar to create AST nodes. Finally, the AST nodes will be used by the Generator to generate the assembly.

Lexer and Tokens

The available Tokens are: int_value, int_type, function, open_parenthesis, close_parenthesis, open_bracket, close_bracket, return, semicolon and end_of_file. These can be found in the file tokens.h:

#ifndef TOKEN_H
#define TOKEN_H

#include <stdlib.h>

enum token_type {
	T_INT_TYPE,
	T_INT_VALUE,
	T_FUNCTION,
	T_OPAR,
	T_CPAR,
	T_OBRA,
	T_CBRA,
	T_RETURN,
	T_SEMICOLON,
	T_END_OF_FILE
};

/*
 * Tokens for the parser
 */
typedef struct token_base
{
	enum token_type type;
	void * token_pointer;
} token_base;

typedef struct token_int_type
{
	token_base base;
} token_int_type;

typedef struct token_int_value
{
	token_base base;
	int value;
} token_int_value;

typedef struct token_function
{
	token_base base;
	char * name;
} token_function;

typedef struct token_opar
{
	token_base base;
} token_opar;

typedef struct token_cpar
{
	token_base base;
} token_cpar;

typedef struct token_obra
{
	token_base base;
} token_obra;

typedef struct token_cbra
{
	token_base base;
} token_cbra;

typedef struct token_return
{
	token_base base;
} token_return;

typedef struct token_semicolon
{
	token_base base;
} token_semicolon;

typedef struct token_eof
{
	token_base base;
} token_eof;

/*
 * Init functions for the tokens
 */
void init_token_int_type(token_int_type * token);
void init_token_int_value(token_int_value * token, int  value);
void init_token_function(token_function * token, char * name);
void init_token_opar(token_opar * token);
void init_token_cpar(token_cpar * token);
void init_token_obra(token_obra * token);
void init_token_cbra(token_cbra * token);
void init_token_return(token_return * token);
void init_token_semicolon(token_semicolon * token);
void init_token_eof(token_eof * token);

/*
 * Release functions for the tokens
 */
void free_token_int_type(token_int_type * token);
void free_token_int_value(token_int_value * token);
void free_token_function(token_function * token);
void free_token_opar(token_opar * token);
void free_token_cpar(token_cpar * token);
void free_token_obra(token_obra * token);
void free_token_cbra(token_cbra * token);
void free_token_return(token_return * token);
void free_token_semicolon(token_semicolon * token);
void free_token_eof(token_eof * token);
#endif //TOKEN_H

These Tokens are returned by the method next from the Lexer in the file lexer.h:

#ifndef LEXER_H
#define LEXER_H

#include <stdio.h>
#include "token.h"

typedef struct lexer
{
	FILE * f;
} lexer;

void init_lexer(lexer * l, const char * filename);
void destroy_lexer(lexer * l);
struct token_base * next(lexer * l);

#endif //LEXER_H

Grammar and AST nodes

The available AST nodes are: id, int, function and return. These can be found in the file ast.h:

#ifndef AST_H
#define AST_H

#include <stdlib.h>

enum ast_type {
	A_ID,
	A_INT,
	A_FUNCTION,
	A_RETURN
};

/*
 * AST nodes
 */
typedef struct ast_base
{
	enum ast_type type;
	void * ast_pointer;
} ast_base;

typedef struct node_id
{
	ast_base base;
	char * value;
} node_id;

typedef struct node_int
{
	ast_base base;
	int value;
} node_int;

typedef struct node_function
{
	ast_base base;
	char * name;
	ast_base * entry_point;
} node_function;

typedef struct node_return
{
	ast_base base;
	ast_base * value;
} node_return;

/*
 * Init functions for the AST nodes
 */
void init_node_id(node_id * node, char * value);
void init_node_int(node_int * node, int value);
void init_node_function(node_function * node, char * name, ast_base * entry_point);
void init_node_return(node_return * node, ast_base * value);

/*
 * Release functions for the AST nodes
 */
void free_node_id(node_id * node);
void free_node_int(node_int * node);
void free_node_function(node_function * node);
void free_node_return(node_return * node);
#endif //AST_H

The AST is returned by the method build_ast from the Grammar in the file grammar.h:

#ifndef GRAMMAR_H
#define GRAMMAR_H

#include <stdio.h>
#include "lexer.h"
#include "ast.h"

typedef struct grammar
{
	lexer * l;
} grammar;

void init_grammar(grammar * g, lexer * l);
ast_base * build_ast(grammar * g);
void destroy_grammar(grammar * g);


/*
 * Read functions to build AST parts
 */
ast_base * read_function_ast_node(grammar * g);
ast_base * read_function_body(grammar * g);
#endif //GRAMMAR_H

Generation of assembly

The Generator has a Grammar that will be used to generate the assembly with the method generate_code:

#ifndef GENERATOR_H
#define GENERATOR_H

#include <stdio.h>
#include "grammar.h"

typedef struct generator
{
	grammar * g;
	FILE * f;
} generator;

// API
void init_generator(generator * gen, grammar * gra, const char * out);
void destroy_generator(generator * g);
void generate_code(generator * g);

// Internals
void __generate_code(generator * g, ast_base * ast);
void __generate_code_for_main(generator * g, ast_base * ast);
void __generate_code_for_function(generator * g, node_function * ast);
void __generate_code_for_return(generator * g, node_return * ast);
void __generate_code_for_int(generator * g, node_int * ast);

#endif //GENERATOR_H

Example of the current functionality

The code to be compiled into ARM assembly is:

int main()
{
	return 2;
}

Compile the example with our compiler (ACC):

./bin/acc example.c -o example.s

The assembly generated is:

	.text
	.global main
main:
	mov r0, #2
	bx lr

Use GCC to translate that assembly into a executable binary:

gcc example.s -o example

Execute and check the result:

$ ./example
$ echo $?
2

Future

As you can see this project is going to take a long time to be completed. My plan is to work adding small features at a time. The next step is to add conditionals (if and else). Afterwards, I will add operators such as <, <=, >, >=, == and !=. Everytime I achieve a new goal in this project I will create a post like this one to show the feature and how it has been implemented.

Unity: unit test for C

Fri, 05 Feb 2016 13:50:46 +0200

Introduction

Unity is one of the available frameworks to create unit tests for C. In this example, I will use CMake to configure the project and build.

All the code and configuration files used in this post are available in this repo in GitHub.

Can you do unit test in C? And what is Unity?

Some people do not know you can do unit tests in C and people must do unit tests in any language. My choice to do unit tests in C is Unity for several reasons:

No need to install any package in your distro. Just add the three files to your project and start using it.
Great documentation and examples in the repo and their website.
Awesome course about unit test in embedded systems on Udemy. I did it myself, totally worth it if you are interested in these topics.

Structure of the project

In the root of the project we have the CMakeLists.txt file and three folders: lib, src and unittest.

lib: Contains the Unity files.
src: Contains the code we want to test. In this case, a functions header file.
unittest: Contains the test we want to run.

$ ls -R
.:
CMakeLists.txt  lib  src  unittest

./lib:
unity.c  unity.h  unity_internals.h

./src:
functions.h

./unittest:
test_functions.c

The project to test

The configuration will be done in the CMakeLists.txt file:

project(Unity_example)
cmake_minimum_required(VERSION 2.8)

INCLUDE_DIRECTORIES(src)
INCLUDE_DIRECTORIES(lib)
ADD_EXECUTABLE(run_test unittest/test_functions.c lib/unity.c)

The code we want to test is:

int sum(int a, int b)
{
        return a + b;
}

Finally, the unit test:

#include "functions.h"
#include "unity.h"

void test_two_plus_two(void)
{
        TEST_ASSERT_EQUAL_INT(4, sum(2, 2));
}

void test_minus_two_plus_two(void)
{
        TEST_ASSERT_EQUAL_INT(0, sum(-2, 2));
}

int main(void)
{
        UNITY_BEGIN();
        RUN_TEST(test_two_plus_two);
        RUN_TEST(test_minus_two_plus_two);
        return UNITY_END();
}

As you can see in the code above, you need a main in the unit test file that contains: UNITY_BEGIN and UNITY_END calls and a RUN_TEST call for each test we want to run.

Conclusion

Everybody should be doing unit tests, it does not matter what language do they use. I do use this framework among others (that I will introduce in future posts) for testing C applications. Unity is a must have tool for any C developer, even for embedded software. You can check it in their website and course.

Google Mock for C++ testing

Fri, 22 Jan 2016 13:50:46 +0200

Introduction

This post is a continuation from a previous post called Unit test with Google Test for C++, but in this post we are going to use Google Mock, that extends the functionality of Google Test.

Google Mock is one of the available frameworks for C++ to mock objects in unit tests. In this example I will use the same technologies as for the previous one: CMake to configure the project and build it, for the dependency manager I will use the new and shiny conan.

All the code and configuration files used in this post are available in this repo in GitHub.

Why do we need to mock objects?

To answer that question I will introduce the two classes I will use during this post.

The first class is called Producer and it will extract the domain of a given URL:

#ifndef PRODUCER_H
#define PRODUCER_H

#include <string>

class Producer {
public:
	virtual ~Producer(){}
	virtual std::string getDomainFromUrl(const std::string & url) const = 0;
};

#endif /*PRODUCER_H*/

The second class is called Consumer and it will calculate the level of the domain of a given URL:

#ifndef CONSUMER_H
#define CONSUMER_H

#include <string>
#include <algorithm>
#include "producer.h"

class Consumer {
public:
	Consumer(Producer * p): producer(p){}
	int countLevelOfDomain(const std::string & url) const {
		std::string domain = producer->getDomainFromUrl(url);
		return std::count(domain.begin(), domain.end(), '.') + 1;
	}
private:
	Producer * producer;
};

#endif /*CONSUMER_H*/

In this case we want to create a unit test for the Consumer class, but we need to implement a Producer class for that, right? Well, no. We do not need an implementation of the Producer class because we can mock it. Moreover, in that case the unit test would not be only for the Consumer class, but for the Producer too, and that is not what we are looking for.

Unit test with Google Mock

The following unit test is written using the Google Mock framework:

#include <string>
#include <gtest/gtest.h>
#include <gmock/gmock.h>
#include "producer.h"
#include "consumer.h"

class ProducerMock : public Producer {
public:
	MOCK_CONST_METHOD1( getDomainFromUrl, std::string(const std::string & url) );
};

TEST(Consumer, CalculateDomainLevel) {
	const std::string url = "http://www.fantasticdomain.com/site/index.html";
	std::string domain = "fantasticdomain.com";
	ProducerMock mock;
	Consumer consumer(&mock);
	EXPECT_CALL(mock, getDomainFromUrl(url))
		    .WillOnce(::testing::Return(domain));
	int domainLevel = consumer.countLevelOfDomain(url);
	EXPECT_EQ(domainLevel, 2);
}

int main(int argc, char **argv) {
	testing::InitGoogleMock(&argc, argv);
	return RUN_ALL_TESTS();
}

On the one hand, the ProducerMock class is inheriting from the Producer class. However, the interesting part is

MOCK_CONST_METHOD1( getDomainFromUrl, std::string(const std::string & url) );

The line above is telling Google Mock to mock the method getDomainFromUrl from the Producer class and it will receive a const std::string by reference as parameter and it will return a std::string by value.

On the other hand, there is the EXPECT_CALL macro that receives two parameters:

EXPECT_CALL(mock, getDomainFromUrl(url))
                   .WillOnce(::testing::Return(domain));

The first parameter is the mocked object and second one is the method that will be called and the parameter that will be passed. The result of the expansion of the macro will call a method called WillOnce that will make the mocked object answer once to the method. Finally, ::testin::Return(domain) specifies the return of that call, which will be domain.

The execution of the test will be the following:

$ ./bin/run_test 
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from Consumer
[ RUN      ] Consumer.CalculateDomainLevel
[       OK ] Consumer.CalculateDomainLevel (0 ms)
[----------] 1 test from Consumer (0 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (0 ms total)
[ PASSED ] 1 test.

Conclusion

Google Mock, or any mocking framework, is an important tool for developers because it allows to create unit tests for interoperability of different objects. Furthermore, you can create unit tests that are focused on a single class without worring if the class to which it depends is working properly.

Clang tools

Thu, 21 Jan 2016 13:50:46 +0200

Introduction

Clang is a compiler front end for the C, C++, Objective-C and Objective-C++ programming languages. It uses LLVM as its back end. There are also several awesome tools build on top of Clang and I am going to show the three of them I use the most.

As usual, I am working from an Arch Linux computer. Therefore, I can install Clang and the tools from the repository (clang, clang-tools-extra). For other distributions you can find the information in the documentation.

As always, all the code used in this post is available in this repo.

The videos are made with asciinema, that means you can copy from the video.

Clang-format

This tool is perfect to make sure that you are following a specific code style and it is fully configurable.

It can integrated in your editor or IDE. In my case, I have it integrated in my Vim configuration. You can see how it works in the following video:

Or it can be used from the command line. The original state of the code is:

#include <iostream>
#include <vector>

int main(void) {
        std::vector<int> v = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};
  for (std::vector<int>::iterator it = v.begin(); it != v.end(); ++it) {
                std::cout << *it << std::endl;}
return 0;
}

After formatting:

#include <iostream>
#include <vector>

int main(void) {
  std::vector<int> v = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};
  for (std::vector<int>::iterator it = v.begin(); it != v.end(); ++it) {
    std::cout << *it << std::endl;
  }
  return 0;
}

Clang-tidy

This tool is really useful to have it integrated with your version control system. I like to have it configurated to run as a pre-commit hook. In case it detects any problem the commit fails. Therefore, you only commit code that has been aproved by the tool. As I use CMake I have to add a definition to generate the json file needed from the tool.

cmake . -DCMAKE_EXPORT_COMPILE_COMMANDS=ON

After that the code can be analyzed for all checks as follows:

$ clang-tidy -checks='*' src/tidy.cpp 
1811 warnings generated.
/home/maitesin/workspace/blog/clang_tools_2016_01_29/tidy_example/src/tidy.cpp:5:16: warning: redundant boolean literal supplied to boolean operator [readability-simplify-boolean-expr]
        if (isGood == true) {
                      ^
            isGood
Suppressed 1810 warnings (1810 in non-user code).
Use -header-filter='.*' to display errors from all non-system headers.

You can deal on your own with these problems, or let it fix them for you using the -fix flag.

Clang-modernize

Last but not least, this tool is made to update old code to C++11. This tool is not bullet proof, but it can make your job easier if you have to migrate the old code to newer standards. It has an option called -risk that can have three values: safe, reasonable or risky, by default it uses reasonable.

Old code:

#include <iostream>
#include <vector>

int main(void) {
        int hardcoded[] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};
        std::vector<int> v (hardcoded, hardcoded + sizeof(hardcoded) /
                            sizeof(int));
        for (std::vector<int>::iterator it = v.begin(); it != v.end(); ++it) {
                std::cout << *it << std::endl;
        }
        return 0;
}

Modern code:

#include <iostream>
#include <vector>

int main(void) {
        int hardcoded[] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};
        std::vector<int> v (hardcoded, hardcoded + sizeof(hardcoded) /
                            sizeof(int));
        for (auto & elem : v) {
                std::cout << elem << std::endl;
        }
        return 0;
}

Conclusion

These three tools make your live easier as a developer. You will be sure you code is following the code style from your company/project, it will ensure that you are committing code that is using the new features from C++11 and it will help you migrate the old code to use the features from C++11. I strongly recommend to include them in your workflow, because it will boost your work performance.

NOTE: This post is based on Clang version 3.7.1. In version 3.8, clang-tidy will include some checks from clang-modernize. So, it will enforce you to use features from C++11.

rr debugger, an improvement over GDB

Fri, 15 Jan 2016 13:50:46 +0200

Introduction

rr is a debugging tool from Mozilla that enhances the behaviour of GDB. It can be found in GitHub, but my recomendation is to go first to the website of the project. In the website you can find really useful information and documentation.

As usual, I am working from an Arch Linux computer. Therefore, I can install rr from the AUR repository (rr). For other distributions you can find the information in the documentation.

As always, all the code used in this post is available in this repo.

The videos are made with asciinema, that means you can copy from the video.

Before starting debugging

It is recommended to have your CPU frequency governor in ‘performance’ mode instead of ‘powersave’. Because of that, it is recommended to set it:

sudo cpufreq-set -g performance

To set it back to ‘powersave’ mode just run:

sudo cpufreq-set -g powersave

How rr works

rr works in two phases. In the first one, it records the execution of a program. In the second one, it replays the execution of that program as many times as you need. Moreover, during the replay phase you can go forward and backward in the execution.

rr adds some new commands to GDB such as reverse-next or reverse-continue. I think these commands do not need more explanation.

Example of usage

In this example I will use the following code:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char * argv[]) {

	if (argc != 2) {
		fprintf(stderr, "Error: Bad number of parameters\n");
		fprintf(stderr, "Usage: %s <how many times>", argv[0]);
		return -1;
	}

	int num = atoi(argv[1]);
	int input = 0;
	int acum = 0;
	for (int i = 0; i < num; ++i) {
		printf("Next one: ");
		scanf("%d", &input);
		acum += input;
	}

	printf("Total acumulated: %d\n", acum);
	return 0;
}

It will be compiled with the -g to produce debugging information as you would do for GDB:

gcc main.c -g -o app

Record of the execution

In the following video it will be shown how to record the execution of an application.

Replay of the execution

In the folowwing video it will be shown how to replay the execution of an application.

Conclusion

Personally, I consider rr a must have tool for debugging. The ability to go backwards in the execution at any time is extremely useful when you are trying to find the exact moment where something starts to go wrong. Moreover, the fact that you do not have to keep providing any kind of input to the app is really good to focus just in the execution.

Cling a C++ interpreter

Fri, 08 Jan 2016 13:50:46 +0200

Introduction

Cling is an interactive C++ interpreter, built on top of Clang and LLVM compiler infrastructure. It can be found in GitHub (Note: Lately there has not been that much development on it).

Since I use Arch Linux I am lucky to have cling in the AUR repository (cling-git). For other ones you can use the cling-all-in-one repo that contains a script to download all dependencies, compile and build cling for you.

As always, all the code used in this post is available in this repo

The videos are made with asciinema, that means you can copy from the video.

The basics of Cling

In this section I will show the basic commands we can run on cling besides C/C++ code.

Command	Explanation
.I path	to add includes. Like -I option at compiling time
.L file	to load file. This is important to try classes or functions that you already have
.rawInput	Toggle wrapping and printing the execution results of the input
.help	Shows a nice help with these and other commands that can be used

Examples of usage

In this section examples of usage using the basics from the previous section will be shown.

Load an existing file with a function we want to test

In this first example the following header will be loaded to use it:

//Extremely useful function
bool is_greater(int a, int b) {
        return a > b;
}

The header above can be used from Cling as shown in the following video:

Include a folder and load a template class from that folder

This example will include a folder that contains a class that will be loaded from Cling:

template <typename T>
class MyClass {
public:
        MyClass(const T & t) : total(t) {}
        T inc();
        T dec();
private:
        T total;
};

#include "my_class.h"

template <typename T>
T MyClass<T>::inc() {
        return ++total;
}

template <typename T>
T MyClass<T>::dec() {
        return --total;
}

The class above can be loaded from Cling as shown in the following video:

Write a function on-the-fly

Cling allows us to write functions or classes on-the-fly:

Conclusion

I do not consider Cling as my main way to work in C/C++, but it is my first option when I want to try something quick in a function or a class that I am interested in using. It is not a perfect tool for everything, but it is faster than setting up a whole example when you are only interested in something specific.

LD_PRELOAD to fix unsecure libraries

Fri, 01 Jan 2016 13:50:46 +0200

Introduction

I have seen LD_PRELOAD used in several cases. From using it to allow programs that link to a newer version of the libstdc++, to cracks for applications that hijack some calls and provide the expected result to tell the application they have a valid license.

The aim of this post is to show how to find these dangerous calls in applications that you are running which you cannot fix (i.e. you do not have access to the source code). Imagine that one of these applications uses the library call strcpy, as we know that call is dangerous and we have more secure alternatives such as strncpy.

All the code used in this post is available in this repo

As an example of these applications I will use the following code:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>


int main(int argc, char *argv[])
{
	if (argc <= 1)
	{
		printf("Bad number of parameters.\nUsage: %s <string up to 512 characters>\n", argv[0]);
		return 1;
	}
	char s[512];
	strcpy(s, argv[1]);
	printf("You introduced: %s\n", s);
	return 0;
}

An example of its execution is the following:

$ ./app Wololo
You introduced: Wololo

As you can imagine, we will have problems if the user gives a string longer than 511 characters to the application because it will override more than just the string. Actually, it can even be an entry point for an exploit (it depends on the compiling flags).

$ ./app $(perl -e 'print "A"x1000')
You introduced: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
[1] 6625 segmentation fault (core dumped) ./app $(perl -e 'print "A"x1000')

Find which dangerous calls are happenning in the application

To help us in this task we will use ltrace. Remember, we do not have access to the code.

$ ltrace ./app Wololo
__libc_start_main(0x4005f6, 2, 0x7ffc587856b8, 0x4006a0 <unfinished ...>
strcpy(0x7ffc587853c0, "Wololo")                                        = 0x7ffc587853c0
printf("You introduced: %s\n", "Wololo"You introduced: Wololo
)                                = 23
+++ exited (status 0) +++

In the output above we can see that the call strcpy is used. It is dangerous, so we want to use the call strncpy instead.

Using a more secure call than strcpy

We can create our own version of the strcpy that actually calls strncpy:

#include<string.h>

char * strcpy(char * s1, const char * s2)
{
        strncpy(s1, s2, 511); //Copy up to the first 511 characters
        s1[511] = '\0'; // Set always the latest character to '\0'
        return s1;
}

Now we have to compile it as a shared object (library to link):

gcc -shared -fPIC our_patch.c -o our_patch.so

Using LD_PRELOAD to call our strcpy

LD_PRELOAD will be used to load out strcpy instead of the one provided by the standard library.

$ LD_PRELOAD=$PWD/our_patch.so ./app $(perl -e 'print "A"x1000')
You introduced: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA�@

Note that it only copies up to the first 511 characters of the string s2.

Conclusion

I want to point out the versatility of the LD_PRELOAD, for example think about how can this help mitigate a 0-day exploit until the code is fixed.

This use of LD_PRELOAD is quite common in competitions like a CTF (in the attach/defense style) where you are provided with a server (with some services running) and you have to keep alive your services as much time as you can, but usually the services are an older version with known issues you need to patch on the fly ;)

Google Test for C++ testing

Wed, 23 Dec 2015 13:50:46 +0200

Introduction

Google Test is one of the available Frameworks to create unit test for C++. In this example I will use CMake to configure the project and build. Furthermore, for the dependency manager I will use the new and shiny conan.

Before starting, why use a dependency manager such as conan or software to configure and build such as CMake? Because these technologies are widely use it in real projects.

All the code and configuration files used in this post are available in this repo in GitHub.

Step 1 Install conan, configure project and gather dependencies

First of all we need to have install conan from pip2 doing:

pip2 install conan

Now with conan installed we do not have to worry about installing Google Test in our system.

Next step will be preparing the conanfile.txt to gather de dependencies:

[requires]
gtest/1.7.0@lasote/stable

[options]
gtest:shared=True

[generators]
cmake

Once we have conan ready we only need to run it to download the dependency and configure them to have them ready for CMake:

conan install .

This will output something similar to:

WARN: Migration: Updating settings.yml with new gcc versions
Requirements
    gtest/1.7.0@lasote/stable
    Package for gtest/1.7.0@lasote/stable in
    /home/maitesin/.conan/data/gtest/1.7.0/lasote/stable/package/ca89189bc59ff53842d6beea76549f289b7b88bd
    Generated conaninfo.txt
Generated conanbuildinfo.cmake

Step 2 configuring CMake

The configuration will be done in the CMakeLists.txt file:

project(Google_test_example)
cmake_minimum_required(VERSION 2.8)

include(conanbuildinfo.cmake)
CONAN_BASIC_SETUP()

INCLUDE_DIRECTORIES(src)
ADD_EXECUTABLE(run_test src/test.cpp)
TARGET_LINK_LIBRARIES(run_test ${CONAN_LIBS})

Step 3 code and unit test

The code will be held in the src folder. It will contain two files: functions.h and test.cpp.

The function(s) we want to test will be in the header file function.h:

int int_addition(int a, int b) {
            int c = a + b;
            return c;
}

The test(s) we want to run will be in the source file test.cpp:

#include "gtest/gtest.h"
#include "functions.h"

TEST(IntAddition, Negative) {
            EXPECT_EQ(-5, int_addition(-2, -3)) << "This will be shown in case it fails";
            EXPECT_EQ(-3, int_addition(5, -8));
}

TEST(IntAddition, Positive) {
            EXPECT_EQ(4, int_addition(1, 3));
            EXPECT_EQ(9, int_addition(4, 5));
}

int main(int argc, char **argv) {
            testing::InitGoogleTest(&argc, argv);
            return RUN_ALL_TESTS();
}

Step 4 putting all together

What is left to do is actually build the project and run the test. In order to do this we need to run:

cmake .

This will output something similar to:

-- The C compiler identification is GNU 5.3.0
-- The CXX compiler identification is GNU 5.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/maitesin/workspace/blog/google_test_2015_12_22

There will be now a Makefile generated from CMake with everything ready to compile and link all the sources and dependencies together.

make

This will output something like:

Scanning dependencies of target run_test
[ 50%] Building CXX object CMakeFiles/run_test.dir/src/test.cpp.o
[100%] Linking CXX executable bin/run_test
[100%] Built target run_test

This will generate an executable in the bin folder, and we will be able to run them with the command:

./bin/run_test

This will result with the following output:

[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from IntAddition
[ RUN      ] IntAddition.Negative
[       OK ] IntAddition.Negative (0 ms)
[ RUN      ] IntAddition.Positive
[       OK ] IntAddition.Positive (0 ms)
[----------] 2 tests from IntAddition (0 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran. (0 ms total)
[ PASSED ] 2 tests.

More advanced example

These are the basics of how to use Google Test to create unit test for your application. In the website of the project there are plenty of more advanced examples.

Finally, if you want to see how is used Google Test in one of my own project you can have a look to the repository tries. For each of the three data structures (Trie, TST and Radix Tree) there are two folders: lib (where the source code of the data structure is stored) and gtest (where the unit test are stored).

ARM C Compiler (ACC)

Mon, 01 Jan 0001 00:00:00 +0000

ACC is a pet project with the purpose of improving my knowledge of three topics. The C language, compilers and the ARM assembly. I think this is a good exercise to go deep into both topics.

The ACC is a LALR(1) parser, that means it is a Look-Ahead Left-to-Right parser.

The current Grammar of the ACC is the following:

S -> int main "(" ")" "{" E "}"
E -> return I;
     | if "(" B ")" "{" E "}"
     | if "(" B ")" "{" E "}" else "{" E "}"
I -> [0-9]+
B -> I < I
     | I <= I
     | I == I
     | I >= I
     | I > I
     | ! B
     | "(" B ")"

Future work for the project

~~Main function~~
~~Return statement~~
~~Int value~~
~~Boolean operations~~
~~Conditionals~~
Variable (int)
Addition and substraction operations
Multiplication and division operations
Comments
Functions
String
Array
Pointer
Struct
Loop
Include
…
Self-contained Compiler

Example of the usage of the ACC

C code:

int main()
{
    if (1 < 2)
    {
	    if (4 > 10)
	    {
		    return 1;
	    }
	    else {
		    return 2;
	    }
    }
    return 0;
}

Assembly generated:

	.text
	.global main
main:
	mov r0, #1
	mov r1, #2
	cmp r0, r1
	bge if_else_0
	mov r0, #4
	mov r1, #10
	cmp r0, r1
	ble if_else_1
	mov r0, #1
	bx lr
if_else_1:
	mov r0, #2
	bx lr
if_else_0:
	mov r0, #0
        bx lr

Prefix Tree Comparison

Mon, 01 Jan 0001 00:00:00 +0000

Trie, Ternary Search Tree (TST) and Radix Tree is aimed to compare the performance of these three data structures. The three data structure are an implementation of the abstract data type dictionary, but they are enhanced with an extra method called “keys”. That method returns the keys with that share a provided prefix. This can useful to use it in tasks such as autocomplete searches.

The study performed was using std::map and std::unordered_map as baseline for the performance.

Robert Sedgewick claims that a Ternary Search Tree (TST) is a better solution than a Radix Tree since it does not waste space if the amount of data is not dense enough. and Steven S. Skiena.

However, Steven S. Skiena holds that a Suffix Tree is the best approach, but since both data structures do not fulfill the same problems. I decided to use a Radix Tree regarding the close it gets to an Suffix Tree implementation.

Operations supported

The opeartions supported by Trie, Ternary Search Tree (TST) and Radix Tree are:

Operations	Description
clear	Removes all the content of the data structure
find	Returns the value stored in the data structure for a specific key provided
insert	Adds a new pair of key and value to the data structure
size	Returns the amount of elements stored in the data structure
show	Prints the content of the data structure in the dot format
erase	Removes the value stored in the data structure for a specific key provided
contains	Returns true if there is a value associated to the key provided
keys	Returns a std::vector with all the keys in the data structure that have the provided prefix. If no prefix is provided returns a std::vector with all the keys in the data structure
lcp	Returns the longest common prefix of all keys stored in the data structure

Scenarios tested

Several test scenarios have been have developed to compare the performance of these five data structures.

Memory
- Random values
- Dictionaries
Insert
- Random values
Remove
- Random values
  - Found
  - Not found
- Dictionaries
  - Found
  - Not found
Seach
- Random values
  - Found
  - Not found
- Dictionaries
  - Found
  - Not found
Get Keys
- Random values
- Dictionaries
Get Keys with Prefix
- Random values
- Dictionaries
Longest Common Path
- Random values
- Dictionaries

Description of the structures

Trie

A trie is an ordered tree data structure that is used to store a dynamic set or associative array where the keys are usually strings. Unlike a binary search tree, no node in the tree stores the key associated with that node; instead, its position in the tree defines the key with which it is associated. All the descendants of a node have a common prefix of the string associated with that node, and the root is associated with the empty string. Values are not necessarily associated with every node. Rather, values tend only to be associated with leaves, and with some inner nodes that correspond to keys of interest.

Ternary Search Tree (TST)

Ternary Search Tree is a type of trie (sometimes called a prefix tree) where nodes are arranged in a manner similar to a binary search tree, but with up to three children rather than the binary tree’s limit of two. Like other prefix trees, a ternary search tree can be used as an associative map structure with the ability for incremental string search. However, ternary search trees are more space efficient compared to standard prefix trees, at the cost of speed.

Radix Tree

Radix Tree (also radix trie or compact prefix tree) is a data structure that represents a space-optimized trie in which each node that is the only child is merged with its parent. The result is that the number of children of every internal node is at least the radix r of the radix trie, where r is a positive integer and a power x of 2, having x >= 1. Unlike in regular tries, edges can be labeled with sequences of elements as well as single elements. This makes radix trees much more efficient for small sets (especially if the strings are long) and for sets of strings that share long prefixes.

Results

All results can be seen here

Resume Application

Mon, 01 Jan 0001 00:00:00 +0000

This resume is an interactive application written in Go and using the Bubble Tea library it can be found in https://github.com/maitesin/tui-cv/.

The idea for this project was inspired by S0ulshake’s CV, however, I preferred to use other technologies, since I do not know JavaScript.

I have packaged the application inside a Docker image, so people can run it on their computers with the following command.

docker run -it --rm maitesin/resume

Yaus, Yet Another URL Shortener

Mon, 01 Jan 0001 00:00:00 +0000

YAUS, Yet Another URL Shortener, was first written in Python3 using the Flask web framework, and deployed in Heroku: https://yaus.dev. The code of that version can be seen here: https://github.com/maitesin/yaus/tree/e938b0d218849f50df7ab5a4a46923faf235762b. However, the project has been rewritten in Go and I have applied some Domain Driven Design (DDD) and Command Query Segregation (CQS) patterns.

The purpose of this project is for me to tinker and practice some development methods or design I want to test. It also helps me keep up to date with the Heroku platform.

Oscar Forner

Unit test with HTTP client in Go

Introduction

Code with HTTP Client

Testing the code

Generate mock for RoundTripper

Example of the test

Execution

Go embed for migrations

Introduction

What is embed?

Go-Migrate to run migrations

Go-Migrate CLI tool

Install

Create new migration

Go-Migrate with Go Embed

Extra setup

Docker for Postgres

Migration files

Execution

Conclusion

Links

Image manipulation in the terminal

Introduction

Crop Images

Add Transparency

The End?

No Naked Boolean Parameters

Introduction

Wrapper functions

Named booleans

Bitmask

Typed booleans

Conclusion

Sorting algorithms

Introduction

Quicksort

How does it works?

Optimizations

When to use it?

Merge Sort

How does it works?

Optimizations

When to use it?

Heapsort

How does it works?

Optimizations

When to use it?

Insertion Sort

How does it works?

Optimizations

When to use it?

Conclusion

Python's for loop explained

Introduction

Iterator protocol

Default implementation

Alternative implementation

Closing notes

Software development using BASH

Introduction

Formatter

Linter (Static Analysis)

Unit Test Framework

Conclusion

Interactive curriculum in Rust

Introduction

How to run my CV application

My two cents about Go

Introduction

Pros

Standard library

Unit test and benchmarking

Channels

Cons

Build system

Different versions of the compiler

Other

Runtime

Generics