Blog blog("Baptiste Wicht"); (Posts about clang)

Decrease DLL neural network compilation time with C++17

Baptiste Wicht — Wed, 07 Feb 2018 10:39:02 GMT

Just last week, I've migrated my Expression Templates Library (ETL) library to C++17, it is now also done in my Deep Learning Library (DLL) library. In ETL, this resulted in a much nicer code overall, but no real improvement in compilation time.

The objective of the migration of DLL was two-fold. First, I also wanted to simplify some code, especially with if constexpr. But I also especially wanted to try to reduce the compilation time. In the past, I've already tried a few changes with C++17, with good results on the compilation of the entire test suite. While this is very good, this is not very representative of users of the library. Indeed, normally you'll have only one network in your source file not several. The new changes will especially help in the case of many networks, but less in the case of a single network per source file.

This time, I decided to test the compilation on the examples. I've tested the eight official examples from the DLL library:

mnist_dbn: A fully-connected Deep Belief Network (DBN) on the MNIST data set with three layers
char_cnn: A special CNN with embeddings and merge and group layers for text recognition
imagenet_cnn: A 12 layers Convolutional Neural Network (CNN) for Imagenet
mnist_ae: A simple two-layers auto-encoder for MNIST
mnist_cnn: A simple 6 layers CNN for MNIST
mnist_deep_ae: A deep auto-encoder for MNIST, only fully-connected
mnist_lstm: A Recurrent Neural Network (RNN) with Long Short Term Memory (LSTM) cells
mnist_mlp: A simple fully-connected network for MNIST, with dropout
mnist_rnn: A simple RNN with simple cells for MNIST

This is really representative of what users can do with the library and I think it's a much better for compilation time.

For reference, you can find the source code of all the examples online.

Results

Let's start with the results. I've tested this at different stages of the migration with clang 5 and GCC 7.2. I tested the following steps:

The original C++14 version
Simply compiling in c++17 mode (-std=c++17)
Using the C++17 version of the ETL library
Upgrading DLL to C++17 (without ETL)
ETL and DLL in C++17 versions

I've compiled each example independently in release_debug mode. Here are the results for G++ 7.2:

Example	0	1	2	3	4	5	6	7	8
C++14	37.818	32.944	33.511	15.403	29.998	16.911	24.745	18.974	19.006
-std=c++17	38.358	32.409	32.707	15.810	30.042	16.896	24.635	19.134	19.027
ETL C++17	36.045	31.000	30.942	15.322	28.840	16.747	24.151	18.208	18.939
DLL C++17	35.251	32.577	32.854	15.653	29.758	16.851	24.606	19.098	19.146
Final C++17	32.289	31.133	30.939	15.232	28.753	16.526	24.326	18.116	17.819
Final Improvement	14.62%	5.49%	7.67%	1.11%	4.15%	2.27%	1.69%	4.52%	6.24%

The difference by just enabling c++17 is not significant. On the other hand, some significant gain can be obtained by using the C++17 version of ETL, especially for the DBN version and for the CNN versions. Except for the DBN case, the migration of DLL to C++17 did not bring any significant advantage. When everything is combined, the gains are more important :) In the best case, the example is 14.6% faster to compile.

Let's see if it's the same with clang++ 5.0:

Example	0	1	2	3	4	5	6	7	8
C++14	40.690	34.753	35.488	16.146	31.926	17.708	29.806	19.207	20.858
-std=c++17	40.502	34.664	34.990	16.027	31.510	17.630	29.465	19.161	20.860
ETL C++17	37.386	33.008	33.896	15.519	30.269	16.995	28.897	18.383	19.809
DLL C++17	37.252	34.592	35.250	16.131	31.782	17.606	29.595	19.126	20.782
Final C++17	34.470	33.154	33.881	15.415	30.279	17.078	28.808	18.497	19.761
Final Improvement	15.28%	4.60%	4.52%	4.52%	5.15%	3.55%	3.34%	3.69%	5.25%

First of all, as I have seen time after time, clang is still slower than GCC. It's a not a big difference, but still significant. Overall, the gains are a bit higher on clang than on GCC, but not by much. Interestingly, the migration of DLL to C++17 is less interesting in terms of compilation time for clang. It seems even to slow down compilation on some examples. On the other hand, the migration of ETL is more important than on GCC.

Overall, every example is faster to compile using both libraries in C++17, but we don't have spectacular speed-ups. With clang, we have speedups from 3.3% to 15.3%. With GCC, we have speedup from 1.1% to 14.6%. It's not very high, but I'm already satisfied with these results.

C++17 in DLL

Overall, the migration of DLL to C++17 was quite similar to that of ETL. You can take a look at my previous article if you want more details on C++17 features I've used.

I've replaced a lot of SFINAE functions with if constexpr. I've also replaced a lot of statif_if with if constexpr. There was a large number of these in DLL's code. I also enabled all the constexpr that were commented for this exact time :)

I was also thinking that I could replace a lot of meta-programming stuff with fold expressions. While I was able to replace a few of them, most of them were harder to replace with fold expressions. Indeed, the variadic pack is often hidden behind another class and therefore the pack is not directly usable from the network class or the group and merge layers classes. I didn't want to start a big refactoring just to use a C++17 feature, the current state of this code is fine.

I made some use of structured bindings as well, but again not as much as I was thinking. In fact, a lot of time, I'm assigning the elements of a pair or tuple to existing variables not declaring new variables and unfortunately, you can only use structured bindings with auto declaration.

Overall, the code is significantly better now, but there was less impact than there was on ETL. It's also a smaller code base, so maybe this is normal and my expectations were too high ;)

Conclusion

The trunk of DLL is now a C++17 library :) I think this improve the quality of the code by a nice margin! Even though, there is still some work to be done to improve the code, especially for the DBN pretraining code, the quality is quite good now. Moreover, the switch to C++17 made the compilation of neural networks using the DLL library faster to compile, from 1.1% in the worst case to 15.3% in the best case! I don't know when I will release the next version of DLL, but it will take some time. I'll especially have to polish the RNN support and add a sequence to sequence loss before I will release the 1.1 version of DLL.

I'm quite satisfied with C++17 even if I would have liked a bit more features to play with! I'm already a big fan of if constexpr, this can make the code much nicer and fold expressions are much more intuitive than their previous recursive template counterpart.

I may also consider migrating some parts of the cpp-utils library, but if I do, it will only be through the use of conditionals in order not to break the other projects that are based on the library.

How I made my Deep Learning Library 38% faster to compile (Optimization and C++17 if constexpr)

Baptiste Wicht — Thu, 21 Sep 2017 17:44:34 GMT

My Deep Learning Library (DLL) project is a C++ library for training and using artificial neural networks (you can take a look at this post about DLL if you want more information).

While I made a lot of effort to make it as fast as possible to train and run neural networks, the compilation time has been steadily going up and is becoming quite annoying. This library is heavily templated and all the matrix operations are done using my Expression Templates Library (ETL) which is more than template-heavy itself.

In this post, I'll present two techniques with which I've been able to reduce the total compilation of the DLL unit tests by up to 38%.

Compiler benchmark GCC and Clang on C++ library (ETL)

Baptiste Wicht — Mon, 07 Aug 2017 07:16:21 GMT

It's been a while since I've done a benchmark of different compilers on C++ code. Since I've recently released the version 1.1 of my ETL project (an optimized matrix/vector computation library with expression templates), I've decided to use it as the base of my benchmark. It's a C++14 library with a lot of templates. I'm going to compile the full test suite (124 test cases). This is done directly on the last release (1.1) code. I'm going to compile once in debug mode and once in release_debug (release plus debug symbols and assertions) and record the times for each compiler. The tests were compiled with support for every option in ETL to account to maximal compilation time. Each compilation was made using four threads (make -j4). I'm also going to test a few of the benchmarks to see the difference in runtime performance between the code generated by each compiler. The benchmark will be compiled in release mode and its compilation time recorded as well.

I'm going to test the following compilers:

GCC-4.9.4
GCC-5.4.0
GCC-6.3.0
GCC-7.1.0
clang-3.9.1
clang-4.0.1
zapcc-1.0 (commercial, based on clang-5.0 trunk)

All have been installed directly using Portage (Gentoo package manager) except for clang-4.0.1 that has been installed from sources and zapcc since it does not have a Gentoo package. Since clang package on Gentoo does not support multislotting, I had to install one version from source and the other from the package manager. This is also the reason I'm testing less versions of clang, simply less practical.

For the purpose of these tests, the exact same options have been used throughout all the compilers. Normally, I use different options for clang than for GCC (mainly more aggressive vectorization options on clang). This may not lead to the best performance for each compiler, but allows for comparison between the results with defaults optimization level. Here are the main options used:

In debug mode: -g
In release_debug mode: -g -O2
In release mode: -g -O3 -DNDEBUG -fomit-frame-pointer

In each case, a lot of warnings are enabled and the ETL options are the same.

All the results have been gathered on a Gentoo machine running on Intel Core i7-2600 (Sandy Bridge...) @3.4GHz with 4 cores and 8 threads, 12Go of RAM and a SSD. I do my best to isolate as much as possible the benchmark from perturbations and that my benchmark code is quite sound, it may well be that some results are not totally accurate. Moreover, some of the benchmarks are using multithreading, which may add some noise and unpredictability. When I was not sure about the results, I ran the benchmarks several time to confirm them and overall I'm confident of the results.

Compilation Time

Let's start with the results of the performance of the compilers themselves:

Compiler	Debug	Release_Debug	Benchmark
g++-4.9.4	402s	616s	100s
g++-5.4.0	403s	642s	95s
g++-6.3.0	399s	683s	102s
g++-7.1.0	371s	650s	105s
clang++-3.9.1	380s	807s	106s
clang++-4.0.1	260s	718s	92s
zapcc++-1.0	221s	649s	108s

Note: For Release_Debug and Benchmark, I only use three threads with zapcc, because 12Go of RAM is not enough memory for four threads.

There are some very significant differences between the different compilers. Overall, clang-4.0.1 is by far the fastest free compiler for Debug mode. When the tests are compiled with optimizations however, clang is falling behind. It's quite impressive how clang-4.0.1 manages to be so much faster than clang-3.9.1 both in debug mode and release mode. Really great work by the clang team here! With these optimizations, clang-4.0.1 is almost on par with gcc-7.1 in release mode. For GCC, it seems that the cost of optimization has been going up quite significantly. However, GCC 7.1 seems to have made optimization faster and standard compilation much faster as well. If we take into account zapcc, it's the fastest compiler on debug mode, but it's slower than several gcc versions on release mode.

Overall, I'm quite impressed by the performance of clang-4.0.1 which seems really fast! I'll definitely make more tests with this new version of the compiler in the near future. It's also good to see that g++-7.1 also did make the build faster than gcc-6.3. However, the fastest gcc version for optimization is still gcc-4.9.4 which is already an old branch with low C++ standard support.

Runtime Performance

Let's now take a look at the quality of the generated code. For some of the benchmarks, I've included two versions of the algorithm. std is the most simple algorithm (the naive one) and vec is the hand-crafted vectorized and optimized implementation. All the tests were done on single-precision floating points.

Dot product

The first benchmark that is run is to compute the dot product between two vectors. Let's look first at the naive version:

dot (std)	100	500	1000	10000	100000	1000000	2000000	3000000	4000000	5000000	10000000
g++-4.9.4	64.96ns	97.12ns	126.07ns	1.89us	25.91us	326.49us	1.24ms	1.92ms	2.55ms	3.22ms	6.36ms
g++-5.4.0	72.96ns	101.62ns	127.89ns	1.90us	23.39us	357.63us	1.23ms	1.91ms	2.57ms	3.20ms	6.32ms
g++-6.3.0	73.31ns	102.88ns	130.16ns	1.89us	24.314us	339.13us	1.47ms	2.16ms	2.95ms	3.70ms	6.69ms
g++-7.1.0	70.20ns	104.09ns	130.98ns	1.90us	23.96us	281.47us	1.24ms	1.93ms	2.58ms	3.19ms	6.33ms
clang++-3.9.1	64.69ns	98.69ns	128.60ns	1.89us	23.33us	272.71us	1.24ms	1.91ms	2.56ms	3.19ms	6.37ms
clang++-4.0.1	60.31ns	96.34ns	128.90ns	1.89us	22.87us	270.21us	1.23ms	1.91ms	2.55ms	3.18ms	6.35ms
zapcc++-1.0	61.14ns	96.92ns	125.95ns	1.89us	23.84us	285.80us	1.24ms	1.92ms	2.55ms	3.16ms	6.34ms

The differences are not very significant between the different compilers. The clang-based compilers seem to be the compilers producing the fastest code. Interestingly, there seem to have been a big regression in gcc-6.3 for large containers, but that has been fixed in gcc-7.1.

dot (vec)	100	500	1000	10000	100000	1000000	2000000	3000000	4000000	5000000	10000000
g++-4.9.4	48.34ns	80.53ns	114.97ns	1.72us	22.79us	354.20us	1.24ms	1.89ms	2.52ms	3.19ms	6.55ms
g++-5.4.0	47.16ns	77.70ns	113.66ns	1.72us	22.71us	363.86us	1.24ms	1.89ms	2.52ms	3.19ms	6.56ms
g++-6.3.0	46.39ns	77.67ns	116.28ns	1.74us	23.39us	452.44us	1.45ms	2.26ms	2.87ms	3.49ms	7.52ms
g++-7.1.0	49.70ns	80.40ns	115.77ns	1.71us	22.46us	355.16us	1.21ms	1.85ms	2.49ms	3.14ms	6.47ms
clang++-3.9.1	46.13ns	78.01ns	114.70ns	1.66us	22.82us	359.42us	1.24ms	1.88ms	2.53ms	3.16ms	6.50ms
clang++-4.0.1	45.59ns	74.90ns	111.29ns	1.57us	22.47us	351.31us	1.23ms	1.85ms	2.49ms	3.12ms	6.45ms
zapcc++-1.0	45.11ns	75.04ns	111.28ns	1.59us	22.46us	357.32us	1.25ms	1.89ms	2.53ms	3.15ms	6.47ms

If we look at the optimized version, the differences are even slower. Again, the clang-based compilers are producing the fastest executables, but are closely followed by gcc, except for gcc-6.3 in which we can still see the same regression as before.

Logistic Sigmoid

The next test is to check the performance of the sigmoid operation. In that case, the evaluator of the library will try to use parallelization and vectorization to compute it. Let's see how the different compilers fare:

sigmoid	10	100	1000	10000	100000	1000000
g++-4.9.4	8.16us	5.23us	6.33us	29.56us	259.72us	2.78ms
g++-5.4.0	7.07us	5.08us	6.39us	29.44us	266.27us	2.96ms
g++-6.3.0	7.13us	5.32us	6.45us	28.99us	261.81us	2.86ms
g++-7.1.0	7.03us	5.09us	6.24us	28.61us	252.78us	2.71ms
clang++-3.9.1	7.30us	5.25us	6.57us	30.24us	256.75us	1.99ms
clang++-4.0.1	7.47us	5.14us	5.77us	26.03us	235.87us	1.81ms
zapcc++-1.0	7.51us	5.26us	6.48us	28.86us	258.31us	1.95ms

Interestingly, we can see that gcc-7.1 is the fastest for small vectors while clang-4.0 is the best for producing code for larger vectors. However, except for the biggest vector size, the difference is not really significantly. Apparently, there is a regression in zapcc (or clang-5.0) since it's slower than clang-4.0 at the same level as clang-3.9.

y = alpha * x + y (axpy)

The third benchmark is the well-known axpy (y = alpha * x + y). This is entirely resolved by expressions templates in the library, no specific algorithm is used. Let's see the results:

saxpy	10	100	1000	10000	100000	1000000
g++-4.9.4	38.1ns	61.6ns	374ns	3.65us	40.8us	518us
g++-5.4.0	35.0ns	58.1ns	383ns	3.87us	43.2us	479us
g++-6.3.0	34.3ns	59.4ns	371ns	3.57us	40.4us	452us
g++-7.1.0	34.8ns	59.7ns	399ns	3.78us	43.1us	547us
clang++-3.9.1	32.3ns	53.8ns	297ns	3.21us	38.3us	466us
clang++-4.0.1	32.4ns	59.8ns	296ns	3.31us	38.2us	475us
zapcc++-1.0	32.0ns	54.0ns	333ns	3.32us	38.7us	447us

Even on the biggest vector, this is a very fast operation, once vectorized and parallelized. At this speed, some of the differences observed may not be highly significant. Again clang-based versions are the fastest versions on this code, but by a small margin. There also seems to be a slight regression in gcc-7.1, but again quite small.

Matrix Matrix multiplication (GEMM)

The next benchmark is testing the performance of a Matrix-Matrix Multiplication, an operation known as GEMM in the BLAS nomenclature. In that case, we test both the naive and the optimized vectorized implementation. To save some horizontal space, I've split the tables in two.

sgemm (std)	10	20	40	60	80	100
g++-4.9.4	7.04us	50.15us	356.42us	1.18ms	3.41ms	5.56ms
g++-5.4.0	8.14us	74.77us	513.64us	1.72ms	4.05ms	7.92ms
g++-6.3.0	8.03us	64.78us	504.41us	1.69ms	4.02ms	7.87ms
g++-7.1.0	7.95us	65.00us	508.84us	1.69ms	4.02ms	7.84ms
clang++-3.9.1	3.58us	28.59us	222.36us	0.73ms	1.77us	3.41ms
clang++-4.0.1	4.00us	25.47us	190.56us	0.61ms	1.45us	2.80ms
zapcc++-1.0	4.00us	25.38us	189.98us	0.60ms	1.43us	2.81ms

sgemm (std)	200	300	400	500	600	700	800	900	1000	1200
g++-4.9.4	44.16ms	148.88ms	455.81ms	687.96ms	1.47s	1.98s	2.81s	4.00s	5.91s	9.52s
g++-5.4.0	63.17ms	213.01ms	504.83ms	984.90ms	1.70s	2.70s	4.03s	5.74s	7.87s	14.905
g++-6.3.0	64.04ms	212.12ms	502.95ms	981.74ms	1.69s	2.69s	4.13s	5.85s	8.10s	14.08s
g++-7.1.0	62.57ms	210.72ms	499.68ms	974.94ms	1.68s	2.67s	3.99s	5.68s	7.85s	13.49s
clang++-3.9.1	27.48ms	90.85ms	219.34ms	419.53ms	0.72s	1.18s	1.90s	2.44s	3.36s	5.84s
clang++-4.0.1	22.01ms	73.90ms	175.02ms	340.70ms	0.58s	0.93s	1.40s	1.98s	2.79s	4.69s
zapcc++-1.0	22.33ms	75.80ms	181.27ms	359.13ms	0.63s	1.02s	1.52s	2.24s	3.21s	5.62s

This time, the differences between the different compilers are very significant. The clang compilers are leading the way by a large margin here, with clang-4.0 being the fastest of them (by another nice margin). Indeed, clang-4.0.1 is producing code that is, on average, about twice faster than the code generated by the best GCC compiler. Very interestingly as well, we can see a huge regression starting from GCC-5.4 and that is still here in GCC-7.1. Indeed, the best GCC version, in the tested versions, is again GCC-4.9.4. Clang is really doing an excellent job of compiling the GEMM code.

sgemm (vec)	10	20	40	60	80	100
g++-4.9.4	264.27ns	0.95us	3.28us	14.77us	23.50us	60.37us
g++-5.4.0	271.41ns	0.99us	3.31us	14.811us	24.116us	61.00us
g++-6.3.0	279.72ns	1.02us	3.27us	15.39us	24.29us	61.99us
g++-7.1.0	273.74ns	0.96us	3.81us	15.55us	31.35us	71.11us
clang++-3.9.1	296.67ns	1.34us	4.18us	19.93us	33.15us	82.60us
clang++-4.0.1	322.68ns	1.38us	4.17us	20.19us	34.17us	83.64us
zapcc++-1.0	307.49ns	1.41us	4.10us	19.72us	33.72us	84.80us

sgemm (vec)	200	300	400	500	600	700	800	900	1000	1200
g++-4.9.4	369.52us	1.62ms	2.91ms	7.17ms	11.74ms	22.91ms	34.82ms	51.67ms	64.36ms	111.15ms
g++-5.4.0	387.54us	1.60ms	2.97ms	7.36ms	12.11ms	24.37ms	35.37ms	52.27ms	65.72ms	112.74ms
g++-6.3.0	384.43us	1.74ms	3.12ms	7.16ms	12.44ms	24.15ms	34.87ms	52.59ms	70.074ms	119.22ms
g++-7.1.0	458.05us	1.81ms	3.44ms	7.86ms	13.43ms	24.70ms	36.54ms	53.47ms	66.87ms	117.25ms
clang++-3.9.1	494.52us	1.96ms	4.80ms	8.88ms	18.20ms	29.37ms	41.24ms	60.72ms	72.28ms	123.75ms
clang++-4.0.1	511.24us	2.04ms	4.11ms	9.46ms	15.34ms	27.23ms	38.27ms	58.14ms	72.78ms	128.60ms
zapcc++-1.0	492.28us	2.03ms	3.90ms	9.00ms	14.31ms	25.72ms	37.09ms	55.79ms	67.88ms	119.92ms

As for the optimized version, it seems that the two families are reversed. Indeed, GCC is doing a better job than clang here, and although the margin is not as big as before, it's still significant. We can still observe a small regression in GCC versions because the 4.9 version is again the fastest. As for clang versions, it seems that clang-5.0 (used in zapcc) has had some performance improvements for this case.

For this case of matrix-matrix multiplication, it's very impressive that the differences in the non-optimized code are so significant. And it's also impressive that each family of compilers has its own strength, clang being seemingly much better at handling unoptimized code while GCC is better at handling vectorized code.

Convolution (2D)

The last benchmark that I considered is the case of the valid convolution on 2D images. The code is quite similar to the GEMM code but more complicated to optimized due to cache locality.

sconv2_valid (std)	100x50	105x50	110x55	115x55	120x60	125x60	130x65	135x65	140x70
g++-4.9.4	27.93ms	33.68ms	40.62ms	48.23ms	57.27ms	67.02ms	78.45ms	92.53ms	105.08ms
g++-5.4.0	37.60ms	44.94ms	54.24ms	64.45ms	76.63ms	89.75ms	105.08ms	121.66ms	140.95ms
g++-6.3.0	37.10ms	44.99ms	54.34ms	64.54ms	76.54ms	89.87ms	105.35ms	121.94ms	141.20ms
g++-7.1.0	37.55ms	45.08ms	54.39ms	64.48ms	76.51ms	92.02ms	106.16ms	125.67ms	143.57ms
clang++-3.9.1	15.42ms	18.59ms	22.21ms	26.40ms	31.03ms	36.26ms	42.35ms	48.87ms	56.29ms
clang++-4.0.1	15.48ms	18.67ms	22.34ms	26.50ms	31.27ms	36.58ms	42.61ms	49.33ms	56.80ms
zapcc++-1.0	15.29ms	18.37ms	22.00ms	26.10ms	30.75ms	35.95ms	41.85ms	48.42ms	55.74ms

In that case, we can observe the same as for the GEMM. The clang-based versions are much producing significantly faster code than the GCC versions. Moreover, we can also observe the same large regression starting from GCC-5.4.

sconv2_valid (vec)	100x50	105x50	110x55	115x55	120x60	125x60	130x65	135x65	140x70
g++-4.9.4	878.32us	1.07ms	1.20ms	1.68ms	2.04ms	2.06ms	2.54ms	3.20ms	4.14ms
g++-5.4.0	853.73us	1.03ms	1.15ms	1.36ms	1.76ms	2.05ms	2.44ms	2.91ms	3.13ms
g++-6.3.0	847.95us	1.02ms	1.14ms	1.35ms	1.74ms	1.98ms	2.43ms	2.90ms	3.12ms
g++-7.1.0	795.82us	0.93ms	1.05ms	1.24ms	1.60ms	1.77ms	2.20ms	2.69ms	2.81ms
clang++-3.9.1	782.46us	0.93ms	1.05ms	1.26ms	1.60ms	1.84ms	2.21ms	2.65ms	2.84ms
clang++-4.0.1	767.58us	0.92ms	1.04ms	1.25ms	1.59ms	1.83ms	2.20ms	2.62ms	2.83ms
zapcc++-1.0	782.49us	0.94ms	1.06ms	1.27ms	1.62ms	1.83ms	2.24ms	2.65ms	2.85ms

This time, clang manages to produce excellent results. Indeed, all the produced executables are significantly faster than the versions produced by GCC, except for GCC-7.1 which is producing similar results. The other versions of GCC are falling behind it seems. It seems that it was only for the GEMM that clang was having a lot of troubles handling the optimized code.

Conclusion

Clang seems to have recently done a lot of optimizations regarding compilation time. Indeed, clang-4.0.1 is much faster for compilation than clang-3.9. Although GCC-7.1 is faster than GCC-6.3, all the GCC versions are slower than GCC-4.9.4 which is the fastest at compiling code with optimizations. GCC-7.1 is the fastest GCC version for compiling code in debug mode.

In some cases, there is almost no difference between different compilers in the generated code. However, in more complex algorithms such as the matrix-matrix multiplication or the two-dimensional convolution, the differences can be quite significant. In my tests, Clang have shown to be much better at compiling unoptimized code. However, and especially in the GEMM case, it seems to be worse than GCC at handling hand-optimized. I will investigate that case and try to tailor the code so that clang is having a better time with it.

For me, it's really weird that the GCC regression, apparently starting from GCC-5.4, has still not been fixed in GCC 7.1. I was thinking of dropping support for GCC-4.9 in order to go full C++14 support, but now I may have to reconsider my position. However, seeing that GCC is generally the best at handling optimized code (especially for GEMM), I may be able to do the transition, since the optimized code will be used in most cases.

As for zapcc, although it is still the fastest compiler in debug mode, with the new speed of clang-4.0.1, its margin is quite small. Moreover, on optimized build, it's not as fast as GCC. If you use clang and can have access to zapcc, it's still quite a good option to save some time.

Overall, I have been quite pleased by clang-4.0.1 and GCC-7.1, the most recent versions I have been testing. It seems that they did quite some good work. I will definitely run some more tests with them and try to adapt the code. I'm still considering whether I will drop support for some older compilers.

I hope this comparison was interesting :) My next post will probably be about the difference in performance between my machine learning framework and other frameworks to train neural networks.

Partial type erasing in Deep Learning Library (DLL) to improve compilation time

Baptiste Wicht — Wed, 15 Mar 2017 06:43:44 GMT

In a previous post, I compared the compilation time on my Deep Learning Library (DLL) project with different compilers. I realized that the compilation times were quickly going unreasonable for this library, especially for compiling the unit cases which clearly hurts the development of the library. Indeed, you want to be able to run the unit tests reasonably quickly after you integrated new changes.

Reduce the compilation time

The first thing I did was to split the compilation in three executables: one for the unit tests, one for the various performance tests and one for the various other miscellaneous tests. With this, it is much faster to compile only the unit test cases.

But this can be improved significantly more. In DLL a network is a variadic template containing the list of layers, in order. In DLL, there are two main different ways of declaring a neural networks. In the first version, the fast version, the layers directly know their sizes:

using network_t =
    dll::dbn_desc<
        dll::dbn_layers<
            dll::rbm_desc<28 * 28, 500, dll::momentum, dll::batch_size<64>>::layer_t,
            dll::rbm_desc<500    , 400, dll::momentum, dll::batch_size<64>>::layer_t,
            dll::rbm_desc<400    , 10,  dll::momentum, dll::batch_size<64>, dll::hidden<dll::unit_type::SOFTMAX>>::layer_t>,
        dll::trainer<dll::sgd_trainer>, dll::batch_size<64>>::dbn_t;

auto network = std::make_unique<network_t>();
network->pretrain(dataset.training_images, 10);
network->fine_tune(dataset.training_images, dataset.training_labels, 10);

In my opinion, this is the best way to use DLL. This is the fastest and the clearest. Moreover, the dimensions of the network can be validated at compile time, which is always better than at runtime. However, the dimensions of the network cannot be changed at runtime. For this, there is a different version, the dynamic version:

using network_t =
    dll::dbn_desc<
        dll::dbn_layers<
            dll::dyn_rbm_desc<dll::momentum>::layer_t,
            dll::dyn_rbm_desc<dll::momentum>::layer_t,
            dll::dyn_rbm_desc<dll::momentum, dll::hidden<dll::unit_type::SOFTMAX>>::layer_t>,
        dll::batch_size<64>, dll::trainer<dll::sgd_trainer>>::dbn_t;

auto network = std::make_unique<network_t>();

network->template layer_get<0>().init_layer(28 * 28, 500);
network->template layer_get<1>().init_layer(500, 400);
network->template layer_get<2>().init_layer(400, 10);
network->template layer_get<0>().batch_size = 64;
network->template layer_get<1>().batch_size = 64;
network->template layer_get<2>().batch_size = 64;

network->pretrain(dataset.training_images, 10);
network->fine_tune(dataset.training_images, dataset.training_labels, 10);

This is a bit more verbose, but the configuration can be changed at runtime with this system. Moreover, this is also faster to compile. On the other hand, there is some performance slowdown.

There is also a third version that is a hybrid of the first version:

using network_t =
    dll::dyn_dbn_desc<
        dll::dbn_layers<
            dll::rbm_desc<28 * 28, 500, dll::momentum, dll::batch_size<64>>::layer_t,
            dll::rbm_desc<500    , 400, dll::momentum, dll::batch_size<64>>::layer_t,
            dll::rbm_desc<400    , 10,  dll::momentum, dll::batch_size<64>, dll::hidden<dll::unit_type::SOFTMAX>>::layer_t>,
        dll::trainer<dll::sgd_trainer>, dll::batch_size<64>>::dbn_t;

auto network = std::make_unique<network_t>();
network->pretrain(dataset.training_images, 10);
network->fine_tune(dataset.training_images, dataset.training_labels, 10);

Only one line was changed compared to the first version, dbn_desc becomes dyn_dbn_desc. What this changes is that all the layers are automatically transformed into their dynamic versions and all the parameters are propagated at runtime. This is a form a type erasing since the sizes will not be propagated at compilation time. But this is simple since the types are simply transformed from one type to another directly. Behind the scene, it's the dynamic version using the front-end of the fast version. This is almost as fast to compile as the dynamic version, but the code is much better. It executes the same as the dynamic version.

If we compare the compilation time of the three versions when compiling a single network and 5 different networks with different architectures, we get the following results (with clang):

Model	Time [s]
1 Fast	30
1 Dynamic	16.6
1 Hybrid	16.6
5 Fast	114
5 Dynamic	16.6
5 Hybrid	21.9

Even with one single network, the compilation time is reduced by 44%. When five different networks are compilation, time is reduced by 85%. This can be explained easily. Indeed, for the hybrid and dynamic versions, the layers will have the same type and therefore a lot of template instantiations will only be done once instead of five times. This makes a lot of difference since almost everything is template inside the library.

Unfortunately, this also has an impact on the runtime of the network:

Model	Pretrain [s]	Train [s]
Fast	195	114
Dynamic	203	123
Hybrid	204	122

On average, for dense models, the slowdown is between 4% and 8%. For convolutional models, it is between 10% and 25%. I will definitely work on trying to make the dynamic and especially the hybrid version faster in the future, most on the work should be on the matrix library (ETL) that is used.

Since for test cases, a 20% increase in runtime is not really a problem, tests being fast already, I decided to add an option to DLL so that everything can be compiled by default in hybrid model. By using a compilation flag, all the dbn_desc are becoming dyn_dbn_desc and therefore each used network is becoming a hybrid network. Without a single change in the code, the compilation time of the entire library can be significantly improved, as seen in the next section. This can also be used in user code to improve compilation time during debugging and experiments and can be turned off for the final training.

On my Continuous Integration system, I will build the system in both configurations. This is not really an issue, since my personal machine at home is more powerful than what I have available here.

Results

On a first experiment, I measured the difference before and after this change on the three executables of the library, with gcc:

Model	Unit [s]	Perf [s]	Misc [s]
Before	1029	192	937
After	617	143	619
Speedup	40.03%	25.52%	33.93%

It is clear that the speedups are very significant! The compilation is between 25% and 40% faster with the new option. Overall, this is a speedup of 36%! I also noticed that the compilation takes significantly less memory than before. Therefore, I decided to rerun the compiler benchmark on the library. In the previous experiment, zapcc was taking so much memory that it was impossible to use more than one thread. Let's see how it is faring now. The time to compile the full unit tests is computed for each compiler. Let's start in debug mode:

Debug	-j1	-j2	-j3	-j4
clang-3.9	527	268	182	150
gcc-4.9.3	591	303	211	176
gcc-5.3.0	588	302	209	175
zapcc-1.0	375	187	126	121

This time, zapcc is able to scale to four threads without problems. Moreover, it is always the fastest compiler, by a significant margin, in this configuration. It is followed by clang and then by gcc for which both versions are about the same speed.

If we compile again in release mode:

Release	-j1	-j2	-j3	-j4
clang-3.9	1201	615	421	356
gcc-4.9.3	1041	541	385	321
gcc-5.3.0	1114	579	412	348
zapcc-1.0	897	457	306	306

The difference in compilation time is very large, it's twice slower to compile with all optimizations enabled. It also takes significantly more memory. Indeed, zapcc was not able to compile with 4 threads. Nevertheless, even the results with three threads are better than the other compilers using four threads. zapcc is clearly the winner again on this test, followed by gcc4-9 which is faster than gcc-5.3 which is itself faster than clang. It seems that while clang is better at frontend than gcc, it is slower for optimizations. Note that this may also be an indication that clang performs more optimizations than gcc and may not be slower.

Conclusion

By using some form of type erasing to simplify the templates types at compile time, I was able to reduce the overall compilation time of my Deep Learning Library (DLL) by 36%. Moreover, this can be done by switching a simple compilation flag. This also very significantly reduce the memory used during the compilation, allowing zapcc to to compile with up to three threads, compared with only one before. This makes zapcc the fastest compiler again on this benchmark. Overall, this will make debugging much easier on this library and will save me a lot of time.

In the future, I plan to try to improve compilation time even more. I have a few ideas, especially in ETL that should significantly improve the compilation time but that will require a lot of time to implement, so that will likely have to wait a while. In the coming days, I plan to work on the performance of DLL, especially for stochastic gradient descent.

If you want more information on DLL, you can check out the dll Github repository.

Use clang-tidy for static analysis and integration in Sonarqube

Baptiste Wicht — Sat, 11 Mar 2017 08:54:00 GMT

clang-tidy is an extensive linter C++. It provides a complete framework for analysis of C++ code. Some of the checks are very simple but some of them are very complete and most of the checks from the clang-static-analyzer are integrated into clang-tidy.

Usage

If you want to see the list of checks available on clang-tidy, you can use the list-checks options:

clang-tidy -list-checks

You can then choose the tests you are interested in and perform an analysis of your code. For, it is highly recommended to use a Clang compilation database, you can have a look at Bear to generate this compilation database if you don't have it yet. The usage of clang-tidy, is pretty simple, you set the list of checks you want, the header on which you want to have warnings reported and the list of source files to analyse:

clang-tidy -checks='*' -header-filter="^include" -p . src/*.cpp

You'll very likely see a lot of warnings. And you will very likely see a lot of false positives and a lot of warnings you don't agree too. For insance, there are a lot of warnings from the CPP Core Guidelines and the Google Guidelines that I don't follow in my coding. You should not take the complete list of tests as rule, you should devise your own list of what you really want to fix in your code. If you want to disable one check X, you can use the - operation:

clang-tidy -checks='*,-X' -header-filter="^include" -p . src/*.cpp

You can also enable the checks one by one or parts of them with *:

clang-tidy -checks='google-*' -header-filter="^include" -p . src/*.cpp

One problem with the clang-tidy tool is that it is utterly slow, especially if you enable the clang-static-analyzer checks. Moreover, if you use it like it is set before, it will only use one thread for the complete set of files. This may not be an issue on small projects, but this will definitely be a big issue for large projects and template-heavy code (like my ETL project). You could create an implicit target into your Makefile to use it on each file independently and then use the -j option of make to make them in parallel, but it not really practical.

For this, I just discovered that clang propose a Python script, run-clang-tidy.py that does it all for us! On Gentoo, it is installed at /usr/share/run-clang-tidy.py.

run-clang-tidy.py -checks='*' -header-filter="^include" -p . -j9

This will automatically run clang-tidy on each file from the compilation database and use 9 threads to perform the checks. This is definitely much faster. For me, this is the best way to run clang-tidy.

One small point I don't like is that the script always print the list of enabled checks. For, this I changed this line in the script:

invocation = [args.clang_tidy_binary, '-list-checks']

with:

invocation = [args.clang_tidy_binary]

This makes it more quiet.

One thing I didn't mention is that clang-tidy is able to fix some of the errors directly if you use the -fix option. Personally, I don't like this, but for a large code base and a carefully selected set of checks, this could be really useful. Note that not all the checks are automatically fixable by clang-tidy.

Results

I have run clang-tidy on my cpp-utils library and here some interesting results. I have not run all the checks, here is the command I used:

/usr/share/clang/run-clang-tidy.py -p . -header-filter '^include/cpp_utils' -checks='cert-*,cppcoreguidelines-*,google-*,llvm-*,misc-*,modernize-*,performance-*,readility-*,-cppcoreguidelines-pro-type-reinterpret-cast,-cppcoreguidelines-pro-bounds-pointer-arithmetic,-google-readability-namespace-comments,-llvm-namespace-comment,-llvm-include-order,-google-runtime-references' -j9 2>/dev/null  | /usr/bin/zgrep -v "^clang-tidy"

Let's go over some warnings I got:

include/cpp_utils/assert.hpp:91:103: warning: consider replacing 'long' with 'int64' [google-runtime-int]
void assertion_failed_msg(const CharT* expr, const char* msg, const char* function, const char* file, long line) {
                                                                                                      ^

I got this one several times. It is indeed more portable to use int64 rather than long.

include/cpp_utils/aligned_allocator.hpp:53:9: warning: use 'using' instead of 'typedef' [modernize-use-using]
        typedef aligned_allocator<U, A> other;
        ^

This one is part of the modernize checks, indicating that one should use using rather than a typedef and I completely agree.

include/cpp_utils/aligned_allocator.hpp:79:5: warning: use '= default' to define a trivial default constructor [modernize-use-default]
    aligned_allocator() {}
    ^
                        = default;

Another one from the modernize checks that I really like. This is completely true.

I don't agree that every constructor with one argument should be explicit, sometimes you want implicit conversion. Nevertheless, this particular case is very interesting since it is variadic, it can have one template argument and as thus it can be implicitly converted from anything, which is pretty bad I think.

test/array_wrapper.cpp:15:18: warning: C-style casts are discouraged; use reinterpret_cast [google-readability-casting]
    float* mem = (float*) malloc(sizeof(float) * 8);
                 ^
                 reinterpret_cast<float*>(         )

On this one, I completely agree, C-style casts should be avoided and much clearer C++ style casts should be preferred.

/home/wichtounet/dev/cpp_utils_test/include/cpp_utils/aligned_allocator.hpp:126:19: warning: thrown exception type is not nothrow copy constructible [cert-err60-cpp]
            throw std::length_error("aligned_allocator<T>::allocate() - Integer overflow.");
                  ^

This is one of the checks I don't agree with. Even though it makes sense to prefer exception that are nothrow copy constructible, they should be caught by const reference anyway. Moreover, this is here an exception from the standard library.

/home/wichtounet/dev/cpp_utils_test/include/cpp_utils/aligned_allocator.hpp:141:40: warning: do not use const_cast [cppcoreguidelines-pro-type-const-cast]
        free((reinterpret_cast<void**>(const_cast<std::remove_const_t<T>*>(ptr)))[-1]);
                                       ^

In general, I agree that using const_cast should be avoided as much as possible. But there are some cases where they make sense. In this particular case, I don't modify the object itself but some memory before the object that is unrelated and I initialize myself.

I also had a few false positives, but overall nothing too bad. I'm quite satisfied with the quality of the results. I'll fix these warnings in the coming week.

Integration in Sonarqube

The sonar-cxx plugin just integrated support for clang-tidy in main. You need to build the version yourself, the 0.9.8-SNAPSHOT version. You then can use something like this in your sonar-project.properties file:

sonar.cxx.clangtidy.reportPath=clang-tidy-report

and sonar-cxx will parse the results and integrate the issues in your sonar report.

Here is an example:

You can see two of the warnings from clang-tidy :)

For now, I haven't integrate this in my Continuous Integration system because I'm still having issues with clang-tidy and the compilation database. Because the compilation contains absolute paths to the file and to the current directory, it cannot be shared directly between servers. I have to find a way to fix that so that clang-tidy can use on the other computer. I'll probably wait till the sonar-cxx 0.9.8 version is released before integrating all this in Sonarqube, but this is a great news for this plugin :)

Conclusion

clang-tidy is C++ linter that can analyze your code and checks for hundreds of problems in it. With it, I have found some very interesting problems in the code of my cpp_utils library. Moreover, you can now integrate it Sonarqube by using the sonar-cxx plugin. Since it is a bit slow, I'll probably not integrate it in my bigger projects, but I'll integrate at least in the cpp_utils library when sonar-cxx 0.9.8 will be released.

Disappointing zapcc performance on Deep Learning Library (DLL)

Baptiste Wicht — Thu, 09 Mar 2017 12:41:06 GMT

One week ago, zapcc 1.0 was released and I've observed it to be much faster than the other compilers in terms of compile time. This can be seen when I tested it on my Expression Templates Library (ETL). It was almost four times faster than clang 3.9 and about 2.5 times faster than GCC.

The ETL library is quite heavy to compile, but still reasonable. This is not the case for my Deep Learning Library (DLL) where compiling all the test cases takes a very long time. I have to admit that I have been going overboard with templates and such and I have now to pay the price. In practice, for the users of the library, this is not a big problem since only one or two neural networks will be compiled (and it will take hours to train), but in the test cases, there are hundreds of them and this is a huge pain. Anyway, enough with the ramble, I figured it would be very good to test zapcc on it and see what I can gain from using it.

In this article, when I speak of a compiler thread, I mean an instance of the processor, so it's really a process in the Linux world.

Results

However, I soon realized that I would have more issues than I thought. The first problem is the memory consumed by zapcc. Indeed, it is based on clang and I always had problem with huge memory consumption from clang on this library and zapcc has even bigger memory consumption because some information is cached between runs. The amount of memory that zapcc is able to cache can be configured in the configuration file. By default, it can use 1.5Go of memory. When zapcc goes over the memory limit, it simply wipes out its caches. This means that all the gain for the next compilation will be lost, since the cache will have to be rebuilt from scratch. This is not a hard limit for the compilation itself. Indeed, if the compilation itself takes 3Go, it will still be able to complete it, but it is likely that the cache will be wiped after the compilation.

When I tried compiling using several threads, it soon used all my memory and crashed. The same occurs with clang but I can still compile with 3 or 4 threads without too much issues on this computer. The same also occurs with GCC but it can still handle 4 or 5 threads (depending on the order of the compilation units).

The tests are performed on my desktop computer at work, which is not really good... I have 12Go of RAM (I had to ask for extra...) and an old Sandy Bridge processor, but at least I have an SSD (also had to ask for extra).

I started with testing with only one compiler thread. For zapcc, I set the maximum memory limit to 8Go. Even with such a limit, the zapcc server restarted more than 10 times during the compilation of the 84 test cases. After this first experiment, I increased the number of threads to 2 for each compiler, using 4Go limit for zapcc. The limit is for each server and each parallel thread will spawn a new server, so the effective limit is the number of threads times the limit. Even with two threads, I was unable to finish a compilation with zapcc. This is quite disappoint for me since clang is able to run with 4 threads in parallel. Moreover, a big problem with that is that the servers are not always killed when there is no no more memory, they just hang and use all the memory of the computer, which is evidently really inconvenient for service processes. When this happens with clang or gcc, the compiler simply crashes and the memory is released and make is interrupted. Since zapcc is not able to work with more than one thread on this computer, the results are the ones with one thread. I was also surprised to be able to compile the library with clang and four threads, this was not possible before clang-3.9.

Compiler	-j1	-j2	-j3	-j4
gcc-4.9.3	2250.95	1256.36	912.67	760.84
gcc-5.3.0	2305.37	1279.49	918.08	741.38
clang-3.9	2047.61	1102.93	899.13	730.42
zapcc-1.0	1483.73	1483.73	1483.73	1483.73
Difference against Clang	-27.55%	+25.69%	+39.37%	+50.77%
Speedup VS GCC-5.3	-35.66%	+13.75%	+38.09%	+50.03%
Speedup VS GCC-4.9	-34.08%	+15.30%	+38.50%	+48.75%

If we look at the results with only one thread, we can see that there still are some significant improvements when using zapcc, but nowhere near as good as what was seen in the compilation of ETL. Here, the compilation time is reduced by 34% compared to gcc and by 27% compared to clang. This is not bad, since it is faster than the other compilers, but I would have expected better speedups. We can see that g++-4.9 is slightly faster than g++-5.3, but this is not really a significant difference. I'm actually very surprised to find that clang is faster than g++ on this experiment. On ETL, it is always very significantly slower and before, it was also significantly slower on DLL. I was so used to this, that I stopped using it on this project. I may have to reconsider my position when working on this project.

Let's look at the results with more than two threads. Even with two threads, every compiler is faster than zapcc. Indeed, zapcc is slower than Clang by 25% and slower than GCC by about 15%. If we use more threads, the other compilers are becoming even faster and the slowdowns of zapcc are more important. When using four threads, zapcc is about 48% slower than gcc and about 50% slower than clang. This is really showing one big downside of zapcc that has a very large memory consumption. When it is used to compile really heavy template code, it is failing very early to use more processes. And even when there is enough memory, the speedups are not as great as for relatively simpler code.

One may argue that this is not a fair comparison since zapcc does not have the same numbers of threads. However, considering that this is the best zapcc can do on this machine, I would argue that this is a fair comparison in this limited experimental setting. If we were to have a big machine for compilation, which I don't have at work, the zapcc results would likely be more interesting, but in this specific limited case, it shows that zapcc suffers from its high memory consumption. It should also be taken into account that this experiment was done with almost nothing else running on the machine (no browser for instance) to have as much memory as possible available for the compilers. This is not a common use case. Most of the days, when I compile something, I have my browser open, which makes a large difference in memory available, and several other applications (but consoles and vim instances do not really consume memory :D).

This experiment made me realize that the compilation times for this library were quickly becoming crazy. Most of the time, the complete test suite is only compiled on my Continuous Integration machine at home which has a much faster processor and much more RAM. Therefore, it is relatively fast since it uses more threads to compile. Nevertheless, this is not a good point that the unit tests takes so much time to compile. I plan to split the test cases in several sets. Because, currently the real unit tests are compiled with the performance tests and other various tests. I'll probably end up generating three executables. This will help greatly during development. Moreover, I also have a technique to decrease the compilation time by erasing some template parameters at compilation time. This is already ready, but has currently a runtime overhead that I will try to remove and then use this technique everywhere to get back to reasonable compilation times. I'll also try to see if I can find obvious compilation bottlenecks in the code.

Conclusion

To conclude, while zapcc brings some very interesting compilation speedups in some cases like in my ETL library, it also has some downsides, namely huge memory consumption. This memory consumption may prevent the use of several compiler threads and render zapcc much less interesting than other compilers.

When trying to compile my DLL library on a machine with 12Go of RAM with two zapcc threads, it was impossible for me to make it complete. While zapcc was faster with one thread than the other compilers, they were able to use up to four threads and in the end zapcc was about twice slower than clang.

I knew that zapcc memory consumption was very large, but I would have not have expected something so critical. Another feature that would be interesting in zapcc would be to set a max memory hard limit for the server instead of simply a limit on the cache they are able to keep in memory. This would prevent hanging the complete computer when something goes wrong.

I had a good surprise with clang that was actually faster than GCC and also able to work with four threads in parallel. This was not the case with previous version of clang. On ETL, it is still significantly slower than GCC though.

For now, I'll continue using clang on this DLL project and use zapcc only on my ETL project. I'll also focus on improving the compilation time on this project and make it reasonable again.

Release of zapcc 1.0 - Fast C++ compiler

Baptiste Wicht — Thu, 02 Mar 2017 13:50:04 GMT

If you remember, I recently wrote about zapcc C++ compilation speed against gcc 5.4 and clang 3.9 in which I was comparing the beta version of zapcc against gcc and clang.

I just been informed that zapcc was just released in version 1.0. I though it was a good occasion to test it again. It will be compared against gcc-4.9, gcc-5.3 and clang-3.9. This version is based on the trunk of clang-5.0.

Again, I will use my Expression Template Library (ETL) project. This is a purely header-only library with lots of templates. I'm going to compile the full test cases. This is a perfect example for long compilation times.

The current tests are made on the last version of the library and with slightly different parameters for compilation, therefore the absolute times are not comparable, but the speedups should be comparable.

Just like last time, I have configured zapcc to let is use 2Go RAM per caching server, which is the maximum allowed. Moreover, I killed the servers before each tests.

Debug results

Let's start with a debug build, with no optimizations enabled. Every build will use four threads. This is the equivalent of doing make -j4 debug/bin/etl_test without the link step.

Compiler
g++-4.9.3	190.09s
g++-5.3.0	200.92s
clang++-3.9	313.85
zapcc++	81.25
Speedup VS Clang	3.86
Speedup VS GCC-5.3	2.47
Speedup VS GCC-4.9	2.33

The speedups are even more impressive than last time! zapcc is almost four times fast than clang-3.9 and around 2.5 times faster than GCC-5.3. Interestingly, we can see that gcc-5.3 is slighly slower than GCC-4.9.

It seems that they have the compiler even faster!

Release results

Let's look now how the results are looking with optimizations enabled. Again, every build will use four threads. This is the equivalent of doing make -j4 release_debug/bin/etl_test without the link step.

Compiler
g++-4.9.3	252.99
g++-5.3.0	264.96
clang++-3.9	361.65
zapcc++	237.96
Speedup VS Clang	1.51
Speedup VS GCC-5.3	1.11
Speedup VS GCC-4.9	1.06

We can see that this time the speedups are not as interesting as they were. Very interestingly, it's the compiler that suffers the more from the optimization overhead. Indeed, zapcc is three times slower in release mode than it was in debug mode. Nevertheless, it still manages to beat the three other compilers, by about 10% for Gcc and 50% than clang, which is already interesting.

Conclusion

To conclude, we have observed that zapcc is always faster than the three compilers tested in this experiment. Moreover, in debug mode, the speedups are very significant, it was almost 4 times faster than clang and around 2.5 faster than gcc.

I haven't seen any problem with the tool, it's like clang and it should generate code of the same performance, but just compile it much faster. One problem I have with zapcc is that it is not based on an already released version of clang but on the trunk. That means it is hard to be compare with the exact same version of clang and it is also a risk of running into clang bugs.

Although the prices have not been published yet, it is indicated on the website that zapcc is free for non-commercial entities. Which is really great.

If you want more information, you can go to the official website of zapcc

C++ Compiler benchmark on Expression Templates Library (ETL)

Baptiste Wicht — Sun, 11 Dec 2016 13:17:30 GMT

In my Expression Templates Library (ETL) project, I have a lot of template heavy code that needs to run as fast as possible and that is quite intensive to compile. In this post, I'm going to compare the performance of a few of the kernels produced by different compilers. I've got GCC 5.4, GCC 6.20 and clang 3.9. I also included zapcc which is based on clang 4.0.

These tests have been run on an Haswell processor. The automatic parallelization of ETL has been turned off for these tests.

Keep in mind that some of the diagrams are presented in logarithmic form.

Vector multiplication

The first kernel is a very simple one, simple element-wise multiplication of two vectors. Nothing fancy here.

For small vectors, clang is significantly slower than gcc-5.4 and gcc6.2. On vectors from 100'000 elements, the speed is comparable for each compiler, depending on the memory bandwidth. Overall, gcc-6.2 produces the fastest code here. clang-4.0 is slightly slower than clang-3.9, but nothing dramatic.

Vector exponentiation

The second kernel is computing the exponentials of each elements of a vector and storing them in another vector.

Interestingly, this time, clang versions are significantly faster for medium to large vectors, from 1000 elements and higher, by about 5%. There is no significant differences between the different versions of each compiler.

Matrix-Matrix Multiplication

The next kernel I did benchmark with the matrix-matrix multiplication operation. In that case, the kernel is hand-unrolled and vectorized.

There are few differences between the compilers. The first thing is that for some sizes such as 80x80 and 100x100, clang is significantly faster than GCC, by more than 10%. The other interesting fact is that for large matrices zapcc-clang-4.0 is always slower than clang-3.9 which is itself on par with the two GCC versions. In my opinion, it comes from a regression in clang trunk but it could also come from zapcc itself.

The results are much more interesting here! First, there is a huge regression in clang-4.0 (or in zapcc for that matter). Indeed, it is up to 6 times slower than clang-3.9. Moreover, the clang-3.9 is always significantly faster than gcc-6.2. Finally, there is a small improvement in gcc-6.2 compared to gcc 5.4.

Fast-Fourrier Transform

The following kernel is the performance of a hand-crafted Fast-Fourrier transform implementation.

On this benchmark, gcc-6.2 is the clear winner. It is significantly faster than clang-3.9 and clang-4.0. Moreover, gcc-6.2 is also faster than gcc-5.4. On the contrary, clang-4.0 is significantly slower than clang-3.9 except on one configuration (10000 elements).

1D Convolution

This kernel is about computing the 1D valid convolution of two vectors.

While clang-4.0 is faster than clang-3.9, it is still slightly slower than both gcc versions. On the GCC side, there is not a lot of difference except on the 1000x500 on which gcc-6.2 is 25% faster.

And here are the results with the naive implementation:

Again, on the naive version, clang is much faster than GCC on the naive, by about 65%. This is a really large speedup.

2D Convolution

This next kernel is computing the 2D valid convolution of two matrices

There is no clear difference between the compilers in this code. Every compiler here has up and down.

Let's look at the naive implementation of the 2D convolution (units are milliseconds here not microseconds):

This time the difference is very large! Indeed, clang versions are about 60% faster than the GCC versions! This is really impressive. Even though this does not comes close to the optimized. It seems the vectorizer of clang is much more efficient than the one from GCC.

4D Convolution

The final kernel that I'm testing is the batched 4D convolutions that is used a lot in Deep Learning. This is not really a 4D convolution, but a large number of 2D convolutions applied on 4D tensors.

Again, there are very small differences between each version. The best versions are the most recent versions of the compiler gcc-6.2 and clang-4.0 on a tie.

Conclusion

Overall, we can see two trends in these results. First, when working with highly-optimized code, the choice of compiler will not make a huge difference. On these kind of kernels, gcc-6.2 tend to perform faster than the other compilers, but only by a very slight margin, except in some cases. On the other hand, when working with naive implementations, clang versions really did perform much better than GCC. The clang compiled versions of the 1D and 2D convolutions are more than 60% faster than their GCC counter parts. This is really impressive. Overall, clang-4.0 seems to have several performance regressions, but since it's not still a work in progress, I would not be suprised if these regressions are not present in the final version. Since the clang-4.0 version is in fact the clang version used by zapcc, it's also possible that zapcc is introducing new performance regressions.

Overall, my advice would be to use GCC-6.2 (or 5.4) on hand-optimized kernels and clang when you have mostly naive implementations. However, keep in mind that at least for the example shown here, the naive version optimized by the compiler never comes close to the highly-optimized version.

As ever, takes this with a grain of salt, it's only been tested on one project and one machine, you may obtain very different results on other projects and on other processors.

zapcc C++ compilation speed against gcc 5.4 and clang 3.9

Baptiste Wicht — Mon, 05 Dec 2016 17:46:09 GMT

A week ago, I compared the compilation time performance of zapcc against gcc-4.9.3 and clang-3.7. On debug builds, zapcc was about 2 times faster than gcc and 3 times faster than clang. In this post, I'm going to try some more recent compilers, namely gcc 5.4 and clang 3.9 on the same project. If you want more information on zapcc, read the previous posts, this post will concentrate on results.

Again, I use my Expression Template Library (ETL). This is a purely header-only library with lots of templates. I'm going to compile the full test cases.

The results of the two articles are not directly comparable, since they were obtained on two different computers. The one on which the present results are done has a less powerful and only 16Go of RAM compared to the 32Go of RAM of my build machine. Also take into account that that the present results were obtained on a Desktop machine, there can be some perturbations from background tasks.

Just like on the previous results, it does not help using more threads than physical cores, therefore, the results were only computed on up to 4 cores on this machine.

The link time is not taken into account on the results.

Debug build

Let's start with the result of the debug build.

Compiler	-j1	-j2	-j4
g++-5.4.0	469s	230s	130s
clang++-3.9	710s	371s	218s
zapcc++	214s	112s	66s
Speedup VS Clang	3.31	3.31	3.3
Speedup VS GCC	2.19	2.05	1.96

The results are almost the same as the previous test. zapcc is 3.3 times faster to compile than Clang and around 2 times faster than GCC. It seems that GCC 5.4 is a bit faster than GCC 4.9.3 while clang 3.9 is a bit slower than clang 3.7, but nothing terribly significant.

Overall, for debug builds, zapcc can bring a very significant improvement to your compile times.

Release build

Let's see what is the status of Release builds. Since the results are comparable between the numbers of threads, the results here are just for one thread.

This is more time consuming since a lot of optimizations are enabled and more features from ETL are enabled as well.

Compiler	-j1
g++-5.4.0	782s
clang++-3.9	960s
zapcc++	640s
Speedup VS Clang	1.5
Speedup VS GCC	1.22

On a release build, the speedups are much less interesting. Nevertheless, they are still significant. zapcc is still 1.2 times faster than gcc and 1.5 times faster than clang. Then speedup against clang 3.9 is significantly higher than it was on my experiment with clang 3.7, it's possible that clang 3.9 is slower or simply has new optimization passes.

Conclusion

The previous conclusion still holds with modern version of compilers: zapcc is much faster than other compilers on Debug builds of template heavy code. More than 3 times faster than clang-3.9 and about 2 times faster than gcc-5.4. Since it's based on clang, there should not be any issue compiling projects that already compile with a recent clang. Even though the speedups are less interesting on a release build, it is still significantly, especially compared against clang.

I'm really interested in finding out what will be the pricing for zapcc once out of the beta or if they will be able to get even faster!

For the comparison with gcc 4.9.3 and clang 3.7, you can have a look at this article.

If you want more information about zapcc, you can go to the official website of zapcc

zapcc - a faster C++ compiler

Baptiste Wicht — Sat, 26 Nov 2016 12:17:50 GMT

Update: For a comparison against more modern compiler versions, you can read: zapcc C++ compilation speed against gcc 5.4 and clang 3.9

I just joined the private beta program of zapcc. Zapcc is a c++ compiler, based on Clang which aims at being much faster than other C++ compilers. How they are doing this is using a caching server that saves some of the compiler structures, which should speed up compilation a lot. The private beta is free, but once the compiler is ready, it will be a commercial compiler.

Every C++ developer knows that compilation time can quickly be an issue when programs are getting very big and especially when working with template-heavy code.

To benchmark this new compiler, I use my Expression Template Library (ETL). This is a purely header-only library with lots of templates. There are lots of test cases which is what I'm going to compile. I'm going to compare against Clang-3.7 and gcc-4.9.3.

I have configured zapcc to let is use 2Go RAM per caching server, which is the maximum allowed. Moreover, I killed the servers before each tests.

Debug build

Let's start with a debug build. In that configuration, there is no optimization going on and several of the features of the library (GPU, BLAS, ...) are disabled. This is the fastest way to compile ETL. I gathered this result on a 4 core, 8 threads, Intel processor, with an SSD.

The following table presents the results with different number of threads and the difference of zapcc compared to the other compilers:

Compiler	-j1	-j2	-j4	-j6	-j8
g++-4.9.3	350s	185s	104s	94s	91s
clang++-3.7	513s	271s	153s	145s	138s
zapcc++	158s	87s	47s	44s	42s
Speedup VS Clang	3.24	3.103	3.25	3.29	3.28
Speedup VS GCC	2.21	2.12	2.21	2.13	2.16

The result is pretty clear! zapcc is around three times faster than Clang and around two times faster than GCC. This is pretty impressive!

For those that think than Clang is always faster than GCC, keep in mind that this is not the case for template-heavy code such as this library. In all my tests, Clang has always been slower and much memory hungrier than GCC on template-heavy C++ code. And sometimes the difference is very significant.

Interestingly, we can also see that going past the physical cores is not really interesting on this computer. On some computer, the speedups are interesting, but not on this one. Always benchmark!

Release build

We have seen the results on a debug build, let's now compare on something a bit more timely, a release build with all options of ETL enabled (GPU, BLAS, ...), which should make it significantly longer to compile.

Again, the table:

Compiler	-j1	-j2	-j4	-j6	-j8
g++-4.9.3	628s	336s	197s	189s	184s
clang++-3.7	663s	388s	215s	212s	205s
zapcc++	515s	281s	173s	168s	158s
Speedup VS Clang	1.28	1.38	1.24	1.26	1.29
Speedup VS GCC	1.21	1.30	1.13	1.12	1.16

This time, we can see that the difference is much lower. Zapcc is between 1.2 and 1.4 times faster than Clang and between 1.1 and 1.3 times faster than GCC. This shows that most of the speedups from zapcc are in the front end of the compiler. This is not a lot but still significant over long builds, especially if you have few threads where the absolute difference would be higher.

We can also observe that Clang is now almost on par with GCC which shows that optimization is faster in Clang while front and backend is faster in gcc.

You also have to keep in mind that zapcc memory usage is higher than Clang because of all the caching. Moreover, the server are still up in between compilations, so this memory usage stays between builds, which may not be what you want.

As for runtime, I have not seen any significant difference in performance between the clang version and the zapcc. According to the official benchmarks and documentation, there should not be any difference in that between zapcc and the version of clang on which zapcc is based.

Incremental build

Normally, zapcc should shine at incremental building, but I was unable to show any speedup when changing a single without killing the zapcc servers. Maybe I did something wrong in my usage of zapcc.

Conclusion

In conclusion, we can see that zapcc is always faster than both GCC and Clang, on my template-heavy library. Moreover, on debug builds, it is much faster than any of the two compilers, being more than 2 times faster than GCC and more than 3 times faster than clang. This is really great. Moreover, I have not seen any issue with the tool so far, it can seamlessly replace Clang without problem.

It's a bit weird that you cannot allocate more than 2Go to the zapcc servers.

For a program, that's really impressive. I hope that they are continuing the good work and especially that this motivates other compilers to improve the speed of compilation (especially of templates).

If you want more information, you can go to the official website of zapcc

Compile integer Square Roots at compile-time in C++

Baptiste Wicht — Wed, 02 Jul 2014 19:05:11 GMT

For one of my projects, I needed to evaluate a square root at compile-time. There are several ways to implement it and some are better than the others.

In this post, I'll show several versions, both with Template Metaprogramming (TMP) and constexpr functions.

Naive version

The easiest way to implement it is to enumerate the integers until we find two integers that when multiplied are equal to our number. This can easily be implemented in C++ with class template and partial specialization:

template <std::size_t N, std::size_t I=1>
struct ct_sqrt : std::integral_constant<std::size_t, (I*I<N) ? ct_sqrt<N,I+1>::value : I> {};

template<std::size_t N>
struct ct_sqrt<N,N> : std::integral_constant<std::size_t, N> {};

Really easy, isn't it ? If we test it with 100, it gives 10. But, if we try with higher values, we are going to run into problem. For instance, when compiled with 289, here is what clang++ gives me:

src/sqrt/tmp.cpp:5:64: fatal error: recursive template instantiation exceeded maximum depth of 256
struct ct_sqrt : std::integral_constant<std::size_t, (I*I<N) ? ct_sqrt<N,I+1>::value : I > {};
                                                               ^
src/sqrt/tmp.cpp:5:64: note: in instantiation of template class 'ct_sqrt<289, 257>' requested here
struct ct_sqrt : std::integral_constant<std::size_t, (I*I<N) ? ct_sqrt<N,I+1>::value : I > {};
                                                               ^
src/sqrt/tmp.cpp:5:64: note: in instantiation of template class 'ct_sqrt<289, 256>' requested here
struct ct_sqrt : std::integral_constant<std::size_t, (I*I<N) ? ct_sqrt<N,I+1>::value : I > {};
                                                               ^
src/sqrt/tmp.cpp:5:64: note: in instantiation of template class 'ct_sqrt<289, 255>' requested here
struct ct_sqrt : std::integral_constant<std::size_t, (I*I<N) ? ct_sqrt<N,I+1>::value : I > {};
                                                               ^
src/sqrt/tmp.cpp:5:64: note: in instantiation of template class 'ct_sqrt<289, 254>' requested here
struct ct_sqrt : std::integral_constant<std::size_t, (I*I<N) ? ct_sqrt<N,I+1>::value : I > {};
                                                               ^
src/sqrt/tmp.cpp:5:64: note: in instantiation of template class 'ct_sqrt<289, 253>' requested here
struct ct_sqrt : std::integral_constant<std::size_t, (I*I<N) ? ct_sqrt<N,I+1>::value : I > {};
                                                               ^
src/sqrt/tmp.cpp:5:64: note: (skipping 247 contexts in backtrace; use -ftemplate-backtrace-limit=0 to see all)
src/sqrt/tmp.cpp:5:64: note: in instantiation of template class 'ct_sqrt<289, 5>' requested here
struct ct_sqrt : std::integral_constant<std::size_t, (I*I<N) ? ct_sqrt<N,I+1>::value : I > {};
                                                               ^
src/sqrt/tmp.cpp:5:64: note: in instantiation of template class 'ct_sqrt<289, 4>' requested here
struct ct_sqrt : std::integral_constant<std::size_t, (I*I<N) ? ct_sqrt<N,I+1>::value : I > {};
                                                               ^
src/sqrt/tmp.cpp:5:64: note: in instantiation of template class 'ct_sqrt<289, 3>' requested here
struct ct_sqrt : std::integral_constant<std::size_t, (I*I<N) ? ct_sqrt<N,I+1>::value : I > {};
                                                               ^
src/sqrt/tmp.cpp:5:64: note: in instantiation of template class 'ct_sqrt<289, 2>' requested here
struct ct_sqrt : std::integral_constant<std::size_t, (I*I<N) ? ct_sqrt<N,I+1>::value : I > {};
                                                               ^
src/sqrt/tmp.cpp:11:18: note: in instantiation of template class 'ct_sqrt<289, 1>' requested here
    std::cout << ct_sqrt<289>::value << std::endl;
                 ^
src/sqrt/tmp.cpp:5:64: note: use -ftemplate-depth=N to increase recursive template instantiation depth
struct ct_sqrt : std::integral_constant<std::size_t, (I*I<N) ? ct_sqrt<N,I+1>::value : I > {};
                                                               ^

And it is only to compute the square root for 289, not a big number. We could of course increase the template depth limit (-ftemplate-depth=X), but that would only get us a bit farther. If you try with g++, you should see that this works, that is because g++ has a higher template depth limit (900 for 4.8.2 on my machine) where clang has a default limit of 256. It can be noted too that with g++ no context is skipped, therefore the error is quite long.

Now that C++11 gives us constexpr function, we can rewrite it more cleanly:

constexpr std::size_t ct_sqrt(std::size_t n, std::size_t i = 1){
    return n == i ? n : (i * i < n ? ct_sqrt(n, i + 1) : i);
}

Much nicer :) And it works perfectly with 289. And it works quite well up to a large number. But it still fails once we git large numbers. For instance, here is what clang++ gives me with 302500 (550*550):

src/sqrt/constexpr.cpp:8:36: error: constexpr variable 'result' must be initialized by a constant expression
static constexpr const std::size_t result = ct_sqrt(SQRT_VALUE);
                                   ^        ~~~~~~~~~~~~~~~~~~~
src/sqrt/constexpr.cpp:5:38: note: constexpr evaluation exceeded maximum depth of 512 calls
    return n == i ? n : (i * i < n ? ct_sqrt(n, i + 1) : i);
                                     ^
src/sqrt/constexpr.cpp:5:38: note: in call to 'ct_sqrt(302500, 512)'
src/sqrt/constexpr.cpp:5:38: note: in call to 'ct_sqrt(302500, 511)'
src/sqrt/constexpr.cpp:5:38: note: in call to 'ct_sqrt(302500, 510)'
src/sqrt/constexpr.cpp:5:38: note: in call to 'ct_sqrt(302500, 509)'
src/sqrt/constexpr.cpp:5:38: note: in call to 'ct_sqrt(302500, 508)'
src/sqrt/constexpr.cpp:5:38: note: (skipping 502 calls in backtrace; use -fconstexpr-backtrace-limit=0 to see all)
src/sqrt/constexpr.cpp:5:38: note: in call to 'ct_sqrt(302500, 5)'
src/sqrt/constexpr.cpp:5:38: note: in call to 'ct_sqrt(302500, 4)'
src/sqrt/constexpr.cpp:5:38: note: in call to 'ct_sqrt(302500, 3)'
src/sqrt/constexpr.cpp:5:38: note: in call to 'ct_sqrt(302500, 2)'
src/sqrt/constexpr.cpp:8:45: note: in call to 'ct_sqrt(302500, 1)'
static constexpr const std::size_t result = ct_sqrt(SQRT_VALUE);
                                            ^

Again, we run into the limits of the compiler. And again, the limit can be change with fconstexpr-backtrace-limit=X. With g++, the result is the same (without the skipped part, which makes the error horribly long), but the command to change the depth is -fconstexpr-depth=X.

So, if we need to compute higher square roots at compile-time, we need a better version.

Binary Search version

To find the good square root, you don't need to iterate through all the numbers from 1 to N, you can perform a binary search to find the numbers to test. I found a very nice implementation by John Khvatov (source).

Here is an adaptation of its code:

#define MID(a, b) ((a+b)/2)
#define POW(a) (a*a)

template<std::size_t res, std::size_t l = 1, std::size_t r = res>
struct ct_sqrt;

template<std::size_t res, std::size_t r>
struct ct_sqrt<res, r, r> : std::integral_constant<std::size_t, r> {};

template <std::size_t res, std::size_t l, std::size_t r>
struct ct_sqrt : std::integral_constant<std::size_t, ct_sqrt<res,
        (POW(MID(r, l)) >= res ? l : MID(r, l)+1),
        (POW(MID(r, l)) >= res ? MID(r, l) : r)>::value> {};

With smart binary search, you can reduce A LOT the numbers that needs to be tested in order to find the answer. It very easily found the answer for 302500. It can find the square root of almost all integers, until it fails due to overflows. I think it is really great :)

Of course, we can also do the constexpr version:

static constexpr std::size_t ct_mid(std::size_t a, std::size_t b){
    return (a+b) / 2;
}

static constexpr std::size_t ct_pow(std::size_t a){
    return a*a;
}

static constexpr std::size_t ct_sqrt(std::size_t res, std::size_t l, std::size_t r){
    return
        l == r ? r
        : ct_sqrt(res, ct_pow(
            ct_mid(r, l)) >= res ? l : ct_mid(r, l) + 1,
            ct_pow(ct_mid(r, l)) >= res ? ct_mid(r, l) : r);
}

static constexpr std::size_t ct_sqrt(std::size_t res){
    return ct_sqrt(res, 1, res);
}

Which is a bit more understandable. It works the same way than the previous one and is only limited by numeric overflow.

C++14 Fun

In C++14, the constraints on constexpr functions have been highly relaxed, we can now use variables, if/then/else statements, loops and so on... in constexpr functions making them much more readable. Here is the C++14 version of the previous code:

static constexpr std::size_t ct_sqrt(std::size_t res, std::size_t l, std::size_t r){
    if(l == r){
        return r;
    } else {
        const auto mid = (r + l) / 2;

        if(mid * mid >= res){
            return ct_sqrt(res, l, mid);
        } else {
            return ct_sqrt(res, mid + 1, r);
        }
    }
}

static constexpr std::size_t ct_sqrt(std::size_t res){
    return ct_sqrt(res, 1, res);
}

I think this version is highly superior than the previous version. Don't you think ?

It performs exactly the same as the previous. This can only be done in clang for now, but that will come eventually to gcc too.

Conclusion

As you saw, there are several ways to compute a square root at compile-time in C++. The constexpr versions are much more readable and generally more scalable than the template metaprogramming version. Moreover, now, with C++14, we can write constexpr functions almost as standard function, which makes really great.

I hope that is is helpful to some of you :)

All the sources are available on Github: https://github.com/wichtounet/articles/tree/master/src/sqrt

Build OpenCV with libc++ on Gentoo and static-libs

Baptiste Wicht — Tue, 10 Jun 2014 12:09:29 GMT

When you build C++ projects with CLang, you have the choice between using the stdlibc++ that is provided along G++ and the new libc++ that is provided by CLang.

libc++ is another implementation of the C++ Standard Library. This implementation is dual-licensed under the MIT license and UIUC license. It is especially targeting C++11 and has already 100% support for C++14. This last point is the reason that I use libc++ on several of my projects. Moreover, it is also the default on Mac OS X.

The problem with linking with another library is that you can only works with libraries that have been compiled with libc++ support. For instance, if you want to use Boost dynamic libraries, you'll have to compile Boost from sources with libc++.

For one of my project, I'm using OpenCV and libc++. To simplify the installation of OpenCV, I created a new ebuild with a libcxx use flag to selectively build the library with libc++. This requires LLVM/CLang on the build machine. Moreover, by default, the Gentoo ebuild does not have support for building the static libraries. The reason for that is that OpenCV build is not able to build dynamic and static libraries. I added a static-libs use flag that build the static libraries by building OpenCV a second time after the first. That will likely double the compile time (unless ccache is used). Anyhow, it is simple easier than to build that by hand on several machine.

The ebuild is available on my overlay. You can add the overlay to your machine by modifying /etc/layman/layman.cfg:

overlays: http://www.gentoo.org/proj/en/overlays/repositories.xml
          http://github.com/wichtounet/wichtounet-overlay/raw/master/repository.xml

Then, you can add it to layman:

layman -S
layman -a wichtounet

For now, I have created an ebuild for opencv-2.4.8-r1. If someone is interested in other versions, I'd be glad to create new ebuilds.

I hope that this ebuild will be helpful.

Software Reliability Presentation

Baptiste Wicht — Sat, 07 Jun 2014 14:45:33 GMT

Software Reliability

In behalf of my school (College of Engineering and Architecture of Fribourg), I presented a shoft presentation about Software Reliability. In this presentation, I outline the main issues about the subject and propose some solutions:

Software Validation
Defensive Programming
Software Analysis Tools

In the Software Analysis Tools, I present three tools: cppcheck, Valgrind and the Clang Static analyzer. Several examples are presented for each tools as well as some recommendations for using them. A short presentation of SonarQube is also performed.

I thought that it could be of some interest to some of the readers, so here it is:

Don't hesitate if you have any comments or questions about the presentation ;)

The source code for the examples is available on Github.

Install and Use CLang Static Analyzer on a CMake project

Baptiste Wicht — Wed, 09 Apr 2014 14:39:11 GMT

I recently started a bit of work on my compiler (eddic) again. I started by adapting it to build on CLang with libc++. There was some minor adaptions to make it compile, but nothing really fancy. It now compiles and runs fine on LLVM/Clang 3.4 with the last version of libc++. I'm gonna use some features of C++14 in it and I plan to refactor some parts to make it more STL-correct. I also plan to use only CLang on eddic right now, since C++14 support of GCC is not released right now.

I decided it was a good time to try again the CLang static analyzer.

Installation

If, like me, you're using Gentoo, the static analyzer is directly installed with the sys-devel/clang package, unless you disabled the static-analyzer USE flag.

If your distribution does not ship the static analyzer directly with CLang, you'll have to install it manually. To install it from sources, I advise you to follow the Official Installations instruction.

Usage

The usage of CLang static analyzer can be a bit disturbing at first. Most static analysis tools generally takes the sources directly and do their stuff. But that is not how Clang Static Analyzer works. It works as a kind of monitor in top of building the program, using scan-build. When you are analyzing a program, you are also building the program.

For instance, if you are compiling a source file like that:

clang [clang-options] source_file.cpp

you can perform static analysis like that:

scan-build [scan-build-options] clang [clang-options] source_file.cpp

scan-build works by replacing calls to the compiler by calls to ccc-analyzer . This works generally well, but there are some cases where that things get a bit more complicated. That is the case of CMake where the paths to the compiler are hardcoded in the generated makefiles.

For that, you have to run cmake and make with scan-build:

export CCC_CC=clang
export CCC_CXX=clang++
scan-build cmake -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER=clang .
scan-build make

This can take a very long time. On eddic, it is about three times slower than a normal compilation. An important point to note about performance, is that you can run compilations in parallel (-j option of make) and that it is supported by scan-build quite well.

Once analysis is performed, the found bugs are put into an HTML report. By default, the HTML report is created in /tmp/, but you can specificy the folder with -o option of scan-build.

You can enable or disable checker with the -enable-checker and -disable-checker options of scan-build.

Results on eddic

Several versions of Clang ago, I tried the static analyzer on eddic, but it failed on several source files without producing any results. Moreover, at this time, I don't think there was any nice HTML report at this time.

I ran it again on eddic with the last versions. Here is a picture of the generated report:

As you can see, 14 bugs have been found. Unfortunately, none of them is a real bug on my code, but they are not all false positives neither. For instance, here is some unreachable code report:

It is indeed an unreachable statement, but it is expected, since it is an assert to ensure that the code is unreachable. But that proves that the analysis works ;)

Even if it didn't found anything, this time it worked much better than the last time I checked and the HTML results are just really good.

I hope you found this article interesting. If you happen to have interesting results on your codebase with the CLang static analyzer, I'd be glad to hear about them ;)

GCC 4.7 vs CLang 3.1 on eddic

Baptiste Wicht — Mon, 12 Nov 2012 08:28:44 GMT

Now that eddic can be compiled with CLang, I wanted to compare the differences in compilation time and in performance of the generated executable between those two compilers. The tests are done using GCC 4.7.2 and CLang 3.1 on Gentoo.

Compilation Time

The first thing that I tested has been the compilation time of the two compilers to compile eddic with different flags. I tested the compilation in debug mode and with -O2 and -O3.

The most interesting fact in these results is that CLang is much faster than GCC. It takes twice less times to compile eddic with CLang in debug mode than with GCC. The impact on optimizations on CLang's compilation is also more important than on GCC. For both compilers, -O3 does not seems to add a lot of overhead.

Runtime performance

Then, I tested the performance of the generated executable. I tested it on three things, the whole test suite and two test cases that I know are the slowest for the EDDI Compiler. For each case, I took the slowest value of 5 consecutive executions.

The difference are very small. In -02, GCC performs a bit better, but in -O3, the performance are equivalent. I was a bit disappointed by the results, because I thought that there would be higher differences. It seems that CLang is not as far from GCC that some people would like to say. It also certainly depends on the program being compiled.

Conclusion

It is clear that CLang is much faster than GCC to compile eddic. Moreover, the performance of the generated executable are almost similar.

I will continue to use CLang as my development compiler and switches between the two when I'm doing performance benchmarking. I will try to update the benchmark once new versions of GCC / CLang are available.

eddic compiles with CLang 3.1

Baptiste Wicht — Thu, 01 Nov 2012 08:11:05 GMT

I finally added support for compiling eddic with LLVM CLang 3.1 !

The current development version can be completely compiled with CLang. Starting with the version 1.1.4, all versions of eddic will be support GCC and CLang.

The changes have not been as painful as I first thought.

The main problem that I has was about a static const variable of a class that had no user-constructor. GCC allows that, but it is not standard compliant and CLang was complaining.
Another problem that I encountered was about the used of bit flags and Template Meta Programming. I simplified that by the use of a simple type traits and it worked. I don't really know why this does not worked at first.
The remaining effort was to fix the several warnings that CLang had.

CLang also fixed a bug in my code with a warning on a assignment that was not supposed to be an assignment, thanks CLang.

The most interesting fact about CLang is that is it twice faster to build eddic than GCC. I think I'm gonna use it during development to fasten the compile time. Moreover, even if I only worked two days with it, it seems that the error messages are indeed better than the GCC's ones.

I haven't tried to compare the performances of eddic in both cases, but I will do that in the future, soon after the 1.1.4 version is released.

I tried the CLang static analyzer on eddic but it didn't found any bugs. Moreover, it crashed on several of my files. I didn't found why for now, but I will continue to investigate, perhaps I'm not using it correctly.

I expect to publish the next version of eddic in the next two weeks. This version has much more improvements that I thought at first and I have less time to work now that I'm working on my Master thesis.

More informations on CLang: The official site.