Blog blog("Baptiste Wicht"); (Posts about Linux)

Budgetwarrior: Track assets and portfolio, savings rates and auto-completion

Baptiste Wicht — Thu, 12 Oct 2017 17:40:14 GMT

This last month, I've been reading quite a few blogs about personal finance and I've decided to integrate more features into budgetwarrior. This post is about three new features that I've integrated. It's not yet a new release, so if you want to test this version, you'll have to compile it from the master branch on Git.

As it was last time, the values on my screenshots have all been randomized.

If you have several assets with different distributions, I believe it is a great value to have them all shown at the same time. Especially if you want to change the distribution of your portfolio or if you plan big changes in it.

Track assets

The first feature I've added is a feature to precisely track each of your assets independently. And you can also track the allocation of your portfolio in terms of stocks, bonds and cash. The tool also lets you set the desired distribution of your assets and will compute the difference that you should make in order to comply to your desired distribution.

First, you need to define all your asset classes (your accounts, funds, and stocks, ...) and their distribution with budget asset add. It also supports to set a currency. The default currency is now CHF, but you can set it in the configuration file, for instance default_currency=USD. You can see your assets using budget asset:

You can then set the value of your assets using budget asset value add. The system will save all the values of your assets. For now, only the last value is used in the application to display. In the future, I plan to add new reports for evolution of the portfolio over time. You can see your current net worth with the budget asset value:

The different currencies will all be converted to the default currency.

Savings rate

The second change I did is to compute the savings rate of each month and year. The savings rate is simply the portion of your income that you are able to save each month. The savings rate for a year is simple the average of the savings rate of each month.

The savings rate of the month can be seen with budget overview month:

The saving rates of each month can also be seen in the overview of the year with budget overview year:

This shows the savings rate of each month, the average of the year and the average of the current year up to the current month.

The savings rate is a very important metric of your budget. In my case, it's currently way too low and made me realize I really need to save more. Any savings rate below 10% is too low. There are no rule as too much it should be, but I'd like to augment mine to at least 20% next year.

Auto-completion

The last feature is mostly some quality-of-life improvement. Some of the inputs in the console can now be completed. It's not really auto-completion per se, but you can cycle through the list of possible values using the UP and DOWN.

This makes it much easier to set some values such as asset names (in budget asset value add for instance), account names and objective types and sources. I'm trying to make the input of values easier.

Conclusion

I don't know exactly what else will be integrated in this feature, but I may already improve some visualization for asset values. If I learn something new about personal finance that I may integrate in the tool, I'll do it as well.

If you are interested by the sources or want to install this version, you can download them on Github: budgetwarrior.

The new features are in the master branch.

If you have a suggestion for a new features or you found a bug, please post an issue on Github, I'd be glad to help you.

If you have any comment, don't hesitate to contact me, either by letting a comment on this post or by email.

budgetwarrior 0.4.2 - Budget summary and improved fortune reports

Baptiste Wicht — Thu, 14 Sep 2017 18:42:39 GMT

Almost three years ago, I published the version 0.4.1 of budgetwarrior. Since then, I've been using this tool almost every day to manage my personal budget. This is the only tool I use to keep track of my expenses and earnings and it makes a great tool for me. I recently felt that it was missing a few features and added them and polished a few things as well and release a new version with all the new stuff. This new version is probably nothing fancy, but a nice upgrade of the tool.

Don't pay too much attention to the values in the images since I've randomized all the data for the purpose of this post (new feature, by the way :P).

New summary view

I've added a new report with budget summary:

This view gives concise information about the current state of your accounts. It also gives information about your yearly and monthly objectives. Finally, it also gives information about the last two fortune values that you've set. I think this make a great kind of dashboard to view most of the information. If your terminal is large enough, the three parts will be shown side by side.

Improved fortune report

I've made a few improvements to the budget fortune view:

It now display the time between the different fortune values and it compute the average savings (or avg losses) per day in each interval and in average from the beginning of the first value.

Various changes

The balance does not propagate over the years anymore. This should mainly change the behaviour of budget overview. I don't think it was very smart to propagate it all the time. The balance now starts at zero for each year. If you want the old system, you can use the multi_year_balance=true option in the .budgetrc configuration file.

The recurring expenses do not use an internal configuration value. This does not change anything for the behaviour, but means that if you sync between different machines, it will avoid a lot of possible conflicts :)

Fixed a few bugs with inconsistency between the different views and reports. Another bug that was fixed is that budget report was not always displaying the first month of the year correctly, this is now fixed.

The graphs display in budget report are now automatically adapted to width of your terminal. Finally, the budget overview command also displays more information about the comparison with the previous month.

Installation

If you are on Gentoo, you can install it using layman:

layman -a wichtounet
emerge -a budgetwarrior

If you are on Arch Linux, you can use this AUR repository.

For other systems, you'll have to install from sources:

git clone --recursive git://github.com/wichtounet/budgetwarrior.git
cd budgetwarrior
make
sudo make install

Conclusion

A brief tutorial is available on Github: Starting guide.

If you are interested by the sources, you can download them on Github: budgetwarrior.

If you have any suggestion for a new feature or an improvement to the tool or you found a bug, please post an issue on Github, I'd be glad to help you. You can post a comment directly on this post :)

If you have any other comment, don't hesitate to contact me, either by letting a comment on this post or by email.

I hope that this application can be useful to some of you command-line adepts :)

How to fix mdadm RAID5 / RAID6 growing stuck at 0K/s ?

Baptiste Wicht — Sat, 12 Aug 2017 10:28:43 GMT

I just started growing again my RAID6 array from 12 to 13 disks and I encountered a new issue. The reshape started, but with a speed of 0K/s. After some searching, I found a very simple solution:

echo max > /sys/block/md0/md/sync_max

And the reshape started directly at 50M/s :)

The solution is the same if you are growing any type of RAID level with parity (RAID5, RAID6, ...).

Normally, the issues I have are related to speed not very good. I've written a post in the post about how to speed up RAID5 / RAID6 growing with mdadm. Although RAID5 / RAID6 growing, or another reshape operation, will never be very fast, you can still speed up the process a lot from a few days to a few hours. Currently, my reshape is working at 48M/s and I'm looking at around 16 hours of reshape, but I have 13 disks of 3To, so it's not so bad.

I hope this very simple tip can be helpful to some of you :)

Migrated from owncloud 5 to Nextcloud 11

Baptiste Wicht — Fri, 03 Mar 2017 08:08:58 GMT

For several years now I've been using Owncloud running on one of my servers. I'm using simply using as a simple synchronization, I don't use any of the tons of fancy features they keep adding. Except from several synchronization issues, I haven't had too much issues with it.

However, I have had a very bad time with updates of Owncloud. The last time I tried, already long ago, was to upgrade from 5.0 to 6.0 and I never succeeded without losing all the configuration and having to do the resync. Therefore, I've still an Owncloud 5.0 running. From this time, I had to say that I've been lazy and didn't try again to upgrade it. Recently, I've received several mails indicating that this is a security threat.

Since I was not satisfied with updates in Owncloud and its security has been challenged recently, I figured it would be a good moment to upgrade to Nextcloud which is a very active fork of Owncloud that was forked by developers of Owncloud.

I haven't even tried to do an upgrade from such an old version to the last version of Nextcloud, it was doomed to fail. Therefore, I made a new clean installation. Since I only use the sync feature of the tool, it does not really matter, it is just some time lost to sync everything again, but nothing too bad.

I configured a new PostgreSQL on one of my servers for the new database and then installed Nextcloud 11 on Gentoo. It's a bit a pain to have a working Nginx configuration for Nextcloud, I don't advice to do it by hand, better take one from the official documentation, you'll also gain some security. One very bad thing in the installation process is that you cannot choose the database prefix, it's set like Owncloud. The problem with that is that you cannot install both Owncloud and Nextcloud on the same database which would be more practical for testing purpose. It's a bit retarded in my opinion, but not a big problem in the end. Other than these two points, everything went well and it was installation pretty nicely. Then, you should have your user ready to go.

As for the interface, I don't think there is a lot to tell here. Most of it is what you would except from this kind of tool. Moreover, I very rarely use the web interface or any of the feature that are not the sync feature. One thing that is pretty cool I think is the monitoring graphs in the Admin section of the interface. You can the number of users connected, the memory used and the CPU load. It's pretty useful if you share your Nextcloud between a lot of different users.

I didn't have any issue with the sync either. I used the nextcloud-client package on Gentoo directly and it worked perfectly directly. It took about 10 minutes to sync everything again (about 5GB). I'll have to do the same thing on my other computer as well, but I don't think I'll have any issue.

So far, I cannot say that this is better than Owncloud, I just hope the next upgrade will fare better than they did on Owncloud. Moreover, I also hope that the security that they promise is really here and I won't have any problem with it. I'll see in the future!

For more information about nextcloud and owncloud compared to each other, you can read this in-depth comparison, by David Feldman.

Simplify Deep Learning Library usage on Linux and Windows!

Baptiste Wicht — Fri, 29 Apr 2016 10:48:18 GMT

No, I'm not dead ;) I've been very busy with my Ph.D (and playing Path of Exile, let's be honest...) and haven't had time to write something here in a long time.

Until now, there was too way to use my Deep Learning Library (DLL) project:

Write a C++ program that uses the library
Install DLL and write a configuration file to define your network and the problem to solve

The first version gives you all the features of the tool and allows you to build exactly what you need. The second version is a bit more limited, but does not require any C++ knowledge. However, it still does require a recent C++ compiler and build system.

Due to the high C++ requirements that are not met by Visual Studio and the fact that I don't work on Windows, this platform is not supported by the tool. Until now!

I've added a third option to use DLL in the form of a Docker image to make the second option even easier and allow the use of DLL on Windows. All you need is Docker, which is available on Linux, Mac and Windows. This is still limited to the second option in that you need to write a configuration describing the network, but you need to build DLL and don't need to install all its dependencies.

Usage

To install the image, you can simply use docker pull:

docker pull wichtounet/docker-dll

Then, to run it, you have to create a folder containing a dll.conf file and mount in the container at /dll/data/. There are some examples in the image repository. For instance, on Linux from the cloned repository:

docker run -v ${pwd}/rbm_mnist/:/dll/data/ wichtounet/docker-dll

or on Windows:

docker run -v /c/Users/Baptiste/rbm_mnist/:/dll/data wichtounet/docker-dll

This will automatically run the actions specified in the configuration file and train your network.

Conclusion

I would really have thought this would be harder, but it turned out that Docker is a very good solution to deploy multiplatform demo tools :)

As of now, there is only support for mnist data format in the tool in this form, but I plan to add basic CSV support as well in the near future.

I hope that this will help people willing to try the library with a simpler usage.

How to speed up RAID (5-6) growing with mdadm ?

Baptiste Wicht — Sat, 07 Mar 2015 14:01:56 GMT

Yesterday, I added my 11th disk to my RAID 6 array. As the last time it took my more than 20 hours, I spent some time investigating how to speed things up and this post contains some tips on how to achieve good grow performances. With these tips, I have been able to reach a speed of about 55K in average during reshape. It did finish in about 13 hours.

First, take into account that some of these tips may depend on your configuration. In my case, this server is only used for this RAID, so I don't care if the CPU is used a lot during rebuild or if other processes are suffering from the load. This may not be the case with your configuration. Moreover, I speak only of hard disks, if you use SSD RAID, there are probably better way of tuning the rebuild (or perhaps it is fast enough). Finally, you have know that a RAID reshape is going to be slow, there is no way you'll grow a 10+ RAID array in one hour. G

In the examples, I use /dev/md0 as the raid array, you'll have to change this to your array name.

The first 3 tips can be used even after the rebuild has started and you should see the differences in real-time. But, these 3 tips will also be erased after each reboot.

Increase speed limits

The easiest thing to do is to increase the system speed limits on raid. You can see the current limits on your system by using these commands:

sysctl dev.raid.speed_limit_min
sysctl dev.raid.speed_limit_max

These values are set in Kibibytes per second (KiB/s).

You can put them to high values:

sysctl -w dev.raid.speed_limit_min=100000
sysctl -w dev.raid.speed_limit_max=500000

At least with these values, you won't be limited by the system.

Increase stripe cache size

By allowing the array to use more memory for its stripe cache, you may improve the performances. In some cases, it can improve performances by up to 6 times. By default, the size of the stripe cache is 256, in pages. By default, Linux uses 4096B pages. If you use 256 pages for the stripe cache and you have 10 disks, the cache would use 10*256*4096=10MiB of RAM. In my case, I have increased it to 4096:

echo 4096 > /sys/block/md0/md/stripe_cache_size

The maximum value is 32768. If you have many disks, this may well take all your available memory. I don't think values higher than 4096 will improve performance, but feel free to try it ;)

Increase read-ahead

If configured too low, the read-ahead of your array may make things slower.

You can see get the current read-ahead value with this command:

blockdev --getra /dev/md0

These values are in 512B sector. You can set it to 32MB to be sure:

blockdev --setra 65536 /dev/md0

This can improve the performances, but don't expect this to be a game-changer unless it was configured really low at the first place.

Tip: Reshape stuck at 0K/s

If reshape starts, but with a speed of 0K/s, you can try to issue this simple command:

echo max > /sys/block/md0/md/sync_max

And the reshape should start directly at your maximum speed.

The solution is the same if you are growing any type of RAID level with parity (RAID5, RAID6, ...).

Bonus: Speed up standard resync with a write-intent bitmap

Although it won't speed up the growing of your array, this is something that you should do after the rebuild has finished. Write-intent bitmaps is a kind of map of what needs to be resynced. This is of great help in several cases:

When the computer crash (power shutdown for instance)
If a disk is disconnected, then reconnected.

In these case, it may totally avoid the need of a rebuild which is great in my opinion. Moreover, it does not take any space on the array since it uses space that is not usable by the array.

Here is how to enable it:

mdadm --grow --bitmap=internal /dev/md0

However, it may cause some write performance degradation. In my case, I haven't seen any noticeable degradation, but if it is the case, you may want to disable it:

mdadm --grow --bitmap=none /dev/md0

Bonus: Monitor rebuild process

If you want to monitor the build process, you can use the watch command:

watch cat /proc/mdstat

With that you'll see the rebuild going in real-time.

You can also monitor the I/O statistics:

watch iostat -k 1 2

Bonus: How to grow a RAID 5-6 array

As a sidenote, this section indicates how to grow an array. If you want to add the disk /dev/sdl to the array /dev/md0, you'll first have to add it:

mdadm --add /dev/md0 /dev/sdl

This will add the disk as a spare disk. If you had 5 disks before, you'll want to grow it to 6:

mdadm --grow --backup-file=/root/grow_md0_backup_file --raid-devices=6 /dev/md0

The backup file must be on another disk of course. The backup file is optional but improves the chance of success if you have a power shutdown or another form of unexpected shutdown. If you know what you're doing, you can grow it without backup-file:

mdadm --grow --raid-devices=6 /dev/md0

This command will return almost instantly, but the actual reshape won't likely be finished for hours (maybe days).

Once the rebuild is finished, you'll still have to extend the partitions with resize2fs. If you use LVM on top of the array, you'll have to resize the Physical Volume (PV) first:

pvresize /dev/md0

and then extend the Logical Volume (s) (LV). For instance, if you want to add 1T to a LV named /dev/vgraid/work:

vgextend -r -L+1T /dev/vgraid/work

The -r option will automatically resize the underlying filesystem. Otherwise, you'd still have to resize it with resize2fs.

Conclusion

These are the changes I have found that speed up the reshape process. There are others that you may test in your case. For instance, in some systems disabling NCQ on each disk may help.

I hope that these tips will help you doing fast rebuilds in your RAID array :)

Linux tip: Force systemd networkd to wait for DHCP

Baptiste Wicht — Wed, 01 Oct 2014 18:18:55 GMT

Recently, I started using systemd-networkd to manage my network. It works really good for static address configuration, but I experienced some problem with DHCP. There is DHCP client support integrated into systemd, so I wanted to use this instead of using another DHCP client.

(If you are not familiar with systemd-networkd, you can have a look at the last section of this article)

The problem with that is that services are not waiting for DHCP leases to be obtained. Most services (sshd for instance), are waiting for network.target, however, network.target does not wait for the DHCP lease to be obtained from the server. If you configured ssh on a specific IP and this IP is obtained with DHCP, it will fail at startup. The same is true for NFS mounts for instance.

Force services to wait for the network to be configured

The solution is to make services like sshd waits for network-online.target instead of network.target. There is a simple way in systemd to override default service files. For a X.service, systemd will also parse all the /etc/systemd/X.service.d/*.conf files.

For instance, to make sshd be started only after DHCP is finished

/etc/systemd/system/sshd.service.d/network.conf:

[Unit]
Wants=network-online.target
After=network-online.target

However, by default, network-online.target does not wait for anything. You'll have to enable another service to make it work:

systemctl enable systemd-networkd-wait-online

And another note, at least on Gentoo, I had to use systemd-216 for it to work:

emerge -a "=sys-apps/systemd-216"

And after this, it worked like a charm at startup.

Force NFS mounts to wait for the network

There is no service file for nfs mounts, but there is a target remote-fs.target that groups the remote file systems mounts. You can override its configuration in the same as a service:

/etc/systemd/system/remote-fs.target.d/network.conf:

[Unit]
Wants=network-online.target
After=network-online.target

Conclusion

Here we are, I hope this tip will be useful to some of you ;)

Appendix. Configure interface with DHCP with systemd

To configure an interface with DHCP, you have to create a .network file in /etc/systemd/network/. For instance, here is my /etc/systemd/network/local.network file:

[Match]
Name=enp3s0

[Network]
DHCP=v4

and you have to enable systemd-networkd:

systemctl enable systemd-networkd

budgetwarrior 0.4.1 - Expense templates and year projection

Baptiste Wicht — Sun, 21 Sep 2014 18:53:53 GMT

I've been able to finish the version 0.4.1 of budgetwarrior before I though :)

Expense templates

The "most useful" new feature of this release is the ability to create template for expenses.

For that, you can give an extra parameter to budget expense add:

budget expense add template name

This will works exactly the same as creating a new expense expect that it will be saved as a template. Then, the next time you do:

budget expense add template name

A new expense will be created with the date of the day and with the name and amount saved into the template. You can create as many templates as you want as long as they have different names. You can see all the templates you have by using 'budget expense template'. A template can be deleted the exact same as an expense with 'budget expense delete id'.

I think this is very useful for expense that are made several times a month, for instance a coffee at your workplace. The price should not change a lot and it is faster to just use the template name rather than entering all the information again.

Year prediction

You can now see what would be next year if you changed a bit your expenses. For instance, how much would you still have at the end of the year if you increased your house expenses by 20% and reduced your insurances by 5% ?

The 'budget predict' can be used for that purpose. You can enter a multiplier for each account in your budget and a new year will be "predicted" based on the expenses of the current year multiplied by the specified multiplier:

I think that this feature can be very useful if you want to estimate how your budget will be for moving to a more expensive house or another insurance for instance.

Various changes

Two accounts can be merged together with the 'budget account migrate' command. This command will move all expenses from an account to another and adapt the amount of the target account. The source account will be deleted. This supports migrated accounts.

The 'budget wish list' command will now display the mean accuracy of your predictions.

You don't need Boost anymore for this project. The only remaining dependency is libuuid. I will perhaps remove it in the next version since the UUID are not used in the application for now.

The command 'budget gc' will clean the IDs of all your data in order to fill the holes and make all the IDs contiguous. It is mostly a feature for order-freaks like me who do not like to have holes in a sequence of identifiers ;)

There was a bug in the monthly report causing the scale to be displayed completely moved, it is now fixed:

Installation

If you are on Gentoo, you can install it using layman:

layman -a wichtounet
emerge -a budgetwarrior

If you are on Arch Linux, you can use this AUR repository.

For other systems, you'll have to install from sources:

git clone git://github.com/wichtounet/budgetwarrior.git
cd budgetwarrior
make
sudo make install

Conclusion

If you are interested by the sources, you can download them on Github: budgetwarrior.

If you have a suggestion for a new features or you found a bug, please post an issue on Github, I'd be glad to help you.

If you have any comment, don't hesitate to contact me, either by letting a comment on this post or by email.

A Mutt journey: Search mails with notmuch

Baptiste Wicht — Sat, 02 Aug 2014 15:24:12 GMT

In the previous installment in the Mutt series, I've talked about my Mutt configuration. In this post, I'll talk about notmuch and how to use it to search through mails.

By default, you can search mails in Mutt by using the / key. By doing that, you can only search in the current folder. This is very fast, but this is not always what you want. When you don't know in which folder the mail you are looking for is, you don't want to test each folder. By default, there are no feature to achieve global searching in Mutt.

That is where notmuch comes to the rescue. notmuch is a very simple tool that allows you to search through your mail. As its name indicates, it does not do much. It doesn't download your mails, you have to have them locally, which is perfect if you use offlineimap. It does not provide a user interface, but you can query it from the command line and it can be used from other tools. It should be available in most of the distributions.

Configuration

The configuration of notmuch is fairly simple. You can write your .notmuch-config directly or run notmuch setup that will interactively help you to fill the configuration.

Here is my configuration:

[database]
path=/data/oi/Gmail/

[user]
name=Baptiste Wicht
primary_email=baptiste.wicht@gmail.com

[new]
tags=inbox
ignore=

[search]
exclude_tags=deleted;

[maildir]
synchronize_flags=true

It needs of cours the place where your mails are stored. Then, some information about you. The next section is to specify which tags you want to add to new mails. Here, I specified that each new mail must be tagged with inbox. You can add several tags to new mails. In the [search] section, the excluded tags are specified.

Usage

Once you have configured notmuch, you can run notmuch new to process all existing mails. The first run may take some time (in minutes, it is still quite fast), but the subsequent runs will be very fast. You should run notmuch after each offlineimap run. I personally run it in a shell script that is run by cron. You could also use one of the hooks of offlineimap to run notmuch.

Once indexing has been done, you can start searching your mails. The first option to search mail is simply to use notmuch search <query> from the command line. This will directly displays the results. Search is instant on my mails.

If you use mutt-kz like me, notmuch support is directly integrated. You can type X, and then type your query like notmuch://?query=X and the results will be displayed as a normal Mutt folder. You can open mails directly from here and you can also edit the mails as if you were in their source folders. This is really practical.

If you use mutt, you can have the same experience, by using the notmuch-mutt patch (here <http://notmuchmail.org/notmuch-mutt/>). In several distributions, there is an option to build it with this support or another package to add the feature.

Another feature of notmuch is its ability to tag mails. It automatically tags new mails and deleted mails. But you can also explicitely tag messages by using notmuch tag. For instance, to tag all messages from the notmuch mailing list:

notmuch tag +notmuch -- tag:new and to:notmuch@notmuchmail.org

I personally don't use this feature since I use imapfilter and IMAP folders to sort my mail, but it can be very useful. You can run these commands in the cronjob and always have you tags up to date. Tags can then be used in notmuch to search or to create virtual folder in Mutt.

Conclusion

That is already more or less everything that there is to know about notmuch. It does not do a lot of thing, but it does them really well.

That concludes the series of posts on Mutt. If you have any question on my Mutt configuration, I'd be glad to extend on the comments.

A Mutt journey: Mutt configuration

Baptiste Wicht — Tue, 22 Jul 2014 20:14:32 GMT

If you've followed my Mutt posts, you'll know that I'm filtering my mails with imapfilter and downloading them with offlineimap.

In this post, I'll share my Mutt configuration. I'm not using Mutt directly, but mutt-kz which is a fork with good notmuch integration. For this post, it won't change anything.

Configuration

The complete configuration is made in the .muttrc file. Mutt configuration supports the source command so that you can put some of your settings in another files and source them from the .muttrc file. You'll see that the configuration can soon grow large and therefore, splitting it in several files will save you a lot of maintenance issues ;)

First, let's tell Mutt who we are:

set from = "baptiste.wicht@gmail.com"
set realname = "Baptiste Wicht"

Receive mail

As I'm using offlineimap to get my mails, there is no IMAP settings in my configuration. But you need to tell Mutt where the mails are:

set folder = /data/oi/

set spoolfile = "+Gmail/INBOX"
set postponed = "+Gmail/drafts"

source ~/.mutt/mailboxes

The spoolfile and postponed are specifying the inbox and draft mailboxes. The .mutt/mailboxes file is generated by offlineimap.

By default, Mutt will ask you to move read messages from INBOX to another mailbox (set by mbox). I personally let my read messages in my inbox and move them myself in a folder. For that, you have to disable the move:

set move = no

If you move a mail from a mailbox to another, Mutt will ask for confirmation, you can disable this confirmation:

set confirmappend = no

If you use Mutt, you want to read plaintext messages rather than monstruous HTML. You can tell Mutt to always open text plain if any:

alternative_order text/plain text/html

If the mail has no text/plain part, you can manage to read HTML in Mutt in an almost sane format. First, you need to tell Mutt to open html messages:

auto_view text/html

And then, you need to tell it how to open it. Mutt reads a mailcap file to know how to open content. You can tell Mutt where it is:

set mailcap_path = ~/.mailcap

And then, you have to edit the .mailcap file:

text/html; w3m -I %{charset} -T text/html; copiousoutput;

That will use w3m to output the message inside Mutt. It works quite well. You can also use linx if you prefer.

Send mail

You need to indicate Mutt how to send mail:

set smtp_url = "smtp://baptistewicht@smtp.gmail.com:587/"
set smtp_pass = "SECRET"

Some people prefer to use another SMTP client instead of Mutt builtin SMTP support, you can also do that by setting sendmail to the mailer program.

It is generally a good idea to enforce the charset of sent mail:

set send_charset="utf-8"

You can choose another charset if you prefer ;)

You need to configure vim to correctly handle mail editing:

set editor='vim + -c "set textwidth=72" -c "set wrap" -c "set spell spelllang=en"'

It sets the width of the text, enable wrap and configure spelling.

By default, Mutt will ask you if you want to include the body of the message you reply to in your answer and the reply subject. You can make that faster by using these two lines:

set include=yes
set fast_reply

Once mail are sent, they are copied in your outgoing mailbox. If you use GMail, the STMP server already does that for you, therefore you should disable this behavior:

set copy = no

Appearance

Many things can also be configured in the appearance of Mutt. If you like the threaded view of GMail, you want to configure Mutt in a similar way:

set sort = 'threads'
set sort_aux = 'reverse-last-date-received'

It is not as good as the GMail view, but it does the job :)

You can make reading mail more comfortable using smart wrapping:

set smart_wrap

A mail has many many headers and you don't want to see them all:

ignore *
unignore From To Reply-To Cc Bcc Subject Date Organization X-Label X-Mailer User-Agent

With that, you just configure which headers you're interested in.

If you're using the sidebar patch (and you should be ;), you can configure the sidebar:

set sidebar_visible = yes
set sidebar_width = 35
set sort_sidebar = desc

color sidebar_new yellow default

It makes the sidebar always visible with a width of 35 and sort the mailboxes. The last line makes yellow the mailboxes that have unread mails.

The index_format allows you to set what will shown for every mail in the index view:

set index_format = "%4C %Z %{%b %d} %-15.15L %?M?(#%03M)&(%4l)? %?y?{%.20y}? %?g?{%.20g} ?%s (%c)"

This is a classical example that display the sender, the flags, the date, the subject, the size of the mail and so on. You will need to look at the Reference to have more information on what you can do with the format variables. There are plenty of information that can be shown.

You can also configure the text that is present on the status bar:

set status_chars  = " *%A"
set status_format = "───[ Folder: %f ]───[%r%m messages%?n? (%n new)?%?d? (%d to delete)?%?t? (%t tagged)? ]───%>─%?p?( %p postponed )?───"

The example here displays the current folder, the number of mails in it with some details on deleted and unread mails and finally the number of postponed mail. Again, if you want more information, you can read the reference.

You can configure Mutt so that the index view is always visible when you read mails. For instance, to always show 8 mails in the index:

set pager_index_lines=8

Another important thing you can configure is the colors of Mutt. I'm not gonna cover everything, since Mutt is very powerful on this part. For instance, here are some examples from my configuration:

color index red white "~v~(~F)!~N" # collapsed thread with flagged, no unread color index yellow white "~v~(~F~N)" # collapsed thread with some unread & flagged color index_subject brightred default "~z >100K" color header blue default "^(Subject)"

Unless you are really wanting to spend time on this part, I recommend to pick an existing theme. I took a Solarized theme here. It looks quite good and works well. There other themes available, you'll surely find the one that looks best for you.

Bindings

Bindings are always very important. If like me, you're a vim aficionado, you'll want your Mutt bindings to be as close as possible to vim. The default settings are quite good, but not always close to vim.

Something that is important to know when you configure Mutt bindings is that they are relative to the current view open (index, pager,browser,attach, ...). You can bind a keystroke to a different action in each view. You can also select several views in which the keystroke is valid.

If you are using the sidebar patch (and again, you should ;) ), you'll want to configure fast bindings for it. Here are mine:

bind index,pager \Ck sidebar-prev
bind index,pager \Cj sidebar-next
bind index,pager \Cl sidebar-open
bind index,pager \Cn sidebar-scroll-up
bind index,pager \Cv sidebar-scroll-down
bind index,pager \Ct sidebar-toggle

I use Ctrl+j,k to move inside the sidebar. I use Ctrl+l to open a folder and Ctrl+n,v to scroll up and down. The last one is to toggle between multiple sidebars for instance if you use notmuch.

I find l very good to open messages in the index too:

bind index l display-message
bind index gg first-entry
bind index G last-entry
bind index h noop               # Disable h

gg and G are used to go to the first and last element. Here I disabled h which had a not often used command.

The pager is the view where you read mail:

bind pager h exit
bind pager gg top
bind pager G bottom
bind pager J next-line
bind pager K previous-line

In this view, I use h to get out of the pager, gg and G as usual. As I always let the index open, I already use j and k to move in the index, so I chose J and K to move in the pager.

The browser is the view where you select folders for instance:

bind browser l select-entry
bind browser L view-file
bind browser gg first-entry
bind browser G last-entry
bind browser h exit

Again, I use l and h to go back and forth and gg and G to go first and last. j and k are already used here to go up and down.

In the attach view:

bind attach h exit
bind attach e edit-type # Edit MIME Types
bind attach l view-attach

I use h to exit and l to view an attachment.

That is it for my bindings, but you configure a lot more of them.

Conclusion

This is the end of this post. I have covered my complete Mutt configuration here. My .muttrc is available online.

If you have comments on my configuration, you're welcome to let a comment on this post ;)

In the next blog post about my "Mutt journey", I'll talk about notmuch and this will likely be the last post on this series.

pm 0.1.1 - A simple workspace manager for Git projects

Baptiste Wicht — Sun, 20 Jul 2014 18:52:18 GMT

In the last month, I've developped a very simple tool in Python: pm. This tool allows to check the status of all the Git repositories inside a repository. I've just released the first version of this tool: pm-0.1.1

Those who are following this blog will perhaps wonder why Python and not C++ :) The reason is quite simple, I wanted to improve my skills in Python. And what is better than to develop a project from scratch.

Features

The main feature of this application is to show the status of every projects in a directory. The status of your projects can be queried by using pm status. On my computer this gives something like that:

The state of each branch of each project is shown. There different possible status (they are cumulative): * Behind remote: Commits are available on the remote repository * Ahead of remote: Some local commits are no pushed * Diverged: Behind and Ahead * Uncomitted changes: Some changes are not committed * Clean: Indicates that everything is committed, pushed and pull.

By default, the directory is ~/dev/ but you can change it by passing the repository to the command, if you pass a relative directory, it will be relative to home. For instance, here is the status of my doc repositories:

Another feature that can be useful is that it is able to check the status of submodules with the -s option:

As you can see it supports recursive submodules. For each submodule it will indicate if there are new commits available or not.

pm is not only able to show status of the projects, it can also fetch the status of branches from remote by using pm fetch. All the remote branches are fetched from remote. It can also automatically update the projects that are behind remote (equivalent of git pull) with pm update. Only projects that can be fast-forwarded are updated.

Installation

Thanks to pip, installation of pm is quite simple:

pip install pm

If you don't want to use pip, you can install it by hand:

wget https://github.com/wichtounet/pm/archive/0.1.1.tar.gz
tar xf 0.1.1.tar.gz
cd 0.1.1
python setup.py install

For those interested, source code is available on Github.

If you have any suggestion for the tool or on the source code, post a comment to this post ;)

A Mutt Journey: Download mails with offlineimap

Baptiste Wicht — Wed, 16 Jul 2014 17:54:40 GMT

In the series of posts about Mutt, I recently presented how I was filtering my email. In this post, I'll show how I download my emails locally using offlineimap. This is the perfect companion for Mutt.

With Mutt, you can easily directly query an IMAP server and keep the views up to date with it. There are a few problem with this approach:

First, you wont' be able to read your mails when you'are offline. It is rarely an issue in these days, but it can be useful.
Opening an IMAP folder with a large number of mails (>1000) can be quite slow. I've several large folders and it was a pain opening them.
When Mutt synchronizes with the state of the IMAP server, you'll encounter a freeze. If you want to synchronize often, it is quite boring.

Having your mails offline on your computers solves all these problems. Moreover, it is also a good way to have a backup of your mails. I'm gonna talk here about the usage for Mutt, but you can use offlineimap just for backup or for migration reasons. The downside is that you have to store it locally. My mails takes around 5GB on my computer.

offlineimap is a very simple tool to synchronize emails from IMAP servers. It only supports IMAP, but in those days it is not a limitation. The synchronization is made both ways, it will upload your local changes to the IMAP server. It is very powerful when paired with a MUA such as Mutt.

To use offlineimap, you have to put your configuration in the ~/.offlineimaprc. You can synchronize several accounts at once, in this post, we'll focus on one, but the process is the same for several accounts. I'll focus on Gmail too, but again it is the same with a bit different parameters for other mail accounts.

Configuration

First, we have to declare the account:

[general]
accounts = Gmail

[Account Gmail]
localrepository = Gmail-Local
remoterepository = Gmail-Remote

accounts is the list of accounts that we have, here only one. Then, in account, repositories are just names of the repositories we'll declare now.

The local repository has to be configured:

[Repository Gmail-Local]
type = Maildir
localfolders = /data/oi/Gmail/
sep = /

The first important point is localfolders that sets where the mail will be put on your computer. sep defines the separator used for nested IMAP folders. I recommend / since Mutt will nest them automatically if / is used as separator.

Then, the remote repository has to be configured:

[Repository Gmail-Remote]
type = Gmail
remoteuser = USER
remotepass = PASSWORD
realdelete = no
folderfilter = lambda folder: folder not in ['[Gmail]/All Mail',
                                             '[Gmail]/Important',
                                             '[Gmail]/Starred',
                                             ]
sslcacertfile = /etc/ssl/certs/ca-certificates.crt

remotepass and remoteuser are your user names and password. You can also use remotepassfile to read the password from a file. realdelete=no indicates that we only want to remove all the labels of deleted mails. For Gmail, it means that the mail will still be in the All Mail folder. The last line (sslcacertfile) is mandatory for recent versions of offlineimap. The folderfilter is a function that filters some folders. In my case, I do not want to get the "All Mail", "Important" and "Starred" of my Gmail account because it is only a duplicata of the mails in other labels. What is pretty cool with offlineimap is that you can write Python directly in it for some of the configuration options. Here is rule for filter is plain Python, so you can complicated filtering if you want.

Last, but not least, offlineimap can generates a list of mailboxes (one for each folder in every account). It is pretty useful since Mutt can then read this file and you'll find your mailboxes directly configured in Mutt :)

This code will generate a file ~/.mutt/mailboxes that you can source in your Mutt configuration and get the complete list of available mailboxes. This will be kept up to date if you add new IMAP folders on the server for instance.

[mbnames]
enabled = yes
filename = ~/.mutt/mailboxes
header = "mailboxes "
peritem = "+%(accountname)s/%(foldername)s"
sep = " "
footer = "\n"

Translate names

You may have seen in the previous section some weird folder name like "[Gmail]/All mail", this is how Gmail names folders that are not labels. This is quite ugly and will create odd looking folders on your computer. You can configure offlineimap to rename these names to better ones. For that, you'll need to rule (in Python ;) ), one to translate from remote to local and one to do the reverse.

Here is what I did:

[Repository Gmail-Local]
nametrans = lambda folder: {'drafts':   '[Gmail]/Drafts',
                            'sent':     '[Gmail]/Sent Mail',
                            'important':'[Gmail]/Important',
                            'spam':     '[Gmail]/Spam',
                            'starred':  '[Gmail]/Starred',
                            'trash':    '[Gmail]/Trash',
                            'archive':  '[Gmail]/All Mail',
                            }.get(folder, folder)

[Repository Gmail-Remote]
nametrans = lambda folder: {'[Gmail]/Drafts':    'drafts',
                            '[Gmail]/Sent Mail': 'sent',
                            '[Gmail]/Starred':   'flagged',
                            '[Gmail]/Important':   'important',
                            '[Gmail]/Spam':   'spam',
                            '[Gmail]/Trash':     'trash',
                            '[Gmail]/All Mail':  'archive',
                            }.get(folder, folder)

I simply renamed all "[Gmail]" folders into something more readable and that makes more sense to me. It is not limited to special Gmail folders of course, this can also be applied to rename a folder X into a folder Y in the same. As it is Python, you can do sophisticated stuff if necessary.

Speed up things

If you happen to sync your mails often, you may want to speed things up. There are several ways to do that.

The first thing you can do is use several connections to the server. You can set maxconnections to a number higher than 1 in the remote repository configuration. I tested several values and for Gmail 2 was the fastest choice. You can try some values with your server to see what value is good.

Instead of plain old text files for the status of the mails, offlineimap can use a sqlite backend. This is much faster since the complete file is not rewritten for each update of the flags. For that behaviour, you have to set status_backend = sqlite in the Account configuration.

Another thing you can do is reduce the I/O involved during sync by setting general.fsync to false. With that, offlineimap won't have to wait for disk operation completion after each operation.

You can run offlineimap in quick mode with -q option. With this option, change in flags of remote messages will not be updated locally. Changes on the local side will be uploaded corectly. It is generally a good idea is to run offlineimap in quick mode often (every X minutes) and run it in normal mode once or twice a day.

You can also specify which folder to sync with the -f option. Sometimes it is enough to sync INBOX for instance. It may be much faster.

Conclusion

Now that you have fully configured offlineimap, you can make it run by hand or in a cron job. I personally run it every 5 minutes, you can choose your favourite frequency according to your workflow. I think I'll reduce the frequency further, it is more comfortable to get mails only by batch and not too much of them.

If you're interested, you can take a look at my .offlineimaprc configuration.

If you want more information about this awesome tool, you can take a look at the reference documentation.

This is it for this part of this series. In the next post, I'll present my Mutt configuration and how I use it.

A Mutt Journey: Filter mails with imapfilter

Baptiste Wicht — Tue, 08 Jul 2014 18:58:32 GMT

About a month ago, I decided to switch to Mutt to read my emails. I kept my GMail account, but I don't use the web interface anymore. It took me a long time to prepare a complete enviromnent.

Currently, i'm using:

imapfilter to filter mails
offlineimap to download my mails
notmuch to quickly search all my mails

And of course Mutt. To be precise, I use mutt-kz, a fork of mutt with very good notmuch integration.

I'll try to explain each part of my environment in a series of articles on this blog. The first one will be about imapfilter.

imapfilter is a mail filtering utility. It connects to a remote server using IMAP and is then able to move, copy or delete mails around. You can use it for several tasks:

Delete unwanted mail
Move mails into folders according to rules

What is pretty cool is that the configuration is entirely made in Lua. It is quite easy to write rules and then apply them to several mailboxes as if you were programming.

Another advantage of imapfilter is that it works at the server level. Therefore, even if you use your web client from time to time or check your mail on your phone, the changes will still be viewable.

The configuration is done in the ~/.imapfilter/config.lua file. The configuration is quite easy, you have to declare an IMAP object as the account.

local account = IMAP {
    server = 'imap_sever',
    username = 'username',
    password = 'password',
    ssl = 'ssl3',
}

As the configuration is in Lua, you can easily get the password from another file. For instance, here is my account declaration:

local account = IMAP {
    server = 'imap.gmail.com',
    username = 'baptiste.wicht@gmail.com',
    password = get_imap_password(".password.offlineimaprc"),
    ssl = 'ssl3',
}

-- Utility function to get IMAP password from file
function get_imap_password(file)
    local home = os.getenv("HOME")
    local file = home .. "/" .. file
    local str = io.open(file):read()
    return str;
end

It gets the password by reading a file in the home directory.

Once, you have the account, you can check the status of a folder with the check_status() function. For instance:

account.INBOX:check_status()
account['[Gmail]/Trash']:check_status()

You can run imapfilter simply by launching imapfilter on the command line. Once imapfilter is run, it will print the status of the folder you choses:

38 messages, 0 recent, 6 unseen, in baptiste.wicht@gmail.com@imap.gmail.com/INBOX.
70 messages, 0 recent, 67 unseen, in baptiste.wicht@gmail.com@imap.gmail.com/[Gmail]/Trash.

Several functions are important:

select_all() on a folder allows you to get messages from an account to them perform action on them
contain_subject('subject') on a list of mails allows you to keep only the mails that contains 'subject' in their subject
contain_from('from') on a list of mails allows you to keep only the mails that comes from 'from'
contain_to('to') on a list of mails allows you to keep only the mails that are addressed to 'to'
delete_messages() on a collection of mails deletes all of them
move_messages(folder) on a collection of mails moves all of them to another folder.

You can also mix different IMAP accounts, you don't have to use only one.

For instance, if you would delete all the mail coming from me, you could do:

mails = account.INBOX:select_all()
filtered = mails:contains_from("baptiste.wicht@gmail.com")
filtered:delete_messages()

Or you could move all the mails containing Urgent in the subject line to an IMAP folder:

mails = account.INBOX:select_all()
filtered = mails:contains_subject("Urgent")
filtered:move_messages(account["urgent_mails"])

If you want some more examples, you can take a look at my imapfilter configuration.

The best way to start using it is to look at examples, there are plenty of them in the internet, especially in Github dotfiles repositories.

The reference documentation is available using 'man imapfilter_config', there is plenty more to see.

For more information, you can also consult the offical site.

That is it for this part of the mutt series. In the next post about mutt, I'll talk about how I use offlineimap to get my mails.

budgetwarrior 0.4 - Enhanced wish list and aggregate

Baptiste Wicht — Sun, 06 Jul 2014 08:59:55 GMT

I've just released a new version of my command-line budget manager: budgetwarrior 0.4.

Enhanced aggregate overview

The aggregate overviews have been greatly improved. First, there is now a budget overview month command that groups all expenses of amonth together. Here is a possible output:

It also possible to use --full option to also aggregate together the different accounts:

Another new option is --no-group that disables the grouping by categories:

Moreover, the separator of categories can now be configured with --separator=.

All these options can also be set in the configuration with these options:

aggregate_full : If set to true, does the same as the --full option.
aggregate_no_group : If set to true, does the same as the --no-group option.
aggregate_separator : Sets the separator for grouping.

Enhanced wish list

The wishes management has also been improved.

First, each wish can now be set an Urgency and Importance level. This is now shown in wish status as simple indicators:

Moreover, the accuracy of the estimation compared to the paid amount is shown in wish list:

Various changes

Objective status now shows more information about the status of the objectives:

The versioning module has been improved. The versioning sync does now perform a commmit as well as pull/push. versioning push, versioning pull and versioning status commands have been added.

budget version command shows the version of budgetwarrior.

Aliases a now available to make shorted commands:

budget sync -> budget versioning sync
budget aggregate -> budget overview aggregate

Installation

If you are on Gentoo, you can install it using layman:

layman -a wichtounet
emerge -a budgetwarrior

If you are on Arch Linux, you can use this AUR repository.

For other systems, you'll have to install from sources:

git clone git://github.com/wichtounet/budgetwarrior.git
cd budgetwarrior
make
sudo make install

Conclusion

If you are interested by the sources, you can download them on Github: budgetwarrior.

If you have a suggestion or you found a bug, please post an issue on Github.

If you have any comment, don't hesitate to contact me, either by letting a comment on this post or by email.

Software Reliability Presentation

Baptiste Wicht — Sat, 07 Jun 2014 14:45:33 GMT

Software Reliability

In behalf of my school (College of Engineering and Architecture of Fribourg), I presented a shoft presentation about Software Reliability. In this presentation, I outline the main issues about the subject and propose some solutions:

Software Validation
Defensive Programming
Software Analysis Tools

In the Software Analysis Tools, I present three tools: cppcheck, Valgrind and the Clang Static analyzer. Several examples are presented for each tools as well as some recommendations for using them. A short presentation of SonarQube is also performed.

I thought that it could be of some interest to some of the readers, so here it is:

Don't hesitate if you have any comments or questions about the presentation ;)

The source code for the examples is available on Github.

budgetwarrior 0.3.1 - Git versioning and easier creation

Baptiste Wicht — Sat, 10 May 2014 14:45:43 GMT

I've finished a new version of budgetwarrior: budgetwarrior 0.3.1

Changes

The most interesting change is the ability to estimate the date when it is a good time to buy something from the wish list. This is done with the budget wish estimate command:

This command gives you two dates for each wish in your list. The first is the date wating for each yearly objectives to be fullfilled. The second one considers only the monthly objectives. For now on, no estimation of expenses is made for the future months. It means that the estimation is made as if there were no expenses in the future months. I'll try to improve that by considering averages of expenses in the previous months to make it more reliable.

Still on the wish module, you can now mark your wishes as paid instead of deleting them. This helps you keep track of the prices of your wishes. This is done with the budget wish paid id command. Finally, the totals of the unpaid wishes and of the paid wishes is displayed in budget wish list.

Another helpful change is the ability to set a date relative to today's date when creating an expense or an earning. For instance, you can create an expense one month before (-1m) or in one year ((+1y) or yesterday (-1d):

Of course, you can also still set the date manually.

The last major change is the addition of a new module: budget versioning. This module helps you manipulate you budget directory with Git. There are two new commands:

budget versioning save: Commit the current changes with a default message (Update).
budget versioning sync: Pull the changes from the remote directory and push the local changes.

This will only works if you have already configured your budget directory to use Git.

Conclusion

I hope you'll found these changes interesting :)

If you are interested by the tool, you can download it on Github: budgetwarrior

If you have a suggestion or you found a bug, please post an issue on the github project: https://github.com/wichtounet/budgetwarrior.

If you have any comment, don't hesitate to contact me, either by letting a comment on this post or by email.

budgetwarrior 0.3.0 - Objective and wish management

Baptiste Wicht — Mon, 03 Feb 2014 08:21:29 GMT

I'm pleased to announce the release of another budgetwarrior release, the version 0.3.0.

Changes

This version contains several important changes.

The first one is the addition of a new module to manage objectives. You can add objective with budget objective add). For instance, you can add an objective saying you want to save 10000$ a year or 200$ a month. When you set your objectives, budget warrior computes how well you complete them. For instance, here is the status of my objectives:

Another module has been added to manage wishes. You can add wishes to budgetwarrior (budget wish add) and then budgetwarrior will tell you if it is a good time to buy them. Here is an example of wish status:

Wish Status

The diagnostics tells you where the money will be taken: On savings, on year savings or on month savings (ideal case). It also checks the objectives to see if the payment doesn't break the fulfillment of some of them.

For complete diagnostics, it is necessary to you register your fortune (budget fortune check), ideally once a month.

Of course, this is only a tool, you should not only use that to decide when to buy something, but it may have a good point of view ;)

Moreover, the version also have other smaller changes:

When you make an error when creating a new item (expense, earning, ...), the tool now lets you retry without losing what you typed before.
Confirmation messages are now shown after each modification command (delete, add and edit).
The license has been changed from Boost to MIT. The sense is almost the same, but the MIT is more well known and I thought it would be easier for people to know what this means.
There have several changes to the code base, but that doesn't impact the usage of the tool.

Conclusion

I hope you'll found these changes interesting :)

If you are interested by the tool, you can download it on Github: budgetwarrior

There is now Gentoo and Arch Linux installation packages available for ease of installation

If you have a suggestion or you found a bug, please post an issue on the github project: https://github.com/wichtounet/budgetwarrior.

If you have any comment, don't hesitate to contact me, either by letting a comment on this post or by email.

budgetwarrior 0.2.1 - Minor changes and Gentoo ebuild

Baptiste Wicht — Thu, 16 Jan 2014 11:41:00 GMT

I've released a new version of budgetwarrior, the release 0.2.1. budgetwarrior is a simple command line application to manage a personal budget.

The version 0.2.1 contains several bug fixes about archived accounts and bug fixes for budget across several years.

The application as well as the source code is available online: https://github.com/wichtounet/budgetwarrior

I've created Gentoo ebuilds for this application. They are available on my Portage overlay: https://github.com/wichtounet/portage-overlay

Gentoo Installation

Edit overlays section of /etc/layman/layman.cfg. Here's an example:

overlays: http://www.gentoo.org/proj/en/overlays/repositories.xml http://github.com/wichtounet/portage-overlay/raw/master/repository.xml

Sync layman

layman -S

Add the overlay:

layman -a wichtounet

Install budgetwarrior

emerge budgetwarrior

Conclusion

If you find any issues with the tool, don't hesitate to post an issue on Github. If you have comments about it, you can post a comment on this post or contact me by email.

Zabbix - Low Level Discovery of cores, CPUs and Hard Disk

Baptiste Wicht — Mon, 30 Dec 2013 08:12:31 GMT

At home, I'm using Zabbix to monitor my servers, it has plenty of interesting features and can be extended a lot by using User Parameter.

In this post, I'm gonna talk about Low Level Discovery (LLD). If you are only interested in the final result, go the Conclusion section, you can download my template containing all the rules ;)

Low Level Discovery (LLD)

LLD is a feature to automatically discover some properties of the monitored host and create items, triggers and graphs.

By default, Zabbix support three types of item discovery:

Mounted filesystems
Network interface
SNMP's OIDs

The first two are very useful, since they will give you by default, for instance, the free space of each mounted file system or the bandwith going in and out of each network interface. As I only monitor Linux servers, I don't use the last one, but it will eventually interest other people.

Another very interesting thing about this feature is that you can extend it by discovering more items. In this article, I will show how to discover CPUs, CPU Cores and Hard Disk.

The most important part of custom discovery is to create a script on the monitored machines that can "discover" something. It can be any executable, the only thing important is that it outputs data in the correct format. I have to say that the format is quite ugly, but that is probably not very important ;) Here is the output of my hard disk discovery script:

{
"data":[
    {"{#DISKNAME}":"/dev/sda","{#SHORTDISKNAME}":"sda"},
    {"{#DISKNAME}":"/dev/sdb","{#SHORTDISKNAME}":"sdb"},
    {"{#DISKNAME}":"/dev/sdc","{#SHORTDISKNAME}":"sdc"},
    {"{#DISKNAME}":"/dev/sdd","{#SHORTDISKNAME}":"sdd"},
    {"{#DISKNAME}":"/dev/sde","{#SHORTDISKNAME}":"sde"},
    {"{#DISKNAME}":"/dev/sdf","{#SHORTDISKNAME}":"sdf"},
    {"{#DISKNAME}":"/dev/sdg","{#SHORTDISKNAME}":"sdg"},
]
}

You can have as many keys for each discovered items, but the format must remains the same. In the item, trigger and graph prototypes, you will then use {#DISKNAME} or {#SHORTDISKNAME} to use the discovered values.

Once you have created your scripts, you have to register it in the zabbix configuration as a user parameter. For instance, if you use the zabbix daemon, you need these lines in /etc/zabbix/zabbix_agentd.conf:

EnableRemoteCommands=1
...
UnsafeUserParameters=1
...
UserParameter=discovery.hard_disk,/scripts/discover_hdd.sh

Now, when you will create the discovery rule, you can use discovery.hard_disk as the key.

A discovery rule in itself is useful without prototypes, you can create three types of prototypes:

Item Prototype: This will create a new item for each discovered entity
Trigger Prototype: This will create a new trigger for each discovered entity.
Graph Prototype: This will create a graph for each discovered entity.

The most useful are by far the item and trigger prototypes. The biggest problem with graphs is that you cannot create an aggregate graph of each discovered items. For instance, if you record the temperature of your CPU cores, you cannot automatically create a graph with the temperature of each discovered cores. For that, you have to create the graph in each host. Which makes, imho, graph prototypes pretty useless. Anyway...

In the next section, I'll show how I have created discovery rules for Hard Disk, CPU and CPU cores.

Discover Hard Disk

The discovery script is really simple:

#!/bin/bash
disks=`ls -l /dev/sd* | awk '{print $NF}' | sed 's/[0-9]//g' | uniq`
echo "{"
echo "\"data\":["
for disk in $disks
do
    echo "    {\"{#DISKNAME}\":\"$disk\",\"{#SHORTDISKNAME}\":\"${disk:5}\"},"
done
echo "]"
echo "}"

It just lists all the /dev/sdX devices, remove the partition number and remove the duplicates, to have only the hard disk at the end.

I've created several item prototypes for each hard disk. Here are some examples using S.M.A.R.T. (you can download the template with all the items in the Conclusion section):

Raw Read Error Rate
Spin Up Time
SSD Life Left
Temperature
...

You may notice that some of them only make sense for SSD (SSD Life Left) and some others do not make any sense for SSD (Spin Up Time). This is not a problem since they will just be marked as Not Supported by Zabbix.

All these datas are collected using the smartctl utility.

I've also created some trigger to indicate the coming failure of an hard disk:

SSD Life Left too low
Reallocated Sector Count too low
...

I've just used the threshold reported by smartctl, they may be different from one disk manufacturers to another. I don't put a lot of faith on these values, since disk generally fail before going to threshold, but it could be a good indicator anyway.

Discover CPUs

Here is the script to discover CPUs:

#!/bin/bash
cpus=`lscpu | grep "CPU(s):" | head -1 | awk '{print $NF}'`
cpus=$(($cpus-1))
echo "{"
echo "\"data\":["
for cpu in $(seq 0 $cpus)
do
    echo "    {\"{#CPUID}\":\"$cpu\"},"
done
echo "]"
echo "}"

It just uses lscpu and parses its output to find the number of CPU and then create an entry for each CPUs.

I just have one item for each CPU: The CPU Utilization.

I haven't created any trigger here.

Discover CPU Cores

Just before, we discovered the CPUs, but it is also interesting to discover the cores. If you don't have Hyperthreading, the result will be the same. It is especially interesting to get the temperature of each core. Here is the script:

#!/bin/bash
cores=`lscpu | grep "Core(s) per socket:" | awk '{print $NF}'`
cores=$(($cores-1))
echo "{"
echo "\"data\":["
for core in $(seq 0 $cores)
do
    echo "    {\"{#COREID}\":\"$core\"},"
done
echo "]"
echo "}"

It works in the same way as the previous script.

I've only created one item prototype, to get the temperature of each core with lm_sensors.

Wrap-Up

Here are all the UserParameter necessary to make the discovery and the items works:

### System Temperature ###
UserParameter=system.temperature.core[*],sensors|grep Core\ $1 |cut -d "(" -f 1|cut -d "+" -f 2|cut -c 1-4
### DISK I/O###
UserParameter=custom.vfs.dev.read.ops[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$4}'
UserParameter=custom.vfs.dev.read.ms[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$7}'
UserParameter=custom.vfs.dev.write.ops[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$8}'
UserParameter=custom.vfs.dev.write.ms[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$11}'
UserParameter=custom.vfs.dev.io.active[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$12}'
UserParameter=custom.vfs.dev.io.ms[*],cat /proc/diskstats | egrep $1 | head -1 y| awk '{print $$13}'
UserParameter=custom.vfs.dev.read.sectors[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$6}'
UserParameter=custom.vfs.dev.write.sectors[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$10}'
UserParameter=system.smartd_raw[*],sudo smartctl -A $1| egrep $2| tail -1| xargs| awk '{print $$10}'
UserParameter=system.smartd_value[*],sudo smartctl -A $1| egrep $2| tail -1| xargs| awk '{print $$4}'
### Discovery ###
UserParameter=discovery.hard_disk,/scripts/discover_hdd.sh
UserParameter=discovery.cpus,/scripts/discover_cpus.sh
UserParameter=discovery.cores,/scripts/discover_cores.sh

(it must be set in zabbix_agentd.conf)

You also need to give zabbix the right to use sudo with smartctl. For that, you have to edit your /etc/sudoers file and add this line:

ALL ALL=(ALL)NOPASSWD: /usr/sbin/smartctl

Conclusion and Download

I hope that this helps some people to use Low Level Discovery in their Zabbix Monitoring Installation.

LLD eases a lot the creation of multiple items discovery for hosts with different hardware or configuration. However, it has some problems for which I have not yet found a proper solution. First, you have to duplicate the client scripts on each host (or at least have them on a share available from each of them). Then, the configuration of each agent is also duplicated in the configuration of each host. The biggest problem I think is the fact that you cannot automatically create graph with the generated items of each discovered entities. For instance, I had to create a CPU Temperature graph in each of my host. If you have few hosts, like many, it is acceptable, but if you have hundreds of hosts, you just don't do it.

All the scripts and the template export file are available in the zabbix-lld repository. For everything to work, you need the lscpu, lm_sensors and smartmontools utilities.

If you have any question or if something doesn't work (I don't offer any guarantee, but it should work on most recent Linux machines), don't hesitate to comment on this post.

Gentoo Tips: Avoid Gnome 3.8 from being emerged automatically

Baptiste Wicht — Sat, 07 Dec 2013 05:24:17 GMT

Since Gnome 3.8 has been out in the portage tree, a lot of problems arise when you try to emerge something. If it was only when you update the system, it would be OK, but this arises every time you try to install something.

For instance, if I try to update vim on my system, it tries to update empathy to version 3.8 and then pulls some other dependencies causing blocks and other USE problems. I personally don't think empathy should be emerged when emerging vim. Fortunately, you can disable this behavior by using emerge in this way:

emerge --ignore-built-slot-operator-deps=y ...

With that, when you emerge vim, it doesn't emerge Gnome 3.8. It is very useful if you want to stay with Gnome 3.6 for the moment.

I already used this tip several times. I hope that this will be useful to other people.

Why and how I completely left Windows for Linux

Baptiste Wicht — Mon, 01 Jul 2013 06:24:25 GMT

For years now, I always kept a dual-boot at home with a Linux system (currently Gentoo) and a Windows system. At work, I only use Gentoo. This week-end, I decided to completely remove it and migrate the applications I used on Windows to my Gentoo system.

Why Windows ?

So first things first, why was I keeping the Windows system ? For several reasons:

Games :) Unfortunately, most of the games I play are not natively compatible with Linux.
Office. I always liked Microsoft Office. As I hate OpenOffice/LibreOffice, I never wanted to remove it For schools we always had several teachers forcing us to use Microsoft document formats.
Hardware support. I always found that hardware support in Windows was great. Most of the time when you add new peripheral, there is nothing, it just works, which is great.
Applications. I always had some applications that I didn't found good enough Linux equivalents for. For instance, Newsleecher, iTunes or TaggedFrog.

On the other hand, I work on Linux for years now and I would like to have to work on Windows again.

What Changed ?

This weekend I upgraded my hardware configuration (Motherboard, CPU and RAM). I was afraid that I had to reinstall my Linux configurations (because of Gentoo compiled with march=native), but I never thought that I would have to reinstall Windows. I turned out the contrary: my Gentoo installation worked just fine and my Windows totally crashed (BSOD at each startup). I finally made it through Windows after disabled AHCI mode on my motherboard, but then activation was invalidated (of course...) and online activation was not working. I decided to install the new chipset drivers and launch the Windows update and after that, Windows decided to boot without any USB support (WTF...). After that, I decided that Windows what not so great at all for hardware support...

Another reason I left Windows is Windows 8. I find that Window 7 was really great, but I really don't like Windows 8 and I would never have upgraded my Seven to it. Moreover, I recently bought Microsoft Office 2013 and it turned that I had to create an account at Microsoft to install it... Seriously ??? And moreover, it turned out to be worse thant Office 2010 (which, again, was great).

So all these reasons made me remove Windows.

How to migrate everything to Linux ?

First, I had no problem with my data. Most of my data are on a personal NAS and the remaining is on Dropbox, so no problem on this side.

I still had some problems to resolve. First of all, I needed my games to run on Linux. I currently play only Diablo III. As I had received a year free of Crossover, I decided to give it a try. Crossover is based on Wine and ensures that some software are running correctly under it and provide technical support. After some tuning, Diablo III was running almost flawlessly on my Gentoo machine :) Problem 1 solved. I will totally buy a license of Crossover, once my free year is over.

I still add some applications to replace. I use iTunes as my main music player and library manager. Some time ago, I tried a lot of programs like Amarok/Rythmbox/Banshee, but I didn't liked them a lot and they were not running very well on large library of music files. This time, I tried Clementine. Even if not very beautiful, it had all the features I needed and worked very well. I decided to stick with it. Another program I like a lot on Windows is TaggedFrog. It is a very simple program allowing to put tags on any file on the system and then search by tag on them. I haven't found a total equivalent. I first tried Tracker that is a Gnome project, but I was not satisfied with the search interface. After that, I tried the very simple TMSU. It is a command-line based tagging manager. All the tagging must be done in command line. In my case, it is not a problem, as I don't mind using the command-line and I don't tag files very often. What is very interesting about TMSU is that it can create a virtual file system (based on FUSE). In this file system, you have all your tags as folder and you can see directly all the files of each tag. Moreover, you can directly make cross search (has tag X and Y and Z) by just going down in the tag folder. It is really great and has everything I needed. Finally, I also needed something to replace Newsleecher. I haven't found something as great (especially no replacement for the Supersearch function), but I installed Sabnzbd which works really well and is very simple. For now, I just use the web interface and haven't installed any other front-end, but that will perhaps change in the future.

I haven't replaced Office for now on. It occurred to me that since I left school, I haven't used it a lot, so that will probably not be a problem anymore. I will change to write the few letters I have to write on Latex and if I have Office documents, I'll probably read them on Google Drive.

Conclusion

Even if I lost a lot of time with all that, I think it is a great think. It makes one less configuration to maintain and some less costs on the future. Moreover, I will save some time, because I won't have to switch between Linux and Windows for different tasks. And now, I have a second SSD ready for something else, either for RAID 1 to ensure redundancy on Linux or to mount on a server, I'll see later.

I will probably have some more problems in the future, but I'm convinced that there will be Linux solutions to it :)

CMakeLatex 1.0.2 - Support for nomenclature and better filters

Baptiste Wicht — Mon, 05 Nov 2012 08:02:19 GMT

I released a new version of CMakeLatex, the version 1.0.2.

First of all, this version restore the support for nomenclature. Then, it also adds filters for makeindex (including makeglossaries and makenomenclature). The filters will hides all the information of the output stream but the errors. The filters for pdflatex are also improved.

CMakeLatex is a CMake script to build Latex documents using CMake / Make. It supports glossary, indexes, bibliographies and nomenclature. It can automatically converts your images to the right format using imagemagick or cairosvg (for SVG to PDF conversion).

You can download it on Github: CMakeLatex Github repository

If you have any idea for improvement, don't hesitate to contact me or to create a Feature Request on Github.

eddic compiles with CLang 3.1

Baptiste Wicht — Thu, 01 Nov 2012 08:11:05 GMT

I finally added support for compiling eddic with LLVM CLang 3.1 !

The current development version can be completely compiled with CLang. Starting with the version 1.1.4, all versions of eddic will be support GCC and CLang.

The changes have not been as painful as I first thought.

The main problem that I has was about a static const variable of a class that had no user-constructor. GCC allows that, but it is not standard compliant and CLang was complaining.
Another problem that I encountered was about the used of bit flags and Template Meta Programming. I simplified that by the use of a simple type traits and it worked. I don't really know why this does not worked at first.
The remaining effort was to fix the several warnings that CLang had.

CLang also fixed a bug in my code with a warning on a assignment that was not supposed to be an assignment, thanks CLang.

The most interesting fact about CLang is that is it twice faster to build eddic than GCC. I think I'm gonna use it during development to fasten the compile time. Moreover, even if I only worked two days with it, it seems that the error messages are indeed better than the GCC's ones.

I haven't tried to compare the performances of eddic in both cases, but I will do that in the future, soon after the 1.1.4 version is released.

I tried the CLang static analyzer on eddic but it didn't found any bugs. Moreover, it crashed on several of my files. I didn't found why for now, but I will continue to investigate, perhaps I'm not using it correctly.

I expect to publish the next version of eddic in the next two weeks. This version has much more improvements that I thought at first and I have less time to work now that I'm working on my Master thesis.

More informations on CLang: The official site.

Linux symbolic links (soft) and hard links

Baptiste Wicht — Fri, 21 Sep 2012 07:16:37 GMT

On Linux, it is possible to create links to existing file. These links can be either symbolic links or hard links. Each of them having advantages and drawbacks. In this small post, we will see the differences between the two kinds of links and how to use them.

Hard Link

An hard link refers directly to the physical location of another file (an inode to be precise).

A hard link has some limitations: it cannot refer to a directory and cannot cross file system boundaries. It means that you can only create hard links to the same file system where the hard link is located.

When the source of the link is moved or removed, the hard link still refer to the source.

Symbolic link are created with the ln command. For instance, to create a link to source_file:

ln source_file link

Symbolic Link

A symbolic link refers to a symbolic path indicating the location of the source file. You can see it as a link to a path (itself refering to an inode).

A symbolic link is less limited. It can refer to a directory and can cross file system boundaries.

However, when the source of the link is moved or removed, the symbolic link is not updated.

Symbolic link are created with the ln command. For instance, to create a symbolic link to source_file:

ln -s source_file link

Deletion

The deletion of a link (hard or symbolic) can be achieved with the rm or unlink commands:

rm link
unlink link

Conclusion

And that's it!

Symbolic and hard links are very useful tools and are very easy to use.

I hope that this blog post helped you understand a little better the differences between the two types of links and how to use them.

Memory Manager in 64bits Intel Assembly on Linux

Baptiste Wicht — Thu, 02 Aug 2012 06:05:30 GMT

For the last version of the EDDI Compiler, it has been necessary to extend the dynamic memory allocator, to support free memory. In this post, we will see how to write a simple Memory Manager in Intel Assembly for Linux.

In the past, we've seen how to write a basic memory allocator, this time, we will write a more complete version.

The implementation is made in 64bits Intel Assembly.

Memory Manager specification

The memory will be allocated by blocks. Each block will contain a header with two information:

A boolean flag indicating if the block is free or not
The size of the block (including the header)

Each time some memory is asked, the blocks are tested one by one until an available one is found. If no available block is found, a new block is allocated after the last one and this block is returned.

The memory manager consists of three functions:

memory_init: Init the memory manager
memory_alloc: Allocate the given number of bytes of memory
memory_free: Release the given block

The parameter is passed in the r14 register. The return value is returned in the rax register.

Global State

This implementation needs two global variables. One for the start address of memory and the other one for the last:

section .data
mem_last dq 0
mem_start dq 0

Init memory Manager

The init function is very simple to implement:

init:
push rbp
mov rbp, rsp
mov rax, 12
xor rdi, rdi
syscall
mov [mem_start], rax
mov [mem_last], rax
leave
ret

We just have to call sys_brk in order to get the location of program break. Then, the start and the last addresses are the same.

Free memory

The free function is the simplest one:

free:
push rbp
mov rbp, rsp
mov qword [r14 - 16], 1
leave
ret

The address to free is passed in the r14 register. We have to go back 16 bytes (size of the control block) to go to the start of the block. The availability flag is set to 1 (the block is free).

The alloc function

The alloc function is the most complex:

alloc:
push rbp
mov rbp, rsp
push rdi
push r10
push r11
push r12
push r13
push r14
add r14, 16
mov r12, [mem_start]
mov r13, [mem_last]
.start:
cmp r12, r13
je .alloc
mov r10, [r12]
mov r11, [r12 + 8]
cmp r10, 1
jne .move
cmp r11, r14
jl .move
mov qword [r12], 0
lea rax, [r12 + 16]
pop r14
pop r13
pop r12
pop r11
pop r10
pop rdi
leave
ret

.move:
add r12, r11
jmp .start

.alloc:
lea rdi, [r12 + r14]
mov rax, 12
syscall
mov [mem_last], rdi
mov qword [r12], 0
mov qword [r12 + 8], r14
lea rax, [r12 + 16]
pop r14
pop r13
pop r12
pop r11
pop r10
pop rdi
leave
ret

As the function is a bit complex, I will detail it in part:

add r14, 16
mov r12, [mem_start]
mov r13, [mem_last]
.start:
cmp r12, r13
je .alloc
mov r10, [r12]
mov r11, [r12 + 8]
cmp r10, 1
jne .move
cmp r11, r14
jl .move
mov qword [r12], 0
lea rax, [r12 + 16]

The necessary number of bytes is passed in the r14 register. We add 16 bytes (size of the control group) to the size as we also need some place for the header. Then, we load the start and last addresses. If both addresses are equal, we need to allocate more memory (detailed later). Then, we check the size and the availability of the current block. If the size is enough to fit the needs and the block is available, we set it to unavailable. We return the address past the control block (16 bytes).

.move:
add r12, r11
jmp .start

To move to the next block, we just have to add the size of the current block to the current block address.

.alloc:
lea rdi, [r12 + r14]
mov rax, 12
syscall
mov [V_mem_last], rdi
mov qword [r12], 0
mov qword [r12 + 8], r14
lea rax, [r12 + 16]

To allocate memory, we compute the new program break and call sys_brk again to set the new program break. The block is then set to not available and the size is set. We return the address past the control block (16 bytes).

The rest of the program is just here to save and restore the registers and compute the stack frames.

Wrap-Up

In this article, we saw how to implement a very simple memory manager in 64bits Intel Assembly on Linux. This memory manager is very simple, but has several drawbacks:

The overhead for small blocks is important. For example, allocating an 8 bytes integer needs a 24 bytes block, thrice the size of the int.
In the worst-case scenario, all of the process memory need to be walked across to find a new free block
The functions are not thread-safe
This algorithm can lead to a lot of memory fragmentation

In the future I will try to make a more powerful version of this memory manager.

Download

All the functions are available online on the Github Repository:

They are also available in 32bits Intel Assembly: