Aikido, programming, system administration, and other things I find interesting

Why is my munin slow and how to speed it up

At $work we are monitoring a network of hundreds of servers, and that means that we end up recording hundreds of thousands of variable values every five minutes. After a while, the server started slowing down, taking more than 300 seconds to collect the data. Since it has a whole-system lock, that means the next […]

Read More

A munin plugin to monitor each CPU core separately

Monitoring each core separately may seem like a waste – after all, we have an overall CPU usage already available under “system” in munin, isn’t that enough? It turns out that it isn’t. Sometimes, when using top on a multicore/multicpu machine, you can see a process pegged at 100%, while other processes are comfortably using […]

Read More

Keeping a bunch of processes running

From time to time, I need some processes that keep running. It they were simple daemons, I could use something like monit, but what if I need X instances of worker A and Y instances of worker B? I whipped up a quick script that makes it pretty easy to do that, when needed: #!/usr/bin/perl […]

Read More

Making an IPv6 addressing plan

By now, hopefully, you not only realize that you need IPv6, but that you need to replace any ISP that is not able to give you IPv6 connectivity with one who can. But now, you have a different problem. Your ISP probably gave you more addresses than exist on the whole IPv4 internet, just for […]

Read More

Using rrdcached with munin on Ubuntu

When you have munin following many hosts with many variables tracked on each, you will inevitably find your munin server is overwhelmed with IO (rising iowait values, the server sometimes having excursions where it just gets overwhelmed with IO). The 1.4.7-1 version of rrdcached did NOT install cleanly on my Ubuntu 12.04 server: the package […]

Read More

Massive data loss in MySQL 5.5 with master-master replication.

If you put two mysql servers into a master<->master replication and then change the port of one of the servers, data magically disappears from the first master server. I’ve been able to repeat that (twice) bith with a mixed (mysql 5.1<-> mysql 5.5) setup and with a pure Ubuntu 12.04, mysql 5.5 <-> mysql 5.5 […]

Read More

Installing munin-async

With the latest versions of munin-async, this article is out of date. Instructions for munin 2.0.17 on Ubuntu 12.04 are here When using munin, one often runs into one of two problems: ┬áthere are so many nodes to update, the update takes more than the update interval some servers may be connected over flaky lines, […]

Read More

Generating EUID IPv6 address from a MAC

When you set up radvd (router advertisement daemon) on a network, all the computers that support IPv6 will adopt addresses in the advertised address space – except the machine that actually advertises the address prefix (thr router, if you will). You can assign an address to the interface yourself, any address will do. For instance, […]

Read More

DNS workgroup update

(DNS WG at #ripe64) RIPE report Ripe runs K-root, 18 instances, enabled IPv6, migrating to Centos6, IPv6 traffic is growing. They have 117 signed zones. DNSSec growth in reverse zones is steady, but they are still at about 1%. They have 77 ccTLDs, and they have IPv6 glue for every root zone. DNSMON will be […]

Read More

DNS debugging and monitoring

(based on Damas & Kerr lecture at RIPE64) DNS has a very limited number of error codes, but mostly it just returns ServFail. Do not use nslookup for debugging, because it doesn’t report what it is trying to do. Use host if all you want is a simple query. Dig is part of bind9, packaged […]

Read More