Superfast NFS Tuning

In the past week at work, I’ve had a need to utilize some directly attached boxes working over NFS to share a storage array, the backblaze storage pod actually. This was necessary as the pods don’t have many compute resources to handle the load required to backup our datasets. Looking into this, I realized that optimizing NFS was an easy and surefire way to ensure it wasn’t taking extra resources on my pod.

Continue reading ↦

If you don't enable CDP, there's something wrong with you.

Hmmm….I wonder what switch port this box is connected to??? [~]> apt-get install cdpr Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: cdpr 0 upgraded, 1 newly installed, 0 to remove and 31 not upgraded. Need to get 17.4 kB of archives. After this operation, 102 kB of additional disk space will be used. Get:1 http://us.archive.ubuntu.com/ubuntu/ trusty/universe cdpr amd64 2.

Continue reading ↦

My First ZFS Experience: Taming 45 drives

At work, we have a couple Backblaze storage pods (version 3 with 4TB drives) that we use for backup purposes. They were obtained before my time because quick, bulk storage was necessary to backup our object storage platform, Swift. Sadly, the boxes were deployed in an unsatisfactory manner whereas all 45 drives were pooled together in one gigantic LVM formation, meaning any one disk could die and data loss would occur.

Continue reading ↦

The proper way to configure interfaces on Ubuntu

Discovered that in recent Ubuntu versions, the following is the “official” way to up and down interfaces cleanly. Edit /etc/network/interfaces adding your entry similar to below: # The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet static address 10.1.24.30 netmask 255.255.254.0 network 10.1.24.0 broadcast 10.1.25.255 gateway 10.1.24.1 # dns-* options are implemented by the resolvconf package, if installed dns-nameservers 10.

Continue reading ↦

Head first into Ansible, or: How I learned to cfgmgmt

I now run an entire Openstack Swift cluster that of course, is in production. The way its currently setup calls for for loops to be used to easily manage it. Of course, those get lame after a “quick” while. So I decided to take my second (the first was pretty bad, it was a Monday) plunge into Ansible. These are the voyages of the Ansible Noobcake. Its five-hour mission: to automate configuration parameters across all swift nodes, to boldly cfg like no man has cfg’ed before.

Continue reading ↦

Handy DNS response filtering with pdns-recursor

In our quest to roll out IPv6 intranet-wide, we ran into a weird issue with how we do VPN requiring us to filter out AAAA records being served to VPN connected clients. At first, this didn’t seem very easy. Then, pdns-recursor to the rescue! On more recent (>=3.1.7 versions), pdns-recursor can take in scripts (in real time, btw) to modify the way it operates. This is super handy and luckily, there is a filter-aaaa script available on github that does exactly what we want!

Continue reading ↦

Roku Streaming Stick and Comcast Gateways

I just bought the Roku streaming stick for my parents place to supplement the cable box on one of the tv’s. The idea was that they would be able to easily connect it to the network and watch plex, netflix, etc.. in the course of under 10 minutes. That was not the case…. First, I plugged this guy in. It’s boot time was over a minute, roughly 2-3 minutes actually. The Roku then first takes you through the prompts to setup your Wi-Fi network and update itself.

Continue reading ↦

Handling InnoDB Corruption

Having your storage ripped from out from under your database is never good. I recently had to go through this where our iscsi storage backend powering our virtual machine vmdk’s magically died and even worse, its write cache was completely cleared out. Upon things coming back, mysqld was in a reboot loop which could only be stopped by enabling a innodb_force_recovery option in my.conf. Here’s some of the output starting up with recovery on:

Continue reading ↦

Encrypted Time Machine Backups on a separate partition

Trying to setup my external with a couple partitions, one for encrypted Time Machine backups, and the other as ext4 for linux stuffs, yielded a no-go. In short, Time Machine wouldn’t let me encrypt my backups which I dumped on the first partition of my disk. Turns out this is because I used the older MBR partition scheme. These are the requirements in order for that check box to be clickable:

Continue reading ↦

Fixing Backupninja: Or how I learned to stop worrying and build from source

I’m tired. So I’m going to make this as short and clear-cut as possible: backupninja is a pretty nice backup program for linux which supports rdiff-backup, rsync, and duplicity. backupninja exists as version 1.0.1 for the latest stable ubuntu and debian packages. There exists a bug in this version of backupninja which disables it from rsyncing. This is a critical bug which disables you from using any host.

Continue reading ↦