Archive for 2009

WTF did I change?

So, it’s been a while since I posted, I thought I’d share my latest heap of hacky shell up here.

Last week we were having a discussion following on from Chris Siebenmann’s post on restoring files from a package, to see what changes had been made. Anyway, we decided that such a feature could be useful, so I promised to hack one together.

It took a week to finish this, because I spent most of my time confused about how apt-get couldn’t just download package without being root. I now know that it defiantly can’t do this. I also know that aptitude can.

# Check we have an argument
if [ -z "$1" ]; then
    echo "Usage $0 /path/to/file"
exit 1
# Check this is actually a text file, if
# not we drop out straight away
if [ `file $1|grep -c text ` -lt 1 ]; then
    echo "The file $1 is not a text file"
    exit 2
# Work out the package name
PACKAGE=`dpkg -S $1| awk -F: '{print $1}'`
# If there's no matching package then
# just exit with an error
if [ $? -gt 0 ]; then
    echo "No package provides this file"
    exit 3
# Download the package
echo "File $1 is provided by $PACKAGE. Downloading to /tmp/$$"
mkdir /tmp/$$
cd /tmp/$$
aptitude download $PACKAGE > /dev/null
# Extract the package
PKGNAME=`find /tmp/$$ -maxdepth 1 -type f -name "*.deb"`
dpkg -x $PKGNAME /tmp/$$
# Diff shit
diff -us /tmp/$$/$1 $1
# If there's a difference then offer to save the diff
if [ `diff -qs $1 /tmp/$$/$1|grep -c identical` -ne 1 ]; then
    echo -n "Save this diff? [yes] "
    read answer
    if [ -n "$answer" ]; then
    if [ "$ANSWER" = "yes" ]; then
        diff -us /tmp/$$/$1 $1 > /tmp/$PACKAGE.$$.diff
        File saved to /tmp/$PACKAGE.$$.diff
# We should be nice and offer to clean up
echo -n "Delete the temporary package files from /tmp/$$? [yes] "
read answer
if [ -n "$answer" ]; then
if [ "$ANSWER" = "yes" ]; then
    rm -fr /tmp/$$/

This script is also available from the RedBrick package repo

Dirvish, Part 2

In my earlier post on dirvish I said I’d have to beat it to do the kind of expire rules I wanted… well I’ve finally found time to do that.

It’s a pretty simple approach, I’ve used the cron script to touch a dotfile if it’s the start of the week/month, and if that file exists then it determines the expire time, which is just passed as an argument to dirvish

This way, the first backup of the month should always be kept for a year, and the first of the week for a month, no matter when they occur.

date=`date +%d`
day=`date +%a`
if [ "$date" -eq 1 ]; then
touch /home/backup/littlebrick/.monthly
if [ $day == "Sun" ]; then
touch /home/backup/littlebrick/.weekly
if [ -f /home/backup/littlebrick/.monthly ]; then
dirvish --vault littlebrick --expire 1year --image-time 00:00
elif [ -f /home/backup/littlebrick/.weekly ]; then
dirvish --vault littlebrick --expire 1month --image-time 00:00
dirvish --vault littlebrick --image-time 00:00

With that done, all that’s left to do is to remove the dotfiles following a successful backup. I used the post-server options for this.

post-server: ; /backup/dirvish/ $DIRVISH_DEST

And, this is the post_backup script

# This script runs after backups, and if the backup was
# successful clears out the month/week flags
#$2 is the DIRVISH_DEST
cd $2
cd ..
#we're now in /home/backup/littlebrick/YYYYMMDD/
success=`grep -c "Status: success" summary`
if [ "$success" -gt 0 ]; then
if [ -f /home/backup/littlebrick/.monthly ]; then
rm /home/backup/littlebrick/.monthly
if [ -f /home/backup/littlebrick/.weekly ]; then
rm /home/backup/littlebrick/.monthly

Laptop Backups With Dirvish

My laptop is something I should *really* backup, since most of my files and stuff I’m working on is there. Since I’m too lazy to do this regularly though I setup an automated system to do this after I re-installed it this week.

My objective was to setup a backup system that would always run automatically, without me having to press any buttons. Incremental backups were a must. Keeping multiple days of backups was also a relatively high priority.

Dirvish is a perl wrapper for rsync. It’s main feature is to automate running incremental rsync backups on a regular basis without too much hassle. Having worked with it alot before it seemed the obvious choice for running my laptop backups.

Before I continue, some terminology:

Server: the machine running dirvish, in this case my desktop
Client: the machine with the files, in this case my laptop (littlebrick)
Bank: A directory where dirvish stores backups (eg. /backup)
Vault: A particular backup source (eg, my laptop)
Branch: A particular run of a vault (usually, daily)


First, I needed to setup my laptop to have a static ip address on the wireless, which uses dhcp.

host littlebrick {
hardware ethernet       00:1b:77:37:af:3d;
fixed-address ;


and then I added this ip to the local dns server.

littlebrick     A

Now I can specify the laptop by hostname in the config files later. You could skip this step and just use the ip or /etc/hosts of course.


Next step was to generate an ssh key, so the server could connect without a password

laptop# ssh-keygen
laptop# cat > /root/.ssh/authorized_keys2
laptop# scp id_rsa root@desktop:~/.ssh/

I set the laptop to allow root ssh (I’ll tidy this up a bit later), and then tested the desktop root user could ssh to the laptop.

Actually, the default in most cases is to allow ssh access as root (either via password or key). I think this is fairly nuts as a default, and usually change it pretty quickly


Once that’s done I started on dirvish. On the server I installed dirvish, and the client already had rsync, so nothing to install there. With that done I started configuring dirvish. All the config for this needs to be done on the server.

Fedora didn’t provide any default master.conf for dirvish, so I copied one I’d written before, and made a few small changes.

littlebrick     00:00
# by default we keep backups for 7 days
expire-default: +7 days
*   *     *   *         1    +1 months
*   *     1   *         *    +1 year
# permissions for logs, etc.
meta-perm: 600

Most of this is fairly self explanatory – the bank is the directory where all the backups will be kept. The exclude rules specify directories that we don’t want to include in the backups.

The expire rules are setup to keep the first backup of the month for a year, and the first of the week for a month, and all other backups for 7 days. I’ve found this to work well in the past. Obviously, it’s flawed here, but I’ll get to that later.

server# mkdir -p /home/backup/littlebrick/dirvish

Then, setup the default.conf in this directory. (the format here is $bank/$vault/dirvish)

client: littlebrick
tree: /
xdev: 0
index: none
image-default: %Y%m%d
image-perm: 700

Most of the options used here are fairly obvious, I want to backup the root of the host littlebrick, without keeping any indexes, and with a permission of 700 on the backup directory, which should be named in a YYYYMMDD format (you can add times and stuff here, but i’ve never felt the need. Keeping it in this simple format makes it easier to script stuff).

The xdev option specifies whether rsync should move from the original file system. Since /home is seperate from /, and has most of my stuff xdev needs to be turned off here. A value of 0 here is false (don’t stick to one file system), all other values (including the word false) will be interpreted as true.

With that done it’s time to test it. Hopefully it’ll “just work” ™.

server# dirvish --vault littlebrick --init

Knowing this would take ages, I went to make some coffee. Before it finishes run get the exact rsync command ran on the client from ps.

With that command edit /root/.ssh/authorized_keys2 (on the client) to look like this:

command="rsync --server --sender -vlHogDtpre.iLs --numeric-ids . /",from="",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ssh-rsa UrCysuKiGZRQzAfypvZNUuOmcPHu
aXq+vBkguC8ausv+wnSXhR9TINA7jDlu6g4pkkl7eCvOelzxvJ6O62kimt6JNWHJTHUMwue/mwSog8HPmACwY/OSOtelAih76v0Pur6dlMZT5cw1oOnU8DBXjt1e3V2Mw5wQxMxorBtXHeYxw== root@littlebrick

Where the command is the rsync command is the one from ps earlier, and the ip address is the ip of the dirvish server. With that done, edit sshd_config and set: “PermitRootLogin forced-commands-only”

Then I just added a script to /etc/cron.daily on the server to run it daily


dirvish-runall will backup any vaults specified in the runall section of /etc/dirvish/master.conf. dirvish-expire deletes any images that have been lying around longer than the expire time.

Still to do….

I find the pattern of keeping the first monthly backup for a year, and the first weekly for a month works well on systems where I expect backups to run every day, but obviously this won’t run daily. I’d like a system whereby I can ensure that the first successful backup can be kept for the month, regardless of what date it is. Dirvish doesn’t support anything this fancy, so it’ll take some scripting.

At the moment backups are set to run about 4am, this means they should generally be finished before I want to leave in the morning, but it’s not that unusual for me to be still working on my laptop at 4am. I’d like a script to stall/cancel backups if I’m still working. I’m thinking I could do this by querying the state of gnome-screensaver…

Fedora 10 vs. Dell D420

When I installed fedora 7 it involved beating thing about to get it to work. As expected, installing fedora 10 was almost flawless.

The sound, graphics, wireless all worked out of the box perfectly.

As with fedora 8 I had to fix the Font DPI, internal speaker, selinux, useless services. The mouse speed is still a problem, but since there’s now no xorg.conf I just changed this in the preferences menu. Like all my other fedora 10 boxes I just removed PackageKit so that it’d fuck off.

Plymouth worked fine once I added vga=0x318 to the kernel line in the grub.conf.

So, overall the re-install has been very painless. Next I need to get it working with my 3G modem, and set up dirvish to to automated backups. I’ll be posting about that over the next few days 🙂

Simple Word Document Viewer

I have an eeepc900. It’s great for reading emails and stuff, but not very powerful. I installed abiword to read the occasional word document but it seems to eat the cpu. What I really wanted was something that could just view the documents as quickly as possible, so I started asking around…

< andrew> has anyone come accross a gui word document viewer for linux?
< phaxx> andrew: antiword tbh.doc | xmessage
< andrew >phaxx: catdoc tbh.doc | less would have a similar affect

Actually, this wasn’t that bad an idea, but I wanted more formatting than catdoc would give, and without the bother of opening a terminal.

I’ve used wvHtml before for processing mutt attachments, it does a pretty good job of converting word documents to html.

$ cat /usr/local/bin/doc-viewer
wvHtml "$@" /tmp/$$.html
exec firefox /tmp/$$.html

This is what I came up with, works excellently. (I have /tmp mounted in ram on the eee, so I wasn’t too bothered about clearing from /tmp). I use doc-viewer as the default program to handle word documents in firefox, so word docs open within the browser 🙂

Solaris, Syslog & Syslog-ng

Recently I setup a Solaris 10 box logging to a syslog-ng loghost, using the standard solaris syslog.

I found loads of docs on how to build syslog-ng on solaris, but I decided I’d rather avoid that – don’t really need any of the extra features here.

# /etc/syslog.conf Configuration file for syslogd.
# First some standard logfiles. Log by facility.
# /var/log/auth.log
*.info /var/log/syslog /var/log/daemon.log /var/log/kern.log /var/log/lpr.log /var/log/mail.log /var/log/user.log
# cron seems to log to /var/cron/log rather than syslog, so
# /var/log/cron.log should be a symlink to that.
# /var/log/cron.log
# Some `catch-all' logfiles.
*.debug;news.none;mail.none;auth.none /var/log/debug
*.info;*.notice;*;warn;auth;cron,daemon.none;mail,news.none /var/log/messages
# Emergencies are sent to everybody logged in.
*.emerg *
# Remote logging
# Send everything except auth.debug to sprout
*.debug; @log.internal

This is the syslog.conf I ended up with. The stock solaris one wasn’t very readable, so I ended up throwing it away and starting again. I took the default syslog.conf from a debian box instead, and used that as a basis for starting with (since we’re all much more familiar with debian I prefered having the logs in debian like locations).

The biggest change I had to make was that where debian had used a wildcard to indicate any level here I had to specify the lowest level to include – including this level also includes any level of a higher priority.

The last line covers sending the logs to our remote syslog-ng host, sending it all lines except auth.debug (that seemed to be VERY long). Log.internal is specified in the /etc/hosts file.

This worked fairly well on the loghost. Solaris uses UDP to send syslog messages, where as we usually use TCP, but setting syslog-ng up to accept these too wasn’t a problem. It doesn’t however send the hostname, so you’ll need to setup syslog-ng to look these up, or you’ll just get ip addresses everywhere.

The man page for syslog.conf is actually fairly helpful – this one bit in particular is worth mentioning..

A filename, beginning with a leading slash, which indicates that messages specified by the selector are to be written to the specified file. The file  is  opened  in append  mode  if it exists. If the file does not exist, logging silently fails for this action.

Hello world!

First (new) blog post, since I, em, lost the old blog. Will be putting up the more useful of the old posts and adding some new ones 🙂