Archive for the ‘Scripts’ Category

The Trouble Getting A 69

As someone who uses the bus a lot I’ve been using the Real Time Information system since it came out. I now use it pretty much every day to plan my trip.

Usually I find the data to be pretty accurate, with one constant exception – the 69 passing my house in the mornings which is constantly late.

A few weeks ago I decided to gather some data and find out just how late it was. There is still no API for accessing the real time information, so I wrote a python script to get the data from the website, and set up a cron job on my raspberry pi to run the script between 8:00 and 8:45.

That was three weeks ago, so I figured there was enough data gathered that it was worth putting it in a spreadsheet for further analysis. Here’s what I’ve learned:

  • At 8am the bus is due between 8:14 and 8:17
  • On most days it actually arrived between 8:30 and 8:35
  • The first three days seem to have had extra delays, arriving between 8:40 and 8:45
  • On one of the 15 days the bus didn’t run

The full data is available here

RPi Munin Temperatures

Just two weeks ago I upgraded my raspberry pi from the original squeeze image to the newer wheezy based raspbian. Something I tried to do was to get temperature information in munin, but it didn’t seem possible

Today’s update from Eben indicated this was now simples. So, I updated and after reboot the temperature was available from /sys/class/thermal/thermal_zone0/temp.

I’ve put together a quick munin plugin to record this. The temperatures end up getting rounded to the nearest degree, but I think that’s ok.

#!/bin/bash

if [ "$1" == "autoconf" ]; then
     if [ -e /sys/class/thermal/thermal_zone0/temp ]; then
         echo "yes"
     else
         echo "no"
         exit 1
     fi
elif [ "$1" == "config" ]; then
     echo "graph_title Raspberry Pi Temperature"
     echo "graph_args --base 1000 -l 0"
     echo "graph_vlabel Temperature in Celsius"
     echo "graph_category Sensors"
     echo "temp.label temp"
else
     temp=$(cat /sys/class/thermal/thermal_zone0/temp)
     echo "temp.value $((temp/1000))"
fi

Fedora, LXDE and the Dell Mini 10

So, last night I won a Dell Mini 10, and decided to try out LXDE rather than XFCE for a change (using the LXCD spin), so it’s been a little more fun to get working than it otherwise would have been ūüėÄ

LXDE

This seems pretty light and cool. The default applications are very light, but most are functional enough to use. The panel comes with just the right amount of gadgets and configurations to do everything I need in the smallest possible amount of screen space.

Software

Browser

The default web browser Midori seems cool, but just doesn’t have enough functionality to use as a firefox/chrome replacement, so I installed chrome. This mostly worked fairly well, but, it couldn’t open downloads because of xdg-open not being at all LXDE friendly.

Hacking up the terrible default path to be sensible and putting a new xdg-open in /usr/local/bin got around this though.


[andrew@Callandra ~]$ cat /usr/local/bin/xdg-open
#!/bin/bash

[ -z "$1" ] && exit 1

if echo "${1}" | egrep -q "^(https?://|www.)"; then
    exec google-chrome "${1}"
    exit 0
fi

pcmanfm "${1}"

Terminal

My biggest criticism of the LXDE terminal is that it was clearly designed by someone who didn’t love terminals like I do. It’s functional, but I can’t live without clickable links.

I tried installing the gnome-terminal, but, it too seemed to fail at opening links. Luckily the xfce terminal worked perfectly ūüôā

For some reason though, F11 is mapped to FullScreen by OpenBox, which is a real pain cause I use that for screen. Eventually I found to disable that in ~/.config/openbox/config

Hardware

The hardware seems mostly good, the screen and keyboard size are most excellent (though, there’s enough room here that there could have been a Page Up/Page down button). The wireless (Broadcom 4312 *shudder*) works fine with kmod-wl from rpmfusion, and the network manager applet works great in LXDE.

Biggest problem with the hardware is the trackpad, which has the buttons integrated (like the ones on the new Mac). This is at best ‘tricky’ to use, it’s possible better drivers would help, I’m still looking.

WTF did I change?

So, it’s been a while since I posted, I thought I’d share my latest heap of hacky shell up here.

Last week we were having a discussion following on from Chris Siebenmann’s post on restoring files from a package, to see what changes had been made. Anyway, we decided that such a feature could be useful, so I promised to hack one together.

It took a week to finish this, because I spent most of my time confused about how apt-get couldn’t just download package without being root. I now know that it defiantly can’t do this. I also know that aptitude can.

#!/bin/bash
#
# Check we have an argument
#
if [ -z "$1" ]; then
    echo "Usage $0 /path/to/file"
exit 1
fi
#
# Check this is actually a text file, if
# not we drop out straight away
#
if [ `file $1|grep -c text ` -lt 1 ]; then
    echo "The file $1 is not a text file"
    exit 2
fi
#
# Work out the package name
#
PACKAGE=`dpkg -S $1| awk -F: '{print $1}'`
#
# If there's no matching package then
# just exit with an error
#
if [ $? -gt 0 ]; then
    echo "No package provides this file"
    exit 3
fi
#
# Download the package
#
echo "File $1 is provided by $PACKAGE. Downloading to /tmp/$$"
mkdir /tmp/$$
cd /tmp/$$
aptitude download $PACKAGE > /dev/null
#
# Extract the package
#
PKGNAME=`find /tmp/$$ -maxdepth 1 -type f -name "*.deb"`
dpkg -x $PKGNAME /tmp/$$
#
# Diff shit
#
diff -us /tmp/$$/$1 $1
#
# If there's a difference then offer to save the diff
#
if [ `diff -qs $1 /tmp/$$/$1|grep -c identical` -ne 1 ]; then
    echo -n "Save this diff? [yes] "
    ANSWER=yes
    read answer
    if [ -n "$answer" ]; then
        ANSWER=$answer
    fi
    if [ "$ANSWER" = "yes" ]; then
        diff -us /tmp/$$/$1 $1 > /tmp/$PACKAGE.$$.diff
        File saved to /tmp/$PACKAGE.$$.diff
    fi
    echo
fi
#
# We should be nice and offer to clean up
#
echo -n "Delete the temporary package files from /tmp/$$? [yes] "
ANSWER=yes
read answer
if [ -n "$answer" ]; then
    ANSWER=$answer
fi
if [ "$ANSWER" = "yes" ]; then
    rm -fr /tmp/$$/
fi
echo

This script is also available from the RedBrick package repo

Dirvish, Part 2

In my earlier post on dirvish I said I’d have to beat it to do the kind of expire rules I wanted… well I’ve finally found time to do that.

It’s a pretty simple approach, I’ve used the cron script to touch a dotfile if it’s the start of the week/month, and if that file exists then it determines the expire time, which is just passed as an argument to dirvish

This way, the first backup of the month should always be kept for a year, and the first of the week for a month, no matter when they occur.


#!/bin/bash
#
date=`date +%d`
day=`date +%a`
#
if [ "$date" -eq 1 ]; then
touch /home/backup/littlebrick/.monthly
fi
#
if [ $day == "Sun" ]; then
touch /home/backup/littlebrick/.weekly
fi
#
if [ -f /home/backup/littlebrick/.monthly ]; then
dirvish --vault littlebrick --expire 1year --image-time 00:00
elif [ -f /home/backup/littlebrick/.weekly ]; then
dirvish --vault littlebrick --expire 1month --image-time 00:00
else
dirvish --vault littlebrick --image-time 00:00
fi
#
dirvish-expire

With that done, all that’s left to do is to remove the dotfiles following a successful backup. I used the post-server options for this.

#/etc/dirvish/master.conf
...
post-server: ; /backup/dirvish/post_backup.sh $DIRVISH_DEST
...

And, this is the post_backup script


#!/bin/bash
#
# This script runs after backups, and if the backup was
# successful clears out the month/week flags
#
#$2 is the DIRVISH_DEST
cd $2
cd ..
#
#we're now in /home/backup/littlebrick/YYYYMMDD/
#
success=`grep -c "Status: success" summary`
#
if [ "$success" -gt 0 ]; then
if [ -f /home/backup/littlebrick/.monthly ]; then
rm /home/backup/littlebrick/.monthly
if [ -f /home/backup/littlebrick/.weekly ]; then
rm /home/backup/littlebrick/.monthly
fi
fi

Laptop Backups With Dirvish

My laptop is something I should *really* backup, since most of my files and stuff I’m working on is there. Since I’m too lazy to do this regularly though I setup an automated system to do this after I re-installed it this week.

My objective was to setup a backup system that would always run automatically, without me having to press any buttons. Incremental backups were a must. Keeping multiple days of backups was also a relatively high priority.

Dirvish is a perl wrapper for rsync. It’s main feature is to automate running incremental rsync backups on a regular basis without too much hassle. Having worked with it alot before it seemed the obvious choice for running my laptop backups.

Before I continue, some terminology:

Server: the machine running dirvish, in this case my desktop
Client: the machine with the files, in this case my laptop (littlebrick)
Bank: A directory where dirvish stores backups (eg. /backup)
Vault: A particular backup source (eg, my laptop)
Branch: A particular run of a vault (usually, daily)

DHCP

First, I needed to setup my laptop to have a static ip address on the wireless, which uses dhcp.

/etc/dhcpd.conf
...
host littlebrick {
hardware ethernet       00:1b:77:37:af:3d;
fixed-address           10.0.2.90;
}
...

DNS

and then I added this ip to the local dns server.

/var/named/master/db.internal
...
littlebrick     A       10.0.2.90
...

Now I can specify the laptop by hostname in the config files later. You could skip this step and just use the ip or /etc/hosts of course.

SSH

Next step was to generate an ssh key, so the server could connect without a password

laptop# ssh-keygen
laptop# cat id_rsa.pub > /root/.ssh/authorized_keys2
laptop# scp id_rsa root@desktop:~/.ssh/

I set the laptop to allow root ssh (I’ll tidy this up a bit later), and then tested the desktop root user could ssh to the laptop.

Actually, the default in most cases is to allow ssh access as root (either via password or key). I think this is fairly nuts as a default, and usually change it pretty quickly

Dirvish

Once that’s done I started on dirvish. On the server I installed dirvish, and the client already had rsync, so nothing to install there. With that done I started configuring dirvish. All the config for this needs to be done on the server.

Fedora didn’t provide any default master.conf for dirvish, so I copied one I’d written before, and made a few small changes.

/etc/dirvish/master.conf
bank:
/home/backup
exclude:
lost+found/
/var/log/
/var/tmp/
/var/run/
/var/lock/
/mnt/
/media/
/sys
/dev/
/proc/
/tmp/
/vmlinuz*
/initrd*
.gvfs
Runall:
littlebrick     00:00
# by default we keep backups for 7 days
expire-default: +7 days
expire-rule:
#    MIN HR    DOM MON    DOW  STRFTIME_FMT
*   *     *   *         1    +1 months
*   *     1   *         *    +1 year
#
# permissions for logs, etc.
meta-perm: 600

Most of this is fairly self explanatory – the bank is the directory where all the backups will be kept. The exclude rules specify directories that we don’t want to include in the backups.

The expire rules are setup to keep the first backup of the month for a year, and the first of the week for a month, and all other backups for 7 days. I’ve found this to work well in the past. Obviously, it’s flawed here, but I’ll get to that later.

server# mkdir -p /home/backup/littlebrick/dirvish

Then, setup the default.conf in this directory. (the format here is $bank/$vault/dirvish)

/home/backup/littlebrick/dirvish.conf
client: littlebrick
tree: /
xdev: 0
index: none
image-default: %Y%m%d
image-perm: 700

Most of the options used here are fairly obvious, I want to backup the root of the host littlebrick, without keeping any indexes, and with a permission of 700 on the backup directory, which should be named in a YYYYMMDD format (you can add times and stuff here, but i’ve never felt the need. Keeping it in this simple format makes it easier to script stuff).

The xdev option specifies whether rsync should move from the original file system. Since /home is seperate from /, and has most of my stuff xdev needs to be turned off here. A value of 0 here is false (don’t stick to one file system), all other values (including the word false) will be interpreted as true.

With that done it’s time to test it. Hopefully it’ll “just work” ™.

server# dirvish --vault littlebrick --init

Knowing this would take ages, I went to make some coffee. Before it finishes run get the exact rsync command ran on the client from ps.

With that command edit /root/.ssh/authorized_keys2 (on the client) to look like this:

command="rsync --server --sender -vlHogDtpre.iLs --numeric-ids . /",from="10.0.2.5",no-port-forwarding,no-X11-forwarding,no-agent-forwarding ssh-rsa UrCysuKiGZRQzAfypvZNUuOmcPHu
lZy/B31Mje8UfooJCurM3JKL0t7U7mbor2Fs43OVZhUp6m1xburbDpFOtzKU0UDMO0MopRZ6q8si6fUP5oqAQAE2
U3Pieq3qJGLYzUB3zzzoDromtsYXbCwWDJsVdVVo6yvKNiSDWBZAo2UeyqRtsPuavVhuHqOxwu2/Fqu
aXq+vBkguC8ausv+wnSXhR9TINA7jDlu6g4pkkl7eCvOelzxvJ6O62kimt6JNWHJTHUMwue/mwSog8HPmACwY/OSOtelAih76v0Pur6dlMZT5cw1oOnU8DBXjt1e3V2Mw5wQxMxorBtXHeYxw== root@littlebrick

Where the command is the rsync command is the one from ps earlier, and the ip address is the ip of the dirvish server. With that done, edit sshd_config and set: “PermitRootLogin forced-commands-only”

Then I just added a script to /etc/cron.daily on the server to run it daily

/etc/cron.daily/00dirvish
dirvish-runall
dirvish-expire

dirvish-runall will backup any vaults specified in the runall section of /etc/dirvish/master.conf. dirvish-expire deletes any images that have been lying around longer than the expire time.

Still to do….

I find the pattern of keeping the first monthly backup for a year, and the first weekly for a month works well on systems where I expect backups to run every day, but obviously this won’t run daily. I’d like a system whereby I can ensure that the first successful backup can be kept for the month, regardless of what date it is. Dirvish doesn’t support anything this fancy, so it’ll take some scripting.

At the moment backups are set to run about 4am, this means they should generally be finished before I want to leave in the morning, but it’s not that unusual for me to be still working on my laptop at 4am. I’d like a script to stall/cancel backups if I’m still working. I’m thinking I could do this by querying the state of gnome-screensaver…

Simple Word Document Viewer

I have an eeepc900. It’s great for reading emails and stuff, but not very powerful. I installed abiword to read the occasional word document but it seems to eat the cpu. What I really wanted was something that could just view the documents as quickly as possible, so I started asking around…

< andrew> has anyone come accross a gui word document viewer for linux?
< phaxx> andrew: antiword tbh.doc | xmessage
< andrew >phaxx: catdoc tbh.doc | less would have a similar affect

Actually, this wasn’t that bad an idea, but I wanted more formatting than catdoc would give, and without the bother of opening a terminal.

I’ve used wvHtml before for processing mutt attachments, it does a pretty good job of converting word documents to html.

$ cat /usr/local/bin/doc-viewer
#!/bin/bash
#
wvHtml "$@" /tmp/$$.html
exec firefox /tmp/$$.html

This is what I came up with, works excellently. (I have /tmp mounted in ram on the eee, so I wasn’t too bothered about clearing from /tmp). I use doc-viewer as the default program to handle word documents in firefox, so word docs open within the browser ūüôā