Archive for the 'UbuntuWeblogsOrg' Category

Debugging notes-to-self

Stuff I wish I’d known for years (and that the rest of the world probably always knew, just not me):

  1. after your buggy program was slowly eating your memory and swap, and you escaped to a console to kill it using Alt+SysRq+R and then Ctrl+Alt+F1, when you get back to your X-session you can make it grab the keyboard again by typing “kbd_mode -s” in an xterm (found here after trying all sorts of incantations in DuckDuckGo)
  2. if that was on a laptop keyboard with “AltGr dead keys”, the SysRq combo might need to be Fn+SysRq+AltGr+R (yes, try that with one hand!), where AltGr is the right Alt key (found after lots of failed attempts – very frustrating if your swap runs out before you get to the console)
  3. in the end, you wouldn’t typically have needed any of the above if only you’d known that “ulimit -v 1000000” limits any process in your bash shell to 1GB of memory, so that your buggy program will die superquick without locking up the system for 10 minutes (found in no time through a most obvious query – it just took a long time before I thought of looking for it at all…)

Ksplice Uptrack: a quick-test on Ubuntu 9.04 Live

I’ve been using Ubuntu 8.04 on my laptop for ages, and never had any reason to upgrade from there – “it just works, I’m done upgrading” is what I’d smugly tell people… Now, I’ve found a big reason to upgrade: Ksplice, which I mentioned the other day, put a new service up:

Ksplice Uptrack is a new service that lets you effortlessly keep your systems up to date and secure, without rebooting.

Once you’ve completed the easy installation process, your system will be set up to receive rebootless updates instead of traditional, disruptive updates.  […]

Ksplice, Inc. is proud to make this service freely available for the latest version of the world’s most popular desktop Linux distribution: Ubuntu 9.04 Jaunty Jackalope.

No more reboots, and still applying security patches as soon as they become available. That’s worth the dist-upgrade hassle.

For now, all I did was running a quick test. I had a USB stick with Ubuntu Netbook Remix 9.04 lying around, so I booted from that, hooked up the wifi (man, connecting is fast with NetworkManager 0.7-something – another reason to upgrade…), downloaded ksplice-uptrack.deb, and installed it on the Live system (you also need network connectivity to fetch some dependencies from the Ubuntu repository). This is what you get:

ksplice-uptrack updates window

There’s a little tray-icon (the one resembling a “K”…) informing you that kernel updates are available, and clicking it opens an update window. Nothing exciting to see here, actually.

ksplice-uptrack in action

Still not very exciting. The whole thing is very understated, almost disappointingly so – I mean, something this cool should look cool, shouldn’t it?

…. and everything still works after this. In fact, I’m typing this post from the Live system with the (supposedly) updated kernel. I tried shutting the lid on my D630, and it nicely went into ACPI suspend. And came back up.

Wicked.

(Small disappointment: it seems Firefox crashed between suspend and resume. Did it a second time, and again Firefox died. Third time: no problems. Not sure if this has anything to do with anything, so for now pretend I didn’t mention it.)

Cool stuff, seriously. This will be in 10.04 by default, I’ve no doubt. In case you’re looking, here’s one guy eager to work on that!

One more thing: in their FAQ they suggest a little test to demonstrate that the thing actually does something. I tried their suggestion and ran their test-thing a couple of times. But I’m off to bed now, so here’s the output, and I’ll leave calculating whether the difference before/after updates is statistically significant to you…

ubuntu@ubuntu:~$ wget -O demo.c http://www.ksplice.com/uptrack/2009-06-demo.c
ubuntu@ubuntu:~$ gcc demo.c -o demo
ubuntu@ubuntu:~$ sudo cpufreq-selector -c 0 -g performance
ubuntu@ubuntu:~$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance
ubuntu@ubuntu:~$ sudo cpufreq-selector -c 1 -g performance
ubuntu@ubuntu:~$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Core(TM)2 Duo CPU     T8100  @ 2.10GHz
stepping        : 6
cpu MHz         : 2101.000
cache size      : 3072 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 lahf_lm ida tpr_shadow vnmi flexpriority
bogomips        : 4189.64
clflush size    : 64
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Core(TM)2 Duo CPU     T8100  @ 2.10GHz
stepping        : 6
cpu MHz         : 2101.000
cache size      : 3072 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
apicid          : 1
initial apicid  : 1
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 lahf_lm ida tpr_shadow vnmi flexpriority
bogomips        : 4189.57
clflush size    : 64
power management:

ubuntu@ubuntu:~$ ./demo
time to write 100 lines is 6(msec)
# ...hmmm, wait, this is a Live system...
ubuntu@ubuntu:~$ sudo mount /dev/sda3 /mnt
ubuntu@ubuntu:~$ cd /mnt/
ubuntu@ubuntu:/mnt$ sudo mkdir test
ubuntu@ubuntu:/mnt$ sudo chmod a+rwx test
ubuntu@ubuntu:/mnt$ cd test/
ubuntu@ubuntu:/mnt/test$ cp /home/ubuntu/demo .
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 49(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 54(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 64(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 60(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 75(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 72(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 62(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 65(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 80(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 52(msec)
ubuntu@ubuntu:/mnt/test$ sudo uptrack-remove --all -y
The following steps will be taken:
Remove [cdoprpi1] Performance regression in filesystem buffer code.
Remove [9xoc5qmo] Possible erroneous memory overcommit in program start.
Remove [ll9q1ymc] Multiple bugs in filesystem core.
Remove [ovniqwxh] CVE-2009-1192: Information leak in the agp subsystem.
Remove [hrxbvh0e] CVE-2009-1265: Integer overflow in the af_rose maximum user frame size.
Remove [uzolzfa2] CVE-2009-1337: kill the wrong capable(CAP_KILL) check.
Remove [xgqc9vy4] VGA console corrupts non-ASCII characters.
Remove [pdfrn6qa] Denial of service by evading CPU time limits.
Remove [c8ueseae] Symbolic link filenames under eCryptfs can produce alarming warnings in dmesg.
Removing [cdoprpi1] Performance regression in filesystem buffer code.
Removing [9xoc5qmo] Possible erroneous memory overcommit in program start.
Removing [ll9q1ymc] Multiple bugs in filesystem core.
Removing [ovniqwxh] CVE-2009-1192: Information leak in the agp subsystem.
Removing [hrxbvh0e] CVE-2009-1265: Integer overflow in the af_rose maximum user frame size.
Removing [uzolzfa2] CVE-2009-1337: kill the wrong capable(CAP_KILL) check.
Removing [xgqc9vy4] VGA console corrupts non-ASCII characters.
Removing [pdfrn6qa] Denial of service by evading CPU time limits.
Removing [c8ueseae] Symbolic link filenames under eCryptfs can produce alarming warnings in dmesg.
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 816(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 805(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 793(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 786(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 785(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 787(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 791(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 787(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 786(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 785(msec)
ubuntu@ubuntu:/mnt/test$ sudo uptrack-upgrade -y
The following steps will be taken:
Install [c8ueseae] Symbolic link filenames under eCryptfs can produce alarming warnings in dmesg.
Install [pdfrn6qa] Denial of service by evading CPU time limits.
Install [xgqc9vy4] VGA console corrupts non-ASCII characters.
Install [uzolzfa2] CVE-2009-1337: kill the wrong capable(CAP_KILL) check.
Install [hrxbvh0e] CVE-2009-1265: Integer overflow in the af_rose maximum user frame size.
Install [ovniqwxh] CVE-2009-1192: Information leak in the agp subsystem.
Install [ll9q1ymc] Multiple bugs in filesystem core.
Install [9xoc5qmo] Possible erroneous memory overcommit in program start.
Install [cdoprpi1] Performance regression in filesystem buffer code.
Installing [c8ueseae] Symbolic link filenames under eCryptfs can produce alarming warnings in dmesg.
Installing [pdfrn6qa] Denial of service by evading CPU time limits.
Installing [xgqc9vy4] VGA console corrupts non-ASCII characters.
Installing [uzolzfa2] CVE-2009-1337: kill the wrong capable(CAP_KILL) check.
Installing [hrxbvh0e] CVE-2009-1265: Integer overflow in the af_rose maximum user frame size.
Installing [ovniqwxh] CVE-2009-1192: Information leak in the agp subsystem.
Installing [ll9q1ymc] Multiple bugs in filesystem core.
Installing [9xoc5qmo] Possible erroneous memory overcommit in program start.
Installing [cdoprpi1] Performance regression in filesystem buffer code.
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 61(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 56(msec)
ubuntu@ubuntu:/mnt/test$ ./demo
time to write 100 lines is 47(msec)
ubuntu@ubuntu:/mnt/test$

Ksplice Trophée du Libre

I’ve repeatedly been whining here about how kernel-update reboots kill productivity, but I also think that delaying security updates is the worse alternative.  So I was very excited to learn about Ksplice, through the LWN announcement of the “Trophées du Libre”. Ksplice is the 2009 winner in the Security category.

A quick snippet from the project page:

Ksplice enables running systems to stay secure without the disruption of rebooting.  Specifically, Ksplice creates rebootless updates that are based on traditional source code patches.  These updates are as effective as traditional updates, but they can be applied seamlessly, with no downtime.

Ksplice currently supports updating the Linux kernel, but the core technology applies to any operating system or to user space applications.

A quick search tells me even ZDNet had already heard of this project over a year ago, so I’m half ashamed that it’s news to me, but I’m too excited to keep it to myself :)

MiserWare MicroMiser: Miserable Licensing

Edit dd. 25 May: Miserware are offering the software under a more permissive license now. In short: they ask that you run your benchmarking procedures by them before publishing results. This seems a reasonable compromise, that protects them from poorly conducted testing taking the limelight. I will be playing with it…


Through the Ubuntu Weblogs feed I read a post at The Open Sourcerer about a piece of software by a company called MiserWare (yeah, the name inspired the pun even before I concluded as much…). They’ve released a private beta of MicroMiser, a closed-source package that promises a substantial power-consumption reduction on x86 Linux systems. Quoting from their website:

MicroMiser typically lowers total system energy use by 10-35% even when a system is 100% utilized.

That’s interesting. Even better, Alan wrote a follow-up post at The Open Sourcerer which shows that on his Ubuntu Server system the software estimates its own power savings to be close to 65% (!). Now, assuming for the moment that it’s not a bug in the calculations in the beta (hey, I’m in a mild mood tonight), I’d say those 65% would be really worthwile.

Alan was also kind enough to send me an invitation for the beta, and so – always the curious type – I was starting to make a plan on how to test this thing: use a dummy installation (we don’t trust closed-source betas, do we?), current-monitoring at the wall-socket, that sort of thing.Then, I figured I’d download the package first to see what was in it. A kernel module? Lots of ugly scripts? Who knows…

I don’t, anyway. I never got to the download step. In the step before that I was presented with a rather restrictive license. I quote verbatim (only the change to bold-type is mine, all text is original):

5.    Confidentiality

The Software and Documentation, all related Intellectual Property, and any information learned or discovered by You about the performance of the Software in the course of use under this License constitutes proprietary trade secret information owned solely by Licensor (collectively, “Confidential Information”).  You agree that You will not, without the express prior written consent of Licensor, (1) use the Confidential Information other than to use the Software as authorized by this License Agreement; (2) disclose any Confidential Information to any third party; or (3) fail to use best efforts to safeguard the Confidential Information from unauthorized use, copying, or disclosure. You acknowledge that a breach of this Section 5 may cause the Licensor irreparable harm and damages that are difficult to ascertain.  Therefore, the Licensor, upon a disclosure or threatened disclosure of any Confidential Information, will be entitled to injunctive relief (without the requirement of posting bond), without limiting its other remedies under this License Agreement, in equity or at law.  The obligations of this Section shall survive this License Agreement without limitation in duration.  By clicking the “I Agree” button below, You consent to having any information that You provide to the Licensor processed and stored in the United States.

Well, I’m still not a lawyer, nor am I a native speaker, but I think that this basically means I would not be allowed to tell you what I would read on my current monitor. What the….

I also think that this means that the post reporting the 65% savings is violating the license.

After a long half second of thinking about it, I did not agree, so I can’t tell you anything about the software – by the way, even if I had agreed, I still wouldn’t be able to tell you anything! My advice? Don’t waste your time with bloody legalese and read some real information instead: Intel has a very informative website on the topic at LessWatts.org.

The superiority of the distro

Shawn wrote a nice piece pointing out why developers (in particular for embedded systems) are better off running Linux than Windows. In summary, it’s all about the tools that ship with it. If we’re nitpicking then, what we’re actually talking about here is not Linux proper (the kernel), and not GNU/Linux (the operating system), but rather the distribution (whichever one that is). It’s the software stack as a whole that’s making the difference.

This is exactly what I usually tell people who ask why I’m running “this Linux thing”. It’s better suited to my needs on every level of the software stack. Of course, that’s usually way too vague to compel someone who had to ask to begin with. Examples please? My new favourite example is a security thing.

The other week I was catching up with a few long-overdue admin tasks on my parents’ PCs, mostly security updates. They’re (still) on MS Windows, and you should have seen the number of pop-ups when I logged on as admin: I don’t know how many apps, all reporting they’d like permission to fetch and install updates. Compare that to the elegance of the little warning star in the Gnome menu, or the output of a quick “aptitude upgrade”.

Big deal? I think so. I’m quite sure that that single interface to all software updates is not just more elegant, but that it’s also directly keeping systems more secure. Quoting from an old eWeek item:

For example, for the nine highest-profile Windows malicious code incidents as of March 2003, Microsofts patches predated major outbreaks by an average of 305 days, yet most firms hadnt applied the patches.

That is not a statement about Windows, or Linux. It’s a statement about human nature.

Here’s another item, from Ars:

Secunia cited data from Microsoft showing that third-party software vulnerabilities are the ones that are most frequently exploited, and said that its own data showed that users simply don’t update as frequently as they should.

Having a clear, simple, and non-crappy upgrade manager vastly diminishes these problems, because all people are lame, and the number of people that will not apply updates promptly will go up at least quadratically with the number of steps the update takes (and that’s a conservative estimate). That’s why distros will win from bare operating systems with apps dropped on top of them. It’s also why I’m going to press mom and dad to please let me replace their systems…

Is this benefit intrinsically tied to free software? In theory, no. I actually tried to pitch this idea to the Ideastorm crowd at some point. But maybe it would not work so well in practice: if MS would try to turn Windows into a distro, or would try to press other vendors into using their update manager, the anti-monopolistic regulators would be all over them in no time. So in practice, one might say this is a benefit of free software. Yay.

Dell shipping revamped Ubuntu 8.04 with the Mini 10

Update: it looks like I’m sucking in a lot of search engine traffic relating to Poulsbo / GMA500 on Linux – that was unintentional because there’s no information here. Please look at Adam Williamson’s work on a native driver for Poulsbo instead.

Update 2: I came across some Ubuntu-specific notes which should be useful too.

I was a bit surprised by Dell’s announcement just now: it seems they’ve gone through some real efforts to make Ubuntu 8.04 look all 2009ish on their latest netbook entry-level notebook. Let me try and be all 2009ish too, and include some of their Flash-terrorism YouTube footage:

Don’t worry,  I haven’t suddenly become a sucker for fancy “iconography” (apparently that has a whole different meaning “in the office of the CTO” – don’t take this badly, Dell, I am a fan!), nor have I finally grown fond of NetworkManager, but seriously: WWAN support is pretty cool, and having it neatly integrated is even better.

Actually, I’m just really pleased to see what importance Dell seem to be assigning their Ubuntu programme these days. This little cheapling has Intel’s Poulsbo chipset (just like the Mini 12), which is great for power consumption, but apparently a real chore when it comes to Linux support. As far as I’m aware, no other vendor is even trying to support it. In addition, also to keep the price low, there’s the ever troublesome Broadcom wifi chip, “official” Linux-support for which was supposedly also negotiated by Dell. I’m guessing there was no way they’d get all that working on 9.04, but to make up for that they backported the NM improvements and updated the “iconography” (sorry).

I’m just saying, that’s a lot of effort when you have a business partner called Microsoft that’s giving you Windows XP basically for free…

Ok, enough Ubuntu/Dell fanboyism for one day. Good night!

P.S.: Rik – if you happen to read this – I didn’t know they had a 6-cell battery coming (I told the guy to get a Samsung instead…)!

Fully restricting rsync options server-side

Let me tell you up front: the following is a respin of information I found elsewhere. And it was very well written. Normally, then, I wouldn’t blog this, and rather add a link in my RSS feed in the sidebar – after all that’s the most compact form of “code reuse” and of upping the PageRank for a good site I found. What makes things different today? Well, it took me a crazy number of  search-engine queries to find the info.

Maybe I’m just stoopid, but let’s assume I’m not. I hope this respin ranks a bit better for keywords, so I can help some other lost souls find the site that I found. If you want the expert story, click straight through to the source – actually, that whole site is simply excellent, and well worth browsing thoroughly if you’re looking to learn cool sysadmin stuff and more.

I learned (at least) two new things today:

  • how to extract the rsync command that your locally-executed rsync sends to the remote machine’s ssh-server
  • how to make the ssh-server execute that exact command, regardless what someone tries to feed it from the local machine

But first, let me explain (to myself) why I wanted to know these things.

The issue with remote backups

You want off-site backups, because, well, that’s rule #3. But, by rule #2, you also want to backup often, and there’s only one way to guarantee that that will work out: automation. There’s a problem with that though: to attain automation, your remote backups will need some unprotected authentication token, e.g. an ssh-certificate with an empty passphrase.

Obviously, you want to restrict what that dangerous key lets you do on the remote system. Simply put, you don’t want someone that managed to break into your backup client to be able to erase both your backups and the originals. The solutions I had seen so far included creating a separate backup-user on the server, and providing a restricted shell of some sort. That’s one way of doing things, but it’s not easy to set up:

  • you only want to allow a select few commands, say rsync for transport, and perhaps some scripts to prepare the backup
  • ideally you want to have read-only access so that the client performing the backup cannot damage files, which might even occur without malicious intent, say by a wrong string of rsync options
  • but maybe you want to run some sort of hotcopy command on some database you’re using, and this does require write access
  • do you create yet another user for that?
  • and are you sure your shell is really as restricted as you think? No tricks to break out of it?
  • aaaaaahhrghh…..

Right. I’m *that* mistrusting, and especially when it comes to my own competence. I’d definitely bork that restricted shell setup. Please give me something dead simple.

Figuring out what your local rsync needs from the remote rsync

Okay. Assume we’ll always call the server with the exact same rsync command, perhaps something like

bash$ rsync -avz remote_host:/var/backup/ /var/remote_host_backup

(On a side-note, I’m still doubting myself every time: trailing slash or no trailing slash? Terrible.)

Now, you can see what rsync command will get executed on the remote host if you add another -v:

bash$ rsync -avvnz remote_host:/var/backup/ /var/remote_host_backup

where I also added an -n to have a dry-run. The first line of output reads something like

opening connection using ssh remote_host rsync --server --sender -vvnlogDtprz . /var/backup

…which runs off the page here because I didn’t pay WordPress for my own custom CSS yet, but you can try this yourself anyway. What we’re interested in is the part that starts at “rsync”: this is what is executed on the remote host.

Using sshd with the command=”” option

Remember we’re using a passphrase-less ssh-certificate for the sake of automation. On the server, that requires an entry like this in $HOME/.ssh/authorized_keys:

ssh-rsa AAYourVeryLongPublicKeyThingy== plus arbitrary comment/username at the end

The sshd manpage tells you you can insert quite a few options at the start of this line. You should really consider all of those options, but the cool one for now is the command=”” option. Between the quotes we put the result of the previous section minus the -n (or you’ll have only dry runs…).

command="rsync --server --sender -vlogDtprz . /var/backup" ssh-rsa AAYourVeryLongPublicKeyThingy== plus arbitrary comment/username at the end

…that’s probably running off the page big time now. Sorry. And I didn’t even add all the other restrictive options you ought to consider.

The beauty of this is that sshd will now ignore whatever abuse you’re feeding it from the ssh client. Whenever you authenticate using this specific certificate, it will only run that exact command.

Let me put this  yet another way. The only way to successfully talk to the server with that certificate is to say what it expects you to say: you can only run the matching local rsync command or the two rsync instances will not understand eachother. All the options are fixed, client-side and server-side.

This is what you want. Or, it is what I wanted, anyway.

What about running scripts before the actual rsync?

Okay, I learned a third thing. This was in the rsync manpage: your remote rsync “can be any program, script, or command sequence you’d care to run, so long as it does not corrupt the standard-in & standard-out that rsync is using to communicate”.

In other words: you can run any database hotcopy command on the server, as long as it cares to shut up, so that to the client, it looks as if only rsync was called. Your authorized_keys entry now looks somewhat like this:

command="do_db_hotcopy >> /var/log/hotcopy.log 2>&1 ; rsync --server --sender -vlogDtprz . /var/backup" ssh-rsa AAYourVeryLongPublicKeyThingy== plus arbitrary comment/username at the end

… where you’re being careful to make sure the only output sent comes from rsync. This works for me; I could imagine a long script might cause your local rsync to time-out in some way, so ymmv.

One more thing

I’ll shut up soon, too, but there was actually also a fourth thing… how do you make sure your local rsync command uses the restricted, passphraseless key under all circumstances? When I’m actually logged in myself, often ssh-agent is keeping my less-restricted key available. The problem with this is that ssh will prefer using that key, but when I use that, my fancy hotcopy (from the previous section) never gets called.

To fix this, my backup script on the client contains an extra -e option to rsync, which is self-explanatory, but that’s not enough: ssh still prefers the key held by ssh-agent. The full solution (as the ssh-agent manpage more or less documents) is thus:

#! /bin/bash
unset SSH_AUTH_SOCK
rsync -avz -e 'ssh -i .ssh/restricted_key' remote_host:/var/backup/ /var/remote_host_backup

Sometime soon I might respin this whole thing with rdiff-backup (…you want to keep multiple states of your backup, because, well, that’s rule #4 :P). I just need to figure out how client-server communication works for that.