Let me tell you up front: the following is a respin of information I found elsewhere. And it was very well written. Normally, then, I wouldn’t blog this, and rather add a link in my RSS feed in the sidebar – after all that’s the most compact form of “code reuse” and of upping the PageRank for a good site I found. What makes things different today? Well, it took me a crazy number of search-engine queries to find the info.
Maybe I’m just stoopid, but let’s assume I’m not. I hope this respin ranks a bit better for keywords, so I can help some other lost souls find the site that I found. If you want the expert story, click straight through to the source – actually, that whole site is simply excellent, and well worth browsing thoroughly if you’re looking to learn cool sysadmin stuff and more.
I learned (at least) two new things today:
- how to extract the rsync command that your locally-executed rsync sends to the remote machine’s ssh-server
- how to make the ssh-server execute that exact command, regardless what someone tries to feed it from the local machine
But first, let me explain (to myself) why I wanted to know these things.
The issue with remote backups
You want off-site backups, because, well, that’s rule #3. But, by rule #2, you also want to backup often, and there’s only one way to guarantee that that will work out: automation. There’s a problem with that though: to attain automation, your remote backups will need some unprotected authentication token, e.g. an ssh-certificate with an empty passphrase.
Obviously, you want to restrict what that dangerous key lets you do on the remote system. Simply put, you don’t want someone that managed to break into your backup client to be able to erase both your backups and the originals. The solutions I had seen so far included creating a separate backup-user on the server, and providing a restricted shell of some sort. That’s one way of doing things, but it’s not easy to set up:
- you only want to allow a select few commands, say rsync for transport, and perhaps some scripts to prepare the backup
- ideally you want to have read-only access so that the client performing the backup cannot damage files, which might even occur without malicious intent, say by a wrong string of rsync options
- but maybe you want to run some sort of hotcopy command on some database you’re using, and this does require write access
- do you create yet another user for that?
- and are you sure your shell is really as restricted as you think? No tricks to break out of it?
Right. I’m *that* mistrusting, and especially when it comes to my own competence. I’d definitely bork that restricted shell setup. Please give me something dead simple.
Figuring out what your local rsync needs from the remote rsync
Okay. Assume we’ll always call the server with the exact same rsync command, perhaps something like
bash$ rsync -avz remote_host:/var/backup/ /var/remote_host_backup
(On a side-note, I’m still doubting myself every time: trailing slash or no trailing slash? Terrible.)
Now, you can see what rsync command will get executed on the remote host if you add another -v:
bash$ rsync -avvnz remote_host:/var/backup/ /var/remote_host_backup
where I also added an -n to have a dry-run. The first line of output reads something like
opening connection using ssh remote_host rsync --server --sender -vvnlogDtprz . /var/backup
…which runs off the page here because I didn’t pay WordPress for my own custom CSS yet, but you can try this yourself anyway. What we’re interested in is the part that starts at “rsync”: this is what is executed on the remote host.
Using sshd with the command=”" option
Remember we’re using a passphrase-less ssh-certificate for the sake of automation. On the server, that requires an entry like this in $HOME/.ssh/authorized_keys:
ssh-rsa AAYourVeryLongPublicKeyThingy== plus arbitrary comment/username at the end
The sshd manpage tells you you can insert quite a few options at the start of this line. You should really consider all of those options, but the cool one for now is the command=”" option. Between the quotes we put the result of the previous section minus the -n (or you’ll have only dry runs…).
command="rsync --server --sender -vlogDtprz . /var/backup" ssh-rsa AAYourVeryLongPublicKeyThingy== plus arbitrary comment/username at the end
…that’s probably running off the page big time now. Sorry. And I didn’t even add all the other restrictive options you ought to consider.
The beauty of this is that sshd will now ignore whatever abuse you’re feeding it from the ssh client. Whenever you authenticate using this specific certificate, it will only run that exact command.
Let me put this yet another way. The only way to successfully talk to the server with that certificate is to say what it expects you to say: you can only run the matching local rsync command or the two rsync instances will not understand eachother. All the options are fixed, client-side and server-side.
This is what you want. Or, it is what I wanted, anyway.
What about running scripts before the actual rsync?
Okay, I learned a third thing. This was in the rsync manpage: your remote rsync “can be any program, script, or command sequence you’d care to run, so long as it does not corrupt the standard-in & standard-out that rsync is using to communicate”.
In other words: you can run any database hotcopy command on the server, as long as it cares to shut up, so that to the client, it looks as if only rsync was called. Your authorized_keys entry now looks somewhat like this:
command="do_db_hotcopy >> /var/log/hotcopy.log 2>&1 ; rsync --server --sender -vlogDtprz . /var/backup" ssh-rsa AAYourVeryLongPublicKeyThingy== plus arbitrary comment/username at the end
… where you’re being careful to make sure the only output sent comes from rsync. This works for me; I could imagine a long script might cause your local rsync to time-out in some way, so ymmv.
One more thing
I’ll shut up soon, too, but there was actually also a fourth thing… how do you make sure your local rsync command uses the restricted, passphraseless key under all circumstances? When I’m actually logged in myself, often ssh-agent is keeping my less-restricted key available. The problem with this is that ssh will prefer using that key, but when I use that, my fancy hotcopy (from the previous section) never gets called.
To fix this, my backup script on the client contains an extra -e option to rsync, which is self-explanatory, but that’s not enough: ssh still prefers the key held by ssh-agent. The full solution (as the ssh-agent manpage more or less documents) is thus:
#! /bin/bash unset SSH_AUTH_SOCK rsync -avz -e 'ssh -i .ssh/restricted_key' remote_host:/var/backup/ /var/remote_host_backup
Sometime soon I might respin this whole thing with rdiff-backup (…you want to keep multiple states of your backup, because, well, that’s rule #4 :P). I just need to figure out how client-server communication works for that.