SVN Part 2

Now that we can log in automatically (SVN Part 1), we can do some cool stuff. This summer I managed a SVN server for a company that has multiple programs, each of which had different developers working on them. Each of these programs is stored as a repository in SVN. For example, developer A needs access to repository 1 and 2 and developer B needs access to repository 1 and 3, etc. To make it even trickier, in repository 1, developer A should have access to the entire repository, while developer B should only have access to the directory devB in repository 1. Also, every time a developer makes a change to a file, the change needs to be logged with their username.

This is where the RSA keys come in. Each repository should be owned by a different user. For example, /usr/bin/svnserver/repository1 is owned by user repo1, /usr/bin/svnserver/repository2 by repo2, etc. Put any developer who needs access to [repo]'s RSA pubkey in the /home/[repo]/.ssh/authorized_keys file, where [repo] is the name of the repository's owner (repo1, repo2). Configure authorized_keys like:

command="/usr/bin/svnserve -t -r /var/svn/ --tunnel-user=[developer's username]",no-port-forwarding,no-pty,no-agent-forwarding,no-X11-forwarding ssh-rsa [developer's RSA pubkey]== [developer's username]

This enables developers to check out [repository] using: svn+ssh://[repo]@[server]/[repository]. While we are sending the information over SSH, the developer does not have a SSH account and can do nothing but use SVN to check out (svn co) and update (svn ci) repositories owned by a user (repo1, repo2) whose authorized_keys file contains the developer's pubkey. The last column in the authorized_keys file is the comment line. SVN automatically uses this line as the comment when a developer commits code to the SVN repository.

The last part, giving only partial access to a repository, coming in Part 3.

SVN Part 1

Subversion is another awesome piece of software for keeping files in sync across multiple computers. Not quite suited for backing up pictures or anything like that -- but hey, its great for big coding projects! Usually with big projects, there are different levels of access. For example, Bob's algorithm may be super secret and only he and Bill should be able to see the code for it. But, Barry may be logging in to the same SVN server and need access to other code. There's an app for that. I mean, there's a feature for that. Also, perhaps not everyone should have access to the server's file, or even an account on the server. Well, there's a feature for that, too. Anyway, here's the procedure/setup I came up with after lots of research/trial-and-(hopefully not catastrophic) error.

Make sure at least YOU can ssh into the server (or better yet, be sitting in front of it and skip the next few steps). Its handy to use an RSA key so you can log in automatically and don't have to type your password every time you want to execute a subversion command. So:

sh-keygen -t rsa

will generate you a key. Then:

rsync -a ~/.ssh/id_rsa.pub server:~/.ssh/

Before we switch over to the server, if you're using a non-standard port, in /etc/ssh/ssh_config, add:

Host [server]
Port [port #]
Host * [this line should be there already, it just tells you where to put the 2 preceeding lines]

Next, ssh over to the server and:

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
rm ~/.ssh/id_rsa.pub

The >> instead of just > puts the key at the END of the file instead of overwriting, so if you already have a computer set up in your authorized_keys file, this will preserve that setup.

Now you should be able to log in with no password using ssh [server].

Part 2 coming soon! :P

APGT Linux Commands, Editors, and Shell Programming

The rsync scipts will be in the next edition of A Practical Guide to Linux Commands, Editors, and Shell Programming! The author and I go way back... ;)

Anyway, its (again) a handy utility. My roommate used to play all 90's music (and not the good stuff) on his Mac. The rsync utility comes default in Mac OS X (not to say that it didn't before...) so it wasn't too hard to write a script to sync up his iTunes with my music database:


rsync \
-av \
--compress \
--port=[redacted for security :P] \
--stats \
--exclude=Movies \
--exclude=Podcasts \
carter@192.168.1.110:'/media/disk/My\ Music/iTunes/iTunes\ Music/' \
'/Volumes/External Space/Music/Music_from_Max/'


So there -- taking the music from my disk (its only in iTunes so when I boot Vista it syncs up with my iPhone... Vista? iPhone? now I'm embarrassed...) and putting it on his disk. There are a few issues we ran in to. First of all, his username is capitalized on the Mac -- Carter -- and lowercase on my computer. So thats why, even though its almost the same user, we have carter@192... . Also, the port=### option doesn't seem to work on rsync on the Mac. I run ssh on a non-standard port, but even with the --port= option it kept defaulting to 22. So, we edited /etc/ssh/ssh_config and put my port in there. Also, we're doing this with rsa keys so the script could run on a crontab! Then, every time I download music from this millenium, he would get it.

Finally. No more Red Hot Chili Peppers.

Backup software

I keep hearing people say: "man, I really should back up my computer" with this look of dread on their faces. Its really not so bad. Here are a few ways, ranging from the absolute easiest to the most nerdly-satisfying :) Oh, and they're all free (at least for the most basic option)

1. Dropbox: for the novice who needs to back up some documents, a couple pictures maybe, or even some music. All in all, less than 2 GB ("500 songs", in Apple-speak). 2 GB is free. If you're willing to pay, you can back up your whole system with Dropbox -- 50 GB is $100/yr and 100 GB is $200/yr. The best part about Dropbox is that its got version control (covered in "subversion" later on) so its great for programming projects or group projects -- its also got a "share folder" feature so everyone can be working from the same files. ALSO: Dropbox works for Mac, Linux and Windows. Awesome! My favorite new piece of software. The downside (or upside, depending) is that its all got to be in one folder called "Dropbox" on Linux or "My Dropbox" on Windows. I put mine on my desktop for easy access. If you want to just sync your whole desktop or documents across a couple computers (Windows only), try www.sugarsync.com. They now offer (because of competition from Dropbox) a free 2 GB option.

2. Picasa web: Backing up pictures can be very space-consuming. If you use Picasa (now for Mac, Windows and Linux) and have a google account, you can back up to the google servers. They give you 1 GB for free (pictures only) which will hold a lot of pictures at medium resolution. Now, you could just use 1 GB of your Dropbox space, but most cameras now take pictures that are really high resolution (and take up more space on your disk) -- Picasa reduces the size of your photos before it uploads them, so its a simple way to make sure you still have a decent quality copy of your pictures even if you disk fails.

These are the 2 most basic options that cover most things. However, some people are concerned with privacy. These sites keep things private (Dropbox is password protected unless you "share" a folder with someone, and Picasa has 2 options to keep things private -- "unlisted", which means the link isn't public, or "sign in required to view" which means even if you have the link, only people you explicitly share it with can see your pictures).

Some people are even more concerned with privacy and are resolutely "anti-cloud". Cloud computers are, generally, computers that you can't see (because you don't know where they are) that have your data on them. So, webmail (gmail, hotmail, yahoo, etc.) is a cloud-based service. There are some people (you know who you are) who don't use cloud-based services. This makes life very difficult. So, for these people, there are a few other options. I've gotta write about these later because I have to finish an rsync script first ;)

rsync

The rsync utility is very cool. Its cool because its fast, effective and the syntax for the command is relatively simple. Its a great way for the uber-paranoid to avoid the Cloud. [Side note: if you are not uber-paranoid and do not mind the Cloud, check out Dropbox <-- shameless referral link. But you get extra space with that link, too! 2 GB -> 2.25 GB]

You can get rsync to do automatic, incremental backups for you, although I'm still working on that part. The best I have so far is to sync 1 or more computers with your server computer (I sync my computers at home using my dad's computer in San Francisco [static IP] as the server computer).

To do it (assuming the server is running an rsync deamon [more on that here], and your server name is in the /etc/hosts file), each computer (except for the server computer) needs 2 identical scripts. Mine are:


rsync \
--verbose \
--archive \
--compress \
--update \
coffee:~max/test1 ~/rsync/


and


rsync \
--verbose \
--archive \
--compress \
--update \
~/rsync/test1 coffee:~max/

This keeps the folder "test1" in my home directory on coffee (the server computer) and my ~/rsync directory on my home computers. The options are:
--verbose = tell me whats going on as its happening!
--archive = this is a cocktail of options: recurse into the directories, copy symlinks as symlinks and preserve permissions.
--compress = use compression to speed up the transmission, but use up more CPU
--update = don't overwrite newer files on the receiver (server) computer

The rest of the syntax is the same as a copy command. If you use a remote computer, preface its directory structure with the name of the computer followed by ":".

Another very very useful option when testing all this is --dry-run. rsync will go through all the steps it would have taken to make the sync without actually transferring any files. Especially useful if you use the --delete flag, which deletes files on the server computer that are no longer present on the source computer. Because of rsync's trailing slash issues, its easy to delete the contents of an entire directory with one wrong "/".

Here's a simple script if you're just using rsync to backup:

#!/bin/sh
BUNAME=$(date +%A)
rsync \
--verbose \
--archive \
--compress \
--update \
--backup \
--backup-dir=~max/$BUNAME/ \
~/rsync/test1/ coffee:~max/test1/


Because of the --backup and --backup-dir=... options, when rsync is going to change a file, it first makes a copy of the file to the directory specified by $BUNAME (which is defined as the day of the week i.e. Thursday) and then overwrites the file in the main directory (test1, in this case). This way, if you accidentally change a file you didn't mean to, you have backups of it. You could go so far as to make hourly directories within the day-of-the-week directories with some simple shell scripting...