Backing-Up Your Data With Raspberry Pi

Among information technology bad practices not backing-up your data is the one you are going to regret more as something goes wrong. For too many years I’ve been relying solely on the good health of my disk drives, that means I was relying mostly on good luck, and on manually copying data, mostly photos and videos, on different supports. I decided so to buy myself a bigger disk drive with the sole purpose of keeping a backup copy of my data.

Hardware set-up

First things first, I bought an external 2TB USB disk. The disk size is enough to backup my 1TB NAS disk and other data I actually keep on the Raspberry Pi server or the desktop PC. More importantly the disk is externally powered since the Raspberry couldn’t power it by itself.

I formatted the , with the XDS file-system, by using the tool provided with my NAS disk, after attaching the new disk to the NAS spare USB port. This step is not strictly necessary but it will allow me, in future, to share the backup disk just by connecting it to the NAS USB port.

I then connected the backup disk to the Raspberry Pi 3 and configured the /etc/fstab file in order to mount it on start-up

/dev/sdb1 /media/backup xfs rw,defaults 0 0

Software set-up: rdiff-backup

While I was looking for a backing-up software solutions I was mostly thinking to some directory mirroring tool like RSync but, like often happens, I stumbled in a more complete solution when I did read this page describing a backup system based on Raspberry Pi and rdiff-backup.

Rdiff-backup is a command line based backup tool, it provide all basic backing-up features like differential backups or time based restore. More importantly, to me, it’s based on standards tools like Rsync directory mirroring and tar archives, meaning that archives are easily readable without the need of rdiff-backup itself.

I installed rdiff-backup with apt-get tool:

sudo apt-get install rdiff-backup

Then created some target folder in the backup drive

sudo mkdir /media/backup/public

sudo mkdir /media/backup/nas

sudo chown pi:pi /media/backup/public/

sudo chown pi:pi /media/backup/nas/

Then I manually started the initial backup, knowing the operation was going to take some time I started a independent shell session with the screen command.

screen

rdiff-backup --exclude /media/public/BTdownload /media/public/ /media/backup/public/

The backup command just needs source and destination folders path, many optional switches are available like the one I used, “--exclude”, which exclude from the backup process some sub-folder.

Once the initial backup completed successfully I used the crontab command to schedule the backup command.

crontab -e

Since most of backed-up data are photos and videos I produce during week-ends I scheduled it weekly ad Tuesday (Monday is already SD-card backup day)

0 5 * * 2 rdiff-backup --exclude /media/public/BTdownload /media/public/ /media/backup/public/

Some notes on performances

The initial backup of my photos and videos collection, about 300 GB, took most of the day to complete. The poor performance is mostly because I keep my data on a Samba share. Rdiff-backup documentation discourages backing up networked disks through in favor of the better optimized SSH protocol, unfortunately my NAS disk doesn't support this option. Not a big problem in my case anyway, because future differential backups will need less time, of course, and the backup process will be performed automatically between two always-on devices.

Backing-up the Raspberry Pi 3

Recently I had to repeat installation of my Raspberry Pi 3 server. Probably because of some SD card corruption problem I started experiencing unexpected loss of active services, first GitLab then MiniDLNA, a and failures while trying reinstalling or reconfiguring them. I so downloaded latest Raspbian image and went with a complete installation. Not a big deal, since I’ve been following my own instructions on this blog but still a time consuming process.

Once the Raspberry server was operative again I started looking for a simple backup solution in the case something broke again by itself or, not unlikely, I broke something by myself.

I quickly found in the Internet this discussion page where, among other solutions, it was proposed a handy shell script to completely backup Raspberry SD Card. Following discussion links I landed on this GitHub page where the same script is available in its latest version.

Installation

Installing the script is just matter of downloading and unzipping it or, as alternative, cloning it with git command. I did choose the latter since I’m going to need git in future.

sudo apt-get install gitmkdir scriptcd script/git clone https://github.com/aweijnitz/pi_backup.git

configuring the script

The PI backup script is a fine example of shell programming, it needs only a couple of arrangements to fit the system where is installed.

First functions stopServices() and startServices must be edited by uncommenting commands to stop and start services running on the Raspberry. I added command to stop and start MiniDLNA service:

...sudo service minidlna stop...sudo service minidlna start...

Then I edited the backup path. In the same scrip section it’s possible to set the number of old backups to keep and if backed-up images must be compressed.

...# Setting up directoriesSUBDIR=backupMOUNTPOINT=/media/usbdiskDIR=$MOUNTPOINT/$SUBDIRRETENTIONPERIOD=1 # days to keep old backupsPOSTPROCESS=0 # 1 to use a postProcessSucess function after successfull backupGZIP=0 # whether to gzip the backup or not...

GitLab on the Raspberry Pi 3

Software version control systems (VCS) are among essential (almost life-saving) tools when programming in team. Also while working by themselves they can reveal a big deal useful. I had Mercurial and Mercurial-server installed on my desktop computer time ago and used them to backup and synchronize my programming experiments between the netbook and the desktop computer.

Since I installed Mercurial I went trough a couple of desktop full-reinstall, of course, I also had to reinstall and reconfigure the version control system tools. So, when I bought the Raspberry Pi 3, I did put using it as VCS server on top of my personal wish-list.

Why Git? Why GitLab?

I must say I have no complaints against Mercurial, it always worked without any problem, but I’ve become a GitHub user so, passing to Git also at home seemed me the natural thing to do. At the beginning I was thinking about a plain bare-bone Git installation, without any graphical user interface, just like it was for Mercurial-server. While I was looking for a suitable git-on-raspberry how-to page I literally stumbled on this GitLab Installation how-to.

GitLab is a Git server full featured with web interface, projects and users management. GitLab is Open-source, or at least exists a community version, but what triggered my decision is that it’s quite easy to install. Installing GitLab is not harder than installing a bare-bone Git server, so … here I am.

Installing GitLab

Installing GitLab has been a simple four-step process, at the end of installation process GitLab was ready-and-running without any need of configuration.

First I installed some required dependencies. Because of previous installations on my Raspberry the only one I had to install has been Postfix mail server.

sudo apt install curl openssh-server ca-certificates postfix apt-transport-https

Then I added GitLab repositories and keys to Raspbian sources:

curl https://packages.gitlab.com/gpg.key | sudo apt-key add -
sudo curl -o /etc/apt/sources.list.d/gitlab_ce.list "https://packages.gitlab.com/install/repositories/gitlab/raspberry-pi2/config_file.list?os=debian&dist=jessie" && sudo apt-get update

I then installed GitLab with a simple apt-get command

sudo apt-get install gitlab-ce

At last I executed the GitLab reconfiguration command in order to make configuration effective.

sudo gitlab-ctl reconfigure

I edited GitLab configuration file (located in /etc/gitlab/gitlab.rb) only to change two configuration details: I placed GitLab repository folder on Raspberry external USB disk

git_data_dirs({"default" => "/media/usbdisk/gitlab"})

And I set GitLab web server in order to work on a HTTP port different from default one.

external_url 'http://raspberrypi3:8887'

After any change of GitLab configuration file the reconfigure command must be executed in order to see them effective. In spite of changing GitLab web port SFPG picture gallery I had installed on the Raspberry stopped working, I’ll have to solve this in the near future.

First login

On the very first access to GitLab web page user is asked to change administrator password

Setting-up the Raspberry PI 3 as a home server

It has been some time since last time I wrote, unfortunately my job stole most of my limited free time keeping me from experimenting and so posting about it. At last I managed to collect enough free time to complete Raspberry PI 3 installation and replace the older model I was using as home server.

Just like before? Not exactly

When I started configuring and installing the Raspberry PI 3 I was hoping it would have been a simple repetition of operation I already performed on the older Raspberry. Most of it has been that way but in some cases I experienced some relevant difference.

Configuring a static IP address

I've been configuring static IP address on Linux since I installed my first Pentium III based Linux, so I really didn't expect any problem here. Once I configured “/etc/networking/interfaces file” I instead noticed the Raspberry was visible on the network with two different addresses. After some searching on the Internet I discovered it's because of a different way DACP client works on latest Raspbian release. The problem can be solved in two ways: first is configuring DHCP client in order to let it set a static address on network interface alternatively it's possible disable DHCP client for one or more network interface. I chose the latter, at the end, since the Raspberry is going to be a DHCP server so there is no deal in keeping DHCP client active. I disabled DHCP client from assigning address to both wired and wireless network interface by adding the following line to “/etc/dhcpcd.conf” configuration file.

denyinterfaces wlan0 eth0

DHCP server and wireless access point

Like I did on my first Raspberry server I configured the new one to work as DHCP server and a wireless access point. It seem there are no relevant changes since first time I did it, I simply had to follow my own instructions.

Installing applications

Installing apt-ger based applications like LAMP sever and MiniDLNA server has been a quite easy task, while to install other applications like RPI-Monitor I had to look for the updated download link on the Internet. I also installed the updated version of SFPG picture gallery, it works but picures thumbnails don't show. I'll look for a solution later.

Power supply and cables

Once I had the Raspberry PI 3 installed and configured on my desktop I pit it in place of the old one and … nothing was working. It took me some time of testing and pinging before I noticed the board power led wasn't properly lighted-up. The phone/tablet charger I used to supply the old Raspberry wasn't up to the Raspberry PI 3 power requirement. I replaced the power supply with the one I used for desktop test but it wasn't enough to make the Raspberry PI 3 working, I had to replace micro-USB power cable with a shorter one in order to have it working, I think it's better I'll buy a dedicated Raspberry PI 3 power supply soon.

New toy on the desk: Raspberry PI 3

Just before Christmas I've been to a “traditional” electronics and surplus fair, here in Genoa, and bought myself, among other things, a Raspberry PI 3 (version B) board. My goal is to replace, as home server, the Raspberry PI I bought two years ago in order to take advantage of the more computing power offered by the new board. Additionally the older board will be set free for more “experimental” experiments. As “accessories” to my new Raspberry board I bought a (clear) plastic case, a 16GB micro SD card and en external 2.5'' USB hard disk.

Installation and first tests

I first downloaded latest Raspbian release, the “Lite” version since I'm going to use it as a headless server. Like I did last time I copied the disk image on the 16GB SD card using the “dd” command.

sudo dd if=2016-11-25-raspbian-jessie-lite.img of=/dev/sdd

All worked fine but I had to fix a couple of things. The image I copied at the first tentative didn't boot, I had to remove all memory card partitions using Gparted then repeat the copy process. The disk copied after the second tentative works fine, I don't know if problem was because a failure in the first copy or because of how the card was pre-formatted.

The latest Raspbian release has SSH demon disabled by default to enable it I had just to add an empty “ssh” named file on the memory card root folder.

touch /media/maxx/boot/ssh