DIASER - Documentation

This short HOWTO will help you install DIASER beta2

rpm, Makefile or deb installation

rpm dependencies will be automatically installed with yum.

Makefile as root will allow installation; make, make install.

deb package still requires extra dependencies, see section 5.

System Requirements

Note: if installed using the Makefile or using the rpm then the command to run is ]$diaser not ]$./diaser from V1b2.

1) Perl v5.8.8 or above, (Perl v5.10.0 is recommended for best performance) enabled workstation. Optional for --bandwidth, gnuplot v4.2.

2) 3 x Linux, currently tested with RHEL, Fedora 12, Ubuntu Server 9.10 and LTS 8.04.3, storage nodes (you can use VM's) with root access for initial setup and Perl v5.8.8 or above with services; sshd, crontab, iptables ssh port open, ntpd, rsync (non daemon).

3) LAN or WAN connection between each server, the 3 machines must be able to ping one another.

4) SSH server on each node, not necessarily port 22.

5) Perl modules

Workstation: Net::SSH::Perl, Net::SFTP, Getopt::Long, AppConfig, Term::ReadKey and Data::Password.

i.e. as root ]#yum -y install perl-Net-SSH-Perl or cpan>install Net::SSH::Perl

Nodes: Perl Core and File::Find (installed as default with most distributions).

rpm will attempt to install Perl module dependencies automatically.

IMPORTANT Notes for: Debian 5, Ubuntu Server 9.10 and 8.04.3 LTS

Use of Ubuntu nodes is either or at the moment, i.e. Ubuntu or say Fedora. I plan to allow mixed nodes later.

First only use the installer with Ubuntu Server, otherwise LTS will need a Perl Upgrade to v5.10.0. You can use LTS for nodes and proceed to set them up as storage.

The installation of Net::SFTP and Net::SSH::Perl is not as straightforward as hoped, this may change in future releases, for now these instructions do work, borrowed from http://www.csclug.org/?q=node/128

a. Install the following with synaptic/apt-get: libyaml-dev libgmp3-dev libpari-dev libmath-gmp-perl libmath-bigint-gmp-perl

b. Manually install pari from
ftp://megrez.math.u-bordeaux.fr/pub/pari/unix/ I'm using pari-2.3.4.tar.gz

c. Install using cpan: YAML

d. Update cpan by typing: install CPAN 5. Re-Initialize cpan by typing:
reload CPAN 6.

e. Grab Math::Pari from cpan.perl.org and manually install

f. Install using cpan: Math::GMP Math::BigInt Math::BigInt::FastCalc Math::BigRat Net::SSH::Perl Net::SFTP

6) Enough disk space. here is a simple rough guide. Use the --bandwidth option to find out your available bandwidth between nodes.

    Bandwidth and capacity lookup table
    ===================================
    BW      Hours GB (Decimal)
    Mbit/s  1    2   3    4    5     6   
    1       0.45 0.9 1.35 1.8  2.25  2.7 
    10      4.5  9   13.5 18   22.5  27  
    100     45   90  135  180  225   270 
    1000    450  900 1350 1800 2250  2700

    Disk space lookup table
    =======================
    BW      Month   1xYr    2xYr
    Mbit/s
    1       20GiB   240GiB  480GiB
    10      67GiB   804GiB  9.6TiB
    100     542GiB  6.5TiB  78TiB
    1000    5.2TiB  62.4TiB 748.8TiB

assuming best transfer rates at full capacity, 2 x phase operation @ 3 hours each, average Differential est. 500M, a prune function will allow clipping of the volumes to free up space.

Configuration

For a question driven configuration tool run the command:

]$diaser --configure

or

edit the file diaser.conf contains user settings, please edit as required.
Main settings:

number of years you want the system to run for
time of daily operation
node IP addresses
SSH ports
dry run or not
time out and maximum transfer rate
home account directories
fill DIASER settings
time zone compensation
sudo accounts or not
backup volume names and location

The minimum recommended length of deployment is 2 years.

Start Installation

Run ./diaser --install or diaser --install as an ordinary user, node root passwords will be requested and make a note of your chosen diaser password.

Other options can be viewed with the diaser --help switch.

Upgrades

Product upgrades will apply fixes and updates, use the --upgrade switch.

Data Entry point / filling DIASER

This machine contains directory d0 and FULL01 and is your machine A. Machine A is assumed to be your most robust and closest from a network perspective to your main backup facility. Volumes on your file store are assumed to be read/write by user id: $your_diaser_uid. Fills Full01 for your FULL copy and d$ for all subsequent Differentials or constant volumes. The script fill_diaser automates the loading of DIASER using your chosen settings created by the --configure, --modify or when you edit diaser.conf.

See the fill DIASER flow chart for a detailed overview, Diagrams to the left.

I have included an example of my Bacula volume creation in helper_scripts/bacula-dir.conf.extract.

Data Retrieval

Either use the simple tool provided using the --retrieve option, which also has additional command line options or login to nodes directly and use scp. The retrieval tool will walk you through a set of questions then list files for you to pick and transfer. The file will retain it's name and be located in the diaser_rel directory.

SE Linux and AppArmor

No problems observed during either installation or operation.

Need more than one installation per machine?

Share disk space with other organisations or groups by using a different account name and staggering or alternating the transfer times (phases) or lowering the LMB - lowest maximum bandwidth between nodes. See diaser.conf. diaser will allow the use of more than one configuration file and you can force diaser to read a configuration file.