BackupPC-users

[BackupPC-users] Bandwidth Consumption

2009-07-08 22:35:39
Subject: [BackupPC-users] Bandwidth Consumption
From: Adam Goryachev <mailinglists AT websitemanagers.com DOT au>
To: "General list for user discussion, questions and support" <backuppc-users AT lists.sourceforge DOT net>
Date: Thu, 09 Jul 2009 12:30:44 +1000
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

Just wanted to supply some information on something I setup yesterday in
a spare couple of hours (yeah, like any hours are spare :)

I wanted to determine the amount of bandwidth was being consumed by each
host I backup because sometimes I get close to exceeding the monthly
quota, and also it is a good indication that something is wrong if one
host is doing 5 GB a day (like trying to backup a folder of movies
someone dumped onto their PC yesterday)...

So, here it is:
1) Download and install http://www.intevation.de/iam
It seems to be very much un-maintained/improved on since initial release
many years ago, somewhat in german (I presume).
2) Setup your iptables so that you have an input and output rule
pointing to a chain for each individual entry you want in your reports:

new_chain client1
iptables -A INPUT -p tcp --sport 22 -i eth0 -s backuphost1 -j client1
iptables -A OUTPUT -p tcp --dport 22 -o eth0 -d backuphost1 -j client1

You can have more than one host in the same report row like this:
new_chain client1
iptables -A INPUT -p tcp --sport 22 -i eth0 -s backuphost1 -j client1
iptables -A OUTPUT -p tcp --dport 22 -o eth0 -d backuphost1 -j client1

iptables -A INPUT -p tcp --sport 22 -i eth0 -s backuphost2 -j client1
iptables -A OUTPUT -p tcp --dport 22 -o eth0 -d backuphost2 -j client1
Repeating the last two lines for as many hosts as needed.

You might need different rules to catch the traffic you are interested
in, eg, for rsyncd change the port entry, etc...

3) Follow the rest of the instructions included in the file, and you are
done.

With a couple of tweaks to iam_report:
#!/bin/sh
IAM=/usr/local/iam/iam
DUMP=/var/log/backuppc/ipacct-data
WWWDIR=/var/www/backuppc-accounting

YM_CURRENT=`date '+%Y-%m'`
YM_LAST=`date --date='1 month ago' '+%Y-%m'`
TODAY=`date '+%Y-%m-%d'`
YESTERDAY=`date --date='1 day ago' '+%Y-%m-%d'`

$IAM -f $YM_CURRENT-00 -t $YM_CURRENT-99 -w $WWWDIR/$YM_CURRENT.html $DUMP
$IAM -f $YM_LAST-00 -t $YM_LAST-99 -w $WWWDIR/$YM_LAST.html $DUMP
$IAM -f $TODAY -t $TODAY -w $WWWDIR/$TODAY.html $DUMP
$IAM -f $YESTERDAY -t $YESTERDAY -w $WWWDIR/$YESTERDAY.html $DUMP

ln -sf $WWWDIR/$YM_CURRENT.html $WWWDIR/00-current.html
ln -sf $WWWDIR/$YM_LAST.html $WWWDIR/00-last.html
ln -sf $WWWDIR/$TODAY.html $WWWDIR/00-today.html
ln -sf $WWWDIR/$YESTERDAY.html $WWWDIR/00-yesterday.html

I can now see stats for today, yesterday, this month, and last month,
and of course, all of the daily/monthly history.
Reports look like this:
                   IP traffic (2009-07-08..2009-07-09):
                   ====================================
| traffic   | cost   | description                                     |
+-----------+--------+---------------------------------------------------+
|   28.9 MB |        | filesrv (local Filesrv)                         |
|    1.0 MB |        | flail (local Flail)                             |
+-----------+--------+---------------------------------------------------+
|   29.9 MB |        | WebMan Local Hosts                              |
+-----------+--------+---------------------------------------------------+
|           |        |                                                 |
|  226.6 MB | $ 2.27 | castle (Castle)                                 |
|   76.1 MB | $ 0.76 | forex (Windows Forex)                           |
|   27.8 MB | $ 0.28 | shared (Windows Shared)                         |
+-----------+--------+---------------------------------------------------+
|  330.5 MB | $ 3.31 | WebMan Remote Hosts                             |
+-----------+--------+---------------------------------------------------+

So you can group hosts and see the total for each group, assign a
different cost / MB to each group, etc...

I have no association with IAM, it just showed up on a google search,
seemed to be the easiest to install, and pretty much just worked out of
the box. (PS, I needed to delete line 5 from each of the .py files
because it had a non standard character in it which python complained
about).

For someone who knows/uses python (I don't) they could fairly easily
munge this into putting data into a mysql DB, or generate different
reports, etc quite easily... or of course, you could just script it from
nothing.

The main drawback to this is that you pretty much require your hosts to
have a static IP, or can define some static iptables rule which will
count the traffic for you.

I hope this is useful/helpful to someone else.

If you have some problems setting it up with backuppc, let me know and
I'll try and offer some advise...

Regards,
Adam

- --
Adam Goryachev
Website Managers
www.websitemanagers.com.au
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkpVVlQACgkQGyoxogrTyiVa4ACePBREItUdC5g2g7PovO5FUwKc
jiEAoM5ELp6dJSPlIm/NPeypgwGefm6F
=QTFs
-----END PGP SIGNATURE-----

------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

<Prev in Thread] Current Thread [Next in Thread>
  • [BackupPC-users] Bandwidth Consumption, Adam Goryachev <=