Backup Isilon SMB filer

[ritchi64]

Hi, since I got no reply on my first attemp, I will try another time :

We got a new Isilon with 700TB. We have 30 millions of files on it (small filles). We prefer to use Incremental for ever then the NDMP model because we dont want to take a full backup every week!

Our first try take 3.5h for only scanning 3 millions of files. Too long. We are thinking to buy some SSD to put metadata on it and hope to accelerate the scanning time. Do you guy's have try it with good result????

 

[marclant]

One thing that could help you (if it's not too late), is to make more volumes. That way, by increasing the resourceutil, you can process volumes in parallel, cutting down the time.

 

[ritchi64]

Well it's for shure a way we go but it will be 10 times 3 million of static files. Did someone out there have try to put SSD for metadata in the Isilon cluster to make scanning faster (will it work ???).

 

[DazRaz]

Do you have the latest firmware on the Isilon? I believe there is a bug with an old level that causes TSM to see all the data as new files.

 

[ritchi64]

Yes, we are at the latest Onefs 7.2. it backup full the first time and incremental after that But the scanning time is long and very slow. By the way, I backup true a SMB share directly on the TSM server (eth 10gb). I try from a storage agent but the scan was terrebly slower.

Does TSM as to verify every file in it's database to be shure the file as not to be backup???

It's almost no network trafic between the tsm server and the Isilon!

 

[marclant]

Does TSM as to verify every file in it's database to be shure the file as not to be backup???

Yes and no. Not every file in the database, only the active ones, it does not look at inactive/previous versions.

At a high level, here's what happen:
- TSM Client queries the server to get the list of objects previously backed up
- Using that list, the client compares each objects on the list to the filesystem
- While the comparison is in progress, if files are identified to be backed up, another thread starts sending those to the server, while the first thread continues to compare

When doing the compare:
- Files that are the same are skipped
- Files that are changed are backed up
- Files that no longer exist on filesystem, but on list are expired (marked inactive)
- Files not on the list, but on filesystem are backed up

So, the faster the disks, the faster the scan of the filesystem can be done.

It's sure that the scan is a big part of it with several million files. If you really want to see where the time is being spent, add this line in your client option file:

testflag instrument:detail

Run a manual backup or restart the scheduler and let a schedule backup complete. Remove the testflag from the option file, restart the scheduler again.

In the baclient directory, look for a file named dsminstr.report.p###, the number will always be different because it corresponds to the PID of that run. You can use this presentation to have the explanation of every type of activity in the report: https://www-304.ibm.com/support/docview.wss?uid=swg27021050

With that report, you will be able to see how long each part of the backup takes. "Process Dirs" is the scanning of the disk.

 

[ritchi64]

Thank you very much for your reply. We will surely try that.

By the time, We buy 8 SSD drives for metadata and see almos no improuvement. After that, we try to backup without the security option and it goes 2 times faster (1.5h).

Why?

*Remember, there just old files (3 millions) and nothing to backup, only scan time.

 

[marclant]

The client compares a series of things to determine if a file changed, some of which are:
- size
- modified date
- attributes
- permissions

So if you skip permissions, it's less work for the client during the scan time.

You could do an incremental-by-date during the week, which is faster because it only checks the date of the old backup compared to the files, and do a regular incremental on the weekend to cover for the shortfalls of incremental-by-date. More info on incremental by date here: http://www-01.ibm.com/support/knowl...com.ibm.itsm.client.doc/c_bac_incrbydate.html

 

[ritchi64]

Thank you for your fast reply,
I'll try your suggestion and repost the result soon!

 

[hgoetz]

hello,

mabye we have a solution for you.
there is a trial version available at the website from our partner concat (called dsmISI MAGS) www(dot)concat(dot)de/leistungen/it-infrastrukturen/backup_disaster_recovery/tsm-isilon/

more information www(dot)general-storage(dot)com

best regards

 

[hgoetz]

Hello mikemac,

make sure you understand that MAGS is in effect a wrapper for the Windows TSM client, designed to accelerate backups of NTFS and, particularly, CIFS/SMB file systems with many objects. MAGS does not interfere with what TSM does or does not do. It merely starts many TSM clients in parallel to assure faster backups and restores. All other limitations inherited from the standard “dsmc” client and your TSM server environment still apply. For your server, there are a number of crucial settings you may have to adjust in order to fit a MAGS profile. These are in particular maxnummp settings for the TSM node(s) you plan to use, as well as mountlimit settings for the deviceclass (if applicable) and maxsession from your general TSM server settings. There will be a more elaborate users guide for MAGS in the near future covering all these aspects. For now it will suffice to understand that, with MAGS, probably all of those settings will have to be significantly
increased.

You can use also your snapshots for backup. A quick start guide with detailed information is available in the trial version.

If you are interested in more detail, I can organize a Webex or a call with our developer team or you download and test the free trial version.

One customer example: (only example!)
Isilon production 56 milion files
normal TSM client 32 hours
MAGS (default settings) 6 hours
MAGS (customized settings) 4 hours

Regards
Holger

 

[ritchi64]

Hi mikeymac,
I looked at MAGS and probably call them for pricing. On the other hand, I begin to look at some Isilon snapshot command (isi_stf_test , isi_change_list ) to get a list of file to backup during the week and do a normal incremental on weekend.

To Holger: Did MAGS will try to implement a kind of "snapdiff" with the Isilon snapshot API ? That could remove the scanning time problem and the load on the Isilon cluster.

 

[gvoss]

Hello everyone,

I am looking for information to eliminate the Spectrum Protect backup-archive client scan time of the ISILON using the changedlist. One of the challenges is having multiple daily snapshots - hence a changedlist between each. Has anyone successfully scripted this? The flowchart of the process is simple, however the parsing and concatenation of multiple daily changelist files is a challenge.

copy the ISILON changedlist file using the API
parse the file and format as an parameter passed to the dsmc -selective $changedlist

Thank you,

Greg Voss

 

[gvoss]

Hi, Greg!

I like your idea! As an update on my first few posts, we opted to go with another backup/recovery solution that leverages the Isilon change API.

Good luck!

Mike, I am interested to know more. Who can I contact for further information? Can you share the webpage?

Thank you.

Greg Voss

 

[gvoss]

Hi, Greg!

Many of the major backup/recovery vendors support the Isilon change API:

Commvault
EMC Networker
NetBackup

We went with Commvault.

Hope this helps!

 

[gvoss]

Thank you for the response. I will investigate...

 

[aixfplwiz]

Hi, Greg!

Many of the major backup/recovery vendors support the Isilon change API:

Commvault
EMC Networker
NetBackup

We went with Commvault.

Hope this helps!

Hi Mikeymac,
Of course you still have TSM ? - What version are you guys running with ?
Thanks.

 

[jideck]

Hi, since I got no reply on my first attemp, I will try another time :

We got a new Isilon with 700TB. We have 30 millions of files on it (small filles). We prefer to use Incremental for ever then the NDMP model because we dont want to take a full backup every week!

Our first try take 3.5h for only scanning 3 millions of files. Too long. We are thinking to buy some SSD to put metadata on it and hope to accelerate the scanning time. Do you guy's have try it with good result????

Hi we apply some helping options like (dsm.opt or clopset option) would like network 1G or better 10G to work smooth.

Memoryefficientbackup diskcachemethod (ensure some disk space is available and first run will take longer)
TCPBUFFSIZE 512
RESOURCEUTILIZATION 10 *(change maxnummp for client)
TCPWINDOWSIZE 512
LARGECOMMBuffers Yes
TXNByteLimit 2097152 *change up for many big files
******************************************************************and*******
TESTFLAGS threadstacksize:2048 *to avoid some errors in long paths checked in version 8.1.0

Also I would suggest to split data in 3 separate nodes/FS or run by local script with different complementary path.

 

[marclant]

TESTFLAGS threadstacksize:2048 *to avoid some errors in long paths checked in version 8.1.0

I would recommend to use the latest 8.1.x client instead of the workaround for APAR IT18901 Note, 7.1.x clients not affected by this APAR.

 

[aixfplwiz]

But, of course!

We're at 7.1.7 (soon to be at 7.1.8).

We just stood up an instance of 8.1.1.0 to run parallel with our MUCH older TSM versions. We are working on a method to "cut over" soon, but we have challenges with the big picture.
BTW and side note - does ADSM.org allow direct messages ? I stalked you thru the "interweb" as I was looking for info and experience with Infinidat. Thanks