1. Community Tip: Please Give Thanks to Those Sharing Their Knowledge.
    If you receive helpful answer on this forum, please show thanks to the poster by clicking "LIKE" link for the answer that you found helpful.
  2. Community Tip: Forum Rules (PLEASE CLICK HERE TO READ BEFORE POSTING)
    Click the link above to access ADSM.ORG Acceptable Use Policy and forum rules which should be observed when using this website. Violators may be banned from this website. This notice will disappear after you have made at least 3 posts.

Backup Isilon SMB filer

Discussion in 'Backup / Archive Discussion' started by ritchi64, Dec 15, 2014.

  1. ritchi64

    ritchi64 Active Newcomer

    Joined:
    Aug 31, 2011
    Messages:
    7
    Likes Received:
    0
    Hi, since I got no reply on my first attemp, I will try another time :

    We got a new Isilon with 700TB. We have 30 millions of files on it (small filles). We prefer to use Incremental for ever then the NDMP model because we dont want to take a full backup every week!

    Our first try take 3.5h for only scanning 3 millions of files. Too long. We are thinking to buy some SSD to put metadata on it and hope to accelerate the scanning time. Do you guy's have try it with good result????
     
  2.  
  3. marclant

    marclant ADSM.ORG Moderator

    Joined:
    Jun 16, 2006
    Messages:
    2,568
    Likes Received:
    358
    Occupation:
    Accelerated Value Specialist for Spectrum Protect
    Location:
    Canada
    One thing that could help you (if it's not too late), is to make more volumes. That way, by increasing the resourceutil, you can process volumes in parallel, cutting down the time.
     
  4. ritchi64

    ritchi64 Active Newcomer

    Joined:
    Aug 31, 2011
    Messages:
    7
    Likes Received:
    0
    Well it's for shure a way we go but it will be 10 times 3 million of static files. Did someone out there have try to put SSD for metadata in the Isilon cluster to make scanning faster (will it work ???).
     
  5. DazRaz

    DazRaz ADSM.ORG Senior Member

    Joined:
    Jun 15, 2008
    Messages:
    148
    Likes Received:
    14
    Do you have the latest firmware on the Isilon? I believe there is a bug with an old level that causes TSM to see all the data as new files.
     
  6. ritchi64

    ritchi64 Active Newcomer

    Joined:
    Aug 31, 2011
    Messages:
    7
    Likes Received:
    0
    Yes, we are at the latest Onefs 7.2. it backup full the first time and incremental after that But the scanning time is long and very slow. By the way, I backup true a SMB share directly on the TSM server (eth 10gb). I try from a storage agent but the scan was terrebly slower.

    Does TSM as to verify every file in it's database to be shure the file as not to be backup???

    It's almost no network trafic between the tsm server and the Isilon!
     
  7. marclant

    marclant ADSM.ORG Moderator

    Joined:
    Jun 16, 2006
    Messages:
    2,568
    Likes Received:
    358
    Occupation:
    Accelerated Value Specialist for Spectrum Protect
    Location:
    Canada
    Yes and no. Not every file in the database, only the active ones, it does not look at inactive/previous versions.

    At a high level, here's what happen:
    - TSM Client queries the server to get the list of objects previously backed up
    - Using that list, the client compares each objects on the list to the filesystem
    - While the comparison is in progress, if files are identified to be backed up, another thread starts sending those to the server, while the first thread continues to compare

    When doing the compare:
    - Files that are the same are skipped
    - Files that are changed are backed up
    - Files that no longer exist on filesystem, but on list are expired (marked inactive)
    - Files not on the list, but on filesystem are backed up

    So, the faster the disks, the faster the scan of the filesystem can be done.

    It's sure that the scan is a big part of it with several million files. If you really want to see where the time is being spent, add this line in your client option file:
    Code:
    testflag instrument:detail
    Run a manual backup or restart the scheduler and let a schedule backup complete. Remove the testflag from the option file, restart the scheduler again.

    In the baclient directory, look for a file named dsminstr.report.p###, the number will always be different because it corresponds to the PID of that run. You can use this presentation to have the explanation of every type of activity in the report: https://www-304.ibm.com/support/docview.wss?uid=swg27021050

    With that report, you will be able to see how long each part of the backup takes. "Process Dirs" is the scanning of the disk.
     
  8. ritchi64

    ritchi64 Active Newcomer

    Joined:
    Aug 31, 2011
    Messages:
    7
    Likes Received:
    0
    Thank you very much for your reply. We will surely try that.

    By the time, We buy 8 SSD drives for metadata and see almos no improuvement. After that, we try to backup without the security option and it goes 2 times faster (1.5h).

    Why?

    *Remember, there just old files (3 millions) and nothing to backup, only scan time.
     
  9. marclant

    marclant ADSM.ORG Moderator

    Joined:
    Jun 16, 2006
    Messages:
    2,568
    Likes Received:
    358
    Occupation:
    Accelerated Value Specialist for Spectrum Protect
    Location:
    Canada
    The client compares a series of things to determine if a file changed, some of which are:
    - size
    - modified date
    - attributes
    - permissions

    So if you skip permissions, it's less work for the client during the scan time.

    You could do an incremental-by-date during the week, which is faster because it only checks the date of the old backup compared to the files, and do a regular incremental on the weekend to cover for the shortfalls of incremental-by-date. More info on incremental by date here: http://www-01.ibm.com/support/knowl...com.ibm.itsm.client.doc/c_bac_incrbydate.html
     
  10. ritchi64

    ritchi64 Active Newcomer

    Joined:
    Aug 31, 2011
    Messages:
    7
    Likes Received:
    0
    Thank you for your fast reply,
    I'll try your suggestion and repost the result soon!
     
  11. hgoetz

    hgoetz Newcomer

    Joined:
    Sep 16, 2002
    Messages:
    3
    Likes Received:
    0
    hello,

    mabye we have a solution for you.
    there is a trial version available at the website from our partner concat (called dsmISI MAGS) www(dot)concat(dot)de/leistungen/it-infrastrukturen/backup_disaster_recovery/tsm-isilon/

    more information www(dot)general-storage(dot)com

    best regards
     
  12. mikeymac

    mikeymac ADSM.ORG Moderator

    Joined:
    Jun 20, 2003
    Messages:
    928
    Likes Received:
    28
    Location:
    Syracuse, NY

    How are you accessing the files? Did you put some servers in between the TSM server and the NAS, and mount up the OneFS volumes as drives on those servers? We're about to embark on this little adventure ourselves. We plan on using a combination of Isilon snapshots, Isilon replication and NDMP to TSM to protect it. One thing we had considered was putting some Windows servers up in front of the Isilon, and mounting the shares as drives and backing it up incrementally with the TSM client. Unfortunately, you can't use TSM journaling with a mounted drive, so the poor client has to scan the filesystem every time a backup runs. Oy!
     
  13. hgoetz

    hgoetz Newcomer

    Joined:
    Sep 16, 2002
    Messages:
    3
    Likes Received:
    0
    Hello mikemac,

    make sure you understand that MAGS is in effect a wrapper for the Windows TSM client, designed to accelerate backups of NTFS and, particularly, CIFS/SMB file systems with many objects. MAGS does not interfere with what TSM does or does not do. It merely starts many TSM clients in parallel to assure faster backups and restores. All other limitations inherited from the standard “dsmc” client and your TSM server environment still apply. For your server, there are a number of crucial settings you may have to adjust in order to fit a MAGS profile. These are in particular maxnummp settings for the TSM node(s) you plan to use, as well as mountlimit settings for the deviceclass (if applicable) and maxsession from your general TSM server settings. There will be a more elaborate users guide for MAGS in the near future covering all these aspects. For now it will suffice to understand that, with MAGS, probably all of those settings will have to be significantly
    increased.

    You can use also your snapshots for backup. A quick start guide with detailed information is available in the trial version.

    If you are interested in more detail, I can organize a Webex or a call with our developer team or you download and test the free trial version.

    One customer example: (only example!)
    Isilon production 56 milion files
    normal TSM client 32 hours
    MAGS (default settings) 6 hours
    MAGS (customized settings) 4 hours

    Regards
    Holger
     
  14. mikeymac

    mikeymac ADSM.ORG Moderator

    Joined:
    Jun 20, 2003
    Messages:
    928
    Likes Received:
    28
    Location:
    Syracuse, NY
    Hi, Holger. My reply was to ritchi64. I'm aware of Mags, too. (Deutsche Bank, etc.) I was just curious to know how Ritchi64 was skinning this cat. Thanks!
     
  15. ritchi64

    ritchi64 Active Newcomer

    Joined:
    Aug 31, 2011
    Messages:
    7
    Likes Received:
    0
    Hi mikeymac,
    I looked at MAGS and probably call them for pricing. On the other hand, I begin to look at some Isilon snapshot command (isi_stf_test , isi_change_list ) to get a list of file to backup during the week and do a normal incremental on weekend.

    To Holger: Did MAGS will try to implement a kind of "snapdiff" with the Isilon snapshot API ? That could remove the scanning time problem and the load on the Isilon cluster.
     
  16. gvoss

    gvoss Active Newcomer

    Joined:
    Aug 3, 2011
    Messages:
    5
    Likes Received:
    0
    Hello everyone,

    I am looking for information to eliminate the Spectrum Protect backup-archive client scan time of the ISILON using the changedlist. One of the challenges is having multiple daily snapshots - hence a changedlist between each. Has anyone successfully scripted this? The flowchart of the process is simple, however the parsing and concatenation of multiple daily changelist files is a challenge.

    copy the ISILON changedlist file using the API
    parse the file and format as an parameter passed to the dsmc -selective $changedlist

    Thank you,

    Greg Voss
     
  17. mikeymac

    mikeymac ADSM.ORG Moderator

    Joined:
    Jun 20, 2003
    Messages:
    928
    Likes Received:
    28
    Location:
    Syracuse, NY
    Hi, Greg!

    I like your idea! As an update on my first few posts, we opted to go with another backup/recovery solution that leverages the Isilon change API.

    Good luck!
     
  18. gvoss

    gvoss Active Newcomer

    Joined:
    Aug 3, 2011
    Messages:
    5
    Likes Received:
    0
    Mike, I am interested to know more. Who can I contact for further information? Can you share the webpage?

    Thank you.

    Greg Voss
     
  19. mikeymac

    mikeymac ADSM.ORG Moderator

    Joined:
    Jun 20, 2003
    Messages:
    928
    Likes Received:
    28
    Location:
    Syracuse, NY
    Hi, Greg!

    Many of the major backup/recovery vendors support the Isilon change API:

    Commvault
    EMC Networker
    NetBackup

    We went with Commvault.

    Hope this helps!
     
  20. gvoss

    gvoss Active Newcomer

    Joined:
    Aug 3, 2011
    Messages:
    5
    Likes Received:
    0
     
  21. gvoss

    gvoss Active Newcomer

    Joined:
    Aug 3, 2011
    Messages:
    5
    Likes Received:
    0
    Thank you for the response. I will investigate...
     

Share This Page