ADSM-L

Re: 15,000,000 + files on one directory backup

2005-06-16 20:04:27
Subject: Re: 15,000,000 + files on one directory backup
From: TSM_User <tsm_user AT YAHOO DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Thu, 16 Jun 2005 17:03:55 -0700
Richard, the backup does not take long because of the communication with the 
TSM server. It is the reading of the MFT (Master File Table) on the client 
itself that takes time. Each drive has its own MFT.

Also more importantly TSM will only start one producer thread per logical 
drive.  So if there were seperate physical drives then you chould get a 
seperate set of producer and comsumer threads running on each one.

We had a server with 3 logical drives with about 1 million files on each.  It 
took 3 hours or so to backup with resource utilization set to 2.  When we 
raised it to 10 it took under 1 hour to run.

Kyle

Still at a certain point even with resource utilzation set to 20
"Dearman, Richard" <rdearm1 AT UIC DOT EDU> wrote:
Yes, but the long backup time is because of the time it takes the TSM
client to query the TSM database for backup file candidates and not due
to the actual movement of files from the client to the TSM server. So
how pulling from 10 separate drives increase the query speed.

For instance it takes hours and GB of memory on my client for the client
to query the TSM server for file info for 15 million files even though
the actual backup will end up being 11,000 files consisting of 300mb.
And the backup take over 17+ hours to complete. This is running from a
GigE connection with my client with 4 cpus and 6GB of memory although it
is Win2k.

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Stapleton, Mark
Sent: Thursday, June 16, 2005 12:53 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: 15,000,000 + files on one directory backup

No, because you're pulling data off of 10 separate physical drives. It's
likely to be significantly faster.

--
Mark Stapleton (stapleton AT berbee DOT com)
IBM Certified Advanced Deployment Professional
Tivoli Storage Management Solutions 2005
IBM Certified Advanced Technical Expert (CATE) AIX
Office 262.521.5627



>-----Original Message-----
>From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On
>Behalf Of Dearman, Richard
>Sent: Thursday, June 16, 2005 12:46 PM
>To: ADSM-L AT VM.MARIST DOT EDU
>Subject: Re: 15,000,000 + files on one directory backup
>
>Wouldn't the backup take close to the same amount of time has using one
>mount point. Because the TSM client on that one server still had to go
>through 15 million files whether its doing it in one session or 10
>sessions.
>
>I am experiencing the same problem of an imaging system. I am
>trying to
>go the snapshot image route of the 12 millions 300Gb of files and
>sending them to disk storage pool then off to 3592 tapes nightly. It
>will grow to 1Tb of the next year. I'm not sure how imaging will work
>on such a large file system.
>
>-----Original Message-----
>From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On
>Behalf Of
>TSM_User
>Sent: Thursday, June 16, 2005 12:32 PM
>To: ADSM-L AT VM.MARIST DOT EDU
>Subject: 15,000,000 + files on one directory backup
>
>I replied to the list about using windows mount points. A
>reply was sent
>back about it being the opposit of what they wanted. Well I had this
>thought.
>
>If you have an imaging system (or any other application) that can only
>use one drive letter why not use mount points for that drive. Have a
>server with 10 drives with 100 GB each (D: - M:) Then use mount points
>to get all 1 TB of space behind the D:\. The application will then use
>the D:\ alone. Mean while you can run the backup on all 10 drives.
>
>For details on how to set this up you need to consult the
>Microsoft doc.
>There is plenty on MS's website.
>
>Of course if the system is already setup then you'd need to add the
>drives and mount points. Then move the folders under those mount
>points.
>
>__________________________________________________
>Do You Yahoo!?
>Tired of spam? Yahoo! Mail has the best spam protection around
>http://mail.yahoo.com
>
>**************************EMAIL DISCLAIMER***************************
>
>This email and any files transmitted with it may be
>confidential and are
>intended solely for the use of the individual or entity to
>whom they are
>addressed. If you are not the intended recipient or the individual
>responsible for delivering the e-mail to the intended recipient, any
>disclosure, copying, distribution or any action taken or omitted to be
>taken in reliance on it, is strictly prohibited. If you have
>received this
>e-mail in error, please delete it and notify the sender or
>contact Health
>Information Management 312.413.4947.
>
>
>

**************************EMAIL DISCLAIMER***************************

This email and any files transmitted with it may be confidential and are
intended solely for the use of the individual or entity to whom they are
addressed. If you are not the intended recipient or the individual
responsible for delivering the e-mail to the intended recipient, any
disclosure, copying, distribution or any action taken or omitted to be
taken in reliance on it, is strictly prohibited. If you have received this
e-mail in error, please delete it and notify the sender or contact Health
Information Management 312.413.4947.




__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com