ADSM-L

Re: 15,000,000 + files on one directory backup

2005-06-16 20:10:40
Subject: Re: 15,000,000 + files on one directory backup
From: TSM_User <tsm_user AT YAHOO DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Thu, 16 Jun 2005 17:10:21 -0700
Also, with my suggestion you don't need to have 10 different nodes. You can 
have one node.  With resource utilization of 10 I believe you will have up to 4 
producer and 4 consumer threads.  It's been a while since I saw the doc on that 
exact setting to thread count.  If you want more you can set it higher like 30. 
Just remember at some point the available resources on the server will come 
into play so change that setting with care.

My main point was if you can get the data onto 10 logical drives then the TSM 
backup will start seperate producer threads.  When you have all 15,000,000 
files on one drive you get one producer thread running on the whole drive.

K

Egon Blouder <egonle AT NETSCAPE DOT NET> wrote:
Hi Mark,

how can I store any data as filespaces of one node(virtualnode) in that case? I 
don't want to setup 10 different webagents for doing restores.


Thanks
..--

"Stapleton, Mark" wrote:

>You create 10 separate nodes, one for each mountpoint (drive). You can
>then run 10 separate backup threads; each thread has (in theory) 1/10th
>as many files to filter through before a backup begins.
>
>It's done that way a lot in such cases.
>
>--
>Mark Stapleton (stapleton AT berbee DOT com)
>IBM Certified Advanced Deployment Professional
> Tivoli Storage Management Solutions 2005
>IBM Certified Advanced Technical Expert (CATE) AIX
>Office 262.521.5627
>
>
>
>>-----Original Message-----
>>From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On
>>Behalf Of Dearman, Richard
>>Sent: Thursday, June 16, 2005 1:06 PM
>>To: ADSM-L AT VM.MARIST DOT EDU
>>Subject: Re: 15,000,000 + files on one directory backup
>>
>>Yes, but the long backup time is because of the time it takes the TSM
>>client to query the TSM database for backup file candidates and not due
>>to the actual movement of files from the client to the TSM server. So
>>how pulling from 10 separate drives increase the query speed.
>>
>>For instance it takes hours and GB of memory on my client for
>>the client
>>to query the TSM server for file info for 15 million files even though
>>the actual backup will end up being 11,000 files consisting of 300mb.
>>And the backup take over 17+ hours to complete. This is running from a
>>GigE connection with my client with 4 cpus and 6GB of memory
>>although it
>>is Win2k.
>>
>>-----Original Message-----
>>From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On
>>Behalf Of
>>Stapleton, Mark
>>Sent: Thursday, June 16, 2005 12:53 PM
>>To: ADSM-L AT VM.MARIST DOT EDU
>>Subject: Re: 15,000,000 + files on one directory backup
>>
>>No, because you're pulling data off of 10 separate physical
>>drives. It's
>>likely to be significantly faster.
>>
>>--
>>Mark Stapleton (stapleton AT berbee DOT com)
>>IBM Certified Advanced Deployment Professional
>> Tivoli Storage Management Solutions 2005
>>IBM Certified Advanced Technical Expert (CATE) AIX
>>Office 262.521.5627
>>
>>
>>
>>>-----Original Message-----
>>>From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On
>>>Behalf Of Dearman, Richard
>>>Sent: Thursday, June 16, 2005 12:46 PM
>>>To: ADSM-L AT VM.MARIST DOT EDU
>>>Subject: Re: 15,000,000 + files on one directory backup
>>>
>>>Wouldn't the backup take close to the same amount of time has
>>using one
>>>mount point. Because the TSM client on that one server still
>>had to go
>>>through 15 million files whether its doing it in one session or 10
>>>sessions.
>>>
>>>I am experiencing the same problem of an imaging system. I am
>>>trying to
>>>go the snapshot image route of the 12 millions 300Gb of files and
>>>sending them to disk storage pool then off to 3592 tapes nightly. It
>>>will grow to 1Tb of the next year. I'm not sure how imaging will work
>>>on such a large file system.
>>>
>>>-----Original Message-----
>>>From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On
>>>Behalf Of
>>>TSM_User
>>>Sent: Thursday, June 16, 2005 12:32 PM
>>>To: ADSM-L AT VM.MARIST DOT EDU
>>>Subject: 15,000,000 + files on one directory backup
>>>
>>>I replied to the list about using windows mount points. A
>>>reply was sent
>>>back about it being the opposit of what they wanted. Well I had this
>>>thought.
>>>
>>>If you have an imaging system (or any other application) that can only
>>>use one drive letter why not use mount points for that drive. Have a
>>>server with 10 drives with 100 GB each (D: - M:) Then use
>>mount points
>>>to get all 1 TB of space behind the D:\. The application
>>will then use
>>>the D:\ alone. Mean while you can run the backup on all 10 drives.
>>>
>>>For details on how to set this up you need to consult the
>>>Microsoft doc.
>>>There is plenty on MS's website.
>>>
>>>Of course if the system is already setup then you'd need to add the
>>>drives and mount points. Then move the folders under those mount
>>>points.
>>>
>>>__________________________________________________
>>>Do You Yahoo!?
>>>Tired of spam? Yahoo! Mail has the best spam protection around
>>>http://mail.yahoo.com
>>>
>>>**************************EMAIL DISCLAIMER***************************
>>>
>>>This email and any files transmitted with it may be
>>>confidential and are
>>>intended solely for the use of the individual or entity to
>>>whom they are
>>>addressed. If you are not the intended recipient or the individual
>>>responsible for delivering the e-mail to the intended recipient, any
>>>disclosure, copying, distribution or any action taken or
>>omitted to be
>>>taken in reliance on it, is strictly prohibited. If you have
>>>received this
>>>e-mail in error, please delete it and notify the sender or
>>>contact Health
>>>Information Management 312.413.4947.
>>>
>>>
>>>
>>
>>**************************EMAIL DISCLAIMER***************************
>>
>>This email and any files transmitted with it may be
>>confidential and are
>>intended solely for the use of the individual or entity to
>>whom they are
>>addressed. If you are not the intended recipient or the individual
>>responsible for delivering the e-mail to the intended recipient, any
>>disclosure, copying, distribution or any action taken or omitted to be
>>taken in reliance on it, is strictly prohibited. If you have
>>received this
>>e-mail in error, please delete it and notify the sender or
>>contact Health
>>Information Management 312.413.4947.
>>
>>
>>
>

__________________________________________________________________
Switch to Netscape Internet Service.
As low as $9.95 a month -- Sign up today at http://isp.netscape.com/register

Netscape. Just the Net You Need.

New! Netscape Toolbar for Internet Explorer
Search from anywhere on the Web and block those annoying pop-ups.
Download now at http://channels.netscape.com/ns/search/install.jsp

                
---------------------------------
Yahoo! Sports
 Rekindle the Rivalries. Sign up for Fantasy Football