ADSM-L

Re: 15,000,000 + files on one directory backup

2005-06-28 17:54:18
Subject: Re: 15,000,000 + files on one directory backup
From: Egon Blouder <egonle AT NETSCAPE DOT NET>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Tue, 28 Jun 2005 17:54:00 -0400
Hi,

unfortunately your setup isn't totally obvious to me. In fact you have three 
schedules probably as you have to backup remaining files of C:.
Do these different schedules run in parallel or is there only one schedule 
running at one time but serialized?

I'm still looking for those details (dsm.opt (files?),service setup,schedule 
setup) on how to setup multiple parallel schedules for one nodename.


Regards,

..--

"Lepre, James" <JLEPRE AT NECA DOT ORG> wrote:

>No I used the same node name with different schedules.  Yes I am doing
>memory efficient with a resourceU of 2.
>
>Example
>
>One schedule is for c:\DIR1\*.*
>Second Schedule is for C:\DIR2\*.*
>
>They both run at different times throughout the night
>
>-----Original Message-----
>From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
>Dearman, Richard
>Sent: Monday, June 20, 2005 9:45 AM
>To: ADSM-L AT VM.MARIST DOT EDU
>Subject: Re: 15,000,000 + files on one directory backup
>
>Did you setup separate node names for each directory or drive?  Or did
>you use the same node name for each schedule.  And are you doing
>memoryefficient backups on those schedules and is resoureutilization set
>to more than 2 for each schedule?
>
>-----Original Message-----
>From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
>Lepre, James
>Sent: Monday, June 20, 2005 8:20 AM
>To: ADSM-L AT VM.MARIST DOT EDU
>Subject: Re: 15,000,000 + files on one directory backup
>
>We have had the same problem with our Siebel system.  We have a system
>with over 25 million files in it and what we did was break it up.  We
>have multiple schedule running for individual directories.  For example
> Dir1\*.* has 3 million files in it, make it one schedule
> Dir2\*.* has 4.5 million files in it make it 2nd schedule and so on
>
>
>
>-----Original Message-----
>From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
>Dearman, Richard
>Sent: Monday, June 20, 2005 9:09 AM
>To: ADSM-L AT VM.MARIST DOT EDU
>Subject: Re: 15,000,000 + files on one directory backup
>
>I'm having the same problem has everyone else.  Our imaging system is
>going to be 30millioin+ files over 3 disks and 2TB by the end on the
>year.  I was going to attempt to use a snapshot image backup of each
>filesystem which is on a Win2k server.  Has anyone tried this before on
>a server of this size and were you successful?  I'm not very confident
>on doing this on the windows platform.
>
>I understand the problems associated with backing up a server with
>millions of files and a badly written application that stores everything
>in one place on one drive but I need a solution.
>
>-----Original Message-----
>From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
>Ted Byrne
>Sent: Saturday, June 18, 2005 4:57 PM
>To: ADSM-L AT VM.MARIST DOT EDU
>Subject: Re: 15,000,000 + files on one directory backup
>
>I would second Bill's addition of poorly-architected applications to
>Richard's list of issues that should be (but are often not) addressed,
>or
>even considered.  At another customer, we and the customer's sysadmins
>were
>bedeviled by a weblog analysis application (which shall remain nameless)
>that chose to store its data on the filesystem, using the date of the
>log
>data as a directory under which the data was stored (as well as the
>associated reports, I believe).  The explanation we were given was that
>they had chosen to do this for application performance reasons; it was
>apparently quicker that using a DBMS.
>
>This decision, although it made random access of data quicker, had
>horrible
>implications for backup as the log data and reports accumulated over
>time;
>recovery was even worse.  Aggravating the situation was the insistence
>by
>the "application owner" that ALL historical log data absolutely had to
>be
>maintained in this inside-out database format.  Just getting a count of
>files and directories on this drive (via selecting Properties from the
>context menu) took something on the order of 9 hours to complete.  The
>volume of data, in GB, was really not that large - something on the
>order
>of 100 GB.  All of their problems managing the data stemmed entirely
>from
>the large number of files and directories.
>
>When the time came to replace the server hardware and upgrade the
>application, they had extreme difficulty migrating the historical data
>from
>the old server to the new.  They did finally succeeded in copying the
>data
>from the old server to the new, but it took days and days of
>around-the-clock network traffic to complete.
>
>Addressing the ramifications of this type of design decision after the
>fact
>is difficult at best.  If at all possible, we need to prevent it from
>occurring in the first place.
>
>Ted
>
>**************************EMAIL DISCLAIMER***************************
>
>This email and any files transmitted with it may be confidential and are
>intended solely for the use of the individual or entity to whom they are
>addressed.  If you are not the intended recipient or the individual
>responsible for delivering the e-mail to the intended recipient, any
>disclosure, copying, distribution or any action taken or omitted to be
>taken in reliance on it, is strictly prohibited.  If you have received
>this
>e-mail in error, please delete it and notify the sender or contact
>Health
>Information Management 312.413.4947.
>
>
>

__________________________________________________________________
Switch to Netscape Internet Service.
As low as $9.95 a month -- Sign up today at http://isp.netscape.com/register

Netscape. Just the Net You Need.

New! Netscape Toolbar for Internet Explorer
Search from anywhere on the Web and block those annoying pop-ups.
Download now at http://channels.netscape.com/ns/search/install.jsp

<Prev in Thread] Current Thread [Next in Thread>