Hi,
I have now tried different approaches for my large TSM query:
DBI:
Allen Rout confirmed my suspicions in a previous post. This one loads the whole
output of the sql into memory before processing. For my large query I killed
the program when it reached 1GB.
ODBC:
This one seems to kick off the query and start processing immediately. However
it still uses a lot of memory if not as much as DBI. My query topped out at
around 400 MB. I am not sure what is going on under the hood here.
Command pipe:
open OUT, "dsmadmc -noconfirm -tabd -id=xxx -password=xxx select .... |";
This worked well, perl just using 4MB of memory. No memory problems on the
TSM-server either.
Thank you to all who responded!
Best regards,
Hans Chr. Riksheim
-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Andrew Raibeck
Sent: Monday, December 04, 2006 3:42 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: Using the DBI::TSM module, memory problem
I cannot remember if I ever tried using DBI, but I have been able to get Perl
to work with the TSM ODBC driver by using the Roth ODBC package for Perl. Go to
the URL in my sig, then do a search on:
odbc perl
to find an article on using Perl with the TSM ODBC driver. I believe my example
in the article does not have the memory consumption issue that you describe.
Regards,
Andy
Andy Raibeck
IBM Software Group
Tivoli Storage Manager Client Development Internal Notes e-mail: Andrew
Raibeck/Tucson/IBM@IBMUS Internet e-mail: storman AT us.ibm DOT com
IBM Tivoli Storage Manager support web page:
http://www.ibm.com/software/sysmgmt/products/support/IBMTivoliStorageManager.html
The only dumb question is the one that goes unasked.
The command line is your friend.
"Good enough" is the enemy of excellence.
"ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 12/04/2006
06:21:07 AM:
> Hi,
>
> anybody else using the perl module DBI::TSM?
>
> I've recently been using it for reporting purposes and find it quite
> useful. However I have some problems with the perl memory consumption.
> This may very well be caused by my lack of knowledge of DBI, but I
> haven't found a remedy yet.
>
> The problem is that I want to process the output of sql statements
> line by line as it is returned by the TSM-server, but it seems that
> DBI requires the output of the whole sql to be stored in memory before
> I can process it. For large queries the memory consumption will be
> huge.
>
> Example(I know this is a simple one where it could be all handled in
> the sql itself, this is just for explaining my problem):
>
>
> # Set up the database handle:
> $dbh=DBI->connect("DBI::TSM:myserver", 'admin', 'password¨');
>
> # SQL that will return a zillion rows
> $sql='select description as "DESC" from archives where
node_name='MYNODE'';
>
> $sth=$dbh->prepare($sql);
>
> $sth->execute; #This is the one that slurps my memory
>
> while($row_ref)=$sth->fetchrow_hashref){
> #this processes each row that is already stored in memory
> $archive_names{$row_ref->{'DESC'}}='';
> }
>
>
> I guess I could avoid the problem by using "dsmadmc" directly and
> write the output to a file that i could later have perl parse, but
> that would not be a very elegant solution. I want it to work with DBI.
>
>
> Any ideas?
>
>
> Best regards,
>
> Hans Chr. Riksheim
>
>
>
>
>
>
> <html><body><br /><br /><br /><font face="Arial" size="1"><hr>This
> email originates from Steria AS, Biskop Gunnerus' gate 14a, N-0051
> OSLO,
> http://www.steria.no. This email and any attachments may contain
> confidential/intellectual property/copyright information and is only
> for the use of the addressee(s). You are prohibited from copying,
> forwarding, disclosing, saving or otherwise using it in any way if you
> are not the addressee(s) or responsible for delivery. If you receive
> this email by mistake, please advise the sender and cancel it
> immediately. Steria may monitor the content of emails within its
> network to ensure compliance with its policies and procedures. Any
> email is susceptible to alteration and its integrity cannot be
> assured. Steria shall not be liable if the message is altered,
> modified, falsified, or even edited.</font><html><body><br /><br /><br
> /><font face="Arial" size="1"><hr>This email originates from Steria AS,
> Biskop Gunnerus' gate 14a, N-0051 OSLO, http://www.steria.no. This email and
> any attachments may contain confidential/intellectual property/copyright
> information and is only for the use of the addressee(s). You are prohibited
> from copying, forwarding, disclosing, saving or otherwise using it in any way
> if you are not the addressee(s) or responsible for delivery. If you receive
> this email by mistake, please advise the sender and cancel it immediately.
> Steria may monitor the content of emails within its network to ensure
> compliance with its policies and procedures. Any email is susceptible to
> alteration and its integrity cannot be assured. Steria shall not be liable if
> the message is altered, modified, falsified, or even
> edited.</font></body></html></body></html>
|
