Re: Using the DBI::TSM module, memory problem

Hi,
 
I use the 64-bit version of perl which can handle the amount of memory. But the 
memory required is so great that the server starts swapping and the execution 
rate goes to a standstill(together with the rest of the things running on that 
AIX-server). The same will probably happen with DBM.
 
I essentially want to "stream" the output of the sql through a filter while it 
is returned from the TSM-server, not glob it up in memory first and then 
process it later. Have not found out yet with DBI. Some database guy suggested 
use of cursors which make it possible to fetch one row at a time and have the 
database keep track of your progress, but I don't know the first thing about 
cursors and I don't think the TSM database is designed for those needs anyway.
 
I will try the ODBC-module Andrew suggested and let you know if I can make it 
work in a stream/filter fashion.
 
 
Best regards
 
Hans Christian Riksheim
 
 
 
 

________________________________

Fra: ADSM: Dist Stor Manager på vegne av Richard Sims
Sendt: ma 04.12.2006 15:46
Til: ADSM-L AT VM.MARIST DOT EDU
Emne: Re: Using the DBI::TSM module, memory problem



On Dec 4, 2006, at 9:21 AM, Hans Christian Riksheim wrote:

> The problem is that I want to process the output of sql statements
> line by line as it is returned by the TSM-server, but it seems that
> DBI requires the output of the whole sql to be stored in memory
> before I can process it. For large queries the memory consumption
> will be huge.

Hans -

I faced this problem with a reporting program which had to operate on
a very large data set, which caused perl to exhibit flakey problems
(Illegal Instruction) as it exhausted its memory - even with Unix
Resource Limits high.  I initially approached dealing with this via
perl's dbm capabilities, which allows hashes to be on disk rather
than in memory.  Still, that was not enough.  I eventually resolved
the issue by switching to 64-bit perl, which provided the elbow room
necessary.

    Richard Sims


<html><body><br /><br /><br /><font face="Arial" size="1"><hr>This email 
originates from Steria AS, Biskop Gunnerus' gate 14a, N-0051 OSLO, 
http://www.steria.no. This email and any attachments may contain 
confidential/intellectual property/copyright information and is only for the 
use of the addressee(s). You are prohibited from copying, forwarding, 
disclosing, saving or otherwise using it in any way if you are not the 
addressee(s) or responsible for delivery. If you receive this email by mistake, 
please advise the sender and cancel it immediately. Steria may monitor the 
content of emails within its network to ensure compliance with its policies and 
procedures. Any email is susceptible to alteration and its integrity cannot be 
assured. Steria shall not be liable if the message is altered, modified, 
falsified, or even edited.</font></body></html>