Veritas-bu

[Veritas-bu] Cambarra

2004-11-23 15:45:16
Subject: [Veritas-bu] Cambarra
From: george.badinter AT citigroup DOT com (Badinter, George)
Date: Tue, 23 Nov 2004 15:45:16 -0500
I need to bring cambarra server down for a few minutes. Please let me
know if it is possible.

George


-----Original Message-----
From: Alberto, Aldrin [AM] 
Sent: Tuesday, November 23, 2004 9:05 AM
To: Pare, David [AM]; Robinson, Jeffrey [AM]
Cc: Badinter, George; Kosubinsky, James [AM]; Harshe, Gautam [AM];
Pavlik, George
Subject: RE: Cambarra Database restore


Jeff,

The restored database is being brought down and I am creating the list of files 
that need to be copied in /var/cam/db.

Will let you know once done.


Thanks
Aldrin


-----Original Message-----
From: Pare, David [AM] 
Sent: Tuesday, November 23, 2004 7:53 AM
To: Robinson, Jeffrey [AM]
Cc: Badinter, George; Kosubinsky, James [AM]; Alberto, Aldrin [AM];
Harshe, Gautam [AM]; Pavlik, George
Subject: RE: Cambarra Database restore
Importance: High


Has there been any update on the restoration of the CAMBARRA Db?

Please advise.

Thanks,
David

-----Original Message-----
From: Robinson, Jeffrey [AM] 
Sent: Monday, November 22, 2004 11:14 PM
To: Pare, David [AM]
Cc: Badinter, George; Kosubinsky, James [AM]
Subject: Re: Cambarra Database restore


Sorry, there was a break down in communications. 

Jeff



-----Original Message-----
From: Pare, David [AM] <dp26390 AT imcnam.ssmb DOT com>
To: Robinson, Jeffrey [AM] <jr75571 AT imcnam.ssmb DOT com>
CC: Badinter, George <gb50817 AT imcnam.ssmb DOT com>; Kosubinsky, James [AM] 
<jk52460 AT imcnam.ssmb DOT com>
Sent: Mon Nov 22 22:07:10 2004
Subject: Re: Cambarra Database restore

I was under the assumption that the files were being copied tonight and 
restored by tomorrow morning.  
--------------------------------------------------------------------------------------------------------
This message and any attachments are confidential to the ordinary 
user of the e-mail address to which it was addressed and may 
also be privileged. If you receive this message in error please 
immediately delete it and all copies of it from your system, destroy 
any hard copies of it and notify the sender.  You must not, directly 
or indirectly, use, disclose, distribute, print or copy any part of this 
message if you are not the intended recipient. Internet 
communications cannot be guaranteed to be secure or 
error-free as information could be intercepted, corrupted, lost, 
arrive late or contain viruses or unauthorized amendments.  
The sender, therefore, does not accept liability for any errors or 
omissions in the content of this message or any damage or other 
consequences arising as a result of Internet transmission. 
Opinions, conclusions and other information in this message that 
do not relate to the official business of CITIGROUP shall be 
understood as neither given nor endorsed by it. 


-----Original Message-----
From: Robinson, Jeffrey [AM] <jr75571 AT imcnam.ssmb DOT com>
To: Pare, David [AM] <dp26390 AT imcnam.ssmb DOT com>
CC: Badinter, George <gb50817 AT imcnam.ssmb DOT com>; Kosubinsky, James [AM] 
<jk52460 AT imcnam.ssmb DOT com>
Sent: Mon Nov 22 20:33:50 2004
Subject: Re: Cambarra Database restore

I have not gotten the OK to restore anything, tonight I'm backing up the files 
from the corrupted database per the DBA.  

Maybe you or George know more then me?

Jeff



-----Original Message-----
From: Pare, David [AM] <dp26390 AT imcnam.ssmb DOT com>
To: Vemparala, Srini [AM] <sv26230 AT imcnam.ssmb DOT com>; Alberto, Aldrin 
[AM] <aa86662 AT imcnam.ssmb DOT com>; Corbin, Jennifer [AM] <jc27528 AT 
imcnam.ssmb DOT com>; Monticone, Ron [AM] <RM29517 AT imcnam.ssmb DOT com>; 
Alberto, Aldrin [AM] <aa86662 AT imcnam.ssmb DOT com>; Harshe, Gautam [AM] 
<gh79701 AT imcnam.ssmb DOT com>; Quick, Jonathan [AM] <jq26925 AT imcnam.ssmb 
DOT com>; Blank, Alex [AM] <ab87055 AT imcnam.ssmb DOT com>; Raimondi, Victor 
[AM] <VR66514 AT imcnam.ssmb DOT com>; Jonker, Eduard [AM] <ej27287 AT 
imcnam.ssmb DOT com>
CC: *CAM Dba Support Oracle <dbasupportoracle AT imcnam.ssmb DOT com>; 
Costanza, Joseph [AM] <jc91857 AT imcnam.ssmb DOT com>; Montay, Robert [AM] 
<rm87979 AT imcnam.ssmb DOT com>; Sacramone, Mario [AM] <ms08117 AT imcnam.ssmb 
DOT com>; Potter, Brendan [AM] <bp91802 AT imcnam.ssmb DOT com>; *CAM UNIX 
Server Operations <unixserverops AT imcnam.ssmb DOT com>; *CAM NA Desktop 
Support <cam.desktopsupp AT imcnam.ssmb DOT com>
Sent: Mon Nov 22 18:12:31 2004
Subject: RE: Cambarra Database restore

FYI -

Apparently the Total Risk (CAMBARRA) database is in the processes of being 
restored from the 'cold' backup.  The process should take a majority of the 
night, (according to George Badinter), therefore the Database and the Total 
Risk application should be available on Tuesday morning.

In the mean time, Domestic and International Total Risk are unavailable.

Thanks,
David

-----Original Message-----
From: Vemparala, Srini [AM] 
Sent: Saturday, November 20, 2004 7:49 PM
To: Pare, David [AM]; Alberto, Aldrin [AM]; Corbin, Jennifer [AM];
Monticone, Ron [AM]; Alberto, Aldrin [AM]; Harshe, Gautam [AM]; Quick,
Jonathan [AM]; Blank, Alex [AM]; Raimondi, Victor [AM]; Jonker, Eduard
[AM]
Cc: *CAM Dba Support Oracle; Costanza, Joseph [AM]; Montay, Robert [AM];
Sacramone, Mario [AM]; Potter, Brendan [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore



Thanks to Bob! Given the complications, the man power and vendor support 
required to resolve this, Looks like we can resume our recovery efforts on 
Monday morning. 

Thanks to all of you for your efforts in the last 3 days to recover this 
database. Have a nice weekend and see you on Monday!

Let me know if you have any questions.

Regards,

Srini
-----Original Message-----
From: Pare, David [AM] 
Sent: Saturday, November 20, 2004 5:50 PM
To: Vemparala, Srini [AM]; Alberto, Aldrin [AM]; Corbin, Jennifer [AM];
Monticone, Ron [AM]; Alberto, Aldrin [AM]; Harshe, Gautam [AM]; Quick,
Jonathan [AM]; Blank, Alex [AM]; Raimondi, Victor [AM]; Jonker, Eduard
[AM]
Cc: *CAM Dba Support Oracle; Costanza, Joseph [AM]; Montay, Robert [AM];
Sacramone, Mario [AM]; Potter, Brendan [AM]
Subject: RE: Cambarra Database restore


Srini,

Is this "camprdsptr1.nj.ssmb.com" server's entry in the TNSNames.ORA?  I don't 
reacll seeing it in there.

Thanks,
Dave

-----Original Message-----
From: Vemparala, Srini [AM] 
Sent: Saturday, November 20, 2004 5:37 PM
To: Vemparala, Srini [AM]; Alberto, Aldrin [AM]; Corbin, Jennifer [AM];
Pare, David [AM]; Monticone, Ron [AM]; Alberto, Aldrin [AM]; Harshe,
Gautam [AM]; Quick, Jonathan [AM]; Blank, Alex [AM]; Raimondi, Victor
[AM]; Jonker, Eduard [AM]
Cc: *CAM Dba Support Oracle; Costanza, Joseph [AM]; Montay, Robert [AM];
Sacramone, Mario [AM]; Potter, Brendan [AM]
Subject: RE: Cambarra Database restore


At this point we have only one option as mentioned below

1) Re-point TotalRisk (TR) US and International applications to the Barra 
database on "camprdsptr1.nj.ssmb.com" (which is the backup database). Please 
note this is not the CAMBARRA database.

2) Check if both US and International applications come up and work ok based on 
CAMPRDSPTR1 database.

3) If yes, then we can restore the data from the backup.

4) Once the DB is up, Ron can kick off all TR jobs(US & International).

5) If everything looks ok, we can figure out if Reporting data in CAMTR schema 
can be recoverd. This is necessary but the first priority would be bring up TR 
applications.

I think to do this, it is recommended that we get to the work place as this is 
so complicated it cannot be accomplished via remote connections and phone 
conversations. 

Bob - This could be done first thing Monday morning as everyone required to 
work on this would be available. But if you think this is very critical then I 
would suggest doing this Sunday morning.

I would atleast think following perople are needed if we decide to do Sunday 
morning.

Srini Vemparala
Alex Blank
Dave Pare
Ron Monticone
Aldrin Alberto
Jon Quick (Just in case for his expertise)

Please let me know what you think and let me know if you have any questions.

Regards,

Srini

 

-----Original Message-----
From: Vemparala, Srini [AM] 
Sent: Saturday, November 20, 2004 5:07 PM
To: Alberto, Aldrin [AM]
Cc: *CAM Dba Support Oracle; Costanza, Joseph [AM]; Montay, Robert [AM];
*CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


Aldrin,

I have got Alex logged onto Outlook as I could not conference all of us. We can 
discuss what needs to be done from here.

Thanks,

Srini

-----Original Message-----
From: Alberto, Aldrin [AM] 
Sent: Saturday, November 20, 2004 4:10 PM
To: Alberto, Aldrin [AM]; Vemparala, Srini [AM]
Cc: *CAM Dba Support Oracle; Costanza, Joseph [AM]; Montay, Robert [AM];
*CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


Srini,

I have recovered the restored database (with one datafile missing). 

You might want to check the database before it is restored. 


The connection details are below.

BARRA =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = camprdsptr1.nj.ssmb.com)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = CAMBARRA)
   )
  )

Thanks


Aldrin



-----Original Message-----
From: Alberto, Aldrin [AM] 
Sent: Saturday, November 20, 2004 2:01 PM
To: Alberto, Aldrin [AM]; Vemparala, Srini [AM]
Cc: *CAM Dba Support Oracle; Costanza, Joseph [AM]; Montay, Robert [AM];
*CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


Srini,

The export is done, but one table was giving an error due the corruption. Below 
is the table

. . exporting table    RISK_DECOMPOSITION_DET_TEMP
EXP-00008: ORACLE error 376 encountered
ORA-00376: file 6 cannot be read at this time

At this point, we can try and restore the database with missing datafile and 
see if we can salvage some 
data from that table.

Please give me call to discuss ...


Thanks
Aldrin


-----Original Message-----
From: Alberto, Aldrin [AM] 
Sent: Saturday, November 20, 2004 1:15 PM
To: Alberto, Aldrin [AM]; Vemparala, Srini [AM]
Cc: *CAM Dba Support Oracle; Costanza, Joseph [AM]; Montay, Robert [AM];
*CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


Step 1 still  is going on ....

-----Original Message-----
From: Alberto, Aldrin [AM] 
Sent: Saturday, November 20, 2004 11:09 AM
To: Vemparala, Srini [AM]
Cc: *CAM Dba Support Oracle; Costanza, Joseph [AM]; Montay, Robert [AM];
*CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


Srini,

As discussed here is the plan, given that one datafile is missing from the 
backup.

1. Take export of the CAMTR from the current production. (there could be issue 
with this export. will come to know once the export is done)

2. Once step 1 is done, shutdown the Production and have the backup taken.

3. Have the currently restored database up and running with out the CAMTR 
schema (due to missing datafile).

4. Have the restored database copied over to Production.

5. Re-start the production. Import the CAMTR schema from the export in step 1.

Given all the steps this will be time consuming and the database may not be 
available on Monday. Please give me a 
call if any you require any further information.

Thanks,

Aldrin





-----Original Message-----
From: Robinson, Jeffrey [AM] 
Sent: Saturday, November 20, 2004 10:39 AM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


Working on getting discharged, so signing off.

Call me if you need anything.

Jeff

-----Original Message-----
From: Robinson, Jeffrey [AM] 
Sent: Saturday, November 20, 2004 10:32 AM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


Good luck.

I have  changed the code so this will not be a problem in the future, not that 
it helps us now.

And we should still test it, just to be sure it is right. 

Jeff

-----Original Message-----
From: Alberto, Aldrin [AM] 
Sent: Saturday, November 20, 2004 10:28 AM
To: Robinson, Jeffrey [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


Let me see, if I can get the data from that file from the current cambarra 
database .


Thanks
Aldrin


-----Original Message-----
From: Robinson, Jeffrey [AM] 
Sent: Saturday, November 20, 2004 10:15 AM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


Bad news, file did not make it over (issue with the size and tar options).

What's next. 


Jeff
-----Original Message-----
From: Alberto, Aldrin [AM] 
Sent: Saturday, November 20, 2004 10:00 AM
To: Alberto, Aldrin [AM]; Robinson, Jeffrey [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


/oradata/camnabarra/CAMTR_DATA5.dbf

-----Original Message-----
From: Alberto, Aldrin [AM] 
Sent: Saturday, November 20, 2004 9:54 AM
To: Robinson, Jeffrey [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


I have killed the process and mounted the restored database.

Jeff,

There is typo in one of the soft links in /oraindex.

The link should be 'camnabarra' and nor 'cambarra'. Please can you change as I 
don't have permissions to change the link


Thanks

Aldrin




-----Original Message-----
From: Robinson, Jeffrey [AM] 
Sent: Saturday, November 20, 2004 9:11 AM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


Performance may be slow because there are 1961 copies of /bin/ksh 
/export/dbspace/cgamdev1/exp/run_exp.ksh running, could there be something 
wrong with the script?

Jeff

-----Original Message-----
From: Robinson, Jeffrey [AM] 
Sent: Saturday, November 20, 2004 8:40 AM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


Name:    camprdsptr1.nj.ssmb.com
Address:  150.110.177.

Oracle is installed, Brendan use to check the production (db2) backup with this 
server.

It is a very small box, 1 CPU (800MHZ) and 2 GB memory.

Jeff

-----Original Message-----
From: Alberto, Aldrin [AM] 
Sent: Saturday, November 20, 2004 8:29 AM
To: Robinson, Jeffrey [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


Jeff,

On which box should I login to check the restore.

Please hold on today's backup.


Thanks
Aldrin


-----Original Message-----
From: Robinson, Jeffrey [AM] 
Sent: Saturday, November 20, 2004 8:05 AM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


Files are back ad you can log onto the box and scsu to oracle.


Not sure if we'll have enough space for this and this weekends backup of 
camnairtdb1, We'll have to keep an eye on the space and try to adjust as we go.

Also, what do you want to do with today's backup of Cambarra, set to start at 
10AM?
Jeff

-----Original Message-----
From: Alberto, Aldrin [AM] 
Sent: Saturday, November 20, 2004 7:48 AM
To: Robinson, Jeffrey [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


Jeff,

Please can you provide me the access to the SAN, once the database is restored 
so than I can try and bring it up there and then do the 
restore.

Thanks
Aldrin

-----Original Message-----
From: Robinson, Jeffrey [AM] 
Sent: Friday, November 19, 2004 10:18 PM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


Good, my concern was someone made a type creating the list.

Everything is going, I'll check on it when the nurse wakes me up at 6AM to see 
how I slept and if I want to wash up.

Jeff

-----Original Message-----
From: Alberto, Aldrin [AM] 
Sent: Friday, November 19, 2004 10:00 PM
To: Robinson, Jeffrey [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


If those files are in the list, then they are the control files and should have 
been backed up.

Aldrin

-----Original Message-----
From: Robinson, Jeffrey [AM] 
Sent: Friday, November 19, 2004 9:56 PM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


I looked at the save list from last week (it is backed too) and the entries are

/orasys/camnabarracontrol01.ctl
/orasys/camnabarracontrol02.ctl

Is there an option if the backup is no good (this is the first time we are 
testing it)

Jeff

-----Original Message-----
From: Alberto, Aldrin [AM] 
Sent: Friday, November 19, 2004 9:48 PM
To: Robinson, Jeffrey [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


Jeff,

I did not quite understand the last step. The control files are in the list 
that is generated in /var/cam/db

Thanks
Aldrin

-----Original Message-----
From: Robinson, Jeffrey [AM] 
Sent: Friday, November 19, 2004 9:27 PM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore


The compressed tar files are backed up to tape every week, to restore we need to

Restore from tape
uncompress files
setup dir to untar the files to
untar files.

Step 1 is done and working on step 2.

This  may take most of the night.

Looking at the list of files, it looks like you were not saving the control 
files
/orasys/camnabarracontrol01.ctl
/orasys/camnabarracontrol02.ctl

Heck of a way to test the backups!

Jeff


-----Original Message-----
From: Alberto, Aldrin [AM] 
Sent: Friday, November 19, 2004 8:05 PM
To: Alberto, Aldrin [AM]; *CAM UNIX Server Operations
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]
Subject: RE: Cambarra Database restore


Matt,

I just spoke to George. According to him he as to restore the database from 
tape to SAN. Please confirm.

Thanks,

Aldrin


-----Original Message-----
From: Alberto, Aldrin [AM] 
Sent: Friday, November 19, 2004 7:50 PM
To: *CAM UNIX Server Operations
Subject: Cambarra Database restore


Matt,

It looks like the Cambarra database (machine camnadocudb1) is corrupted and may 
need a restore from last week-end. 

This database is backed up  every week on to SAN. I would like to do the 
following.

Get Access to the backup on SAN. Try to start up the database there 
successfully. If database start there successfully restore 
database to the machine  (camnadocudb1 - IP address 169.188.0.75).


I can be reached at 201 218 6052 / 973 324 0727.


Thanks




Aldrin Alberto,
Retail Technology,
100 Stamford Place,Stamford.
(203) 961-6414
Aldrin.Alberto AT citigroup DOT com



<Prev in Thread] Current Thread [Next in Thread>
  • [Veritas-bu] Cambarra, Badinter, George <=