I need to bring cambarra server down for a few minutes. Please let me
know if it is possible.
George
-----Original Message-----
From: Alberto, Aldrin [AM]
Sent: Tuesday, November 23, 2004 9:05 AM
To: Pare, David [AM]; Robinson, Jeffrey [AM]
Cc: Badinter, George; Kosubinsky, James [AM]; Harshe, Gautam [AM];
Pavlik, George
Subject: RE: Cambarra Database restore
Jeff,
The restored database is being brought down and I am creating the list of files
that need to be copied in /var/cam/db.
Will let you know once done.
Thanks
Aldrin
-----Original Message-----
From: Pare, David [AM]
Sent: Tuesday, November 23, 2004 7:53 AM
To: Robinson, Jeffrey [AM]
Cc: Badinter, George; Kosubinsky, James [AM]; Alberto, Aldrin [AM];
Harshe, Gautam [AM]; Pavlik, George
Subject: RE: Cambarra Database restore
Importance: High
Has there been any update on the restoration of the CAMBARRA Db?
Please advise.
Thanks,
David
-----Original Message-----
From: Robinson, Jeffrey [AM]
Sent: Monday, November 22, 2004 11:14 PM
To: Pare, David [AM]
Cc: Badinter, George; Kosubinsky, James [AM]
Subject: Re: Cambarra Database restore
Sorry, there was a break down in communications.
Jeff
-----Original Message-----
From: Pare, David [AM] <dp26390 AT imcnam.ssmb DOT com>
To: Robinson, Jeffrey [AM] <jr75571 AT imcnam.ssmb DOT com>
CC: Badinter, George <gb50817 AT imcnam.ssmb DOT com>; Kosubinsky, James [AM]
<jk52460 AT imcnam.ssmb DOT com>
Sent: Mon Nov 22 22:07:10 2004
Subject: Re: Cambarra Database restore
I was under the assumption that the files were being copied tonight and
restored by tomorrow morning.
--------------------------------------------------------------------------------------------------------
This message and any attachments are confidential to the ordinary
user of the e-mail address to which it was addressed and may
also be privileged. If you receive this message in error please
immediately delete it and all copies of it from your system, destroy
any hard copies of it and notify the sender. You must not, directly
or indirectly, use, disclose, distribute, print or copy any part of this
message if you are not the intended recipient. Internet
communications cannot be guaranteed to be secure or
error-free as information could be intercepted, corrupted, lost,
arrive late or contain viruses or unauthorized amendments.
The sender, therefore, does not accept liability for any errors or
omissions in the content of this message or any damage or other
consequences arising as a result of Internet transmission.
Opinions, conclusions and other information in this message that
do not relate to the official business of CITIGROUP shall be
understood as neither given nor endorsed by it.
-----Original Message-----
From: Robinson, Jeffrey [AM] <jr75571 AT imcnam.ssmb DOT com>
To: Pare, David [AM] <dp26390 AT imcnam.ssmb DOT com>
CC: Badinter, George <gb50817 AT imcnam.ssmb DOT com>; Kosubinsky, James [AM]
<jk52460 AT imcnam.ssmb DOT com>
Sent: Mon Nov 22 20:33:50 2004
Subject: Re: Cambarra Database restore
I have not gotten the OK to restore anything, tonight I'm backing up the files
from the corrupted database per the DBA.
Maybe you or George know more then me?
Jeff
-----Original Message-----
From: Pare, David [AM] <dp26390 AT imcnam.ssmb DOT com>
To: Vemparala, Srini [AM] <sv26230 AT imcnam.ssmb DOT com>; Alberto, Aldrin
[AM] <aa86662 AT imcnam.ssmb DOT com>; Corbin, Jennifer [AM] <jc27528 AT
imcnam.ssmb DOT com>; Monticone, Ron [AM] <RM29517 AT imcnam.ssmb DOT com>;
Alberto, Aldrin [AM] <aa86662 AT imcnam.ssmb DOT com>; Harshe, Gautam [AM]
<gh79701 AT imcnam.ssmb DOT com>; Quick, Jonathan [AM] <jq26925 AT imcnam.ssmb
DOT com>; Blank, Alex [AM] <ab87055 AT imcnam.ssmb DOT com>; Raimondi, Victor
[AM] <VR66514 AT imcnam.ssmb DOT com>; Jonker, Eduard [AM] <ej27287 AT
imcnam.ssmb DOT com>
CC: *CAM Dba Support Oracle <dbasupportoracle AT imcnam.ssmb DOT com>;
Costanza, Joseph [AM] <jc91857 AT imcnam.ssmb DOT com>; Montay, Robert [AM]
<rm87979 AT imcnam.ssmb DOT com>; Sacramone, Mario [AM] <ms08117 AT imcnam.ssmb
DOT com>; Potter, Brendan [AM] <bp91802 AT imcnam.ssmb DOT com>; *CAM UNIX
Server Operations <unixserverops AT imcnam.ssmb DOT com>; *CAM NA Desktop
Support <cam.desktopsupp AT imcnam.ssmb DOT com>
Sent: Mon Nov 22 18:12:31 2004
Subject: RE: Cambarra Database restore
FYI -
Apparently the Total Risk (CAMBARRA) database is in the processes of being
restored from the 'cold' backup. The process should take a majority of the
night, (according to George Badinter), therefore the Database and the Total
Risk application should be available on Tuesday morning.
In the mean time, Domestic and International Total Risk are unavailable.
Thanks,
David
-----Original Message-----
From: Vemparala, Srini [AM]
Sent: Saturday, November 20, 2004 7:49 PM
To: Pare, David [AM]; Alberto, Aldrin [AM]; Corbin, Jennifer [AM];
Monticone, Ron [AM]; Alberto, Aldrin [AM]; Harshe, Gautam [AM]; Quick,
Jonathan [AM]; Blank, Alex [AM]; Raimondi, Victor [AM]; Jonker, Eduard
[AM]
Cc: *CAM Dba Support Oracle; Costanza, Joseph [AM]; Montay, Robert [AM];
Sacramone, Mario [AM]; Potter, Brendan [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
Thanks to Bob! Given the complications, the man power and vendor support
required to resolve this, Looks like we can resume our recovery efforts on
Monday morning.
Thanks to all of you for your efforts in the last 3 days to recover this
database. Have a nice weekend and see you on Monday!
Let me know if you have any questions.
Regards,
Srini
-----Original Message-----
From: Pare, David [AM]
Sent: Saturday, November 20, 2004 5:50 PM
To: Vemparala, Srini [AM]; Alberto, Aldrin [AM]; Corbin, Jennifer [AM];
Monticone, Ron [AM]; Alberto, Aldrin [AM]; Harshe, Gautam [AM]; Quick,
Jonathan [AM]; Blank, Alex [AM]; Raimondi, Victor [AM]; Jonker, Eduard
[AM]
Cc: *CAM Dba Support Oracle; Costanza, Joseph [AM]; Montay, Robert [AM];
Sacramone, Mario [AM]; Potter, Brendan [AM]
Subject: RE: Cambarra Database restore
Srini,
Is this "camprdsptr1.nj.ssmb.com" server's entry in the TNSNames.ORA? I don't
reacll seeing it in there.
Thanks,
Dave
-----Original Message-----
From: Vemparala, Srini [AM]
Sent: Saturday, November 20, 2004 5:37 PM
To: Vemparala, Srini [AM]; Alberto, Aldrin [AM]; Corbin, Jennifer [AM];
Pare, David [AM]; Monticone, Ron [AM]; Alberto, Aldrin [AM]; Harshe,
Gautam [AM]; Quick, Jonathan [AM]; Blank, Alex [AM]; Raimondi, Victor
[AM]; Jonker, Eduard [AM]
Cc: *CAM Dba Support Oracle; Costanza, Joseph [AM]; Montay, Robert [AM];
Sacramone, Mario [AM]; Potter, Brendan [AM]
Subject: RE: Cambarra Database restore
At this point we have only one option as mentioned below
1) Re-point TotalRisk (TR) US and International applications to the Barra
database on "camprdsptr1.nj.ssmb.com" (which is the backup database). Please
note this is not the CAMBARRA database.
2) Check if both US and International applications come up and work ok based on
CAMPRDSPTR1 database.
3) If yes, then we can restore the data from the backup.
4) Once the DB is up, Ron can kick off all TR jobs(US & International).
5) If everything looks ok, we can figure out if Reporting data in CAMTR schema
can be recoverd. This is necessary but the first priority would be bring up TR
applications.
I think to do this, it is recommended that we get to the work place as this is
so complicated it cannot be accomplished via remote connections and phone
conversations.
Bob - This could be done first thing Monday morning as everyone required to
work on this would be available. But if you think this is very critical then I
would suggest doing this Sunday morning.
I would atleast think following perople are needed if we decide to do Sunday
morning.
Srini Vemparala
Alex Blank
Dave Pare
Ron Monticone
Aldrin Alberto
Jon Quick (Just in case for his expertise)
Please let me know what you think and let me know if you have any questions.
Regards,
Srini
-----Original Message-----
From: Vemparala, Srini [AM]
Sent: Saturday, November 20, 2004 5:07 PM
To: Alberto, Aldrin [AM]
Cc: *CAM Dba Support Oracle; Costanza, Joseph [AM]; Montay, Robert [AM];
*CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
Aldrin,
I have got Alex logged onto Outlook as I could not conference all of us. We can
discuss what needs to be done from here.
Thanks,
Srini
-----Original Message-----
From: Alberto, Aldrin [AM]
Sent: Saturday, November 20, 2004 4:10 PM
To: Alberto, Aldrin [AM]; Vemparala, Srini [AM]
Cc: *CAM Dba Support Oracle; Costanza, Joseph [AM]; Montay, Robert [AM];
*CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
Srini,
I have recovered the restored database (with one datafile missing).
You might want to check the database before it is restored.
The connection details are below.
BARRA =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = camprdsptr1.nj.ssmb.com)(PORT = 1521))
)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = CAMBARRA)
)
)
Thanks
Aldrin
-----Original Message-----
From: Alberto, Aldrin [AM]
Sent: Saturday, November 20, 2004 2:01 PM
To: Alberto, Aldrin [AM]; Vemparala, Srini [AM]
Cc: *CAM Dba Support Oracle; Costanza, Joseph [AM]; Montay, Robert [AM];
*CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
Srini,
The export is done, but one table was giving an error due the corruption. Below
is the table
. . exporting table RISK_DECOMPOSITION_DET_TEMP
EXP-00008: ORACLE error 376 encountered
ORA-00376: file 6 cannot be read at this time
At this point, we can try and restore the database with missing datafile and
see if we can salvage some
data from that table.
Please give me call to discuss ...
Thanks
Aldrin
-----Original Message-----
From: Alberto, Aldrin [AM]
Sent: Saturday, November 20, 2004 1:15 PM
To: Alberto, Aldrin [AM]; Vemparala, Srini [AM]
Cc: *CAM Dba Support Oracle; Costanza, Joseph [AM]; Montay, Robert [AM];
*CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
Step 1 still is going on ....
-----Original Message-----
From: Alberto, Aldrin [AM]
Sent: Saturday, November 20, 2004 11:09 AM
To: Vemparala, Srini [AM]
Cc: *CAM Dba Support Oracle; Costanza, Joseph [AM]; Montay, Robert [AM];
*CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
Srini,
As discussed here is the plan, given that one datafile is missing from the
backup.
1. Take export of the CAMTR from the current production. (there could be issue
with this export. will come to know once the export is done)
2. Once step 1 is done, shutdown the Production and have the backup taken.
3. Have the currently restored database up and running with out the CAMTR
schema (due to missing datafile).
4. Have the restored database copied over to Production.
5. Re-start the production. Import the CAMTR schema from the export in step 1.
Given all the steps this will be time consuming and the database may not be
available on Monday. Please give me a
call if any you require any further information.
Thanks,
Aldrin
-----Original Message-----
From: Robinson, Jeffrey [AM]
Sent: Saturday, November 20, 2004 10:39 AM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
Working on getting discharged, so signing off.
Call me if you need anything.
Jeff
-----Original Message-----
From: Robinson, Jeffrey [AM]
Sent: Saturday, November 20, 2004 10:32 AM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
Good luck.
I have changed the code so this will not be a problem in the future, not that
it helps us now.
And we should still test it, just to be sure it is right.
Jeff
-----Original Message-----
From: Alberto, Aldrin [AM]
Sent: Saturday, November 20, 2004 10:28 AM
To: Robinson, Jeffrey [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
Let me see, if I can get the data from that file from the current cambarra
database .
Thanks
Aldrin
-----Original Message-----
From: Robinson, Jeffrey [AM]
Sent: Saturday, November 20, 2004 10:15 AM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
Bad news, file did not make it over (issue with the size and tar options).
What's next.
Jeff
-----Original Message-----
From: Alberto, Aldrin [AM]
Sent: Saturday, November 20, 2004 10:00 AM
To: Alberto, Aldrin [AM]; Robinson, Jeffrey [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
/oradata/camnabarra/CAMTR_DATA5.dbf
-----Original Message-----
From: Alberto, Aldrin [AM]
Sent: Saturday, November 20, 2004 9:54 AM
To: Robinson, Jeffrey [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
I have killed the process and mounted the restored database.
Jeff,
There is typo in one of the soft links in /oraindex.
The link should be 'camnabarra' and nor 'cambarra'. Please can you change as I
don't have permissions to change the link
Thanks
Aldrin
-----Original Message-----
From: Robinson, Jeffrey [AM]
Sent: Saturday, November 20, 2004 9:11 AM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
Performance may be slow because there are 1961 copies of /bin/ksh
/export/dbspace/cgamdev1/exp/run_exp.ksh running, could there be something
wrong with the script?
Jeff
-----Original Message-----
From: Robinson, Jeffrey [AM]
Sent: Saturday, November 20, 2004 8:40 AM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
Name: camprdsptr1.nj.ssmb.com
Address: 150.110.177.
Oracle is installed, Brendan use to check the production (db2) backup with this
server.
It is a very small box, 1 CPU (800MHZ) and 2 GB memory.
Jeff
-----Original Message-----
From: Alberto, Aldrin [AM]
Sent: Saturday, November 20, 2004 8:29 AM
To: Robinson, Jeffrey [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
Jeff,
On which box should I login to check the restore.
Please hold on today's backup.
Thanks
Aldrin
-----Original Message-----
From: Robinson, Jeffrey [AM]
Sent: Saturday, November 20, 2004 8:05 AM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
Files are back ad you can log onto the box and scsu to oracle.
Not sure if we'll have enough space for this and this weekends backup of
camnairtdb1, We'll have to keep an eye on the space and try to adjust as we go.
Also, what do you want to do with today's backup of Cambarra, set to start at
10AM?
Jeff
-----Original Message-----
From: Alberto, Aldrin [AM]
Sent: Saturday, November 20, 2004 7:48 AM
To: Robinson, Jeffrey [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
Jeff,
Please can you provide me the access to the SAN, once the database is restored
so than I can try and bring it up there and then do the
restore.
Thanks
Aldrin
-----Original Message-----
From: Robinson, Jeffrey [AM]
Sent: Friday, November 19, 2004 10:18 PM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
Good, my concern was someone made a type creating the list.
Everything is going, I'll check on it when the nurse wakes me up at 6AM to see
how I slept and if I want to wash up.
Jeff
-----Original Message-----
From: Alberto, Aldrin [AM]
Sent: Friday, November 19, 2004 10:00 PM
To: Robinson, Jeffrey [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
If those files are in the list, then they are the control files and should have
been backed up.
Aldrin
-----Original Message-----
From: Robinson, Jeffrey [AM]
Sent: Friday, November 19, 2004 9:56 PM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
I looked at the save list from last week (it is backed too) and the entries are
/orasys/camnabarracontrol01.ctl
/orasys/camnabarracontrol02.ctl
Is there an option if the backup is no good (this is the first time we are
testing it)
Jeff
-----Original Message-----
From: Alberto, Aldrin [AM]
Sent: Friday, November 19, 2004 9:48 PM
To: Robinson, Jeffrey [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
Jeff,
I did not quite understand the last step. The control files are in the list
that is generated in /var/cam/db
Thanks
Aldrin
-----Original Message-----
From: Robinson, Jeffrey [AM]
Sent: Friday, November 19, 2004 9:27 PM
To: Alberto, Aldrin [AM]
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]; *CAM UNIX Server Operations
Subject: RE: Cambarra Database restore
The compressed tar files are backed up to tape every week, to restore we need to
Restore from tape
uncompress files
setup dir to untar the files to
untar files.
Step 1 is done and working on step 2.
This may take most of the night.
Looking at the list of files, it looks like you were not saving the control
files
/orasys/camnabarracontrol01.ctl
/orasys/camnabarracontrol02.ctl
Heck of a way to test the backups!
Jeff
-----Original Message-----
From: Alberto, Aldrin [AM]
Sent: Friday, November 19, 2004 8:05 PM
To: Alberto, Aldrin [AM]; *CAM UNIX Server Operations
Cc: Vemparala, Srini [AM]; *CAM Dba Support Oracle; Costanza, Joseph
[AM]; Montay, Robert [AM]
Subject: RE: Cambarra Database restore
Matt,
I just spoke to George. According to him he as to restore the database from
tape to SAN. Please confirm.
Thanks,
Aldrin
-----Original Message-----
From: Alberto, Aldrin [AM]
Sent: Friday, November 19, 2004 7:50 PM
To: *CAM UNIX Server Operations
Subject: Cambarra Database restore
Matt,
It looks like the Cambarra database (machine camnadocudb1) is corrupted and may
need a restore from last week-end.
This database is backed up every week on to SAN. I would like to do the
following.
Get Access to the backup on SAN. Try to start up the database there
successfully. If database start there successfully restore
database to the machine (camnadocudb1 - IP address 169.188.0.75).
I can be reached at 201 218 6052 / 973 324 0727.
Thanks
Aldrin Alberto,
Retail Technology,
100 Stamford Place,Stamford.
(203) 961-6414
Aldrin.Alberto AT citigroup DOT com
|