Veritas-bu

Re: [Veritas-bu] Error status 129 - DiskStaging Unit is full -NetBackup 6.0

2007-10-30 12:15:59
Subject: Re: [Veritas-bu] Error status 129 - DiskStaging Unit is full -NetBackup 6.0
From: Rongsheng Fang <unixlifebox AT gmail DOT com>
To: VERITAS-BU AT mailman.eng.auburn DOT edu
Date: Tue, 30 Oct 2007 10:56:06 -0400
This is a instance of defect? It looks like to me 6.0 MP5 already does cleanup of old images when the DSU gets filled up during the backup process. I just had one DSU filled up last night and here are the logs I captured: 

The backup job log from the Admin console:

10/29/2007 22:18:20 - begin writing
10/29/2007 22:19:05 - Warning bpdm (pid=5419) storage unit ee03dbp-disk47 is full: processing disk full condition
10/29/2007 22:19:22 - Critical bpdm (pid=5419) storage unit /data/bk47 is full
10/29/2007 22:19:22 - Critical bpdm (pid=5419) bp_sts_open_image failed: error 31
10/29/2007 22:19:23 - end writing; write time: 0:01:03
Disk storage unit is full (129)

Here is the bpdm log showing it was trying to clean up old images:

22:19:05.159 [5419] <4> write_backup: successfully wrote backup id rptprd-data-vip.prod.advertising.com_1193710548, copy
1, fragment 4, 2000128 Kbytes at 49402.954 Kbytes/sec
22:19:05.175 [5419] <16> ee03dbp-data: STSBasicDisk: Caught exception in STSBasicDisk::CreateImage, FILE=STSBasicDiskIma
geInfo.cpp, FUNCTION=, LINE=260, ERROR=STS_ENOSPC
22:19:05.175 [5419] <32> bp_sts_open_image: sts_create_image failed: error 31
22:19:05.176 [5419] <4> db_getSTUNIT: EMM interface already initialized (using cached master name s01-inf-bkpvip.prod.ro
ot).
22:19:05.279 [5419] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2034: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
22:19:05.280 [5419] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: service: bpdbm
22:19:05.284 [5419] <2> logconnections: BPDBM CONNECT FROM 192.168.9.107.61767 TO 10.100.9.50.13724
22:19:05.599 [5419] <8> process_diskfull: storage unit ee03dbp-disk47 is full: processing disk full condition
22:19:05.614 [5419] <2> make_space_STANDARD: Acquired exclusive use of bpdm disk cleanup lock (ee03dbp-disk47 1092188)
22:19:05.620 [5419] <2> ds_logger: libsts openp() 07/10/29 22:19:05: opening module /usr/openv/lib/libstspibasicdisk.so

22:19:07.269 [10540] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2034: VN_REQUEST_SERVICE_SOCKET: 6 0x00000006
22:19:07.270 [10540] <2> vnet_vnetd_service_socket: vnet_vnetd.c.2048: service: bpdbm
22:19:07.274 [10540] <2> logconnections: BPDBM CONNECT FROM 192.168.9.107.61775 TO 10.100.9.50.13724
22:19:07.353 [5419] <4> ds_logger: found 0 of 523 images
22:19:07.359 [5419] <2> bpdm: firstloop, no candidates, no delay on lock --- returning disk full
22:19:07.360 [5419] <2> bpdm: updating EMM with FINAL freespace = 7855583232
22:19:07.360 [5419] <4> db_modifySTUNIT: EMM interface already initialized (using cached master name s01-inf-bkpvip.prod
.root).
22:19:07.425 [5419] <2> db_modifySTUNIT:
22:19:07.426 [5419] <2> make_space_STANDARD: Released bpdm disk cleanup lock (ee03dbp-disk47 1092188)
22:19:07.427 [5419] <2> process_diskfull: disk is full, calling diskfull_notify script
22:19:07.427 [5419] <2> notify_call: executing - /usr/openv/netbackup/bin/diskfull_notify bpdm /data/bk47/rptprd-data-vi
p.prod.advertising.com_1193710548_C1_F5 </dev/null >/dev/null 2>/dev/null
22:19:07.496 [5419] <2> diskfull_corral: DSU IS FULL (0KB written)
22:19:07.516 [5419] <2> diskfull_corral: Released diskfull cleanup lock   


 Rongsheng

On Oct 30, 2007, at 10:17 AM, Steve Fogarty wrote:

I had the same issue.  This is what I got from my SUN/Symantic Support:

"I did hear back from Symantec. The level 4 engineer at Symantec identified the issue as an instance of a defect that will be resolved in 6.0 MP6.

I've filed Sun Change Request (CR) 6623137 fo document this problem.

The Symantec issue was described as, "backup jobs to disk would end in a status 129 when the disk was full." The DSSU's would fill up, the backups writing to it would exit with a status 129, then the images would subsequently be cleaned up and the backups run fine.
 
The fixes (to bptm and bpdm) should pause backups writing to full dssu's, clean up old images and then resume backups.

The tentative expectation is that 6.0 MP6 will be available in January to February, 2008.

Symantec has tested a hot fix on a Linux platform at a customer site, and can generate Solaris binaries, if you require a hot fix.

I explained to our contact at Symantec that my greatest concern was a misidentification of the issue, as there has been difficulty explaining this issue up till now. She has reviewed the logs and other data provided from your site, and is confident that this issue the issue they addressed for one or two other sites and documented as the bug cited here."

Steve

On 10/30/07, Martin, Jonathan <JMARTI05 AT intersil DOT com> wrote:
Nothing worse than backing up 800GB of data and having it fail at 799GB
because of low / no disk space.  If you enable checkpoint backups it
will retain the backed up bit back to the last 15 or 30 minutes or
whatever you configure.  That way you can restart the backup after you
clear disk space.  Its better than starting over from scratch.

-Jonathan

-----Original Message-----
From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of osonder
Sent: Tuesday, October 30, 2007 9:12 AM
To: VERITAS-BU AT mailman.eng.auburn DOT edu
Subject: [Veritas-bu] Error status 129 - DiskStaging Unit is full
-NetBackup 6.0


Hello Jonathan

You may have a point. Because this happens in the middle of the night.
So I'm not here when this happens. I just looked at the DSSU at the
morning and noticed a free space of 325Gb.

+----------------------------------------------------------------------
|This was sent by oson AT fredrikstad.kommune DOT no via Backup Central.
|Forward SPAM to abuse AT backupcentral DOT com.
+----------------------------------------------------------------------


_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

_______________________________________________

_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu