Thank you all for your suggestions. I think I may have a better explanation now.

1)      600 million new files from the Isilon’s

2)      Turning on TIR for an existing policy that had moved to new storage for the images

3)      Improper exclusions on the backup master

After looking at folder sizes in /usr/openv/netbackup/db/images, I realized my backup server was damn near the same size as the two Isilon nodes. This seems way too large for me. After digging through some config’s I realized my existing ‘exclude’ file for the backup host was named to an old policy. In other words, I think the OS backup of the master server was including the catalog database its self. As it grew, so did the size of the master servers backup, as it was now including itself. The inclusion of the massive Isilon’s into this catalog, then backing that catalog up, resulting in an even larger catalog, became a huge snowball effect.

We do hot catalogs so there’s no need to also be backing up the catalog filesystem. I think this was the ultimate cause I was looking for.

Thanks again everyone.

That makes sense for the 600mil image files, and was sort of my guess, but what about the existing policy that I simply changed the location of the image from DSSU to a Data Domain? Would just that change still warrant a 40GB growth in the catalog over a 24 hour period? We were only backing up < 100 unix hosts, I would guess.

There is an entry per file in the catalog, a million files at 1k each will cause more catalog space than a thousand  1tb files

As of a few weeks ago we saw our NBU7 catalog explode from 80GB to a peak of 238GB. My initial belief was this was part of us standing up a 2^nd media server for the office which was linked to a Data Domain 630. But after reviewing the historical graphs, it is very clear this increase happened just a couple weeks ago. The obvious thing I started doing 2 weeks ago, is backing up our new Isilon cluster and also introduced these backups to a new Data Domain 670. More specifically, our Isilon which contains around 600 million image files that are less than 128k each.

How in the world could 10TB of image data suddenly cause the catalog to triple in size? Yet previously adding 18TB of data from a new media server (the dd630) didn’t even cause more than a 20GB increase in the catalog ?

I’m at quite a loss here to understand how this increase happened so quickly. The real issue here is that the catalog filesystem has filled up 3 times, causing nbpem to crash, and backups to grind to a halt. The last time this happened had nothing to do with the Isilon backups, but happened as a result of changing the media destination from some existing OS backups to go to a Data Domain 670, instead of their old DSSU locations. The isilon also points to this same DD670. Could that be the real issue here? Is there something different about how the catalog stores information when it resides on a Data Domain?