Compression corrupts small jpg files ?!?

[Moebius]

I just noticed a weird behavior with TSM 5.5.1.0 Windows BA client.

If client compression is turned on (compression yes in dsm.opt), jpg files with size smaller than approx 18 KB will be backed up corrupted. They appear ok in the database with their correct size, but restoring them generate null size files (0 KB).

With compression turned off (compression NO in dsm.opt) the same files back up and restore without problems.
Also, it seems that jpg files larger than approx 18 KB are not affected by this problem.

I tried with TSM 5.4.1.0 BA client and it works fine. I also tried backing up other types of compressed files (zip files) with 5.5.1.0 and they backed up and restored fine.

The server is ver. 5.5.1.0. The target storage pool is a disk type, so no hardware compression is involved. I can reproduce the problem every time.

This looks as a very bad bug. How could this go unnoticed? A forum search on this subject gave no results.

Also, can anyone suggest a way to repair the thousands of jpg files that are in my storage pools? I run incremental backups. Changing the compression option for the jpg files (e.g. with the exclude.compression statement) and running another backup won't touch these files as they haven't changed. Forcing full backups would need too much space. I can't figure a way to "refresh" the backed up, corrupted jpg files. Help please!

 

[Moebius]

A further search spotted the problem. This can happen when compression=yes and compressalways=no are in effect, the file is small and grows during compression.
The bug has been addressed with APAR IC57348 (http://www.ibm.com/support/docview.wss?uid=swg1IC57348) and reportedly fixed with interim fix 5.5.1.2.

However, I'm still looking for a way to repair the thousands of 0-byte jpg files that are stored in the pools. Any hint would be appreciated.

 

[ofernandez]

I'm fairly new to TSM (8 months give or take), but that does sound like a huge problem. Have you tried any other image/sound files types around that same size range? Just curious.

 

[wim]

You can't "repair" the files. The only possibility I see is to backup these files again. If syou say that a full takes too much pace, the only possibility I see is a selective backup of these jog files.

 

[Moebius]

have you tried any other image/sound files types around that same size range? Just curious.

I only tried with small size .zip files (besides the .jpg, that is) and the problem did not show. However, according to the cited APAR, the problem may arise with any file whose size increases during the compression.

Another description of this bug is here: http://www-01.ibm.com/support/docview.wss?uid=swg21322625

 

[Moebius]

Ok, so there are both a workaround and a fix for this issue.

Now my problem is that there will probably be many corrupted jpeg's (and possibly other file types) scattered in the TSM storage pools, with no way to find them nor to tell them from the "good" copies.

I have been backing up some 25 servers for years and at some time this year I have upgraded the clients to 5.5.1.0. Both my disk pools and my library are almost full, so I can't afford to run a full backup of the whole system.

Even narrowing the selective backup to the jpeg's only would mean adding too many GB's to the storage. Typically the jpegs do not change and they should mostly be present in one single version in the TSM storage. Running a selective of the jpeg's would add another version, and the older one(s) would stay there for good. This is true not only for the small files that are affected by the issue, but also, unfortunately, for the larger ones.

I considered creating a new, single-version mgmt class for the jpegs and then running the selective, but this (a) would not solve the problem for other file types that could possibly be affected as well by this issue, and (b) would mean to lose the older versions in case some jpeg file has been purposedly modified during its life.

The solution would be of course to spot the "bad" copies and to reback up only the relevant files, but apparently there is no way to do that, since the corrupted copies that would restore to 0 B appear with their original dimensions in the TSM database.

Can anybody come up with a bright idea?

 

[moon-buddy]

It looks like your hands are tied here.

As already mentioned, I think the best way is to backup the files again but this time via a script that will only backup JPG files that are less than 18 kb in size (provided that you are sure that files bigger than 18 kb are NOT really affected). The script would fairly be easy to do if the source is a UNIX type environment but maybe a little challenging on a Windows server.

 

[Phil_02GT]

The script would fairly be easy to do if the source is a UNIX type environment but maybe a little challenging on a Windows server.

I admit that using dos it would be rather difficult but if you use powershell just use this (on one line):

get-childitem -recurse -include *.jpg,*.gif C:\* | where-object {$_.length -gt 18432 } | ft -property fullname > c:\filelist.txt

You can add as many file extension as you need as long as you add a comma in between them.
I think you should be able to use the filelist in TSM

 

[dicksharp]

I just noticed a weird behavior with TSM 5.5.1.0 Windows BA client.

If client compression is turned on (compression yes in dsm.opt), jpg files with size smaller than approx 18 KB will be backed up corrupted. They appear ok in the database with their correct size, but restoring them generate null size files (0 KB).

With compression turned off (compression NO in dsm.opt) the same files back up and restore without problems.
Also, it seems that jpg files larger than approx 18 KB are not affected by this problem.

I tried with TSM 5.4.1.0 BA client and it works fine. I also tried backing up other types of compressed files (zip files) with 5.5.1.0 and they backed up and restored fine.

The server is ver. 5.5.1.0. The target storage pool is a disk type, so no hardware compression is involved. I can reproduce the problem every time.

This looks as a very bad bug. How could this go unnoticed? A forum search on this subject gave no results.

Also, can anyone suggest a way to repair the thousands of jpg files that are in my storage pools? I run incremental backups. Changing the compression option for the jpg files (e.g. with the exclude.compression statement) and running another backup won't touch these files as they haven't changed. Forcing full backups would need too much space. I can't figure a way to "refresh" the backed up, corrupted jpg files. Help please!

If your important Jpeg or Jpg file has become corrupt or damaged then in such cases opt for contemporary <a href="http://www.jpegrecovery.biz" target="_blank">Jpeg recovery</a> software. With the use of application, you can restore the file with ease and less effort.