There are a few things that you will need to check:

 you need to make sure that Oracle is using ADSM's orasbt.dll, not the
internal dll that Oracle shipped with.
   If you have a visual C++ compiler on the machine that Oracle is running, you
can use "process viewer" to check       this.  process viewer -> oracle803
->memory detail-> User Address Space for...
Then look for adsmv3.dll and orasbt.dll.  If you see these dll, you areThen 
look for adsmv3.dll and orasbt.dll.  If you see these dll, you are using
ADSM's dlls.

 RMAN writes out some trace/log info under
$ORACLE_HOME\rdbms80\trace\sbtio.log, sbtio.log must exist.  If it doesn't
exist, create one first.  Then run some backups and see anything is logged in
that file.

 also, check logs and traces under $ORACLE_HOME\database directory.  During
the backup, check to see whether any file like df_something is created or not.

 did you initialize the password?