I figured I'd post this, as we recently performed a manual migration to new hardware, along with an upgrade from 5.4.2 to 6.3.1. The old TSM Server instance name matched the DNS name of the physical server. We did not want that to be the case moving forward, so we renamed the server (with the understanding that we would encounter authentication problems to some degree. One authentication problem cascaded into another, so I'm just going to cover the changes and verifications we performed, from server name change to option file configuration to service creation (and a couple items in between).

We run all of our backups in prompted mode (CAD and Scheduler) with generated passwords.

Easy method for repairing authentication after the rename:
In the TSM Administrative Console, update the PW for all nodes:
update nodename password

On each node:
Stop the Client Acceptor (CAD) Service and Scheduler Services

Verify that the dsm.opt file is configured properly (for CAD and Scheduler prompted backups with the new server name):
TCPSERVERADDRESS newservername or ip address
PASSWORDACCESS GENERATE
SCHEDMODE PROMPTED
MANAGEDSERVICES WEBCLIENT SCHEDULE
NODENAME nodename
TCPCLIENTADDRESS nodename or client ip address

Open Regedit (standard warning to use Regedit with care and to export any keys you are going to edit prior to making changes in the event you need to restore) and navigate to HKLM\Software\IBM\ADSM\Current Version\Nodes. You will see a key for the name of the old TSM Server. Delete this key.

Now launch dsmc.exe in a command prompt
login as the current node, using the new password you specified above on the server. Once logged in, run the following:
q node
Hit enter to accept current node
password


You will see the following output with informative results:
Node Name Platform Policy Domain Days Since Days Since Locked?

If that failed, update the password again and then try logging in again.

If everything is working properly, you can start the Client Acceptor Service, wait a few minutes, and see the following in your dsmwebcl.log and dsmsched.log files:

<timestamp> Next operation scheduled:
<timestamp> ------------------------------------------------------------
<timestamp> Schedule Name: <name of schedule>
<timestamp> Action: Incremental
<timestamp> Objects:
<timestamp> Options:
<timestamp> Server Window Start: <date and time of next start>
<timestamp> ------------------------------------------------------------
<timestamp> Scheduler has been stopped.


If you do not see the above, then there's a likely a problem with the services.

Open a command prompt and navigate to C:\Program Files\Tivoli\TSM\baclient
Run dsmcutil list

Note the names of the Client Acceptor Service and the Scheduler Service. If you are more well versed in the use of dsmcutil than I, then you can probably make the appropriate changes via dsmcutil update. As I was uncomfortable with that, I simply removed the services:
dsmcutil remove /name:"name of service"

At that point, I launched the Backup/Archive GUI on the node, and entered the Setup Wizard, creating new TSM Web Client and TSM Client Scheduler Services. When creating the Scheduler Service, be sure to check the box to use CAD. Once finished, again check your dsmwebcl.log and dsmsched.log files for the lines shown above.