To move to a new hardware, you are essentially doing the same thing as a DR. So you would do the exact same thing as if your data center was destroyed today and had to recover your Spectrum Protect Server elsewhere.
https://www.ibm.com/support/knowledgecenter/SSGSG7_6.2.0/com.ibm.itsm.srv.doc/t_protect_srv.html
Only difference is that you won't need to restore the storage pools because you will still have your library and tapes.
Hi-level:
On original machine:
- make sure no client sessions and no processes are running to get a pristine database backup
- migrate your random disk pool to tape
- backup the database
- take a copy of these files: volume history file, device config file and dsmserv.opt
- halt the server
- turn off the machine
On the new server:
- install the same version of TSM that you had when you did the database backup
- physically connect your tape library to the new server (wiring, drivers, etc.)
- create a new instance with similar structure of old system as far as database and log directories goes, that's assuming the config was good. Now would be a good time to adjust the config to best practices if mistakes were done with the original one:
https://www.ibm.com/support/knowledgecenter/SSGSG7_7.1.6/perf/t_optim_config.html
- restore the database
- fix the paths for the drives, as the device names will be different