I do not know if any one has responded to you on this yet , but, It sounds
like your asking 2 separate questions

1.) How does TSm handle "open files"
2.) How to "Include" the "system object/system state objects" in the DSM.opt
file

Answer:
        #1.) As far as I know, TSM doesn't backup open files, I do believe
that you need to purchase a 3rd party open files agent. I think St. Bernard
has one but I have never used it myself.

        #2.) I have read from the list and brushed over it in the red book
and manuals that the system state can be placed in the dsm.opt file via
simple command line statements referencing the physical name of each system
object, (I.E. COM+, Active Directory, etc....)

Hope this helps, anyone else have input??