TSM Reporting: Client Backup Usage by... department?

[wernercd]

So I'm exploring Tivoli Monitoring and finally got it setup and (so far) purring like a kitten...

Now it's time to explore data retention and reporting on questions I'm being asked.

The #1 question I have is: How much data is each "department" using?

I have ~1100 servers being backed up. Each server is "owned" by a department (Accounting, shipping, etc). Due to consolidation, the servers aren't named consistently and I don't expect that to change any time soon.

I need a way to say "This server is Dept#1... that server is Dept#2".

Then I want to say "Dept#1 is using 8TB of space... up from 7.5TB last month... trending at 500gb a month. At this rate, you are paying for 9TB and you'll need to increase that in 3 months" (paraphrasing what I assume is going to be different reports. The idea is I need to track usage by departments)

The catch is I'm not sure as too a good, recommended, way of breaking down servers in that fashion.

 

[moon-buddy]

Querying each node (server) using select statements to determine how much data is backed up on a daily, weekly or monthly basis is relatively easy.

The complexity is: how you can separate (and report correctly) real data from OS/system data. This may be harder unless real data has been isolated from OS/System data, i.e., on a separate disk drive or mount point.

Having said all this, and assuming the data contributed by the OS/System is negligible, you can do daily select statements to query backups captured from each node (server) and create graphs or charts and summarizing at end of each month.

 

[marclant]

The #1 question I have is: How much data is each "department" using?

I have ~1100 servers being backed up. Each server is "owned" by a department (Accounting, shipping, etc). Due to consolidation, the servers aren't named consistently and I don't expect that to change any time soon.

I need a way to say "This server is Dept#1... that server is Dept#2".

Depending on your current config and number of departments, is to use policy domains. If you currently only have one policy domain, then it's easy, you could create 1 policy domain by department. You do that by copying your current policy domain into a new one. Then update the domain value for each node into their respective department.

Then any command that can include the domain can take advantage of that. For example, checking schedules.

Another option would be to update the Contact field of the node if you are not using it. You could only use it in select queries though:

tsm: SERVER1>select occupancy.node_name,sum(reporting_mb) as MB from -
occupancy join nodes on occupancy.node_name=nodes.node_name where -
nodes.contact='Dept1' group by occupancy.node_name

NODE_NAME: NODE1
  MB: 588.51

NODE_NAME: NODE2
  MB: 0.00

Or for all departments:

tsm: SERVER1>select nodes.contact,sum(reporting_mb) as MB from occupancy join -
nodes on occupancy.node_name=nodes.node_name group by nodes.contact

CONTACT: Dept0
  MB: 0.00

CONTACT: Dept1
  MB: 588.51

CONTACT: Dept3
  MB: 60219.25

 

[marclant]

I re-read your post, if you want to be able to do the department break in Reporting and Monitoring, you will need to use policy domains to do so. It would be the only way to group similar nodes. If you have multiple policy domains, you may need to do something like:
- dept1_std
- dept1_tdpo
- dept1_tsmve
- dept2_std
- dept2_tdpsql
- dept2_7years

Then get clever with your reports to get combine that.

Here are the fields you can filter on:
https://www.ibm.com/developerworks/...torage Manager/page/Client Node Storage table

As you can see, Domain is likely the only one you can customize if you cannot rename the nodes with a department prefix.

 

[wernercd]

Querying each node (server) using select statements to determine how much data is backed up on a daily, weekly or monthly basis is relatively easy....

Yeah, my initial goal is simply raw space used. There is some deduplication (which might complicate things), but we do have servers where we don't back up the C: drive ("standard" is to have "important" stuff on D: and database backups on E:.. same issue: not 100% consistent which means that it's just easier to stay with complete drive space usage.

The question is how can I split ~1100 servers into ~20 departments so that I can report by Department for "billing" purposes.

Depending on your current config and number of departments, is to use policy domains. If you currently only have one policy domain, then it's easy, you could create 1 policy domain by department. You do that by copying your current policy domain into a new one. Then update the domain value for each node into their respective department.
Then any command that can include the domain can take advantage of that. For example, checking schedules.
Another option would be to update the Contact field of the node if you are not using it. You could only use it in select queries though:
...

Option #1: Domains
Option #2: Schedules
Option #3: Contact
Option #4: Email

Two main issues...

1) 3 TSM Servers - each with multiple domains (Production, Test, Development, etc), multiple schedules (DB, NonDB, VMDK etc) and each with slightly different setups (TSM 7.1.3/6.2.3, tape drive vs no tape drive, etc)
Issue #1: Domains/Schedules are already split up... going further down that rabbit hole is unrealistic. I like the idea from a higher idea, but I don't think it'll work in our current setup. I'm also low on the totem pole, so 'drastic changes' like that are way out of my per'vue.

2) VMWare, ESXI, VMDK level backups
Contact/Email - I had been circling that thought in my mind. It wouldn't be hard to add text like Contact: "John Doe - Owner:Accounting" and then through SQL Magic get the "Owner:Accounting" and thus "Accounting". I'd be willing to bite the bullet and update all the Nodes... except: VMDK level backups.

VMDK backups (1/3? 1/2? of the backups) are done via Vmware VMDK level backup. So one "Node" has ~300 "filespaces" and all of them would be tagged "John Doe, Owner:Accounting" which wouldn't be right.

The "department" tag can be done at the "Server" level for "Client Level" backups... but it needs to be done at the "Filespace" level for "VMDK Level" backups.

If I can tag Contact information for 1/2-2/3 of the servers, that could get me a long way... how would the Filespaces get tagged?

 

[marclant]

If I can tag Contact information for 1/2-2/3 of the servers, that could get me a long way... how would the Filespaces get tagged?

Can't tag them, there is no "free text" field for a filespace. Node is the lowest level you can use this trick for. Only thing I can think of, is rename the VMs (which doesn't mean renaming the hostname of the VM), but that means a full backup of the VM which you don't want to do.

 

[marclant]

If you can create a small DB somewhere (MS Access or the like). You could have a table with each node and filespace listed along with department. Then dump the occupancy to a file attached to that DB and do your cross-reference and report there. By your last post, sounds like you may have the skills to do that, but maybe not the time.

 

[wernercd]

Only thing I can think of, is rename the VMs (which doesn't mean renaming the hostname of the VM), but that means a full backup of the VM which you don't want to do.

Yeah, renaming them would be unrealistic (Servers with Archival schedules, servers with millions of files, TB sized disks, etc).

If you can create a small DB somewhere (MS Access or the like). You could have a table with each node and filespace listed along with department. Then dump the occupancy to a file attached to that DB and do your cross-reference and report there.

My thought process atm is moving in that direction: Creating a database schema.table to store ancillary information like this.

I'm going to be looking into adding this information into VMWare VCenter. There are a couple fields I think may work (Agency, System - text and not being used)... Also, there should be the ability to make custom attributes... This would give me a single point to add information and I'd also have access to the UUID to track the VM across changes, Guest OS information to verify OPTION SET matches, etc.

We do have Physical Servers I'll need to consider, but I think at this point the majority are by and large virtual.

I'll have to chew on this. At the least, I'll probably push the pertinent data into Warehouse since I want all the data in one place and then tackle reporting

By your last post, sounds like you may have the skills to do that, but maybe not the time.

I'm considered a jack-of-all (Server admin, programming, reporting, admin, etc) so I'm confident I can do everything . Time is a factor... but also staying within constraints and keeping it simple enough for others (and myself 6 months from now). It really needs to be balanced between "this information is invaluable moving forward" and "we need a system that is manageable done in a realistic time frame". To many moving parts. To much required knowledge. No one else trained. Documentation. Etc. Etc. Etc.