Veritas-bu

Re: [Veritas-bu] bpclntcmd and others ignoring nsswitch.conf?-RESOLVED (yes actually resolved this time)

2011-05-03 14:12:54
Subject: Re: [Veritas-bu] bpclntcmd and others ignoring nsswitch.conf?-RESOLVED (yes actually resolved this time)
From: Girish Jorapurkar <girishsj AT yahoo DOT com>
To: JeffLightner <JLightner AT water DOT com>
Date: Tue, 3 May 2011 11:12:46 -0700 (PDT)
Jeff,

Great! Yes, NB 7.1 (starting with 7.0.1) has started using getaddrinfo (IPv6 
friendly API) on all platforms (not just HP-UX). So, it is somewhat surprising 
that you observed this only on HP-UX. Perhaps, you had ipnodes setting in 
nsswitch.conf on other platforms.

Anyways, I'll get your findings documented in NB tech note; so that other NB 
users can benefit from it - Thanks to you!

Regards,
/Girish

--- On Tue, 5/3/11, Lightner, Jeff <JLightner AT water DOT com> wrote:

> From: Lightner, Jeff <JLightner AT water DOT com>
> Subject: RE: [Veritas-bu] bpclntcmd and others ignoring 
> nsswitch.conf?-RESOLVED (yes actually resolved this time)
> To: "Girish Jorapurkar" <girishsj AT yahoo DOT com>
> Cc: veritas-bu AT mailman.eng.auburn DOT edu
> Date: Tuesday, May 3, 2011, 10:58 PM
> The patch (PHCO_30030) has been
> superseded more than once and we have one of the superseding
> patches PHCO_37369 so would already have the "bug fix" the
> poster at the link mentioned.   
> 
> However, I'd like to thank you as your pointer helped me
> resolve the issue:
> 1)  By downloading and compiling his source code I was
> able to see that I did have the same issue with his binary
> as that poster did.   (That is to say this
> binary like the NetBackup binaries did not resolve from
> /etc/hosts first but rather from DNS.)
> 2)  On reviewing the
> /var/adm/sw/products/PHCO_37369/pfiles/README file that came
> with the later patch I saw the following (which is from
> another patch this one supersedes so is incorporated):
>         PHCO_29287:
>         ( QX:QXCR1000523428
> SR:8606308071 CR:JAGae71107 )
>         If IPv6 is enabled on a system,
> the Internet Services
>         commands use the APIs
> getipnodeXX() or get*info() to
>         resolve the name instead of the
> APIs gethostbyXX().
>         The APIs getipnodeXX() and
> get*info() look into the
>         ipnodes directive of the
> /etc/nsswitch.conf to resolve
>         the name/address.  As the
> NIS/NIS+ backends do not
>         respond to the ipnodes
> directive, the resolution of
>         IPv4 addresses fails for NIS
> and NIS+ sources.
> 
> While we aren't doing anything with NIS/NIS+ the other info
> about IPv6 was informative.  Turns out that in addition
> to the standard "hosts:" line in /etc/nsswitch.conf HP-UX is
> expecting an "ipnodes:" line for binaries that use the above
> APIs.   
> 
> After modifying /etc/nsswtich.conf so it now has both the
> following lines:
> hosts: files [notfound=CONTINUE] dns
> ipnodes: files [notfound=CONTINUE] dns
> 
> The new binary resolved correctly from /etc/hosts and on
> testing so did bpclntcmd.   Based on that I
> restarted the backup that had been going across the primary
> LAN previously and it is currently running on the expected
> backup LAN.
> 
> Presumably this means the 7.1 NetBackup binaries are using
> the above referenced APIs on HP-UX where the 6.5.4 binaries
> didn't.   
> 
> We never saw this issue on HP-UX up through version 6.5.4
> and in testing yesterday I'd copied the 6.5.4 binary and its
> associated libraries to another directory on my HP-UX host
> and it resolved from /etc/hosts.
> 
> I added the ipnodes entry to /etc/nsswitch.conf on the
> other HP-UX 11.11 media server and the HP-UX 11.23 client
> I'd recreated the error on previously.   On
> adding the line both hosts correctly resolved the backup LAN
> IP from /etc/hosts instead of going to DNS.
> 
> FYI:  ITRC is going away so I'm not sure if the link
> will still be accessible in the
> future.   Accordingly I'm including the
> source code that poster had made so it is available from
> this forum.
> 
> His source code file was called:  114249.c and I
> compiled it by running:
> "cc -o testresolver 114249.c".   Usage is
> simply "testresolver <hostname>".   The
> source code is as follows:
> 
> /* Test program to test IP resolution for domain names
>  */
> 
> 
> #include <stdio.h>
> #include <sys/types.h>
> #include <sys/socket.h>
> #include <arpa/inet.h>
> #include <netdb.h>
> #include <string.h>
> 
> 
> /* Function to get IP address for hostname
>  ** hostname - (i/p) - Hostname to find IP address
>  ** buffer   - (o/p) - Buffer filled with IP
> address
>  ** buffer_len - (i/p) - Length of buffer
>  ** Return APR_SUCCESS on success
>  **/
> int get_ip_from_hostname(char *hostname)
> {
>     int status = 0;
>     struct addrinfo hints, *ai, *ai_list;
>     char host_ip[INET_ADDRSTRLEN];
>     const char *ret = NULL;
> 
>     memset(&hints, 0, sizeof(hints));
>     hints.ai_family = AF_INET;
>     hints.ai_socktype = SOCK_STREAM;
>     status = getaddrinfo(hostname, NULL,
> &hints, &ai_list);
>     if (status)
>     {
>         printf("Failed to resolve IP
> address for %s \n ", hostname);
>         printf("Error: %s \n",
> gai_strerror(status));
>         return status;
>     }
> 
>     ai = ai_list;
>     if(ai)
>     {
>         struct sockaddr_in *in =
> (struct sockaddr_in *)ai->ai_addr;
>         host_ip[0] = '\0';
>         ret = inet_ntop(AF_INET,
> &in->sin_addr, host_ip, sizeof(host_ip));
>     }
>     freeaddrinfo(ai_list);
> 
>     if(host_ip)
>     {
>         printf("Hostname \"%s\"
> resolves to IP \"%s\" \n", hostname, host_ip);
>         status = 0;
>     }
>     else
>     {
>         printf("Failed to resolve IP
> address for %s \n", hostname);
>         status = -1;
>     }
> 
>     return status;
> }
> 
> void usage()
> {
> 
>     printf("resolve_hostname <hostname>
> \n");
> 
>     return;
> }
> 
> int main(int argc, char *argv[])
> {
> 
>     char hostname[1024];
> 
>     if(argc<2)
>     {
>         usage();
>         return 0;
>     }
> 
>     strcpy(hostname, argv[1]);
>     get_ip_from_hostname(hostname);
> 
> 
>     return 0;
> }
> 
> 
> -----Original Message-----
> From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
> [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu]
> On Behalf Of Girish Jorapurkar
> Sent: Monday, May 02, 2011 5:02 PM
> To: Patrick; Lightner, Jeff
> Cc: veritas-bu AT mailman.eng.auburn DOT edu
> Subject: Re: [Veritas-bu] bpclntcmd and others ignoring
> nsswitch.conf?-SPOKE TO SOON
> 
> This seems to be a HP-UX bug (getaddrinfo API). Here is
> link describing it:
> 
> https://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1304369816045+28353475&threadId=614359
> 
> > PHCO_30030 fixed the problem
> 
> Let me know if the patch resolves the issue.
> 
> Thanks,
> /Girish
> 
> --- On Mon, 5/2/11, Lightner, Jeff <jlightner AT water DOT com>
> wrote:
> 
> > From: Lightner, Jeff <jlightner AT water DOT com>
> > Subject: Re: [Veritas-bu] bpclntcmd and others
> ignoring nsswitch.conf? -SPOKE TO SOON
> > To: "Patrick" <netbackup AT whelan-consulting.co DOT uk>
> > Cc: veritas-bu AT mailman.eng.auburn DOT edu
> > Date: Monday, May 2, 2011, 9:25 PM
> > The answer is both commands "work" in
> > that they both run.   However, both
> commands
> > are using the IP from DNS to make the connection
> rather than
> > the one from /etc/hosts.   Therefore
> the only
> > IP I ever see in the output is the main LAN IP rather
> than
> > the one for the backup LAN that is in /etc/hosts.
> > 
> > Some testing was done after adding an entry to
> /etc/hosts
> > that doesn't exist in DNS and I found the command
> resolved
> > the correct (backup LAN) IP.   Also
> > afterwards the new entry was commented out of
> /etc/hosts and
> > bpclntcmd -clear_host_cache and it did NOT find
> anything in
> > cache.   This to me suggests that the
> issue
> > is that the NetBackup utilities are looking at
> /etc/hosts
> > but only after they can't find it in DNS but this is
> the
> > opposite of the way nsswitch.conf is configured.
> > 
> > I do not see an issue resolving the backup LAN IPs
> from
> > /etc/hosts from either the RHEL6 master or the Windows
> 2003
> > client.    
> > 
> > Other tests show me that I can repeat this bad
> behavior on
> > another HP-UX 11.11 media server and an HP-UX 11.23
> > client.    Since host level utilities
> (nslookup
> > and ping) return the correct address from /etc/hosts I
> know
> > that nsswitch.conf is working on
> HP-UX.   
> > 
> > To me it appears the issue is the NetBackup binaries
> all
> > have the same flaw on HP-UX.   This
> suggests
> > to me there is a bug in a common object or library
> that is
> > used for the host name resolution used by all these
> > NetBackup binaries but that it is only on the HP-UX
> version
> > of NBU 7.1.
> > 
> > -----Original Message-----
> > From: Patrick [mailto:netbackup AT whelan-consulting.co DOT uk]
> > 
> > Sent: Friday, April 29, 2011 10:00 AM
> > To: Lightner, Jeff
> > Cc: veritas-bu AT mailman.eng.auburn DOT edu
> > Subject: RE: [Veritas-bu] bpclntcmd and others
> ignoring
> > nsswitch.conf? -SPOKE TO SOON
> > 
> > Have you tried running bptestbpcd -client <client
> > name> -verbose -debug from
> > the media server? What are the results? Does
> bpgetconfig -M
> > <client name>
> > work?
> > 
> > Regards,
>
> > Patrick Whelan
> > VERITAS Certified NetBackup Support Engineer for
> UNIX.
> > VERITAS Certified NetBackup Support Engineer for
> Windows.
> > 
> > netbackup AT whelan-consulting.co DOT uk
> > 
> > 
> > -----Original Message-----
> > From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
> > [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu]
> > On Behalf Of Lightner,
> > Jeff
> > Sent: 29 April 2011 13:41
> > To: Rosie Cleary
> > Cc: veritas-bu AT mailman.eng.auburn DOT edu
> > Subject: Re: [Veritas-bu] bpclntcmd and others
> ignoring
> > nsswitch.conf?
> > -SPOKE TO SOON
> > 
> > We actually do that for most clients but this is a MS
> > Exchange cluster name
> > and we didn't need to use a separate name for this
> cluster
> > in 6.5.4.  To me
> > this seems like a bug in the NetBackup tools since I
> can
> > demonstrate OS
> > level tools resolve the expected IP and it is only
> the
> > NetBackup tools that are getting the wrong
> > name.   Also as I noted
> > before this is only happening on my HP-UX media server
> - my
> > master server is
> > getting the correct IP in bpclntcmd for the client.
> > 
> > -----Original Message-----
> > From: Rosie Cleary [mailto:Rosie.Cleary AT nuim DOT ie]
> > Sent: Friday, April 29, 2011 5:06 AM
> > To: Lightner, Jeff
> > Cc: veritas-bu AT mailman.eng.auburn DOT edu
> > Subject: Re: [Veritas-bu] bpclntcmd and others
> ignoring
> > nsswitch.conf?
> > -SPOKE TO SOON
> > 
> > Hi Jeff,
> > 
> > I've just started using dnsmasq on Red Hat Linux which
> acts
> > as a dns
> > forwarder but allows me to override certain IP
> addresses.
> > It's easy to
> > configure and will get around this problem.
> > 
> > Other than that is it out of the question to use a
> slightly
> > different name
> > for client interface that is in the backup network?
> (e.g. 
> > servername-b) You would then put servername-b rather
> than
> > servername in the
> > policy and the traffic would automatically route over
> the
> > backup network.
> > I've used this for all clients in my backup network
> from
> > day one and it's
> > been fine. The disadvantages that I can think of for
> you
> > are
> >   - The name of the client in reports
> and for searches
> > is different from
> > 
> > the normal client name, and if it only applies to one
> or
> > two servers then it
> > will be confusing.
> >   - Backups taken before the change
> will be considered
> > by the server to be
> > of a different client so there are a few extra steps
> to
> > restore from these.
> > 
> > Best regards,
> > 
> > Rosie.
> > 
> > Rosie Cleary
> > Computer Centre
> > National University of Ireland, Maynooth
> > 
> > 
> > Lightner, Jeff  wrote [28/04/2011 20:53]:
> > > Sorry folks - NOT resolved.
> > >
> > > I thought this was resolved because the backup
> started
> > but on checking
> > I
> > > see it is using the primary LAN rather than the
> backup
> > LAN. The
> > addition
> > > of the FQDN on the client did get us past the 59
> error
> > but didn't fix 
> > > the issue I was asking about initially.
> > >
> > > The bpclntcmd is still showing the 10.x primary
> IP
> > instead of the
> > 172.x
> > > backup IP that I have in host file of the media
> > server. We really need 
> > > this backup to go across the backup LAN.
> > >
> > >
> >
> ------------------------------------------------------------------------
> > >
> > > *From:*veritas-bu-bounces AT mailman.eng.auburn DOT edu
> > > [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu]
> > *On Behalf Of 
> > > *Lightner, Jeff
> > > *Sent:* Thursday, April 28, 2011 3:11 PM
> > > *To:* veritas-bu AT mailman.eng.auburn DOT edu
> > > *Subject:* Re: [Veritas-bu] bpclntcmd and others
> > ignoring
> > nsswitch.conf?
> > > -RESOLVED
> > >
> > > *Dan Otto had responded and based on what he
> wrote I
> > resolved the
> > issue.
> > > The below shows the thread between us and is
> reposted
> > here with his
> > > permission.*
> > >
> > > **
> > >
> > > *From:*Lightner, Jeff [mailto:jlightner AT water DOT com]
> > > *Sent:* Thursday, April 28, 2011 2:02 PM
> > > *To:* Daniel Otto
> > > *Subject:* RE: bpclntcmd and others ignoring
> > nsswitch.conf?
> > >
> > > That was it.
> > >
> > > After checking bpcd log on the client we saw that
> it
> > was complaining 
> > > that the FQDN name wasn't a media server. Our
> entry
> > for the server was 
> > > the short name for the media server. Adding the
> FQDN
> > to the line that 
> > > had the backup LAN IP and short name resolved
> the
> > issue.
> > >
> > > It just didn't occur to me to look at the client
> > because I thought 
> > > bpclntcmd was simply trying to resolve from the
> media
> > server.
> > >
> > > I had actually tried adding FQDN of the client to
> the
> > media server 
> > > earlier because we have seen various issues
> regarding
> > short name vs
> > FQDN
> > > since implementing 7.1.
> > >
> > > Thanks for your help.
> > >
> > >
> >
> ------------------------------------------------------------------------
> > >
> > > *From:*Daniel Otto [mailto:dan_otto AT symantec DOT com]
> > > *Sent:* Thursday, April 28, 2011 2:57 PM
> > > *To:* Lightner, Jeff
> > > *Subject:* RE: bpclntcmd and others ignoring
> > nsswitch.conf?
> > >
> > > The 59 is thrown because whatever server hostname
> the
> > client is 
> > > resolving doesn't exist in the client's server
> list
> > hence server
> > access
> > > denied status 59 and should show up as a status
> 46
> > error in bpcd as a 
> > > invalid server. If the media server couldn't
> resolve
> > the client at all 
> > > or getting the wrong IP address you would be
> getting
> > 58/25 or even
> > 54's
> > > type of errors.
> > >
> > >
> >
> ------------------------------------------------------------------------
> > >
> > > *From:*Lightner, Jeff [mailto:jlightner AT water DOT com]
> > > *Sent:* Thursday, April 28, 2011 1:43 PM
> > > *To:* Daniel Otto
> > > *Subject:* RE: bpclntcmd and others ignoring
> > nsswitch.conf?
> > >
> > > I did actually try removing dns from
> nsswitch.conf but
> > it didn't help.
> > >
> > > I haven't looked at the client's bpcd log - I
> did
> > verify the media 
> > > server is in the server list for the client. The
> only
> > other one in the 
> > > list is the master.
> > >
> > >
> >
> ------------------------------------------------------------------------
> > >
> > > *From:*Daniel Otto [mailto:dan_otto AT symantec DOT com]
> > > *Sent:* Thursday, April 28, 2011 2:35 PM
> > > *To:* Lightner, Jeff
> > > *Subject:* RE: bpclntcmd and others ignoring
> > nsswitch.conf?
> > >
> > > Easy fix would be to use the bpcd log on the
> client
> > and whatever 
> > > hostname is getting resolved simply add it to
> the
> > SERVER = on the
> > client
> > > and the 59 should go away. As to why it is not
> using
> > the host 
> > > file...that's interesting.
> > >
> > > Perhaps for a quick test remove the DNS entry
> > altogether from 
> > > switch.conf to see if it then goes to the host
> file.
> > Though I have
> > only
> > > seen Solaris server do funkly things such as
> this.
> > >
> > > Hope this helps,
> > >
> > > Dan O
> > >
> > >
> >
> ------------------------------------------------------------------------
> > >
> > > *From:*veritas-bu-bounces AT mailman.eng.auburn DOT edu
> > > [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu]
> > *On Behalf Of 
> > > *Lightner, Jeff
> > > *Sent:* Thursday, April 28, 2011 2:22 PM
> > > *To:* veritas-bu AT mailman.eng.auburn DOT edu
> > > *Subject:* [Veritas-bu] bpclntcmd and others
> ignoring
> > nsswitch.conf?
> > >
> > > On a UNIX (HP-UX) media server when I run
> bpclntcmd
> > -hn <client> (and 
> > > other bp commands) it appears they're ignoring
> the IP
> > specified for
> > the
> > > client in /etc/hosts.
> > >
> > > The server also exists in DNS but my
> nsswitch.conf
> > clearly says to use 
> > > files (i.e. /etc/hosts) then dns. If I run
> command
> > line tests using 
> > > other tools (e.g. nslookup and ping) the IP seen
> for
> > the client is the 
> > > one in /etc/hosts. However, bpclntcmd is giving
> me the
> > one from DNS.
> > The
> > > IPs are different because we have use a separate
> > subnet for backup
> > LAN.
> > >
> > > I've done a fair amount of searching and have
> read
> > this technote:
> > >
> > >
> > http://www.symantec.com/business/support/index?page=content&id=TECH14111
> > 7&key=15143&actp=LIST
> > >
> > <http://www.symantec.com/business/support/index?page=content&id=TECH1411
> > 17&key=15143&actp=LIST>
> > >
> > > I had actually rebooted the server for another
> reason
> > about an hour
> > and
> > > a half after I changed /etc/hosts yesterday.
> According
> > to all the
> > notes
> > > I found the cache should clear itself about an
> hour
> > after making a 
> > > change but that didn't happen.
> > >
> > > This morning I still saw the issue. I tried
> reissuing
> > the bpclntcmd 
> > > -clear_host_cache (new in 7.x) and bouncing
> NetBackup
> > on the media 
> > > server to no avail.
> > >
> > > Based on the above technote I have tried moving
> the
> > cache directory
> > out
> > > and bouncing NetBackup again to no avail.
> > >
> > > I also tried just for the hell of it stopping
> the
> > pbx_exchange daemon 
> > > and restarting then bouncing NetBackup but that
> didn't
> > help either.
> > >
> > > Interestingly bpclntcmd -hn <client> on
> the
> > RHEL6 Linux master returns 
> > > the correct IP from /etc/hosts file there.
> Running
> > bpclntcmd on the 
> > > client itself with various flags (e.g. -pn,
> -self)
> > find the correct 
> > > master etc...
> > >
> > > My backup is failing with a status 59 - can't
> connect
> > to client and
> > I'm
> > > assuming it is because the media server is
> getting the
> > wrong IP for
> > the
> > > client. (The client has an entry in its hosts
> file
> > specifying the
> > backup
> > > LAN IP of the media server - this is the same as
> it
> > was in 6.5.4
> > except
> > > we were using a Windows media server then.)
> > >
> > > P.S. On HP-UX nslookup DOES look at /etc/hosts
> unlike
> > nslookup on
> > other
> > > *nix flavors so please don't respond that using
> > nslookup for
> > /etc/hosts
> > > isn't valid. As noted ping also works so such an
> > observation wouldn't
> > be
> > > helpful even if it weren't incorrect. The issue
> isn't
> > host name 
> > > resolution in UNIX but rather hostname resolution
> done
> > by NetBackup 
> > > commands.
> > >
> > >
> >
> *_______________________________________________________________________
> > ___________________*
> > >
> > > *Jeff Lightner*****| *UNIX/Linux
> Administrator*****|
> > **DS Waters of 
> > > America, Inc **| 5660 New Northside Drive, Ste
> 250 |
> > Atlanta, GA
> > 30328**
> > > (:(Direct Dial)678-486-3516|(:(Cell)678-772-0018
> |
> > *:jlightner AT water DOT com
> > >
> > > Proud partner. Susan G. Komen for the Cure.
> > >
> > > //Please consider our environment before printing
> this
> > e-mail or 
> > > attachments.//
> > >
> > > ----------------------------------
> > > CONFIDENTIALITY NOTICE: This e-mail may contain
> > privileged or 
> > > confidential information and is for the sole use
> of
> > the intended 
> > > recipient(s). If you are not the intended
> recipient,
> > any disclosure, 
> > > copying, distribution, or use of the contents of
> this
> > information is 
> > > prohibited and may be unlawful. If you have
> received
> > this electronic 
> > > transmission in error, please reply immediately
> to the
> > sender that you 
> > > have received the message in error, and delete
> it.
> > Thank you.
> > > ----------------------------------
> > >
> > >
> > >
> > > _______________________________________________
> > > Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> > 
> > > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> > _______________________________________________
> > Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> > 
> > _______________________________________________
> > Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> > 
> _______________________________________________
> Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>  
> Proud partner. Susan G. Komen for the Cure.
>  
> Please consider our environment before printing this e-mail
> or attachments.
> ----------------------------------
> CONFIDENTIALITY NOTICE: This e-mail may contain privileged
> or confidential information and is for the sole use of the
> intended recipient(s). If you are not the intended
> recipient, any disclosure, copying, distribution, or use of
> the contents of this information is prohibited and may be
> unlawful. If you have received this electronic transmission
> in error, please reply immediately to the sender that you
> have received the message in error, and delete it. Thank
> you.
> ----------------------------------
> 
_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu