Veritas-bu

Re: [Veritas-bu] bpclntcmd and others ignoring nsswitch.conf?-RESOLVED (yes actually resolved this time)

2011-05-08 15:02:46
Subject: Re: [Veritas-bu] bpclntcmd and others ignoring nsswitch.conf?-RESOLVED (yes actually resolved this time)
From: Marianne Van Den Berg <mvdberg AT stortech.co DOT za>
To: "Lightner, Jeff" <JLightner AT water DOT com>
Date: Sun, 8 May 2011 21:02:27 +0200
Hi Jeff

If you don't mind - please set the record straight on the Symantec Ideas page?

https://www-secure.symantec.com/connect/ideas/bug-report-netbackup-name-resolution-hp-ux

Regards

M.

-----Original Message-----
From: veritas-bu-bounces AT mailman.eng.auburn DOT edu 
[mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of 
Lightner, Jeff
Sent: 03 May 2011 07:29 PM
To: Girish Jorapurkar
Cc: veritas-bu AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] bpclntcmd and others ignoring nsswitch.conf?-RESOLVED 
(yes actually resolved this time)

The patch (PHCO_30030) has been superseded more than once and we have one of 
the superseding patches PHCO_37369 so would already have the "bug fix" the 
poster at the link mentioned.

However, I'd like to thank you as your pointer helped me resolve the issue:
1)  By downloading and compiling his source code I was able to see that I did 
have the same issue with his binary as that poster did.   (That is to say this 
binary like the NetBackup binaries did not resolve from /etc/hosts first but 
rather from DNS.)
2)  On reviewing the /var/adm/sw/products/PHCO_37369/pfiles/README file that 
came with the later patch I saw the following (which is from another patch this 
one supersedes so is incorporated):
        PHCO_29287:
        ( QX:QXCR1000523428 SR:8606308071 CR:JAGae71107 )
        If IPv6 is enabled on a system, the Internet Services
        commands use the APIs getipnodeXX() or get*info() to
        resolve the name instead of the APIs gethostbyXX().
        The APIs getipnodeXX() and get*info() look into the
        ipnodes directive of the /etc/nsswitch.conf to resolve
        the name/address.  As the NIS/NIS+ backends do not
        respond to the ipnodes directive, the resolution of
        IPv4 addresses fails for NIS and NIS+ sources.

While we aren't doing anything with NIS/NIS+ the other info about IPv6 was 
informative.  Turns out that in addition to the standard "hosts:" line in 
/etc/nsswitch.conf HP-UX is expecting an "ipnodes:" line for binaries that use 
the above APIs.

After modifying /etc/nsswtich.conf so it now has both the following lines:
hosts: files [notfound=CONTINUE] dns
ipnodes: files [notfound=CONTINUE] dns

The new binary resolved correctly from /etc/hosts and on testing so did 
bpclntcmd.   Based on that I restarted the backup that had been going across 
the primary LAN previously and it is currently running on the expected backup 
LAN.

Presumably this means the 7.1 NetBackup binaries are using the above referenced 
APIs on HP-UX where the 6.5.4 binaries didn't.

We never saw this issue on HP-UX up through version 6.5.4 and in testing 
yesterday I'd copied the 6.5.4 binary and its associated libraries to another 
directory on my HP-UX host and it resolved from /etc/hosts.

I added the ipnodes entry to /etc/nsswitch.conf on the other HP-UX 11.11 media 
server and the HP-UX 11.23 client I'd recreated the error on previously.   On 
adding the line both hosts correctly resolved the backup LAN IP from /etc/hosts 
instead of going to DNS.

FYI:  ITRC is going away so I'm not sure if the link will still be accessible 
in the future.   Accordingly I'm including the source code that poster had made 
so it is available from this forum.

His source code file was called:  114249.c and I compiled it by running:
"cc -o testresolver 114249.c".   Usage is simply "testresolver <hostname>".   
The source code is as follows:

/* Test program to test IP resolution for domain names
 */


#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <string.h>


/* Function to get IP address for hostname
 ** hostname - (i/p) - Hostname to find IP address
 ** buffer   - (o/p) - Buffer filled with IP address
 ** buffer_len - (i/p) - Length of buffer
 ** Return APR_SUCCESS on success
 **/
int get_ip_from_hostname(char *hostname)
{
    int status = 0;
    struct addrinfo hints, *ai, *ai_list;
    char host_ip[INET_ADDRSTRLEN];
    const char *ret = NULL;

    memset(&hints, 0, sizeof(hints));
    hints.ai_family = AF_INET;
    hints.ai_socktype = SOCK_STREAM;
    status = getaddrinfo(hostname, NULL, &hints, &ai_list);
    if (status)
    {
        printf("Failed to resolve IP address for %s \n ", hostname);
        printf("Error: %s \n", gai_strerror(status));
        return status;
    }

    ai = ai_list;
    if(ai)
    {
        struct sockaddr_in *in = (struct sockaddr_in *)ai->ai_addr;
        host_ip[0] = '\0';
        ret = inet_ntop(AF_INET, &in->sin_addr, host_ip, sizeof(host_ip));
    }
    freeaddrinfo(ai_list);

    if(host_ip)
    {
        printf("Hostname \"%s\" resolves to IP \"%s\" \n", hostname, host_ip);
        status = 0;
    }
    else
    {
        printf("Failed to resolve IP address for %s \n", hostname);
        status = -1;
    }

    return status;
}

void usage()
{

    printf("resolve_hostname <hostname> \n");

    return;
}

int main(int argc, char *argv[])
{

    char hostname[1024];

    if(argc<2)
    {
        usage();
        return 0;
    }

    strcpy(hostname, argv[1]);
    get_ip_from_hostname(hostname);


    return 0;
}


-----Original Message-----
From: veritas-bu-bounces AT mailman.eng.auburn DOT edu 
[mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Girish 
Jorapurkar
Sent: Monday, May 02, 2011 5:02 PM
To: Patrick; Lightner, Jeff
Cc: veritas-bu AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] bpclntcmd and others ignoring nsswitch.conf?-SPOKE TO 
SOON

This seems to be a HP-UX bug (getaddrinfo API). Here is link describing it:

https://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1304369816045+28353475&threadId=614359

> PHCO_30030 fixed the problem

Let me know if the patch resolves the issue.

Thanks,
/Girish

--- On Mon, 5/2/11, Lightner, Jeff <jlightner AT water DOT com> wrote:

> From: Lightner, Jeff <jlightner AT water DOT com>
> Subject: Re: [Veritas-bu] bpclntcmd and others ignoring nsswitch.conf? -SPOKE 
> TO SOON
> To: "Patrick" <netbackup AT whelan-consulting.co DOT uk>
> Cc: veritas-bu AT mailman.eng.auburn DOT edu
> Date: Monday, May 2, 2011, 9:25 PM
> The answer is both commands "work" in
> that they both run.   However, both commands
> are using the IP from DNS to make the connection rather than
> the one from /etc/hosts.   Therefore the only
> IP I ever see in the output is the main LAN IP rather than
> the one for the backup LAN that is in /etc/hosts.
>
> Some testing was done after adding an entry to /etc/hosts
> that doesn't exist in DNS and I found the command resolved
> the correct (backup LAN) IP.   Also
> afterwards the new entry was commented out of /etc/hosts and
> bpclntcmd -clear_host_cache and it did NOT find anything in
> cache.   This to me suggests that the issue
> is that the NetBackup utilities are looking at /etc/hosts
> but only after they can't find it in DNS but this is the
> opposite of the way nsswitch.conf is configured.
>
> I do not see an issue resolving the backup LAN IPs from
> /etc/hosts from either the RHEL6 master or the Windows 2003
> client.
>
> Other tests show me that I can repeat this bad behavior on
> another HP-UX 11.11 media server and an HP-UX 11.23
> client.    Since host level utilities (nslookup
> and ping) return the correct address from /etc/hosts I know
> that nsswitch.conf is working on HP-UX.
>
> To me it appears the issue is the NetBackup binaries all
> have the same flaw on HP-UX.   This suggests
> to me there is a bug in a common object or library that is
> used for the host name resolution used by all these
> NetBackup binaries but that it is only on the HP-UX version
> of NBU 7.1.
>
> -----Original Message-----
> From: Patrick [mailto:netbackup AT whelan-consulting.co DOT uk]
>
> Sent: Friday, April 29, 2011 10:00 AM
> To: Lightner, Jeff
> Cc: veritas-bu AT mailman.eng.auburn DOT edu
> Subject: RE: [Veritas-bu] bpclntcmd and others ignoring
> nsswitch.conf? -SPOKE TO SOON
>
> Have you tried running bptestbpcd -client <client
> name> -verbose -debug from
> the media server? What are the results? Does bpgetconfig -M
> <client name>
> work?
>
> Regards,
>
> Patrick Whelan
> VERITAS Certified NetBackup Support Engineer for UNIX.
> VERITAS Certified NetBackup Support Engineer for Windows.
>
> netbackup AT whelan-consulting.co DOT uk
>
>
> -----Original Message-----
> From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
> [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu]
> On Behalf Of Lightner,
> Jeff
> Sent: 29 April 2011 13:41
> To: Rosie Cleary
> Cc: veritas-bu AT mailman.eng.auburn DOT edu
> Subject: Re: [Veritas-bu] bpclntcmd and others ignoring
> nsswitch.conf?
> -SPOKE TO SOON
>
> We actually do that for most clients but this is a MS
> Exchange cluster name
> and we didn't need to use a separate name for this cluster
> in 6.5.4.  To me
> this seems like a bug in the NetBackup tools since I can
> demonstrate OS
> level tools resolve the expected IP and it is only the
> NetBackup tools that are getting the wrong
> name.   Also as I noted
> before this is only happening on my HP-UX media server - my
> master server is
> getting the correct IP in bpclntcmd for the client.
>
> -----Original Message-----
> From: Rosie Cleary [mailto:Rosie.Cleary AT nuim DOT ie]
> Sent: Friday, April 29, 2011 5:06 AM
> To: Lightner, Jeff
> Cc: veritas-bu AT mailman.eng.auburn DOT edu
> Subject: Re: [Veritas-bu] bpclntcmd and others ignoring
> nsswitch.conf?
> -SPOKE TO SOON
>
> Hi Jeff,
>
> I've just started using dnsmasq on Red Hat Linux which acts
> as a dns
> forwarder but allows me to override certain IP addresses.
> It's easy to
> configure and will get around this problem.
>
> Other than that is it out of the question to use a slightly
> different name
> for client interface that is in the backup network? (e.g.
> servername-b) You would then put servername-b rather than
> servername in the
> policy and the traffic would automatically route over the
> backup network.
> I've used this for all clients in my backup network from
> day one and it's
> been fine. The disadvantages that I can think of for you
> are
>   - The name of the client in reports and for searches
> is different from
>
> the normal client name, and if it only applies to one or
> two servers then it
> will be confusing.
>   - Backups taken before the change will be considered
> by the server to be
> of a different client so there are a few extra steps to
> restore from these.
>
> Best regards,
>
> Rosie.
>
> Rosie Cleary
> Computer Centre
> National University of Ireland, Maynooth
>
>
> Lightner, Jeff  wrote [28/04/2011 20:53]:
> > Sorry folks - NOT resolved.
> >
> > I thought this was resolved because the backup started
> but on checking
> I
> > see it is using the primary LAN rather than the backup
> LAN. The
> addition
> > of the FQDN on the client did get us past the 59 error
> but didn't fix
> > the issue I was asking about initially.
> >
> > The bpclntcmd is still showing the 10.x primary IP
> instead of the
> 172.x
> > backup IP that I have in host file of the media
> server. We really need
> > this backup to go across the backup LAN.
> >
> >
> ------------------------------------------------------------------------
> >
> > *From:*veritas-bu-bounces AT mailman.eng.auburn DOT edu
> > [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu]
> *On Behalf Of
> > *Lightner, Jeff
> > *Sent:* Thursday, April 28, 2011 3:11 PM
> > *To:* veritas-bu AT mailman.eng.auburn DOT edu
> > *Subject:* Re: [Veritas-bu] bpclntcmd and others
> ignoring
> nsswitch.conf?
> > -RESOLVED
> >
> > *Dan Otto had responded and based on what he wrote I
> resolved the
> issue.
> > The below shows the thread between us and is reposted
> here with his
> > permission.*
> >
> > **
> >
> > *From:*Lightner, Jeff [mailto:jlightner AT water DOT com]
> > *Sent:* Thursday, April 28, 2011 2:02 PM
> > *To:* Daniel Otto
> > *Subject:* RE: bpclntcmd and others ignoring
> nsswitch.conf?
> >
> > That was it.
> >
> > After checking bpcd log on the client we saw that it
> was complaining
> > that the FQDN name wasn't a media server. Our entry
> for the server was
> > the short name for the media server. Adding the FQDN
> to the line that
> > had the backup LAN IP and short name resolved the
> issue.
> >
> > It just didn't occur to me to look at the client
> because I thought
> > bpclntcmd was simply trying to resolve from the media
> server.
> >
> > I had actually tried adding FQDN of the client to the
> media server
> > earlier because we have seen various issues regarding
> short name vs
> FQDN
> > since implementing 7.1.
> >
> > Thanks for your help.
> >
> >
> ------------------------------------------------------------------------
> >
> > *From:*Daniel Otto [mailto:dan_otto AT symantec DOT com]
> > *Sent:* Thursday, April 28, 2011 2:57 PM
> > *To:* Lightner, Jeff
> > *Subject:* RE: bpclntcmd and others ignoring
> nsswitch.conf?
> >
> > The 59 is thrown because whatever server hostname the
> client is
> > resolving doesn't exist in the client's server list
> hence server
> access
> > denied status 59 and should show up as a status 46
> error in bpcd as a
> > invalid server. If the media server couldn't resolve
> the client at all
> > or getting the wrong IP address you would be getting
> 58/25 or even
> 54's
> > type of errors.
> >
> >
> ------------------------------------------------------------------------
> >
> > *From:*Lightner, Jeff [mailto:jlightner AT water DOT com]
> > *Sent:* Thursday, April 28, 2011 1:43 PM
> > *To:* Daniel Otto
> > *Subject:* RE: bpclntcmd and others ignoring
> nsswitch.conf?
> >
> > I did actually try removing dns from nsswitch.conf but
> it didn't help.
> >
> > I haven't looked at the client's bpcd log - I did
> verify the media
> > server is in the server list for the client. The only
> other one in the
> > list is the master.
> >
> >
> ------------------------------------------------------------------------
> >
> > *From:*Daniel Otto [mailto:dan_otto AT symantec DOT com]
> > *Sent:* Thursday, April 28, 2011 2:35 PM
> > *To:* Lightner, Jeff
> > *Subject:* RE: bpclntcmd and others ignoring
> nsswitch.conf?
> >
> > Easy fix would be to use the bpcd log on the client
> and whatever
> > hostname is getting resolved simply add it to the
> SERVER = on the
> client
> > and the 59 should go away. As to why it is not using
> the host
> > file...that's interesting.
> >
> > Perhaps for a quick test remove the DNS entry
> altogether from
> > switch.conf to see if it then goes to the host file.
> Though I have
> only
> > seen Solaris server do funkly things such as this.
> >
> > Hope this helps,
> >
> > Dan O
> >
> >
> ------------------------------------------------------------------------
> >
> > *From:*veritas-bu-bounces AT mailman.eng.auburn DOT edu
> > [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu]
> *On Behalf Of
> > *Lightner, Jeff
> > *Sent:* Thursday, April 28, 2011 2:22 PM
> > *To:* veritas-bu AT mailman.eng.auburn DOT edu
> > *Subject:* [Veritas-bu] bpclntcmd and others ignoring
> nsswitch.conf?
> >
> > On a UNIX (HP-UX) media server when I run bpclntcmd
> -hn <client> (and
> > other bp commands) it appears they're ignoring the IP
> specified for
> the
> > client in /etc/hosts.
> >
> > The server also exists in DNS but my nsswitch.conf
> clearly says to use
> > files (i.e. /etc/hosts) then dns. If I run command
> line tests using
> > other tools (e.g. nslookup and ping) the IP seen for
> the client is the
> > one in /etc/hosts. However, bpclntcmd is giving me the
> one from DNS.
> The
> > IPs are different because we have use a separate
> subnet for backup
> LAN.
> >
> > I've done a fair amount of searching and have read
> this technote:
> >
> >
> http://www.symantec.com/business/support/index?page=content&id=TECH14111
> 7&key=15143&actp=LIST
> >
> <http://www.symantec.com/business/support/index?page=content&id=TECH1411
> 17&key=15143&actp=LIST>
> >
> > I had actually rebooted the server for another reason
> about an hour
> and
> > a half after I changed /etc/hosts yesterday. According
> to all the
> notes
> > I found the cache should clear itself about an hour
> after making a
> > change but that didn't happen.
> >
> > This morning I still saw the issue. I tried reissuing
> the bpclntcmd
> > -clear_host_cache (new in 7.x) and bouncing NetBackup
> on the media
> > server to no avail.
> >
> > Based on the above technote I have tried moving the
> cache directory
> out
> > and bouncing NetBackup again to no avail.
> >
> > I also tried just for the hell of it stopping the
> pbx_exchange daemon
> > and restarting then bouncing NetBackup but that didn't
> help either.
> >
> > Interestingly bpclntcmd -hn <client> on the
> RHEL6 Linux master returns
> > the correct IP from /etc/hosts file there. Running
> bpclntcmd on the
> > client itself with various flags (e.g. -pn, -self)
> find the correct
> > master etc...
> >
> > My backup is failing with a status 59 - can't connect
> to client and
> I'm
> > assuming it is because the media server is getting the
> wrong IP for
> the
> > client. (The client has an entry in its hosts file
> specifying the
> backup
> > LAN IP of the media server - this is the same as it
> was in 6.5.4
> except
> > we were using a Windows media server then.)
> >
> > P.S. On HP-UX nslookup DOES look at /etc/hosts unlike
> nslookup on
> other
> > *nix flavors so please don't respond that using
> nslookup for
> /etc/hosts
> > isn't valid. As noted ping also works so such an
> observation wouldn't
> be
> > helpful even if it weren't incorrect. The issue isn't
> host name
> > resolution in UNIX but rather hostname resolution done
> by NetBackup
> > commands.
> >
> >
> *_______________________________________________________________________
> ___________________*
> >
> > *Jeff Lightner*****| *UNIX/Linux Administrator*****|
> **DS Waters of
> > America, Inc **| 5660 New Northside Drive, Ste 250 |
> Atlanta, GA
> 30328**
> > (:(Direct Dial)678-486-3516|(:(Cell)678-772-0018 |
> *:jlightner AT water DOT com
> >
> > Proud partner. Susan G. Komen for the Cure.
> >
> > //Please consider our environment before printing this
> e-mail or
> > attachments.//
> >
> > ----------------------------------
> > CONFIDENTIALITY NOTICE: This e-mail may contain
> privileged or
> > confidential information and is for the sole use of
> the intended
> > recipient(s). If you are not the intended recipient,
> any disclosure,
> > copying, distribution, or use of the contents of this
> information is
> > prohibited and may be unlawful. If you have received
> this electronic
> > transmission in error, please reply immediately to the
> sender that you
> > have received the message in error, and delete it.
> Thank you.
> > ----------------------------------
> >
> >
> >
> > _______________________________________________
> > Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
>
> > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> _______________________________________________
> Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
> _______________________________________________
> Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Proud partner. Susan G. Komen for the Cure.

Please consider our environment before printing this e-mail or attachments.
----------------------------------
CONFIDENTIALITY NOTICE: This e-mail may contain privileged or confidential 
information and is for the sole use of the intended recipient(s). If you are 
not the intended recipient, any disclosure, copying, distribution, or use of 
the contents of this information is prohibited and may be unlawful. If you have 
received this electronic transmission in error, please reply immediately to the 
sender that you have received the message in error, and delete it. Thank you.
----------------------------------
_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu