Intermittent can't resolve DNS

Hi there,

The timeout messages in PRTG exactly reflect what the supporters of your mesh-routers found out. If the DNS names cannot be resolved, the requests will run into a timeout or will return a DNS error like "name cannot be resolved". So this appears to be something you need to verify on your used DNS servers.

Although, a traceroute command would just ping the way to the configured DNS server, but won't resolve the name. A DNS Lookup would be more helpful here I guess.

Kind regards,
Felix Saure, Tech Support Team

Felix,

Thanks for the tip. I'll see if I can find a way to do repeated DNS lookups.

I confused whey they are SSL errors. Does a DNS lookup us SSL?

Felix,

Thank you for taking the time to respond. I ran a benchmark from my network to the OpenDNS servers that I am using. The report is showing 100% reliability.

Final benchmark results, sorted by nameserver performance: (average cached name retrieval speed, fastest to slowest)

    4.  2.  2.  6 |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  - Cached Name   | 0.006 | 0.007 | 0.009 | 0.000 | 100.0 |
  - Uncached Name | 0.009 | 0.040 | 0.202 | 0.045 |  83.7 |
  - DotCom Lookup | 0.008 | 0.018 | 0.032 | 0.008 |  95.1 |
  ---<-------->---+-------+-------+-------+-------+-------+
                 f.resolvers.level3.net
            LEVEL3 - Level 3 Parent, LLC, US


  208. 67.220.123 |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  + Cached Name   | 0.006 | 0.007 | 0.009 | 0.001 | 100.0 |
  + Uncached Name | 0.009 | 0.041 | 0.235 | 0.056 | 100.0 |
  + DotCom Lookup | 0.009 | 0.033 | 0.095 | 0.030 | 100.0 |
  ---<-------->---+-------+-------+-------+-------+-------+
                resolver2-fs.opendns.com
               OPENDNS - OpenDNS, LLC, US

I can't seem to determine why the routers are failing to resolve the DNS lookup.

This seems to be inconsistent with the PRTG errors.

Again, any insights on what I might be missing or next steps to diagnose would be appreciated.

Darin

Thanks for your reply,

DNS lookups do not use SSL, but the security sensor in PRTG does. If the host name cannot be resolved properly and points to a different address, the requests are not getting answered and therefore show the SSL Timeout Messages.

You can use the DNS Sensors in PRTG to resolve the domain name and to compare it with a defined IP address. If the resolved address changes, the sensor will let you know. Since you started the test from one of your network devices and not from a Windows Host, the DNS cache also might be different. This depends on the Windows settings as well as on the TTL configured on the DNS server.

If this is different for Windows and the device from which you started the tests, it cannot directly be compared.

Kind regards,
Felix Saure, Tech Support Team

Hi Darin,

Reading your initial post I am confused.

You talk about a open mesh router system - that seems to do some kind of fail-over. Their support tells you now this is due to an DNS issue that the router attempts against cloudtrax.com.

Then you mention two IPs as DNS servers, actually OpenDNS IP addresses, while one is a standard DNS IP and the other one is part of their family shield / parental controls system (namely: 208.67.220.123).

After that you post a lot of errors I don't fully understand - do you SSL monitor the OpenDNS systems?

It seems like you can PING 100% of the time - but you encounter SSL issues, respective certificate check issues. All of it is a bit weird - do those issues happen parallel to your mesh routers having an issue? If so - you might probe something and see results that are both depending on the same bottle-neck respective error source, your internet connection.

The PRTG sensors against an internet DNS would fail simultaneous with your mesh routers connections issues, cause both in theory would depend need to go the same path.

I do very well understand that you can traceroute/nslookup etc. DNS names and IP addresses - at certain points in time - the question in the end is - what can and do the mesh routers do.

You don't mention a vendor etc. - but I assume those devices have detailed log files, probably even SYSLOG and SNMP - probably you could engage SYSLOG or SNMP with PRTG and collect more details.

Is it possible to configure more then this one single test-target on the routers cloudtrax.com - so you have multiple issues before a fail-over? Is it possible to use an IP instead and PING etc. instead?

For my understanding - it seems like you have some kind of route-metric dependency that automatically fails over to another route/router if this target fails - right? Multi-mesh in theory could mean that you would have multiple path ways from source A to target B - while there is a direct (more or less) route between all other endpoints. A more direct route then does not necessarily mean it is the better route, it could be that the theoretical speed between A and B directly is slower / less - associated with higher cost - then going from A to B while taking the path from A to C to D to B what might be more efficient (routing protocols like BGP/EIGRP/IGRP/etc..).

Sorry if I missed some information - I just try to understand your real issue - you might be on the right track - just want to avoid the shier possibility you go dig in to the wrong direction.

Regards

Florian Rossmark

www.it-admins.com

Florian,

Thanks for very much for taking the time to respond. I will answer as many questions as I can in your message below.

Hi Darin,

Reading your initial post I am confused.

You talk about a open mesh router system - that seems to do some kind of fail-over. Their support tells you now this is due to an DNS issue that the router attempts against cloudtrax.com. -Yes

Then you mention two IPs as DNS servers, actually OpenDNS IP addresses, while one is a standard DNS IP and the other one is part of their family shield / parental controls system (namely: 208.67.220.123). - Yes, I did not realize my 2nd DNS was part of the family shield, I have since changed it to 208.67.220.220.

After that you post a lot of errors I don't fully understand - do you SSL monitor the OpenDNS systems? - I was attempting to monitor the DNS with the host name that the router is not able to resolve. I am new to PRTG and I think the default configuration started to SSL monitor OpenDNS. My goal was to monitor OpenDNS's resolution of the host name that was failing in the mesh router.

It seems like you can PING 100% of the time - but you encounter SSL issues, respective certificate check issues. All of it is a bit weird - do those issues happen parallel to your mesh routers having an issue? If so - you might probe something and see results that are both depending on the same bottle-neck respective error source, your internet connection. - I agree with your premise. I have a wired network and the open mesh network attached to the device that is providing a Verizon Fios broadband connection.

The PRTG sensors against an internet DNS would fail simultaneous with your mesh routers connections issues, cause both in theory would depend need to go the same path. -- I was attempting to monitor the DNS resolution of the failing domain name from the wired side of the connection.

The PRTG sensors against an internet DNS would fail simultaneous with your mesh routers connections issues, cause both in theory would depend need to go the same path. -- I agree.

I do very well understand that you can traceroute/nslookup etc. DNS names and IP addresses - at certain points in time - the question in the end is - what can and do the mesh routers do.

You don't mention a vendor etc. - but I assume those devices have detailed log files, probably even SYSLOG and SNMP - probably you could engage SYSLOG or SNMP with PRTG and collect more details. -- vendor is cloudtrax.com. I am logging the syslog with RPTG. The vendor is telling me that my mesh routers are dropping connections because they cannot resolve the host name. I am beginning to wonder if they cannot resolve the host name do to their connectivity issues.

The slslog error is: 6:74:0E:45:60: API[19510]: connection error to https://cloud_ap.cloudtrax.com/cloud-ap/AC:86:74:0E:45:60/checkin: Couldn't resolve host 'cloud_ap.cloudtrax.com' (6)

Is it possible to configure more then this one single test-target on the routers cloudtrax.com - so you have multiple issues before a fail-over? Is it possible to use an IP instead and PING etc. instead? -- I have since found out (from the syslog) that the actual error is: 6:74:0E:45:60: API[19510]: connection error to https://cloud_ap.cloudtrax.com/cloud-ap/AC:86:74:0E:45:60/checkin: Couldn't resolve host 'cloud_ap.cloudtrax.com' (6)

I was confused by the https: in the error message. I thought it was trying to resolve the DNS via SSL. It seems like that is not how DNS works. Sorry if these are newbie questions, I often know just enough to be dangerous. I have used these mesh routers at other locations and they have been very reliable (including running multiple high def ip security cameras over them). In this location, they are dropping approximately hourly. There are 4 routers in the mesh.

For my understanding - it seems like you have some kind of route-metric dependency that automatically fails over to another route/router if this target fails - right? Multi-mesh in theory could mean that you would have multiple path ways from source A to target B - while there is a direct (more or less) route between all other endpoints. A more direct route then does not necessarily mean it is the better route, it could be that the theoretical speed between A and B directly is slower / less - associated with higher cost - then going from A to B while taking the path from A to C to D to B what might be more efficient (routing protocols like BGP/EIGRP/IGRP/etc..).

Sorry if I missed some information - I just try to understand your real issue - you might be on the right track - just want to avoid the shier possibility you go dig in to the wrong direction.

-- Again, thanks for taking the time to respond. I was attempting to resolve the host name in the error message and monitor it PRTG and see if I would get intermittent failures. I have seen posts about people having issues with resolving DNS names on Fios and actually tracking it down to hardware on the Fios Network. I wanted to monitor it over time to see if it was 100% available. During that process, I got wrapped around the axle because the error log had https: resulting in all the SSL errors.

I am still working with the vendor and they are attempting additional configuration changes to the mesh routers to see if they can make them stable. As I said, I have set these up at at three other locations in their default configuration and they are very stable. I am trying to track down what is different at this install that is making them unstable.

Thanks for your thoughts.

-Darin

Regards

Florian Rossmark

Hi Darin,

Summarizing your answers:

• Verizon Fios is in use
• you have 4x mesh routers that somehow target the Verizon Fios
• you root issue is:
  • 6:74:0E:45:60: API[19510]: connection error to https://cloud_ap.cloudtrax.com/cloud-ap/AC:86:74:0E:45:60/checkin: Couldn't resolve host 'cloud_ap.cloudtrax.com' (6)

How DNS works:

DNS is nothing else then in most cases a stateless (UDP) request of DNS server, please tell me an IP address for the name xyz.domain.com - and in most cases a UDP response IP found - it is - 1.1.1.1.

HTTPS in your mix:

This is a protocol that is in use WAY later - it actually states that the traffic to this destination would be secured

What I assume happens on your CloudTrax routers

The following is just an assumption, I did not research it - having said that, I think I am very close to the truth, but please don't take it for granted.

• router requests the defined DNS server (primary DNS server, unless not reachable) to resolve cloud_ap.cloudtrax.com
  • router receives IP address
    • router tries to connect to IP via HTTPS
    • likely a certificate check happens as of the response of the web server
    • since the router targets a AP sub-directory and then a MAC (likely his own) and then access the sub-directory CHECKIN it kinda tells the system it is ALIVE
    • the ALIVE then later be checked by some other members that know the MAC of this router and if they detect he is not, they might do some magic fail-over
  • router receives no IP address
    • router reports your SYSLOG error and some fail over happens

You state you have 4x routers - but if I go it right 1x Internet connection? Quite confusing for me - I still don't get all that multi-mesh, but I don't want to dig somewhere if it is not necessary.

What I would do:

You state you found out other people on Verizon Fios (fiber) state issues related to DNS - well - then let's work on that.

Pretty sure you do have a Windows Domain there and therefor a Windows DNS - if so - can you do the following?

• In Windows DNS
  • configure on the DNS servers them self DNS forwarding (rights click SERVER - Properties - Forwarders) to your OpenDNS and probably even to the Google DNS servers - I would completely bypass your provider (possible they don't allow that, but that's a whole other story)
    • it is possible you either have them already there or your router IP instead
• do a NSLOOKUP on a client against your Windows DNS server(s)
  • NSLOOKUP
  • SERVER <name or IP>
  • google.com
  • paessler.com
  • => should all resolve - repeat for all configured/adjusted DNS servers
• on the ROUTERS
  • instead of using OpenDNS as target DNS servers, you now use the Windows DNS server(s) internally

What happens?

• the Windows DNS server will try to determine the IP of cloudtrax.com - or that specific DNS name.
• it will also cache it according to it's TTL
• the router will request the IP from the internal DNS - that either knows the IP (cache) or will then forward the request to some external (OpenDNS/Google etc.) DNS server and find it out
• you actually might stabilize your DNS resolution on the routers

This is just a theory - it still should be easy to implement and test this. In theory the router might clear the DNS cache for this check-in action constantly, therefor using Windows DNS might just work - while avoiding that it will do a full OpenDNS/Google DNS etc. research again and instead hopefully just using it's own cache (TTL matters).

PS: curios about your results here :-)

Regards

Florian Rossmark

www.it-admins.com