Seeing almost constant "...joins with uptime [xxxxxx] s and last disconnected reason [Heartbeat Loss]. on 2x r710s

  • 1
  • Question
  • Updated 5 months ago
  • Answered
I'm seeing a lot of error messages in my logs
Jun  8 23:08:47 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [110595] s and last disconnected reason [Heartbeat Loss] 

I have two Ruckus r710's one upstairs and one downstairs. The downstairs unit is the master in unleashed. I'm constantly seeing this but it doesn't tell me why its losing its heartbeat. I think I'm having issues because all of a sudden my clients seem to glitch out and lose connection if they are connected to one of the units.

Any suggestions?

Jun  8 23:08:47 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [110595] s and last disconnected reason [Heartbeat Loss] 
Jun  8 23:08:48 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [90809] s and last disconnected reason [Heartbeat Loss] 
Jun  9 01:08:57 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [117805] s and last disconnected reason [Heartbeat Loss] 
Jun  9 01:09:00 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [98020] s and last disconnected reason [Heartbeat Loss] 
Jun  9 03:08:37 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [124985] s and last disconnected reason [Heartbeat Loss] 
Jun  9 03:08:41 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [105201] s and last disconnected reason [Heartbeat Loss] 
Jun  9 05:08:47 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [132195] s and last disconnected reason [Heartbeat Loss] 
Jun  9 05:08:52 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [112412] s and last disconnected reason [Heartbeat Loss] 
Jun  9 07:08:28 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [139376] s and last disconnected reason [Heartbeat Loss] 
Jun  9 07:08:33 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [119592] s and last disconnected reason [Heartbeat Loss] 
Jun  9 09:08:39 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [146587] s and last disconnected reason [Heartbeat Loss] 
Jun  9 09:08:44 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [126803] s and last disconnected reason [Heartbeat Loss] 
Jun  9 11:08:24 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [133983] s and last disconnected reason [Heartbeat Loss] 
Jun  9 11:08:50 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [153798] s and last disconnected reason [Heartbeat Loss] 
Jun  9 13:08:31 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [160979] s and last disconnected reason [Heartbeat Loss] 
Jun  9 13:08:34 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [141193] s and last disconnected reason [Heartbeat Loss] 
Jun  9 15:08:42 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [168190] s and last disconnected reason [Heartbeat Loss] 
Jun  9 15:08:44 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [148403] s and last disconnected reason [Heartbeat Loss] 
Jun  9 17:08:23 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [175370] s and last disconnected reason [Heartbeat Loss] 
Jun  9 17:08:24 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [155583] s and last disconnected reason [Heartbeat Loss] 
Jun  9 19:08:34 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [182581] s and last disconnected reason [Heartbeat Loss] 
Jun  9 19:08:39 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [162798] s and last disconnected reason [Heartbeat Loss] 
Jun  9 21:08:20 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [169979] s and last disconnected reason [Heartbeat Loss] 
Jun  9 21:08:44 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [189791] s and last disconnected reason [Heartbeat Loss] 
Jun  9 23:08:24 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [196971] s and last disconnected reason [Heartbeat Loss] 
Jun  9 23:08:30 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [177190] s and last disconnected reason [Heartbeat Loss] 
Jun 10 01:08:34 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [204181] s and last disconnected reason [Heartbeat Loss] 
Jun 10 01:08:42 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [184401] s and last disconnected reason [Heartbeat Loss] 
Jun 10 03:08:14 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [211361] s and last disconnected reason [Heartbeat Loss] 
Jun 10 03:08:23 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [191581] s and last disconnected reason [Heartbeat Loss] 
Jun 10 05:08:24 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [218571] s and last disconnected reason [Heartbeat Loss] 
Jun 10 05:08:34 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [198792] s and last disconnected reason [Heartbeat Loss] 
Jun 10 07:08:14 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [205972] s and last disconnected reason [Heartbeat Loss] 
Jun 10 07:08:34 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [225781] s and last disconnected reason [Heartbeat Loss] 
Jun 10 09:08:15 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [232962] s and last disconnected reason [Heartbeat Loss] 
Jun 10 09:08:24 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [213182] s and last disconnected reason [Heartbeat Loss] 
Jun 10 11:08:04 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [220362] s and last disconnected reason [Heartbeat Loss] 
Jun 10 11:08:26 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [240173] s and last disconnected reason [Heartbeat Loss] 
Jun 10 13:08:07 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [247354] s and last disconnected reason [Heartbeat Loss] 
Jun 10 13:08:14 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [227572] s and last disconnected reason [Heartbeat Loss] 
Jun 10 15:08:18 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [254565] s and last disconnected reason [Heartbeat Loss] 
Jun 10 15:08:24 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [234782] s and last disconnected reason [Heartbeat Loss] 
Jun 10 17:07:59 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:21:46:60] joins with uptime [261746] s and last disconnected reason [Heartbeat Loss] 
Jun 10 17:08:05 Downstairs syslog: eventd_to_syslog():AP[[email protected]:c5:d3:1a:7f:20] joins with uptime [241963] s and last disconnected reason [Heartbeat Loss] 
Photo of Kevin Vasko

Kevin Vasko

  • 8 Posts
  • 0 Reply Likes

Posted 5 months ago

  • 1
Photo of Michael Brado

Michael Brado, Official Rep

  • 2167 Posts
  • 299 Reply Likes
The long uptimes tell you that the APs are not rebooting.  It appears yoiu have a network issue between the APs
and your controller.  Check intermediate switches/cabling.

Here are a couple related KBAs.
  https://support.ruckuswireless.com/answers/000003945
  https://support.ruckuswireless.com/answers/000001658
Photo of Kevin Vasko

Kevin Vasko

  • 8 Posts
  • 0 Reply Likes
I can only access the top support article (I don't have a subscription as I'm running unleashed).

So I did some more digging and I think what is happening is this.

After doing some more digging I'm thinking it's a problem with the DHCP server or unit doing something funny with its leases. I suspect the ruckus master unit client. I'm using pfsense as my router/dhcp server. It seems to be giving out 2 hour and 24 hour leases. I can't tell you why some clients get 24 and others get 2. It is whatever is setup by default.

The problem seems to stem by the master Ruckus unit not successfully refreshing its DHCP lease of the SAME ip address. I see the unit moving from 192.168.0.62 from yesterday to 192.168.0.74 today (every two hours it changes, so it has moved from 62 to 63, 64,... etc.. This seems to only happen on the master unit. The other unit has a DHCP lease that it keeps renewing and it keeps getting the same IP without a problem.

I feel the issue stems from this scenario.

* Master (Downstairs unit) and slave (upstairs unit) lease their IP and all is well.
* On the downstairs unit, two hours passes, its DHCP lease expires, and then instead of keeping the same IP (e.x 192.168.0.30), a new IP address is requested from the DHCP server
* the Downstairs unit drops from the network and switches IPs
* the slave unit (upstairs unit) loses its heartbeat and rejoins the network as the IP address of the master downstairs has changed

Why this is happening and the master unit is getting new IP addresses beats me. My pfsense is DHCP server leases are full of expired leases from this unit. IThe other Ruckus has not done this once. No other client has done this.

I'm going to test this by setting a static IP address of the downstairs (master) unit. If it stops, I would be willing to bet that is the problem, and why the network drops (and the cause of the heartbeat issues) and clients can't get out to the internet (depending on which AP they are on).

If that is the case, I would suspect the Ruckus master has problems with renewing its DHCP lease (possibly a dhcp client issue on the Ruckus AP firmware?)
I don't have a support license or anything but I'll post back my results when I get home and hopefully the information could be put to good use.