AP's fail to come back online after ZD update

  • 1
  • Question
  • Updated 3 years ago
I am wondering if anyone else has had this same issue and has found a workaround. I updated ZD3000 twice in the past 3 weeks. Each time 1 or 2 AP's have failed to take the update and connect to ZD. We only have around 60 AP in our network and this is quite a big failure rate.
Photo of Gorge Networks

Gorge Networks

  • 8 Posts
  • 0 Reply Likes

Posted 4 years ago

  • 1
Photo of Primož Marinšek

Primož Marinšek, AlphaDog

  • 413 Posts
  • 48 Reply Likes
I've seen this sometimes too. But only when upgrading to 9.6. What I did I manually upgraded the "faulty" APs after a hard reset and a power cycle.

It did the trick. Some APs don't like to upgrade without a fight :)
Photo of Harish

Harish, Employee

  • 21 Posts
  • 4 Reply Likes
We can try CLI command which forces AP to upgrade the firmware from the ZD.

Login to the AP CLI using PUTTY application.

Execute the below command from the AP CLI.

fw force-update
Photo of Primož Marinšek

Primož Marinšek, AlphaDog

  • 413 Posts
  • 48 Reply Likes
Someone should make this a button in the Web GUI :)
Photo of Gorge Networks

Gorge Networks

  • 8 Posts
  • 0 Reply Likes
I can easily upgrade these manually that is not the issue. I was wondering if there was another way to upgrade. It seems to be the mesh AP's that don't come back online. Can I upgrade the furthest mesh AP's through Flexmaster before pushing the update through Zonedirector? I just want to avoid this issue in the future and not send out tech's out to manually service 5 AP's.
Photo of Primož Marinšek

Primož Marinšek, AlphaDog

  • 413 Posts
  • 48 Reply Likes
APs that are ZD controlled can't be provisioned by a FM and if the problem is on MESH APs, then you have a job on your hands if this is what's happening.

Did you power cycle the lost MESH APs before manually cervicing them and they still didn't come back?
Photo of Gorge Networks

Gorge Networks

  • 8 Posts
  • 0 Reply Likes
Sometimes they come back online after a power cycle, and other times I have to manually upgrade them to the new revision. Is this a problem with too many meshed AP's possibly? I know some companies have over 5k AP's to manage, I can't imagine the headache if 5% of these fail to upgrade like I am experiencing. There has to be a workaround somewhere. I don't want to keep sending technicians out to manually upgrade if there is something I can do remotely to remedy this problem.
Photo of Primož Marinšek

Primož Marinšek, AlphaDog

  • 413 Posts
  • 48 Reply Likes
I'd just say you were unlucky with the timing of your implementation and this FW. I've been involved with RW since firmware 3 and went through all of them, and only occasionally they get it a bit wrong. Sometimes a small bug causes bigger problems.
But I'm sure they'll sort it out with 9.7, they always have in the past.
Photo of Matthew Lawson

Matthew Lawson

  • 1 Post
  • 0 Reply Likes
We just pushed 9.7.0.2.9 and had 3 fail. We can ping and manage the AP's via the web interface. We have also tried to force the update via CLI but get a file not found error.
Photo of Michael Brado

Michael Brado, Official Rep

  • 2182 Posts
  • 300 Reply Likes
Does an AP reboot help?

If you can ping the AP, can you SSH in and type "get director", do the APs know the
ZD's IP address.

If you type "set director ip a.b.c.d" (with your ZD's IP), then "reboot", do the APs
show up on the ZD?

It does sound like there was some unusual glitch.
Photo of Monnat Systems

Monnat Systems, AlphaDog

  • 776 Posts
  • 163 Reply Likes
Hi Matthew,

If rebooting or doing the set director command is NOT helping then would advice the following to see if it helps and with the assumption that you have a DHCP server in the network. Here it goes:

Log into one AP and do a reset from GUI. Post Reset, AP will get an IP from DHCP and AP would come to standalone mode and can be managed from remote. Here you directly apply the same firmware version update to AP(keep it handy before you do this, saves time) and then after upgrade do a set director command to ZD.

Best of luck.
Photo of Martin

Martin, Official Rep

  • 213 Posts
  • 49 Reply Likes
Just to know your network a bit better, can you tell us the following

1. Mesh ?
2. how many AP hops ?
3. why to 9.7.0 ? not 9.7.1 or 9.8 ?

Martin
Photo of Com1 NL - Bas Sanders

Com1 NL - Bas Sanders

  • 32 Posts
  • 9 Reply Likes
We have seen similar (bad) results as well after our last two upgrades. In our setup it resulted in having engineers driving all across the country to get the AP's connected again.

In our case the issue was found in the AP's trying to find the zonedirector by DNS. It was 'solved' by replacing the zonedirector address with the IP address vice the FQDN.

Ruckus has confirmed there is a bug that makes the AP's go looney if they have an FQDN configured to connect to in 9.7.x. We have seen the issue in 9.7.0 and 9.7.1.

Most remarkable is that the AP's are actually capable of resolving the IP address but still dont seem to connect to the ZD.

The steps to resolve:
- Configure te Primary and Secondary addresses of the ZD as IP address and not DNS name. Configure under Configure -> Accesspoints -> Accesspoint policies
- Log into the AP using SSH
- Use the command 'set director ip xx.xx.xx.xx' to configure the addres of the ZD
- reboot the AP

Now you should see the AP joining the ZD as per normal. If the firmware doesn't match it will automagically upgrade (or downgrade) the firmware.