Unleashed AP - Suddenlty Factory Reset

  • 1
  • Question
  • Updated 5 days ago
  • Answered
Hello,

i have a strange issue which happend two times now.
A few weeks ago Customer told me that two Outdoor-Mesh-APs (T300 Unleashed) are not accessable any more.
We exchanged the APs and Customer shipped the defective APs to me.
I checked the APs. Strange Thing: APs where set to Factory Defaults. There were not defective. After configuring them they worked again without any issues.

Few days ago same issue at this Site but now with two T301s Unleashed.
T301s were not reachable anymore. Customer exchanged the APs and send me the two faulty ones.
After checking the APs, they were Factory Resetted again.

I ask me how the hack the APs can get Factory Resetted by itself?
This APs are mounted at 15m high. Nobody can reach this AP so easily to reset them.
Maybe an consequenz of an thunderstorm? Overload? Can this be possible? Or maybe a wrong grounding?

I don't really have an idea.
Maybe someone else?

Thanks.
KR
Marco
Photo of Marco Eichstetter

Marco Eichstetter

  • 148 Posts
  • 7 Reply Likes

Posted 2 months ago

  • 1
Photo of Toomas Kadarpik

Toomas Kadarpik

  • 5 Posts
  • 1 Reply Like
Are you using fixed or dynamic ip-s ? We saw problems at least with dynamic management IP-s1
Photo of Marco Eichstetter

Marco Eichstetter

  • 148 Posts
  • 7 Reply Likes
I am using fixed IPs.
Photo of Jeronimo

Jeronimo

  • 170 Posts
  • 19 Reply Likes
Could you let me know firmware version for UAP?

If in 200.2 or 200.3 using fixed IP, Plz upgrade 200.4 and use dynamic IP for aps and management interface IP for master mgmt.

Currently when a UAP which had master role is rebooted, the ap is factory reset after reboot.

And config is synced from new master.

If version 200.2 or 200.3, the ap , at that time, I have seen many times that a rebooted ap which had master role don't find new master.

And when the ap is rebooted, if it have no dhcp server, the ap never works.

If version is 200.2 or 200.3, due to above reason, I had seen factory reset many times.

Regards.
(Edited)
Photo of Marco Eichstetter

Marco Eichstetter

  • 148 Posts
  • 7 Reply Likes
FW is 200.4.
Because there is no DHCP and Customer don't want a DHCP within the Network, i have to use static IPs.
The APs which lost there config are Mesh-APs. So they cannot be Master-APs. Only Root-APs can be a Master.
Photo of Jeronimo

Jeronimo

  • 170 Posts
  • 19 Reply Likes
ok. I got it.

You must deploy dhcp server on the network side.

We worked test many times for this symptom.

If a ap which had master role is rebooted due to some reason such as power issue or kernel panic and so on, the ap is factory-reset.

Then the aps have dynamic ap  from dhcp server after rebooting, the ap join to new master and thier config is synced by new master. 

Plz deploy dhcp server.

And this symptom must fix for static environment.

Regards.
(Edited)
Photo of Marco Eichstetter

Marco Eichstetter

  • 148 Posts
  • 7 Reply Likes
Thanks you for your help.
But i dont get it why a DHCP Server should solve my issue? Also because the problematic AP were never and will never be Master-APs.
Unfortuantelly it is needed that e. g. AP01 has IP xxx.xxx.xxx.1 and AP02 has xxx.xxx.xxx.2.

Maybe from Ruckus can say something regarding this issue.
Sporadically resetting Mesh-AP which are mounted 15 Meter high are not really funny. Customer is also upset and i have to look for an 100% Solution to prevent this behaviour.

Such Kind of Device should never loose his config by itself.
Thanks.
Photo of Michael Brado

Michael Brado, Official Rep

  • 2104 Posts
  • 297 Reply Likes
DHCP provides the default-router to your APs.  Mesh APs reboot if they lose def-gw access.  The APs do NOT lose configuration unless you factory default them, but you cannot use Static IP to create an Unleashed network with Mesh, you must connect your intended Mesh APs to the LAN in order to get configured before you deploy them in Mesh mode.
Photo of Marco Eichstetter

Marco Eichstetter

  • 148 Posts
  • 7 Reply Likes
Michael, Thanks for your reply.
Yes. For creating a Unleashed Network DHCP is needed. This is true.
But there is also the Option to give an AP a static IP. After setting up everything i think it should not a Problem to work with static IPs.

But think about this in my case:
The Mesh AP has a Switch connected to his LAN Interface. This is needed to provide Network connectivity to some LAN Devices.
Of course it can be possible, that the Mesh-AP has some connectivity issues (Interference, DFS Channel, etc.).

Maybe there is an issue that the AP loosing his Mesh Link (DFS Channel, Interference). After that he can't reach the Default Gateway. AP tries to reach GW over LAN (where only a Switch and some LAN Devices are connected) and can't also reach the GW.
Like you say the AP should/will restart after that because for him it seems there is no GW reachable anymore.

What happends then?
Clears the AP his Config because he want's to re-download it from Master, like Jeronimo says?
If yes, this behaviour is not really good for a Mesh AP. Without Mesh-Link-Information that AP will never be able to reach the Master to get his config.

So i always have to unmount the AP, enable DHCP, connect AP to cabled Network to reach Master and get Mesh-Info, disable DHCP, mount AP back at Pole?

Maybe my case is a little bit special but this are setups which are used in "wild-life".
Photo of Jeronimo

Jeronimo

  • 170 Posts
  • 19 Reply Likes
Hi Marco and Micheal.

At our test, we found that only ap changed role(master ap -> membe ap) worked factory-reset and download config from new master.

Regards.
Photo of Michael Brado

Michael Brado, Official Rep

  • 2104 Posts
  • 297 Reply Likes
Answer to what happens then, loses config?  Is NO.  Only if factory defaulted with your Unleashed AP need network re-connect/re-config.
Else it just keeps trying, and yes, over Mesh UPLINK first, or Eth switch if connected (look to become eMAP), else = > reboot, keep trying.

Yes, I'm afraid if you wish to use Static IPs on your Mesh APs, and they have long period of no UPLINK to def-gw, you may under those
curcumstances, need to LAN connect the stranded Mesh AP.

The alternative to having to unmount the AP, is if Unleashed APs advertises an Island-SSID?  (But in another thread, I think we determined
we cannot use the ruckus-<admin-pw> string like ZD/SZ/Standalone APs)  But see KBA-1524.
Photo of Marco Eichstetter

Marco Eichstetter

  • 148 Posts
  • 7 Reply Likes
Ok. Thanks.

I am really sorry but i am still not getting why it should play any role if my Mesh-AP has a static or dynamic IP? If Mesh-AP looses his Mesh Link, a DHCP Server can not solve my Problem. Without Mesh-Link no reachable DHCP Server because this Server is located in my cabled LAN.

I understand, that the Mesh-AP is isolated if Mesh-Link not available. But if interferences are gone, Mesh-Link should come back online again.

But i still asking me how it can be possible, that Customer reports a failed Mesh-AP. Customer got a new AP from me. He only unmounted the defective AP and returned it in my Office. If i connect the failed AP to a PoE Adapter the AP LEDs blink normal and i can see an "ConfigureMe-..." SSID?
So from my perspective it looks like AP lost his Config.
Or do you have any other Explaination for this behaviour?

This happend now with four T300 Unleashed APs (at the same Site) and please understand me that this makes me a little bit nervous.
Customer asked me how this can be possible with such kind of enterprise APs and he is understandable a bit upset.
I hadn't an answer for Customer's question. So i am looking for a Solution before Customer returns all 9 APs to me and searching for another solution...
Photo of Jeronimo

Jeronimo

  • 170 Posts
  • 19 Reply Likes
If It happen same similaty issue when UAP works with mesh, it's seriously problem.

This is because the dhcp server function is useless if the mesh link losts.

I would also test about UAP with mesh link for customer later.
Photo of Marco Eichstetter

Marco Eichstetter

  • 148 Posts
  • 7 Reply Likes
Until now i did not test anything by myself. I wanted to verify here if there is a known Issue.
I think i will have some time for testing tomorrow or over the Weekend.
If you have any Feedback it whould be nice if you can share this with me.
Photo of Jeronimo

Jeronimo

  • 170 Posts
  • 19 Reply Likes
Sure.
Photo of Jeronimo

Jeronimo

  • 170 Posts
  • 19 Reply Likes
I think so Marco.

Must don't lose his config any condition.

Because I had met similary issue on 4 customer site, Our team worked many test, and found that currently UAP have to need dhcp server.

I think that ruckus must fix about this issue.
Photo of Marco Eichstetter

Marco Eichstetter

  • 148 Posts
  • 7 Reply Likes
Yes, definitly yes.
Loosing Config may not be a Problem in a simple Network with DHCP and without Mesh-APs.
But in my Case it is horrible. If a Mesh AP loosing his Config he is not able to access the DHCP because he as no Mesh Information anymore.

So if this is a known issue/bug, there must be a Solution for this from Ruckus.
Photo of Michael Brado

Michael Brado, Official Rep

  • 2106 Posts
  • 297 Reply Likes
No AP should ever lose configuration unless it's been factory defaulted by manual reset button or controller issued remote AP command.
I have never in my 8 years seen it happen otherwise.  Please contact Tech Support before you disconnect any other APs, and provide logs
and network diagram for a detailed analysis.   https://support.ruckuswireless.com/contact-us
Photo of Wopke Weening

Wopke Weening

  • 9 Posts
  • 1 Reply Like
Hi Michael,

I did raise a case with tech support and provided logs. Case # 00557959
Photo of Monnat Systems

Monnat Systems, AlphaDog

  • 759 Posts
  • 162 Reply Likes
whats the current status for this Case # 00557959 ??
Photo of Wopke Weening

Wopke Weening

  • 9 Posts
  • 1 Reply Like
waiting for ruckus support
Photo of Monnat Systems

Monnat Systems, AlphaDog

  • 759 Posts
  • 162 Reply Likes
how old case it is??
this thread is indeed around a month old and issue was reported many times in the forums..
Photo of Monnat Systems

Monnat Systems, AlphaDog

  • 759 Posts
  • 162 Reply Likes
in your case, how do u solve this problem (if temporarily)??
or does it get solved on its own... pls share your fixing procedure whatever it may be so that guys can learn.... Outdoors AP going NUTS r such PIA...
(Edited)
Photo of Wopke Weening

Wopke Weening

  • 9 Posts
  • 1 Reply Like
I reported the case yesterday. With logs of 30 august 2017.
We don't have a real fix. What we do it to reconnect the resetted AP to the same switch as the master AP. It then reloads it's config an can used as meshed AP. It the works fine for a few weeks or days and then resets again.      
Photo of Monnat Systems

Monnat Systems, AlphaDog

  • 759 Posts
  • 162 Reply Likes
problem is removing the AP's from Height locations and reconnecting back to network...
 
Photo of Wopke Weening

Wopke Weening

  • 9 Posts
  • 1 Reply Like
We also see a lot of restarts of the master AP's in our UL networks due to "kernel panics" I submitted a case for that on also (#00552637). We provided logs for hat case also and activated highmem logs on request of Ruckus. I suspect numerous restarts of the master AP form kernel panics triggers somehow the reset of the meshed AP's, but you're the expert.    
Photo of Wopke Weening

Wopke Weening

  • 9 Posts
  • 1 Reply Like
Hi there,

I'm from the Netherlands and we experience the same problems. We have about 20 UL networks each consisting of 4 T300 AP's of which one is connected to a router and the other three AP's are wirelessly meshed. Note that we do use DHCP.
In at least 5 of these 20 networks we experience "spontaneous factory resets" every few weeks or days of some of the meshed AP's.  
DHCP is not the solution for this problem. 
Photo of Monnat Systems

Monnat Systems, AlphaDog

  • 759 Posts
  • 162 Reply Likes
well this is a indeed a major issue and issue should get priortized to P1...
Photo of Jeronimo

Jeronimo

  • 170 Posts
  • 19 Reply Likes
This is very very serious issue.

Especially, in mesh network.

Must fix it asap.
Photo of Marco Eichstetter

Marco Eichstetter

  • 148 Posts
  • 7 Reply Likes
Now I have same issue at another Site, too.
For me it seems that i happends in Mesh-Networks where some Interference present.
If the Mesh-Link of an AP is not always 100% stable, the AP has a higher risk resetting itself to Factory Default.

In a Mesh Setup with high mounted APs a serious issue.
(Edited)
Photo of Stanley Zhou

Stanley Zhou

  • 4 Posts
  • 2 Reply Likes
Hi All:
    In Unleashed AP, only one way the AP will set factory automatically, that's PK fail (There are two master AP, it PK, then one master will become member AP).
    I guess in this case, whether in some reason the mesh AP (T300) gateway is reachable and become root role, then it is elected as master, then come back into the network, it PK by the old master, so it will set factory.
    Can you tell me what's Ip configure for this T300? and what's IP configured for the Master AP? please give us the IP, and gateway infomation.  SDC will try to reproduce it.

   Anyway, in 200.4 patch (200.4.9.13.49) and 200.5 version, we have changed the logic, when Pk fail, if the AP is not gateway mode, it will not set factory, it only reboot. So I think if can customer can try to use 200.4 patch or 200.5 version.
Photo of Jeronimo

Jeronimo

  • 170 Posts
  • 19 Reply Likes
What does PK mean?
Photo of Marco Eichstetter

Marco Eichstetter

  • 146 Posts
  • 7 Reply Likes
Yes, i also asked me, what PK means?
I have Version 200.4.9.13.47 installed in my Networks. Where do i find 200.4.9.13.49 or 200.5?

IP Config:
Default GW: 172.17.5.1
DNS: 172.17.5.1
AP01 ::: Root-AP ::: 172.17.5.61 ::: T301s
AP02 ::: Root-AP ::: 172.17.5.62 ::: T300
AP03 ::: Mesh-AP ::: 172.17.5.63 ::: T300 ::: Factory Resetted
AP04 ::: Mesh-AP ::: 172.17.5.64 ::: T300 ::: Factory Resetted
AP05 ::: Mesh-AP ::: 172.17.5.65 ::: T301s ::: Factory Resetted
AP06 ::: Mesh-AP ::: 172.17.5.66 ::: T301s ::: Factory Resetted
AP07 ::: Mesh-AP ::: 172.17.5.67 ::: T300
AP08 ::: Mesh-AP ::: 172.17.5.68 ::: T300
AP09 ::: Mesh-AP :::172.17.5.69 ::: T300

The LAN Interface of each Mesh-AP is connected either to a IP-Cam or with a Switch (and Multible IP-Cams).
Thanks.
(Edited)
Photo of Wopke Weening

Wopke Weening

  • 9 Posts
  • 1 Reply Like
Not sure what PK means either. I suspect Stanley means the Paraphrase Key (The wireless mesh key).
In our set-up we use networks of 4 T-300 AP's of which only 1 has a cable connection, This one is always the only the master . The other 3 are wirelessly meshed.

I do understand that one of the meshed AP's might loose it's connection with the master AP for any given reason. (Interference, restart of the master etc..)
What I don't get is how the meshed AP after restart would somehow decide it might be master? It can not reach the gateway by any other means than the wireless uplink to the "only real master". Even more so, as we use DHCP it doesn't even get an IP address if no connection can be established to the DHCP server, which it only can reach via the original wireless uplink. IP conflicts are out of the question therefore.     

We also run 200.4.9.13.47 and would very much like get any firmware version that would prevent automatic factory defaults.
Photo of Jeronimo

Jeronimo

  • 170 Posts
  • 19 Reply Likes
We had found another symprom.

When master ap changed to member ap or memeber ap changed to master ap, sometimes the ap is taken old master ip, then old master and new mster are conflicted ip due to have same ip.

We have met this symptom about four times on out LAB and a customer site.

The UAPs have static ip.

If this symptom was happened, the network have been unstabled.

Regards.
Photo of Wopke Weening

Wopke Weening

  • 9 Posts
  • 1 Reply Like
Today I received a text from the customer the AP once more fell into "configure me state" This is the 3d time in 3 weeks and the second time in 4 days. This is riducules bad service. For me It is a 4 hour drive to the customer for the 3 time in 3 weeks :-( This time I will replace the network witch a different brand Wifi solution because I cannot explain this lack of performance anymore. Please Ruckus fix this a.s.a.p. because this is realy bad for our business.  
Photo of Jeronimo

Jeronimo

  • 170 Posts
  • 19 Reply Likes
I tested UAPs all connected wired network without mesh network.

In wired network, similary and diffrerent symptom are also happend.

Regards.
Photo of Michael Brado

Michael Brado, Official Rep

  • 2106 Posts
  • 297 Reply Likes
No, Stanley Zhou, T300 mesh AP cannot become Master AP.  I don't know "PK" acronym in Unleashed terms either.

Everyone needs to please open individual tickets with Tech Support, and provide your logs and network diagrams.

If this is indeed something with APs going to factory default, it needs to be tracked by Tech Support, who can open Bugs with Engineering.

Please open a ticket with Tech Support, or ask for escalation of your existing tickets if no Bug has been filed on your behalf (yet).  Thanks!

https://support.ruckuswireless.com/contact-us
Photo of Wopke Weening

Wopke Weening

  • 9 Posts
  • 1 Reply Like
Escalation and bug fix requested with tech support 5 mins ago. Logs I provided earlier already.
Photo of Marco Eichstetter

Marco Eichstetter

  • 146 Posts
  • 7 Reply Likes
Please share your Solution or Result with us if you have some new Information. Thanks.
Photo of Marco Eichstetter

Marco Eichstetter

  • 148 Posts
  • 7 Reply Likes
Ticket opened with Case No 00556991
Photo of Marco Eichstetter

Marco Eichstetter

  • 148 Posts
  • 7 Reply Likes
Additional Info:
I saw this Problem first with FW 200.4.9.13.47.
Before that i had installed a 200.3 without Factory Resets.
Photo of Michael Brado

Michael Brado, Official Rep

  • 2104 Posts
  • 297 Reply Likes
Thanks Marco, I added that comment to our bug.
Photo of Wopke Weening

Wopke Weening

  • 9 Posts
  • 1 Reply Like
I've seen the same problem with 200.3 sw. 

I only have this experience with T300 meshed networks. This might mean the problem is in the T300 but doesn't have to mean that. We also use R510 UL networks but in those cases the networks are: 1- indoor, 2-fully cabled. So no ceteris paribus comparisons there. 

One thing I also observed is that the "kernel panics" somehow seem to get triggered by "radar detection". The odd thing is that an other network,150 meters away from it doesn't detect radar. As (weather) radars are quite powerful it doesn't make sense that one network does detect radar and the other networks (which are exact copies) do not. I wonder if a false positive on radar detection might trigger a kernel overload?
Photo of Marco Eichstetter

Marco Eichstetter

  • 148 Posts
  • 7 Reply Likes
Ok. I thought about it. It may be that i have seen Factory Resets with 200.3, too.
It may be that my first "factory-resetted" APs were 200.3. Not 100% sure.
But i have the feeling that there are more Resets with 200.4.

Your observation may be true. Yes. My two problematic Sites have also a lot of radar detects. On other Sites where no Factory Reset occours (till now) have no or less radar detects.
Maybe Radard detects (no matter if false positive or not) may trigger "kernel overload" or "Factory Resets".
Photo of Michael Brado

Michael Brado, Official Rep

  • 2106 Posts
  • 297 Reply Likes
The Bug ID is ER-5664: Unleashed 200.4 Frequent AP kernel panics, and the Developers are checking to see if fix is in potential 200.5 Unleashed (GA), or it certainly will HAVE to be!  I filed the bug.
Photo of Marco Eichstetter

Marco Eichstetter

  • 148 Posts
  • 7 Reply Likes
Thanks for Feedback.
Hopefully 200.5 comes fast.
Yesterday the next Mesh-AP (T300) resetted. This is Number 5 within about 3 Month.
Photo of Monnat Systems

Monnat Systems, AlphaDog

  • 759 Posts
  • 162 Reply Likes
looking at entire thread... issue is specific to unleashed based setup, mesh and T300. correct?
do we see this issue on some other model like Unleashed R500 and Mesh...??
Photo of Marco Eichstetter

Marco Eichstetter

  • 148 Posts
  • 7 Reply Likes
Customer Support mentioned that the Bug with ID ER-5664 are also the root cause for my Factory resetted APs.
Bug will be resolved in 200.5.
Customer Support timelined 200.5 for first week of october.
I hope so.
Photo of Jeronimo

Jeronimo

  • 170 Posts
  • 19 Reply Likes
thumb up.
Photo of Marco Eichstetter

Marco Eichstetter

  • 148 Posts
  • 7 Reply Likes
200.5 is released but within the Release Notes i don't find any Information about a resolved "Kernel Panic" or "Factory Reset" Issue.
Did i missed something or is the Issue not solved till now?
Please clarify.
Thanks.