Ruckus Cloud Outage - Network Down - APs and Licenses got deleted

  • 1
  • Question
  • Updated 3 months ago
  • Answered
  • (Edited)
Received alerts this morning at 4am PST all our licenses expired.  Checked online and all our APs got deleted.

Got on the phone with Ruckus and they say its a known outage.

Anyone else still down at this moment?

All our APs got wiped from our cloud portal, end users no longer seeing our SSID brodcast.. we are completely down on wifi


edit: we are in Los Angeles
Photo of WL

WL

  • 9 Posts
  • 0 Reply Likes

Posted 3 months ago

  • 1
Photo of RN

RN

  • 15 Posts
  • 2 Reply Likes
I am also facing the same situation here in Sao Paulo (Brazil)  since 7:00 am BRT (02:00 AM PST).

The case has been opened at Ruckus Support, but there is no estimate of when it will be resolved.

All our APs have been disconnected since 7 am.


Photo of RN

RN

  • 15 Posts
  • 2 Reply Likes
Maybe I have not done a full search, but I found no previous warning / notification that there would be a maintenance on Ruckus Cloud today, either, that the service could be an outage for so long.
Photo of WL

WL

  • 9 Posts
  • 0 Reply Likes
we are still down -- its been about 10-11 hours total now.  have you dialed phone support? any updates
Photo of RN

RN

  • 15 Posts
  • 2 Reply Likes
Yes, I talked to a Ruckus Support engineer by phone early in the morning.

He said they are working to reactivate the service as soon as possible, it is an issue with License Server, the case is escalated, but in fact, nothing has changed since the beginning of this outage.

Still down since 7:00 AM BRT (02:00 AM PST) Over 11 hours now.
(Edited)
Photo of RN

RN

  • 15 Posts
  • 2 Reply Likes
How about you WL in LA? Is your network back - up and running?
Photo of Glen Grotendorst

Glen Grotendorst

  • 2 Posts
  • 0 Reply Likes
We are still down in Florida 5:02 PM Eastern time.  Over 12 Hours now.  Very very bad for us.
Photo of WL

WL

  • 9 Posts
  • 0 Reply Likes
are you guys showing any AP in the portal? or is it still 0?


Photo of RN

RN

  • 15 Posts
  • 2 Reply Likes
Zero! 
Photo of WL

WL

  • 9 Posts
  • 0 Reply Likes
from my observation: it looks like the licenses got readded to my account - tried manually adding the serial numbers to the AP, it gets added but the provisioning server is not up.  so the APs show "never connected to cloud"
Photo of WL

WL

  • 9 Posts
  • 0 Reply Likes
still down - any updates from you guys? its now 13+hours
Photo of RN

RN

  • 15 Posts
  • 2 Reply Likes
Still down here too. After more than 14 hours.

No Ruckus Support information / contact for more than 5 hours - total silence.
Photo of WL

WL

  • 9 Posts
  • 0 Reply Likes
still down for me. 11PM pst... anyone else?

i am still showing zero APs on my portal. end users do not see any SSID broadcasted.

/anyone got any news from support???
Photo of RN

RN

  • 15 Posts
  • 2 Reply Likes
28 hours later and the network still down.
(Edited)
Photo of RN

RN

  • 15 Posts
  • 2 Reply Likes
Any news from folks in Florida and Los Angeles? Is anyone somewhere else in the world still having trouble with Ruckus Cloud?

Has anyone managed to implement any bypass solution to reactivate the network?
Photo of Glen Grotendorst

Glen Grotendorst

  • 2 Posts
  • 0 Reply Likes
It seems we are up and running in Florida.  Not sure about the rest of the World.

Photo of WL

WL

  • 9 Posts
  • 0 Reply Likes
Only 4 of our APs came back online. The remaining 9 are still offline. It says 'never contacted cloud'

We have a total of 3 venues, so only 1 venue is up, but only 30%. not even fully. The other two sites are down, completely

How many APs do you guys have? Phone support is saying they are combing per accounts to manually add and delete APs, per account. There's no blanket fix I don't think

Still down in Los Angeles
Photo of RN

RN

  • 15 Posts
  • 2 Reply Likes
We have 1 venue and 38 APs still offline. We have tried to enter the APs manually, but have received the message that the AP serial number already exists.

The Ruckus Support message remains the same. They hope to resolve in the next few hours and ask for patience.

Still down in Sao Paulo.
(Edited)
Photo of Rajiv Iyer

Rajiv Iyer

  • 7 Posts
  • 0 Reply Likes
Just updating this thread. As of 7 hours back, APs were back online.
Photo of WL

WL

  • 9 Posts
  • 0 Reply Likes
we just got fixed- up now in Los Angeles.

the main issue is the WAPs are pointing to license-expired.ruckus.com, you will likely need an engineer on to ssh into each AP to point it away.

this office location has only 13APs, which is managable.. i cant imagine what can be done with a much higher AP count, if not done manually
Photo of Rajiv Iyer

Rajiv Iyer, Employee

  • 7 Posts
  • 0 Reply Likes

Hi all,

We are deeply sorry for this service outage. Ruckus cloud team is working on resolving this issue.

We are being hit by a Flexera licensing outage. 3.6% of cloud managed APs were impacted yesterday. As of this morning,  50% of the APs are back online. Currently, 1.8% of  APs are still impacted. This number includes APs in North America and Europe data centers.

Thank you,

Ruckus Cloud Team

 


Photo of Rajiv Iyer

Rajiv Iyer, Employee

  • 7 Posts
  • 0 Reply Likes

Ruckus Cloud Service Status update:

  • Ruckus Cloud service is mostly restored. 
  • A very small set of customers are being impacted at this time
  • In North America, 21 APs across 11 customers are still impacted
  • In Europe, 30 APs across 12 customers are still impacted
  • Since Flexera licensing service is showing intermittent connectivity, we have taken steps to prevent this from impacting Ruckus Cloud service

Ruckus Cloud team is continuing to work the issue. Sorry for the inconvenience and thanks for your patience and understanding. 
Photo of RN

RN

  • 15 Posts
  • 2 Reply Likes
Hi Rajiv,

I really appreciate your update, however, the numbers need to be updated.

There are more 38 APs remain completely unavailable since at 7 yesterday morning.

These APs are in Sao Paulo (Brazil) and we connected to the Ruckus Cloud US.
Photo of Rajiv Iyer

Rajiv Iyer, Employee

  • 7 Posts
  • 0 Reply Likes
Ouch. Can you email me ([email protected]) your tenant ID?
Photo of RN

RN

  • 15 Posts
  • 2 Reply Likes
E-mail sent! 

Thank you very much in advance.
Photo of Rajiv Iyer

Rajiv Iyer, Employee

  • 7 Posts
  • 0 Reply Likes
Ricardo - Got it. NOC and customer support teams are working on it.
Photo of WL

WL

  • 9 Posts
  • 0 Reply Likes
@Rajiv,

Appreciate you being on this thread for the update-

Can I suggest an official NOC Alert or Status page for future situations like this?


Photo of Rajiv Iyer

Rajiv Iyer, Employee

  • 7 Posts
  • 0 Reply Likes
Very definitely! That's something we are discussing too. Thanks.
Photo of Rajiv Iyer

Rajiv Iyer, Employee

  • 7 Posts
  • 0 Reply Likes
Update:
------------

We have confirmed that Ruckus Cloud service has been restored for nearly all customers. If you believe, that you are still having issues, please contact Ruckus customer support.

Thanks for your understanding and patience.