ZF7372 High Client Density issue

  • 1
  • Question
  • Updated 4 years ago
  • Doesn't Need an Answer
Hi Everyone,

I'm evaluationg the zf7372 as a potential replacement for some 2942's and 7363's that I use in a very high density environment. We typically budget 75-85 stations per radio on the current AP's we have. From the documentation on the 7372 it looks like it can handle 250 stations per radio and I'd really like to present a case for upgrading to reduce protocol overhead by reducing the number of AP's, but I need to get a proof working in the lab first.

I've setup a testing environment in one of my labs, and have only been able to get ~120 devices to associate before the AP stops accepting associations. The logs on the attached ZD1100 show "User [DEVICE_MAC] fails to join WLAN [WLAN_NAME] from AP[AP_MAC]"

I've done a little more snooping using some of the tools I have and it appears that once I hit this ~120 device limit, the newer devices are able to associate, but are immediately (within ~7ms) sent a deauth frame. This testing environment is an open/WEP64 wlan, so I don't think I'm running into memory or other resource issues on the AP during the negotiation phase, but I could be wrong. I've looked at the AP logs and they show that the AP has plenty of memory remaining (60+MB)

I'm running 9.6.0.0.267 on a zf1100 with the same release on the AP.

The logs on the AP show the following errors once I reach the ~120 device range and start to see the deauth frame response to new clients:

Aug 12 23:35:04 RuckusAP user.info kernel: tac_set_station_key(): tac_set_station_key: new key failed
Aug 12 23:35:04 RuckusAP user.info kernel: net80211_tac_cfg_sta_add(): add station {DEVICE_MAC} session key,failed cipher = 2
Aug 12 23:35:04 RuckusAP user.info kernel: tac_set_station_key(): tac_set_station_key: new key failed
Aug 12 23:35:04 RuckusAP user.info kernel: net80211_tac_cfg_sta_add(): add station {DEVICE_MAC} session key,failed cipher = 2
Aug 12 23:35:04 RuckusAP local2.err syslog: Failed add station in processing MSG_MOBILE_CFG_REQ

I've tried disabling all the features I can to narrow down the issue, but I haven't had any luck (background scanning on/off, dropping multicast packets on/off, client load balancing on/off, client fingerprinting on/off, only one wlan active on the AP, etc). I've also ensured that the device limits are set higher than 120 clients so I don't believe I'm running into an issue there. I also don't see any warnings about reaching 90%+ of the AP's capacity as I generally do when I approach the limits on a 7363 device which also leads me to believe that it's a software issue and not a misconfiguration.

Anyone have some pointers on what might be going on here? It's starting to look like a bug in the AP firmware somewhere in the key management subsystem to me.

~WlanGeek
Photo of WlanGeek

WlanGeek

  • 7 Posts
  • 0 Reply Likes
  • sad

Posted 4 years ago

  • 1
Photo of Keith - Pack Leader

Keith - Pack Leader

  • 860 Posts
  • 50 Reply Likes
Do you have TKIP enabled by any chance? Switch to AES explicit if so (don't use auto). TKIP abuses the CPU.
Photo of WlanGeek

WlanGeek

  • 7 Posts
  • 0 Reply Likes
I'm using wep-64 as most of my device chiptsets have builtin hardware for wep and don't require the use of wpa-supplicant to handle encryption. The CPU utilization seemed to be around 50% according to the logs while I was experiencing this issue. Any chance you could forward the error log to an engineer? This seems like the kinda thing the right individual would look at and either say "hrm...that's interesting...it should never do that" or "hrm...I know just what's going on here".
Photo of Keith - Pack Leader

Keith - Pack Leader

  • 860 Posts
  • 50 Reply Likes
If you get a case opened I can get it to our escalation team (I would open a case for you but I can't ID you in any of our systems using your forum email...)

-K
Photo of Michael Brado

Michael Brado, Official Rep

  • 1856 Posts
  • 266 Reply Likes
WlanGeek, your AP can support more than 100 clients, thou WEP is not a regular
client, but requires wifi chip hardware keycache which is limited to 128 slots, and
with some overhead, 112 slots is a more likely limit for WEP (only) clients. You
should find that you can add additional Open Auth (or WPA) type clients after
you've maxed the WEP clients.

Please also be aware that WEP will not be supported in our (very near) future
ZoneDirector/SCG releases of firmware.
Photo of WlanGeek

WlanGeek

  • 7 Posts
  • 0 Reply Likes
Hi Michael,

I've tried WPA2 with AES and seem to be hitting a limit at ~112 clients with a similar message to the WEP case in the AP's onboard logs. When you said WPA type clients were you refering to all the WPA versions, or only to WPA (and excluding WPA2)?
Photo of WlanGeek

WlanGeek

  • 7 Posts
  • 0 Reply Likes
ttt
Photo of Craig Black

Craig Black

  • 1 Post
  • 0 Reply Likes
Another one here on 9.6.0.0... WPA2 with AES, ~112 clients then the super quick auth / deauth... ZF7982 APs, more than capable of handling this number of clients... In fact, they do, but no more than 112 on the WPA2 WLAN... HELP?!
Photo of WlanGeek

WlanGeek

  • 7 Posts
  • 0 Reply Likes
@Craig, Are you able to get more than 112 clients with some kind of encryption associated with an AP (eg 100 WPA2 and 100 WEP)? I'm starting to wonder if the 250device/radio claim only applies when you're not running any kind of encryption or security and there's still a ~100 device limit when you're running something other than a completely open network. Anyone from Ruckus want to chime in here?
Photo of WlanGeek

WlanGeek

  • 7 Posts
  • 0 Reply Likes
Soo...ruckus...where are you guys? This seems like a pretty simple question to answer. This doesn't give me warm fuzzies about becoming a ruckus customer.
Photo of Keith - Pack Leader

Keith - Pack Leader

  • 860 Posts
  • 50 Reply Likes
@wlangeek - I'm working on getting someone to provide details on how the AP max clients values are obtained and what if any caveats may exist.
Photo of Eizens Putnins

Eizens Putnins

  • 107 Posts
  • 42 Reply Likes
I have seen mentioned in the docs, that 250 associated clients per radio are supported without authentication and encryption.
Iti will be not a surprise, if using WPA/TKIP and even worse, WEP can degrade max connected client quantity. I suppose, WPA2/AES must be better.

I also can't imagine what kind of environment you have, which require high density for devices, supporting WEP-only. Antique devices usually have also 802.11b WLAN cards, which will make such environment not usable much before 100 associations, also they must be at least 10 years old, and must be a candidate for replacement long ago. WEP also normally isn't allowed on corporate networks because of security reasons.
As WEP is not much a security any more, you would be probably in better position using just WEB authentication without encryption (we have used it in a very loaded networks, and it works well), allowing much more clients.

Hope it helps,
Eizens
Photo of David Botha

David Botha, Employee

  • 11 Posts
  • 8 Reply Likes
Hi all - sorry for delayed response here.
The 250+ client limit is for un-encrypted only. For encrypted clients eg WPA/AES, the limitation is just over 100 clients, defined by the size of the encryption block in the WiFi chips used in our APs.

-Dave
Photo of WlanGeek

WlanGeek

  • 7 Posts
  • 0 Reply Likes
Thanks @Dave!

Can you clarify for me whether this encrypted connection limit is ~100 devices per radio, or ~100 devices per access point? I have some ability to force my devices to load balance between the 2.4GHz and 5GHz bands and I might be able to make this work with some creativity on the client side if I could get 100 devices per radio (Eg. 200 per AP - 100 on 2.4GHz and 100 on 5GHz ).

@Eizens: I'm not forced to use WEP for this application, but it was the simplest to implement initially so I went with that. I've done some small scale testing with ~150 devices using WPA2/AES and found that the limit was ~112 devices there as well. I was hoping that the 250 device/radio spec would hold for some of the newer 802.11 security methods as much of the overhaed is handled in user space by WPA Supplicant on the *nix side (thereby dodging hardware encryption engine limitations and being limited instead by CPU, memory, and latency).

-WlanGeek
Photo of David Botha

David Botha, Employee

  • 11 Posts
  • 8 Reply Likes
The 100 encrypted connection limit is per radio, so 200 per AP for a dual-band like 7372. Note that if the encryption table is full (e.g. 100 clients), it is still possible to add un-encrypted connections until the sum total of connections reaches the un-encrypted limit.
Photo of Jon Prouty

Jon Prouty

  • 9 Posts
  • 0 Reply Likes
WlanGeek, I am concerned of your initial step of trying to reduce ap count to open up protocol overhead? Are you talking about the management and control frames? If you are having traffic issues with your ap's separate the ap's into different vlans to reduce the amount of traffic in each vlan. Keep your clients separate from your management vlans and the management traffic should be a negligible issue if one at all. Your clients will have a much easier time if you can spread the clients out between more radios, not fewer radios at higher client density! Remember this is a shared medium and only one client can talk to one radio at a time, throughput of a radio with 250 clients attached to it has got to be a trickle and that would happen even with ac and 80 MHz channels. That is setting yourself up for failure in my opinion. Do you really want 250 clients on each radio? Not this guy!
Photo of Eizens Putnins

Eizens Putnins

  • 107 Posts
  • 42 Reply Likes
Hi, Jon Prouty,
Actually exactly because it is shared medium you want to have less APs supporting more clients -- because if you have multiple APs, on the same channel and near to each other, only one radio anyway can transmit at the time, and there are a lot of additional overheads, primarily related with detection if channel is free to transmit, contention and management traffic of every AP. This overheads may go up to 50% of available bandwidth.
And if you try to say that it is better to use 2 or more APs on different channels, it is normally done already -- we have already situation when there are no unused channels available to use for additional capacity, so you must manage as much efficiently as possible what you have available, and it is much easier with less high-density APs, than with many low density APs.
And with 802.11ac we soon will have situation on 5Ghz quit similar to what we have in 2,4 GHz now -- and efficient management of available spectrum is extremely important to get satisfactory results.
Photo of Jon Prouty

Jon Prouty

  • 9 Posts
  • 0 Reply Likes
Hi Eizens, the way you get by co-channel interference between ap's is properly surveying before you place your access point. If you are running into areas that can no longer place any more access points without causing co-channel interference then yes you have to start replacing older model ap's with newer ap's that can handle the higher densities of clients. The overall fact here that I was trying to point out is the clients will experience better overall performance with fewer clients per radio with all clients (abgnac).
Photo of Eizens Putnins

Eizens Putnins

  • 107 Posts
  • 42 Reply Likes
Hi, Jon,
You probably leave in some happy land, where you can install AP and it will be the only on the channel.
During last 3 years I haven't seen any installation in a city, where I can't see on the every channel at least 3-4 APs, and what is even worth -- most of them are out of your control at all. Only exception are metal buildings, which are shielding everything coming from outside, but than everything which is inside creates even more interference.
Especially much harm cause cheap home 802.11N gear, configured to use 40 MHz channels on 2,4 GHz, and old 802.11b/g APs. You don't need to have 30 dbm level to prevent transmission, it is enough that client can hear -70dbm level of neighbor AP transmission.
The only way to fight this till now is using Ruckus APs with they automatically adjusted directional antenna arrays -- if they doesn't work, my practice shows that no other WiFi gear work.
Recently we had a case -- Wi-Fi (brand new Apple Exterme Air-port and SMB APs from Cisco-Linksys and old 802.11g HP Procurve) provided 10-20 KB/s download speeds in 10 meters of AP in the same room, without any obstacles. Pwner of brand new Apple Air was quit unhappy.
Survey has demonstrated 20x APs with RSSI more that -50 dbm. One was public Hot-spot AP on the next building, actually in front of window, providing -30 dbm RSSI, and there have been at least one AP with RSSI higher that -40 dbm on every channel. There was no non-wifi interference, as we checked using spectrum analyzer, jut spectrum was used up to 80-100% average.
As a last hope I provided Ruckus ZF7982 AP, and never get it back -- as Apple Air have got on spot decent 50 MB/s download speed.

And if you really want more performance, you must get read-off 802.11b clients and disable non-OFDM modulations, this would improve everything except for antique devices.
Photo of Eizens Putnins

Eizens Putnins

  • 107 Posts
  • 42 Reply Likes
It would be really peace of cake, if our Wi-Fi installations would be just islands in a non-wi-fi-ed world, as it was 10 years ago. SOHO equipment unnumbered long ago any enterprise equipment, so whenever we look, we can feel them...
So 2,4 GHz band is going to be abandoned because it is overloaded...
But I can say for sure - it is not the end, no -- it is actually the beginning... Now we have 802.11ac standard in place, so soon most devices will support at least some limited 802.11ac version, and 5GHz band will be as much full of garbage, as 2,4GHz is now.
I expect that actually in most dense enterprise applications 802.11ac APs will use 40 MHz channels, to avoid interference from neighbor APs and outside devices, and real improvement of speed will come not from plain numbers (such as 160MHZ channel or 256 QAM modulation), but from better chipsets, better firmware, better active antennas, better radio-management and so on. And it will not come as easy as we would like.
And surveys have now to be done not just for coverage and inter-AP interference, but for capacity, taking into account Wi-Fi system intelligence and environment.
It is much more complicated, so hopefully new tools are coming to help us to do it soon.
Hope it helps - we have to discuss this issues quit often now, when 802.11ac gear comes more and more around.
Photo of Jon Prouty

Jon Prouty

  • 9 Posts
  • 0 Reply Likes
I do live in the USA, probably qualifies as happy land compared to most!

Thanks for spouting off the last few months of webinars for everyone. Definitely needed that...

And after all of that the fact still remains that fewer clients per radio will give you better overall performance.

Thanks though!
Photo of Keith - Pack Leader

Keith - Pack Leader

  • 860 Posts
  • 50 Reply Likes
I love a vigorous debate and see both sides - but so much depends on the environment. I suspect that local conditions differ quite a lot between the different points of view.

I should try to get a screenshot of the EMF in our support lab and post it here. It's appalling (we're next door to the QA lab). My Mac Pro can barely display all the SSIDs (all rogues to boot). Yet client connections are amazingly stable and responsive. Well, here's the view from my desk where I have only plugged in once (due to a quickly-resolved RADIUS bug. Names pixelated to protect the innocent - but who is Video 54?)



And if you can't buy Ruckus...there is [maybe] another answer

http://www.lessemf.com/paint.html

;) [No endorsement - I've no idea whether this stuff works.]
Photo of Eizens Putnins

Eizens Putnins

  • 107 Posts
  • 42 Reply Likes
I like how you stick to your opinion. There is nothing personal in my comments -- please don't feel offended in any way. I like that you have looked on last Ruckus webinars, I had merely time to look on some recordings only, unfortunately, but I like them -- Ruckus guys as usually know staff quit well and provide it in easy understandable form, which is especially important for new users.
And I have to completely agree -- one client sitting on an AP will have the best performance possible, all for himself. As well as 100 clients distributed to 10 APs, without any interference, would obviously have better performance than 100 clients on 2 APs. This is OK and completely reasonable, and is very correct with lightly loaded networks and clean areas.
Problem is that you usually just never get such situation any more. And if you have 200 uniformly (and heavily) loaded clients in compact area (say, 50x50 m2) and have to decide if you have to use 10x or 2-3x APs, usually 2-3 good APs will actually provide much better overall performance. It is not simple to decide, anyway, as in some conditions it may also be completely opposite -- it may be better having more APs.
Simple explanation is -- you have 3 non-overlapping channels on 2.4Ghz, so adding more APs in same area will just divide same resources between more AP and add a lot of overheads. But as usually, the simple explanation is not correct, as connection speeds for clients have huge impact as well as other factors.
Even more complicated it becomes with Ruckus, as these APs use different antenna patterns for downstream and upstream traffic, and transmission becomes very asymmetric. There is no simple answer any more.
Anyway, having more client supported on AP is usually a huge benefit. This is a typical limitation for SOHO devices, we had replaced during last years a lot of networks in hotels, based on low capacity gear, which became unusable in current conditions.
Most obvious it is in 2,4 GHz, of cause, but 5GHz is getting close, and a lot of enterprise laptops still have 2,4GHz-only cards in our country.
If area is compact and not divided by metal walls, you'll have 3-4 APs on the same channel, you'll have also neighbors, and performance can be actually much less than expected. But it depends on many things, including traffic pattern for clients.
Also some APs would work better in such situation than others (it depends on antennas, primary, but also on hardware and software).
Typical problem recently became that you have a lot of passing-by clients (smartphones), which actually don't want to use your network, but try to connect to it automatically anyway. They can make your AP tables full, if it is not designed to handle it. I had such experience in public areas (especially with open SSIDs), with previous generation Cisco and HP equipment, and the only solution 2 years ago was to replace this gear by working solution from different vendor (surprise?).
We have 3 years old installation with 10x APs producing 200GB traffic per day traffic (and there is about 80 rogue APs around them). Number of associated clients comes to 100 per AP twice a day now, so customer plans to upgrade system.
By the way, in USA you must have even more problems with interference on 2.4GHz as more power and less channels are allowed, but you have much more territory, probably. Also for headquarters with a huge parks around it is really possible to discard any external interference, and have 802.11n in green mode.
It is not the case in our country.
In our office building, there are about 40 companies on 5 floors, even smallest having at least one Wi-Fi AP, and we have 3 multi-tenant buildings in vicinity with a lot of home equipment. Our office list of rogues is 60+ APs, some with RSSI -25dbm. And this is a typical situation, not an exception.
We have just learned in the field during last 3 years that Wi-Fi planning is much more complicated and no webinars or handbooks help much -- you need practical experience, good equipment, you have to try different solutions and you'll get some problems anyway.
There is no going back of cause, and we'll figure how to get most of 802.11ac soon..
Life is life...
Have a good day and as much free of interference channels, as possible!
.
Photo of Jon Prouty

Jon Prouty

  • 9 Posts
  • 0 Reply Likes
I'm not offended and not trying to offend, the internet is hard to relay exactly what you are trying to say. Definitely being aware of your environment with your wifi systems is key which Eizens has demonstrated he is well aware of his environments! I work in education so in most cases they get to control all the airspace around them since a lot of the schools I work with are in unpopulated areas. I have worked with schools in some dense areas and neighboring rf is something you definitely have to be aware of when designing the system. I did one floor of a skyscraper in Seattle a couple years back and that was a fun system to design. I also had to negate a sales guys deal one time when he was trying to sell a 2.4 bridge to hook up two sites in Seattle that were separated by several apartment complexes, there was only just over a hundred ssid's in between the sites in the 2.4 range.

SNR is key in these super dense environments which is why the ruckus ap's do so well by having the adaptive antennas to give the client the best signal quality possible. Having an adaptive controller setup that can adjust channels and power levels of the wifi system to mitigate interfering rf is key as well.

I wish I could leave the 2.4 airspace behind as well but education is full of legacy clients. They will get there one day but until then you always have to use that frequency as your base for installations as it has the larger rf footprint and the fewest channels available.