Alas, my strained relation with Cisco wireless controllers rolls on. My 8540s are on super-wonderful-we-REALLY-tested-it-this-time 220.127.116.11 code, yet *gasp* I’m looking at yet another bug-driven upgrade. Or I can just disable MU-MIMO as listed in the work-around and yet again not use what I paid for! But you don’t care about that rot, as that has nothing to do with the point of this blog. That was just pre-content bitching, as an added bonus.
Let’s get on to the meat and taters of it all.
Take a look at this:
When I’m in the WLC interface, there are various ways to sort for specific APs or groups of APs. The ability to search on Speed is fairly new, and if it’s not obvious it’s talking about the wired connectivity of the AP and is relevant where mGig switch ports are in use. That’s fairly innocuous, yes?
But here’s WHAT COULD HAVE BEEN.
Suppose that Speed search gave another option- for 100 Mbps? Some of you know where this is going…
In a perfect world, all 3000+ access points on this 8540 would connect to their switch ports at Gig or mGig, depending on models of hardware in play. But the world isn’t perfect. Occasionally, some of those thousands of APs connected to hundreds of switches for whatever reason only connect at 100 Mbps. More often than not, that’s indicative of a cabling issue. Once in a great while it’s a switch misconfig or a bad AP in play.
As is, there is no easy way to find those APs that have joined at 100 Mbps in the controller. An AP that connects at 100 Mbps doesn’t trigger a fault. You can’t sort on the speed column, and basically have to wade through almost 50 pages in my case looking for the elusive 100 in a sea of 1000s in the speed column.
Boy it would have been handy if the developers gave us a 100 Mbps option in that Speed search.
I’ve learned that all people are interesting, but sometimes you cross paths with a person in a way that stands out and leaves a particular impression. (Please note- this is the rare non-technical blog, but read on for a couple of quick but fascinating events). In the last several days, I’ve had the good fortune to be in the presence of two fellow human beings that made me go “wow” after talking with them.
1971 World Series Ring Baseball Guy
My wife and I were watching an Auburn Doubledays‘ game a few days ago. We had really good seats, and found ourselves among the staff and scouts for both the Doubledays and the visiting Williamsport Crosscutters. I was on the inside end of 106, The Guy was on the inside end of 105- like so:
There was a lot going on. Radar guns checking pitching speeds, lots of notes being taken on each player, and general “back end” stuff that is the business side of baseball. All of that was cool- but I not as cool as noticing The Guy wearing this ring while he jotted stuff down in a book. It kind of jumps out at you…
During a break in the action, I asked the fellow what that gorgeous ring was all about. Without hesitating, he slipped it off of his finger and handed it to me. While I looked at it, he told me it was the 1971 World Series ring- and I kid you not I could feel the history in it. It had a really weird, magical gravity to it. His father was the PR Director for the Pittsburgh Pirates when they one the series that year. This was the Roberto Clemente team, BTW. Wow.
Things got busy again, and I gave him back his ring after thanking him profusely and having some quick dialogue about how thrilling it must have been for him to be a kid back then. This gent was just a sweetheart, and absolutely didn’t have to entrust a stranger with what is obviously a highly-treasured possession. But I’m glad he did, and man it was cool. Ever curious, I found out out the bigger story on his Dad, which you can read about here.
Underooos, Priceline, ATMs, and Pancake Mix
Fast forward to a few days after the baseball game. My wife and I are at a high school graduation party talking with friends, when the grandfather of the graduate introduced himself. I’d learn that this awesome man was 81, but he could have been in his early 60s. In the course of our discussion, I found myself kicking it old-school with a pretty accomplished gent.
Larry is a fairly humble guy, and very well spoken. We spoke of not only each other’s accomplishments and experiences, but also of life philosophies and the importance of never losing your ability to be a child inside. It was an absolute honor to not only meet Larry, but to spend quality time talking with him as a fellow man-of-action.
People Can Be Awesome…
Thankfully I’ve had the privilege of getting to know a lot of good people through the years. The really special (to me) ones stand out in my memory, from childhood to my military days, into my professional career and beyond, including lots of awesome folks out in the bigger technology community. Sure, I’ve also crossed paths with absolute duds… folks with an evil streak or just dead inside- but thankfully those have been the minority.
You never know where the next Ring Guy or Larry Weiss will turn up- so keep your eyes and your minds open as you interact with people.
You gotta love a good bridge… like this gorgeous unit in Moravia, NY:
If that bridge could talk it would tell you stories from my childhood. It would also talk of my own children flying across it on their bicycles countless times through the years. Lovers holding hands as they stroll, school picnics, photographers marking the change of seasons… Ah, memories.
*Ahem* That’s NOT the kind of bridge we’re here to discuss though (but thanks for indulging me on that). Friends, I’d like you to meet the Siklu EH-600TX. I’m impressed so far, and so feel compelled to share some thoughts with anyone interested.
I have two of them newly in production, forming a point-to-point bridge link, and a third sitting in a spares cabinet waiting to jump into service if ever the need arises. The birds-eye view of my link goes a little something like this:
This unit is absolutely “carrier grade” from a build quality perspective. The EH-600TX works in the unlicensed 60 GHz spectrum, and I added the license that gets you to 1000 Mbps aggregate capacity, from the base 500 Mbps. You can slew your up/down to be asymmetrical if you so desire.
The 60 GHz spectrum is a mysterious place where oxygen absorption of all things needs to be understood (see this quick primer), and the transmitted beams are often described as no bigger than pencils or dimes. Wireless bridges of any sort can be hard to align, and the tighter the beams the more true that statement becomes. Yet Siklu provides excellent directions and an alignment methodology that let a technician who has not done a lot of this sort of work get it right fairly quickly.
Given the weirdness of the 60 GHz spectrum, I really wanted to exercise this link in crappy weather before turning it loose on the network. The skies over Central New York have obliged, as we’ve had a fairly miserable spring and early summer with plenty of Florida-grade Sheets-of-Rain and Walls-of-Water sorts of storms. Through them all, the Siklu link did not blink.
After pressing the new bridges into service for real, I realized I should probably get the latest firmware put on them. As with the alignment process, Siklu provides good directions for this task. And the total time of upgrade-related outage is petty impressive:
In testing via iPerf and other trusted verification metrics, I find that the EH-600TX delivers what it promises for speed and capacity. So far, it’s been a great experience.
This link is replacing a licenced 80 GHz Exalt setup of similar capacity, and it’s so nice to work with “palm-sized” hardware (as Siklu describes it). We also have smaller Exalts and LigoWave bridges in service, but this Siklu is now our big dog from a capacity perspective.
We are running the link as simple as possible, with it functioning as a patch-cable in the air in a simple extension of the LAN. But there is a lot that might be done with the bridge’s three Ethernet ports, and I encourage you to dig more into Siklu’s capabilities if interested.
Time will tell if I made the right choice with the Siklu EH-600TX, but the early verdict is that it’s a winner.
Please note- if you are new to point-to-point bridging, make sure you get educated on rooftop safety, licensed versus unlicensed frequencies, proper mounting procedures, and grounding techniques. None of this is particularly daunting, yet there are plenty of ways of getting it wrong- leading to failed links, damaged downstream equipment, and potential injury or death.
What lies ahead in this blog isn’t so much a rant, as it is an analysis. That sounds classier, and implies critical thought rather than just someone bitching about things. With that in mind, I give you the following image, stolen from Matthew Seymour (on Twitter at @realmattseymour) and sourced from this year’s Aruba Atmosphere conference in Europe:
The topic of the slide is 802.11ax, but that is tangential to the points I want to make here. Read that first caveat- this requires the use of 160MHz channels, which is generally not possible or just not a good idea
I’m not sure who actually presented this session, but I’m assuming it is a trusted voice from a trusted company- Aruba folks generally know their stuff (as evidenced by their growing customer base and industry longevity). When I saw this come across Twitter, something in my mind clicked and this simple thought bubbled up:
Why does the WLAN industry do this idiotic shit to itself?
When I say the WLAN industry, I’m including the IEEE, the Wi-Fi Alliance, and the vendors that provide the hardware and the marketing of wireless networking products. let me state the underlying problem as clearly as I can: the IEEE creates the 802.11 standards, and since 802.11n each standard has had a nonsensical top end- you simply cannot reach the high part of the spec. Evuh. The Wi-Fi Alliance does nothing to bring any sanity to the situation, and WLAN vendors build in configuration options with high-end settings that actually do WLAN damage and that let us create operational situations that are…
just not a good idea
Has anyone TOLD the IEEE that no one is really impressed with the promise of high end specs that can’t be leveraged? That it’s all a big stupid tease? The entire professional WLAN community knows that 160 MHz channels are…
just not a good idea
So why do WLAN vendors present 160 as an option? Why don’t the Wi-Fi Alliance and the vendor community repaint their messaging with reality-based promises of what each new WLAN technology can do? Wi-Fi 6 will STILL be impressive- but market it as if 160 MHz channels don’t exist- and watch the Sun of Truth rise over the wireless landscape.
I’m guessing some of you are thinking “you idiot, the FCC is going to give us more spectrum and then we’ll be rockin’ 160 for sure”. To that I say- pffft. I’ll believe it when I see it- and even then the ability to toggle 160- to even see it in configs- should not be a default.
The argument might also be made that “Maybe people at home can use 160 MHz channels so you should shut up about it already”. Don’t go there, girlfriend. That only amplifies my beef with the Wi-Fi Alliance members who refuse to draw a clean line between Enterprise Grade gear and Wonky Shit That Plays Well at Home But Shouldn’t Be Dragged Into The Enterprise environment.
And that loops us back to the “tip of the iceberg” thing- we have cheap printers that come up in a default 40 MHz wide channels in 2.4 GHz, which also is…
just not a good idea
And an industry-wide trend where pretty much most 5 GHz gear comes up with 80 MHz channels enabled. Which also happens to be…
just not a good idea
We’re at an odd place where all the players involved are obviously aware of all the things that are
just not a good idea
yet we MARKET bad ideas and then have to explain why we can’t really USE those bad ideas which have been marketed.
We kinda need our collective wireless head examined. Thus ends my analysis.
A while back I wrote about the importance of the Good Guy On Other End (GGOOE) for those of us charged with keeping up far-flung network empires. Even when you have decent instrumentation and infinite powers of configuration, sometimes you need someone to move, touch, or add something. It sounds so simple, but even simple tasks asked of the wrong person can result in the failure of those tasks- and possibly the creation of more and bigger problems. Having someone you can trust (even if they aren’t a networking type) on site when you are two, two hundred, or two thousand miles away is huge.
Then of course came The Soon To Be Famous Cocktail Napkin that rocked the IT world to it’s foundation. With one graphic that looked like it could have been inked by a drunken child, I dared to tell the world that “sometimes problems that feel Wi-Fi-ish may not be”, and the cosmos rippled with hyper-galactic truth and acknowledgement, I tellya. OK, maybe it wasn’t quite that epic, but a lot of people seemed to derive value from that simple drawing and the idea behind it.
Now fast forward to the point of this blog. I recently fielded a challenge from an esteemed colleague: “one of our VIPs who telecommutes is having a horrible time with his Wi-Fi at home, I need to get to the bottom of it”. My friend is really smart, but his core strengths are in other facets of IT. He had a tight window for a site visit, and I had a conflict during that window. He floated the idea of buying some sort of wireless tool for a couple of hundred bucks, but I was able to head that off.
We talked a bit deeper about what was going on- and it didn’t sound so “wireless” to me. Evidently there was some ISP issues in the past, and the VIP was using a managed laptop, so this could go in a few directions. Yet because Wi-Fi was involved- and when things got wonky, the VIP was using Wi-Fi- so the specter of Wi-Fi problems still loomed. I took a few minutes to introduce him to a real tool- the NETSCOUT AirCheck G2, with emphasis on how to restart a monitoring session, let the channels all scan a couple of times, and then save the session. The idea was he’d visit the site, do his best to analyze anything possibly in play, and then return the AirCheck to me where I would analyze his captures.
So how did that go?
Again, he’s a smart guy. He was able to prove ISP service interruptions, probably related to crappy cable outside of the house. He also found a pretty significant problem with the YIP’s laptop. And… we were able to exonerate the wireless in this case through almost a dozen captures taken at different points in the home with the AirCheck. He gathered them, I analyzed them, and though THE EXPERT couldn’t be on site to prove that the wireless-feeling problems weren’t really wireless, the right tool in the right hands boiled all of the mystery off of the Wi-Fi side of the discussion. Plenty of signal at sufficient strength and quality inside the home, no real RF encroachment from outside. It also provided a baseline that we can refer to if the situation ever needs to be revisited.
It’s not easy handing off the expensive, beloved tools to someone else. But if you can get past that and you are handing the tools to the right GGOOE, it’s the next best thing to being there.
Another academic year is almost in the can at my day job. As things wind down for staff and students, a lot of IT work is just kicking off for the project frenzy that is the summer months in higher ed. There is a lot to ponder and analyze at the end of every school year, but this go around feels a little different.
On the personal side, I’ve crossed the 21-year mark with my current employer, and am thankful every day that I had the good sense to take a 25% cut in pay way back when to get my foot in the door in hopes that a fantastic career would blossom (it did). I’m also counting the miles I need to drive to see my daughter graduate college, and the months until her wedding later in the year. I’m tallying the time until each of my two sons finish their PhD programs in the near future, and wondering what comes next for all of my kids in the days to come. My wife and I are approaching our 30-year anniversary, we just lost a beloved dog who lived to be 14, and I figure I can now play about 8 good chords on the ukulele. Lots of numbers to think about… thankfully most have good connotations.
But back to work, and work-related numbers.
I’m seeing that our big-honkin’ WLAN served around 30K simultaneous clients at the busiest part of each day. Add remote sites in the US and abroad, and we’re well towards 33K. That equates to millions of aggregate hours of network usage that includes academic instruction, online business applications, gaming, entertainment, and all kinds of stuff whizzing across that sweet, sweet Wi-Fi. We’ve had phenomenally low numbers of trouble tickets (statistical zero, by several measures) and few social media swipes at the WLAN over the last two semesters, and continue to validate that our mantra of “stability above all” has yet again paid off for a lot of people who need to do their work (and play) over the WLAN.
Looking forward, I feel a tension brewing. At some point, our 4,300+ access points will need to be moved off of legacy controllers (Halle-freakin-lujah!), and will grow by hundreds as new buildings come on line and old designs are updated. But those 30K+ clients will need the same stability we’ve mastered on the existing hardware (by never upgrading from one of the few stable code versions we’ve ever found), and new hardware is the devil you don’t know. Then there are the new licensing paradigm numbers- as best I can tell we’ll being paying a lot more but not actually getting a lot more (beyond buzzwords) for those dollars, but I hope I prove myself wrong. I do know that my “Intent” is that the next generation of wireless code won’t suck as bad as the last, and that our vendor at some point realizes that you can only license the shit out of so much before you force customers to find relief elsewhere. Whatever- that is my battle to fight. Meanwhile… a reminder on the most important numbers: those 30K+ clients with their millions of hours of network use just want things to continue working well.
We’ll continue to be peppered with big marketing numbers as .11ax makes it way into town, and as with any past WLAN technology will have to translate it all into “realistic” numbers that come from boiling off hype. We’ll be learning about how new WPA3 security does away with the 4-way handshake of WPA2, how 192-bit security becomes more relevant, and where 5G will or won’t eventually fit in with Wi-Fi 6. mGig and UPOEbring new numbers to reconcile in our own environments. Numbers, numbers, numbers. There is no escaping them. Numbers are part of who we are and what we do. They change with time, and they always have- both at work and at home.
Numbers are funny… just a few bad ones- like hours of downtime- have a way of reputationally negating millions of good ones (like hours of uptime).
You all have your own numbers swimming around in your heads right now. Many of us are facing the same concerns and changes in the days to come, but what they map to for numbers of budget dollars, training hours, or trouble tickets will vary. Learn, evolve, adapt and try to use the various numbers to your advantage as much as possible.
Just remember that your own users probably also want stability above all as you wrestle with all of the numbers in your life. That fact has it’s own number.
As a long-time Ekahau user (pretty sure I was one of the first few American customers way back when), I’ve gotten used to continuous improvement and evolution from Ekahau Site Survey (ESS) suite of tools. There have always been new features right around the corner, and the company has been perhaps the best I’ve ever seen at gathering and acting on user feedback. It’s been a great run. In the recent past, the hot-selling Sidekick provided a unique new dimension to the survey and spectrum analysis processes, and the Ekahau company was purchased by Ookla/Ziff-Davis. Both of those developments are pivotal to what comes next for Ekahau.
And what comes next is called Ekahau Connect.
There’s A LOT here to talk about, starting with ESS getting rebadged as Ekahau Pro, now compatible with both Windows and Mac operating systems. (If you are new to the world of WLAN support, trust me that Mac is a far-better tool platform than Windows- and I am unabashedly NOT an Apple lover.)
Then there is Sidekick’s expanded capabilities- including wireless packet capture leveraging Sidekick’s dual radios (yay!) and the ability to interface with the iPad as a survey platform. This is a pretty big deal, and the light physical weight of the iPad makes for easier, more comfortable surveys.
And… Ekahau does a little catch-up with it’s introduction of Ekahau Cloud. This is one extremely valuable capability that competitor iBwave has had for some time, as I wrote about here. Having used iBwave’s cloud tools, I can assure you that Ekahau’s customers who work in teams are going to love it and there is no doubt that the cloud expertise behind Ookla has some impact here.
And is if this all wasn’t enough for Ekahau Nation, feast your eyes on another new benefit- Ekahau Connect components working together to identify, classify, and locate interferers:
I have been fortunate in that I have been a beta tester for Ekahau’s latest. At the same time, a couple of serious “life happens” events have kept me from being a good beta tester. So for real-world first-hand perspective, I’ll hand you to two two of my favorite people on the No Strings Attached podcast.You’ll be in good hands with Sam and Blake.
The only thing worse than an important feature that’s missing in a network management solution is an important feature that works wrong. Those of us who pay for these systems expect the features that we trade our dollars for to actually, well, work. And bad information is just terrible in that it sends us down time-wasting bad roads.
Meraki- you have a problem. Your topology view doesn’t refresh for hours, days, or weeks after changes are made. And for whatever reason, you have not given us decent LLDP tools that we can invoke on demand. That’s the polite description, which can be abbreviated down to your topology view sucks at times.
In a faraway branch location, we had two switches that daisy-chained off of another switch due to physical layer constraints. Here’s the old topology:
Pay attention to VillaRosa-3, Villino-1,and Vilillino-2. V1 and V2 were daisy-chained off of V3. THEN THEY WERE MOVED TO EACH DIRECTLY CONNECT TO THE MX. And then I got an email… “The switches got moved to ports 6 and 7 on The MX, but the topology isn’t updating. Maybe I should restart the devices…” As we see in the community forums, we may be waiting quite a while for this map to update…
Few things are more basic in network support than simply knowing what connects to what- both physically and logically. Commands like <show cdp neighbors> and <show lldp neighbors> are pretty important, but not available in the Meraki dashboard paradigm. Instead, we need to rely on incomplete or inaccurate graphical information.
As I mentioned above, V1 and V2 moved to direct connects on the MX. So what does the MX say about the switches that are connected to it? Unfortunately, nothing. For whatever reason, Meraki has never seen it as important to give full lldp information for devices connected to an MX:
This is utterly nonsensical for enterprise-grade networking equipment. But it gets worse. Let’s look at one of the switches that has had it’s uplink moved to the MX. Days after the move, the switch still says it’s connected to it’s old uplinked V3 switch:
This is pretty bad. It’s wrong, it’s outdated, it’s misleading, and it’s inexcusable. Let’s have a look at “VillaRosa-3 / 51.
Topology views don’t refresh as they should (if ever)
The MX doesn’t tell who it’s lldp neighbors are- which is a glaring deficiency
The lldp neighbors information on the Meraki switches can’t be trusted
There are no obvious ways to invoke on-demand refreshes for topology and lldp views, we are at the mercy of some undeclared loooooooong refresh timer
That this “feature set” can be this off-kilter defies logic- especially when you consider the cost of the gear and it’s licensing and the importance of lldp to network support.
I’ve heard the cries of anguish. I know the many lives that have been ruined and rendered meaningless since I announced the end of #WIFIQ on Twitter at the end of last year. I too have lived in silent agony since the daily discussion exercise went dark, even though that darkness was of my own making.
And now… I’m happy to announce that #WIFIQ is back, baby!
But going forward, the NEW version of #WIFIQ will not flow from my muscular, impressive fingertips. Ah, the stories those fingertips could tell… *Ahem* It’s time for fresh minds to take up the cause, and I’m tickled to announce that the Wireless LAN Association (WLA) has expressed the desire to bring back the good thing that is #WIFIQ.
I’m not sure exactly how the WLA will originate their questions, or if they will take input like old guy who did #WIFQ did. But I do know the men and women of the WLA are good folks with great minds, and I haven’t a worry in the world that they will do justice to the #WIFIQ concept.
If you don’t yet follow the Wireless LAN Association on Twitter, get yourself hooked up with @WLANAssociation handle and standby for #WIFIQ and related news as they re-prime the pump of wireless knowledge, fellowship, and goodness.