Tales From The Towers

Chapter 64 – Migrate or Migraine, pretty much the same thing

By January 12, 2018 No Comments

This article is going to be specific to migration from Ubiquiti to Mimosa in one area that is perfect for Mimosa and still has hundreds of Ubiquiti outdoor radios operating. We aren’t abandoning Ubiquiti. We opened up new areas last year with them, hundreds of miles of area coincidentally, right next to our Mimosa deployments. A lot of this was decided over 18 months ago when Ubiquiti was still battling firmware and 802.11n compatibility issues and Mimosa was still working on TDMA and hadn’t released a connectorized C5 called the C5c yet. But for this particular area, where all the CPEs are within ¼ mile or so, we decided to go Mimosa because of the A5-360. To understand how dense this area is, we have 1650 homes in an area of 1×1.5 miles and we are currently on over 35% of every single house in the area. In addition, we have CenturyLink and Comcast as competitors so if you don’t think you can compete with cable/DSL, this should change your mind.

Based on the timing of when new areas were opened up and when hardware or firmware was released, antenna availability, antenna testing and so on, we went with one vendor or another at different times in different areas. Both companies were a critical part of our success over the last 3 years along with contributing factors to my lack of sleep and ulcer (just kidding no ulcer, yet). This article also took 3 years to finally come to fruition where we are confident in the decisions that we made and our plan to move forward. At least until next week when we need something new to be competitive (which is why we are testing IgniteNet and Siklu heavily, the next version of firmware comes out, new hardware gets released, or new support tools become available. Basically I’m saying every vendor has strengths and weaknesses and you need to decide which one benefits you  most financially at this time and possibly in the future and don’t lock yourself into a single vendor, ever.

Supporting two vendors also isn’t a big problem when your staff has years of Ubiquiti experience. It’s a little more expensive but the right tool for the right job. Mimosa and Ubiquiti now have phone apps that simplify installations. We have trained 2 installers, neither of whom had any technical experience, successfully. I’m not familiar with Cambium installation process but until their AC products hit the market, it doesn’t interest me yet. My wife kept waiting for Prince Charming but at some point she had to settle for me when he never showed up.

Cambium did a great job and built a quality product that extended 802.11n much farther than I imagined so Kudos to them. Their indoor products are excellent also. They are just really late to the AC game and I can’t wait 2 minutes for my bacon (yes, not the best way to cook bacon but microwave real bacon is better than no bacon at all) to cook in a microwave let alone 2 years or more for AC radios or for firmware to stabilize. We are shooting for 100Mbps everywhere and 200Mbps where we can so we needed at least 802.11ac to pull that off.

Back when Ubiquiti came out with the first AC-Lite, we thought, that’s the way to go, yessiree Bob. Hoo boy, this is going to be a piece of cake to upgrade to 802.11ac. Break out the champagne, we are off and running boys. Now we can overcome the limitations of the Rocket M5 processor easily migrate to our hearts content, and the Cubs are sending me a letter to report to Spring Training next year because they saw me pitch. About then was when like the Coyote, we realized the tunnel was painted. Boom!

We also discovered that Ubiquiti believed the concept of Beta and released hardware/firmware were synonymous. I get it and I’m not going to be the Monday morning quarterback on this one but between umpteen (almost 20 with release candidates on one version) beta versions on a single release, not getting DFS certifications, and beta versions that had more errors than Billy Shindle (119 errors in one baseball season), we decided we needed a plan B so we started testing Mimosa simultaneously. I kind of got tired of waking up to phone calls 24 hours after installation of a new firmware on the Ubiquiti AC-Lite hearing that every user on the AP has been kicked off.

Two years ago Mimosa figured out how to migrate 802.11n radios to their APs in WiFi mode. We jumped on that. We thought we would replace all our underpowered Rocket M5 Ubiquiti APs with Mimosa which I’d been asking for till I looked like I could join “Blue Mans Group”. Then we could just start the migration since Ubiquiti wasn’t quite getting it done. It actually worked great when we had 300 customers in the area last year and we thought the issue was resolved as we migrated clients. Then the word got out that we were doing a pretty good job and everyone and their mom started calling us. Density kept increasing to the point where on one A5-360 AP, we went from 38 users to 62 users in less than 45 days. And it just got mo’ busy from there which caught us off guard as we weren’t spectin’ this speed of growth.

Speeds that started at 350Mbps when were first deployed Mimosa dropped faster than thermometers in Florida. More users came online and WiFi mode was running into collision and interference issues. We had to go to 40MHz channels and then 20MHz with over almost 600 users in an area of 1×1.5sq miles. We quickly needed some type of GPS to reduce the interference or at least polling and get off WiFi mode quickly. We could put up more APs but the interference would just get worse.

Since only about 100 of the users were on Mimosa and since Ubiquiti had 802.11n compatibility humming along, we made the decision to upgrade the remaining Ubiquiti APs that we would get to easily to Prisms and run them simultaneously on the same pole parallel to the Mimosa APs. We then migrated all the Ubiquiti CPE’s back to Ubiquiti APs and then only keep the C5s on Mimosa. This allowed us to use TDMA on Mimosa and Airmax on Ubiquiti. Haven’t used GPS on Ubiquiti there yet as we have several AC-Lites still running.

But the noise, oh the noise! Tasos from RF Elements kept telling me that horn antennas were the next best thing since sliced bread so we changed some of the A5-360 radios over to A5c’s with 30-90 degree horn antennas, depending where they were deployed. That helped significantly and allowed us to start parsing the area which reduced the noise. Some of the Ubiquiti APs got horns or simply stayed on the sectors with RF Armor shield kits and with a reduced number of users. In other areas where we had Ubiquiti omni’s, after running just the A5-360’s for a while, we decided with one exception, that we really needed Mimosa TDMA because we had so many Mimosa and Ubiquiti APs in such a small area, so we kept the Ubiquiti omni’s running and put the A5-360’s next to them. We also have a Prism handling 58 users right now with an omni so they are working pretty well also.

While all this was going on with Ubiquiti, Mimosa had the A5-360 out in WiFi mode and C5s with built in 21dBi antennas and an unknown timeframe for TDMA. This kind of limited the range and didn’t help with the noise issues around the A5-360. It wasn’t a problem early on as we migrated Ubiquiti clients but as the number of users exploded, we needed sectors and higher gain client radios. We found that WiFi compatibility mode on Mimosa had a limit due, not due to processor power, but due to WiFi collisions. The other problem with compatibility mode on Mimosa with WiFi clients when you start loading it down with 40+ clients was that the auto-settings on the Ubiquiti clients mucked things up with ACK settings all over the place. That meant manually setting distances on each client and A5. That works well, just more maintenance at time of installation and management we felt we didn’t need to do (okay, we are lazy). This one area is fairly dynamic with snowbirds and renters so it’s a lot of work when you are adding 3-6 users a day sometimes.

So here is another conundrum. The same thing that makes the A5c radio very cool, 4×4 Mimo, also made it more difficult and expensive to deploy. Early on, KP Performance was the only company with a sector for the 4×4 Mimosa A5c radios, and they had 2 but it was expensive for this deployment. It also didn’t solve the problem of noise off the side and back although it was pretty good. We also found in higher density areas where we are competing with cable/dsl/Dish/Direct who are using indoor 5GHz routers cranked to maximum power, that noise was the ultimate enemy.

MTI then comes out with a good 4×4 antenna for a great price but it picked up even more noise off the side and back. We actually have one running with a user more than a quarter mile away behind it. In some environments, this is a good choice for 4×4, like the one we are using it in up in Northern Arizona. However, in our high-density environment, we needed more isolation because we couldn’t use TDMA yet on all the A5-360’s since some of them we still supporting Ubiquiti clients.

Then RF Elements comes out. Ahh, now we are cooking. Lower price, lower gain but that’s okay in this deployment of high-density and short range, and things start to clear up. The A5cs that we deployed now are cranking back up. It worked so well, we even put 2 A5cs on the same pole with 4 RF Element antennas to handle the density and are running both in TDMA mode. The good part is that as density per AP continues to increase, we can go from having 2 antennas cover each zone in 2×2 to overlapping them and using the 4×4 feature of the AP. We are also back to 40MHz channels instead of 20MHz. And when Mimosa releases MU-MIMO firmware, that will double the capacity of the AP overnight. Keep looking down the path and plan for it.

We were still left to figure out how to handle the higher gain we needed on the client side to fight noise. Although range is short, interference from indoor APs from Comcast and satellite vendors is an issue. The C5cs came out but we didn’t have a cost effective or lightweight antenna to use with them. We tested LanBowan antennas or equivalent models from other vendors since they were very inexpensive and found they varied in quality. But they worked and they were lightweight with a very low wind load. Still, we needed something with higher gain, so we looked again and started using the Ubiquiti RD-5G30-LW antenna.

This is a solidly built antenna and perfect for working with the C5c, sort of. It has RP-SMA jumpers built in, the C5c sort of fits on the sleeve (it’s a little chubby, like me) so the cover doesn’t quite fit but can be wrestled on, the cost is reasonable, performance is solid, and it’s available. The one thing we didn’t like about it was that the LW meant something other than lightweight. It also needed better isolation and there was no ISOBeam option. RF Armor has one we haven’t tested yet but the weight was already an issue for us. We can deal without the ISOBeam although making it compatible with the 620 Isobeam would have been nice, but the weight coupled with the wind load meant that a standard J-Mount was pushing it, especially the longer ones if we had to shoot back over the roof and there was no way a tile mount had a chance (had 2 blow off so we know of what we speak). And no, we can’t add braces because we don’t bolt on top of roofs which eliminates one angle of support. That’s when we discovered the KP Performance Reflector Horn antennas and the testing began.

The KP Performance antennas were a lot lighter than the LW, which we needed, lower cost, and had a lower wind load. They also seem to have better noise reduction. The two biggest issues are that it has to be mounted on the top of a specific diameter J-Mount or pole since it can’t be mounted down the pole because of the mount, and finding cost effective cables. We like LMR-240 with RP-SMA and N-Male but since that added $25 alone per installation, the cost ended up being higher than the LW for cables. Here is a good source: https://www.ebay.com/itm/3Ft-N-Male-Plug-to-RP-SMA-Male-Cable-WIFI-Antenna-Extension-Pigtail-Jumper-1M/201677536995?ssPageName=STRK%3AMEBIDX%3AIT&_trksid=p2060353.m2749.l2649

We also have a new RF Elements parabolic coming in which we will test next. A suggestion for all manufacturers, get a plastic, pointed radome to cut back on the windload, especially for 2’ size dishes.

While this was all going on, Ubiquiti was finally getting a handle on the firmware and migration issues with the AC radios while also releasing the Prism. Now what do we do? Well, this is where you go back to trying to improve the odds. The obvious question to us was which one handled noise better since that was our biggest issue we expected to have with Comcast and Dish/Direct using 5GHz inside the house? We ran a test which got posted in Chapter 62. That pretty much settled that particular decision for this area. We originally planned on going all A5-360’s and we have a few of them still running in the middle of the area but the massive density increase coupled with the noise from cable modems, our Ubiquiti APs, and our A5’s that were still running WiFi mode meant A5’s to RF Elements horn antennas was our only option for some areas.

The horn antennas have a tradeoff between antenna gain/beamwidth and noise but since we were now fighting noise on both sides, that was our higher priority. The horns helped at the AP and the more directional and higher gain antennas on the client side reduced noise from houses around the CPE (and AP). We have some users hitting 90Mbps speed tests but we are pretty sure that’s our bandwidth limitation now since early testing had us over 350Mbps. We only have 400Mbps in the area and are hitting 320-380Mps peak at night which we may upgrade shortly.

When someone asks me what the decisions were behind this mish-mash of Ubiquiti and Mimosa, I can only say it’s complicated. And if you want to know what kind of training is necessary to work through these decisions, don’t play chess, play backgammon. Chess has one move per turn and it’s not random. Backgammon is harder because there is so much out of your control with a single random dice roll that can completely change the entire makeup of your strategy turn by turn. This would be the firmware beta 1, beta 2, beta…, bet 14, and so on. It works, it doesn’t work, doesn’t work, products released without DFS but firmware is coming soon, but wait, more delays due to certification and on and on. Backgammon is no different than keeping track of all the vendors, all the features and bugs, the timing of everything, trying to guess what they are going to do next, and then figuring out if any third-party vendors can change decision processes. A good example of that is RF Elements and KP Performance.

Things have finally settled down from the insane number of customers we were installing weekly and nobody is complaining about speeds or problems viewing video streaming any longer. Although we had to add a couple of Prisms to get Ubiquiti radios off the A5cs, we just decommissioned our first tower Ubiquiti Rocket M. New users are getting hooked up cleanly, Mimosa has the phone app working making it faster and easier for our techs, and life is generally good. Time to start expanding again.