Thursday, October 27, 2016

Because They Can: Spawning of the IoT DDoS

Authored by: Keely Richmond, Sr. Cybersecurity Engineer

On September 13, 2016, the website of a US-based Cybersecurity Journalist (Brian Krebs) was hit by a DDoS attack that knocked his site offline. The resulting investigation uncovered a type of DDoS attack different in composition and scale than that of the typical DDoS attack.  In fact, this type of DDoS attack had only been seen in the wild a handful of times—the earliest known detection: April 2016.

Two pieces of malware (similar, but different), Mirai and Bashlight, were used to enslave enough Internet of Things (IoT) devices to launch a 620Gbps DDoS attack for the sole purpose of silencing a journalist whose articles threaten the livelihood and freedom of criminals.

As expected, an IoT DDoS attack sustaining 620Gbps got the attention of all types of people and organizations and the internet was soon riddled with a flurry of partial facts and theories about who, how, why, and what might be next. 

Then it happened again! 

On September 20, 2016, approximately 150,000 IoT devices (primarily DVRs and CCTVs) were enslaved to launch a 1.1Tbps DDoS attack against OVH (a French hosting company service provider). The target: Minecraft servers. The culprit: Mirai.

By September 30, 2016 the creator of Mirai decided to cut bait and released the monster (source code) into the public domain for anyone to use. This put distance between them and the source code, making the job of law enforcement more difficult because the code was now in the “possession” of virtually everyone. It also put a very powerful weapon in the hands of anyone with a moderate level of technical prowess and malicious intent.

At the same time the source code was published, the list of IoT devices targeted by Mirai (with default login credentials for each) was also published. This accomplished a few things:
  • Put the manufacturers and vendors of those devices on notice that their equipment needs to be secured or securable
  • Informed consumers (at least those who read tech news) of the lurking vulnerabilities in their homes and small businesses
  • Provided a seed list to anyone who wanted to give that source code a test drive
  • Made IoT vulnerability a hot topic for Cybersecurity specialists and service providers 
  • Caught the attention of legislators—domestic and foreign

In addition to releasing the Mirai source code, its creator touted that it had been able to enslave 380,000 IoT devices before the scrutiny following the OVH attack made such endeavors too risky. 

And then…it happened again!

On October 21, 2016 the US-based DNS provider, Dyn, was the third known victim of Mirai. The metrics of the attack have not been released to the public, but speculation puts the volume of the DDoS at 1.2Tbps. The Dyn attack impacted millions of people in the United States and Europe. Traffic destined for sites such as Twitter, Amazon, PayPal, and Netflix could not be resolved if public DNS providers (such as Dyn or Google) were used. Cisco’s OpenDNS, however, held steady because it uses smart caching.

Via a Twitter post, a hacking group called New World Hacker has claimed responsibility for the initial Dyn attack. Their motivation was simply to prove they could do it. Shortly after the attack by New World Hacker ceased, the hacker group Anonymous is said to have performed a second wave of the attack. The participation of both groups is unconfirmed at this time as the Department of Homeland Security continues to investigate.

The Mirai source code had been in the public domain for less than one month when the attack on Dyn was launched.

Now What?

"Perfection is not attainable. But if we chase perfection, we can catch excellence.”- Vince Lombardi

Devising a security strategy for IoT devices is challenging. Not all devices are created equally.  Some are patchable, some are not.  Some have configurable login credentials, some do not.  Some are “smart”, some are not. 

For Subscribers (End Users)
The recommendations offered by researchers are limited and responsibility falls squarely on the owner of the IoT device.
  • Purchase and implement IoT devices that require the password be changed upon initial setup. 
  • If possible, change the default login credentials. Unfortunately, not all devices allow the login credentials to be changed.
  • Upgrade the device firmware as new versions are released. Be mindful, however, that a firmware upgrade may reset all login credentials back to default.
  • Configure your router to point to OpenDNS for DNS resolution.
  • Disable UPnP on routers
  • If the device resides behind a firewall…
    •  Disable unnecessary ports. The Mirai attack looks for devices listening on telnet (especially ports 23 and 2323), SSH, HTTP, SMTP, etc. It’s not realistic to block those ports outright, but limiting their use to specific IPs is a good idea.
    • Monitor traffic on port 48101. Port 48101 is commonly used for the transport of malware.
    • Segment all IoT devices to a specific VLAN and restrict outbound traffic from that VLAN.

 As with any project, we must use the right tool for the job.  Relying on subscribers to build and secure their private networks isn’t the right tool for this job. 

For Service Providers

What tools are available upstream from the subscriber, at the service provider level?
  • DDoS mitigation tools (e.g. Arbor) are effective in monitoring and alerting as bandwidth usage thresholds are exceeded.
  • Ensure your IPS malware signatures are up-to-date. Because the Mirai source code was released, it will be fairly easy for vendors to write a signature.
  • Implement OpenDNS to ensure your DNS traffic is scrutinized with the most up-to-date rules in the industry and to take advantage of smart caching.
  • Use the published list of IoT vendors to seed a DENY rule that restricts traffic from those specific device types. This is feasible, but tricky because blocking all outbound traffic would also prevent those devices from pulling firmware upgrades from their manufacturers.

DDoS attacks are just one of the nefarious acts for which IoT devices are being used. An article published by Brian Krebs on October 13, 2016 explores how they’re also being used to turn consumer-grade routers into SOCKS proxy servers, anonymizing the source of suspect traffic.  Use of these hijacked routes are then offered for sale on the dark net to further the spread of various forms of cybercrime. 

What’s Next?

Device Recall
The IoT devices favored by Mirai use components built by a China-based company named XiongMai. In a statement issued on social media , XiongMai said it would be issuing a recall on millions of devices—mainly network cameras. Details of the possible recall are unknown at this time.

Per Gartner Research, there will be 6 billion IoT devices at our fingertips and on our networks by 2018. New and existing communications standards such as Wi-Fi, Bluetooth Mesh, Low Power WANs, Narrow Band IoT, and ZigBee will grow in usage as the footprint of “things” expands and smart homes grow into smart cities. 

Encouraging or regulating the manufacturers of “things” to adhere to basic security practices is key to standardizing how those “things” can be monitored and managed on our networks.

The European Commission, in an effort to enhance the European Union’s telecommunication laws, is developing requirements to standardize security for IoT devices. At this time, there’s no indication of how security for an IoT device will be defined, measured, or monitored. Though not subject to EU regulations, we will benefit if they are successful in enforcing standardization.

CCI Systems’ Recommendations

CCI Security Engineers have studied the three recent IoT DDoS attacks and devised two lists of recommendations (one for the owners of the IoT devices and one for their service providers), each stated earlier in this article under the “Now What?” section. Ideally, the two lists should be deployed in concert, but it is not realistic to expect subscribers to have the technical knowledge to perform the recommended steps nor for the service provider to have the access or ability to monitor the security measures of privately owned networks.

Cybersecurity = Risk Management! 
  1. Employ security best practices.
  2. Design and implement enforceable security policies. 
  3. Review and incorporate the National Institute of Standards and Technology (NIST) Cybersecurity Framework into your Security Strategy.
For more information regarding your network security, or to discuss the recent DDoS attacks, reach out to us! We’ve got the expertise and know-how to keep your network protected.


ABC7News. 2016. ABC7News. Oct 23. Accessed Oct 24, 2016.
Akamai. 2016. Akamai. Sep 30. Accessed Oct 12, 2016.
Gartner. 2016. Gartner. Oct 12. Accessed Oct 12, 2016.
Krebs, Brian. 2016. KrebsOnSecurity. Sep 30. Accessed Oct 12, 2016.
McCarthy, Kieren. 2016. The Register. Oct 21. Accessed Oct 24, 2016. n.d.
Savage, Marcia. 2016. Network Computing. Oct 12. Accessed Oct 12, 2016.
US-Cert. 2016. USCert. Oct 14. Accessed Oct 14, 2016.
Zeifman, Igal, Dima Bekerman, and Ben Herzberg. 2016. Incapsula. Oct 10. Accessed Oct 12, 2016.
ZigBee. 2016. ZigBee. Oct 12. Accessed Oct 12, 2016.

Thursday, June 16, 2016

The Benefits of DOCSIS 3.1

Author: Drew Kempen, Consulting Systems Engineer

DOCSIS 3.1 has finally arrived! After years of talk and development, we are finally beginning to see the initial roll-outs of this next-generation technology. A few years have passed since the benefits of DOCSIS 3.1 were touted. 

Are those benefits still relevant today? It's worth revisiting.

In order to fully understand the benefits of DOCSIS 3.1, it is necessary to understand the boundaries of DOCSIS 3.0. DOCSIS 3.0 was a transformational technology in its own right and time. It provides the capacity to provide up to 1 GB of data to a service group, the ability to offer a high class of service, and provides many features and functionality that help operators with managing the customer, reporting, and reliability. 

Over time, the strengths of DOCSIS 3.0 become its weakness. The ability to achieve 1 GB downstream with up to 32 QAM becomes a limitation. Long-term bandwidth projections predict that DOCSIS 3.0 will begin to reach maximum capacities as soon as 2019 (without continuing to scale down service group sizes). In addition, competition driving 1 GB classes of service has accelerated the need for something beyond DOCSIS 3.0. The once high service group capacities of 3.0 platforms are now no longer enough. As service groups migrate to smaller and smaller groups of homes passed to manage bandwidth availability, more and more ports are required. The continual scaling of chassis, optics and other equipment to accommodate this growth becomes unsustainable. 

To put this scale into perspective, some operators have said they will need to split nodes from 4 to 10x what they are today over the next 10 years. And this is with the full capacity used on 3.0 chassis. The result would be 10x the CMTS chassis, 10x the optics, and 10x the nodes. Facilities, rack space and power requirements cannot scale with this growth. 

For a time, these inevitabilities were pushing many operators to consider a wholesale infrastructure transition to FTTH and PON technologies. The challenge with this was the complete overhaul of the entire network from video, to data provisioning, to OSP cabling and equipment to CPE. The cost, technology and knowledge change and disruption to the customers (and roadsides) made this a very unattractive option.

Enter DOCSIS 3.1. The first problem solved is the 32 channel limitation. DOCSIS 3.1 provides the ability to bond much larger groups of spectrum together to provide a true 1 GB Class of service and beyond. This also assists in the scaling problem. Whereas before, node segmentation would often be required when groups meet the 32 QAM limitation; the ability to use the full spectrum for data removes that requirement. 

DOCSIS 3.1 also allows for enhanced spectral efficiency. For math purposes, consider that a 3.0 256 QAM channel provides approximately 40 MB of throughput. DOCSIS 3.1 uses Orthogonal Frequency Division Multiplexing (OFDM) technology that allows QAM modulations to reach 1, 2, 4k and beyond. A 1k QAM provides approximately 50 MB of throughput or a 25% increase in the same amount of spectrum. When you combine this capability with distributed access architectures (DAA), we see added improvement resulting in 4k QAM modulation and beyond. Therefore, DOCSIS 3.1 provides the bandwidth with more ‘bang for its buck’. 

High level-comparison features and capabilities of Next-Gen 3.1 platforms vs legacy 3.0. *Numbers may vary slightly by vendor chassis

Originally, the new DOCSIS 3.1 and DAA technologies were designed with smaller and smaller cascades in mind. However, testing has shown that improvements can be made over some of the longer cascades that exist today. For example, it is possible to achieve 1024 QAM where 256 QAM currently exists. This improved performance continues to increase as you get down to smaller and smaller cascades. 

Addressing the Upstream

As data rates increase, the upstream continues to become more and more of a choke point. Studies suggest that the upstream capacity should be 10% of the highest class of service offered. For example, for a 1 GB service to be fully functional, approximately ~100MB of upstream throughput is required. As larger and larger data pipes are brought to each service group, the upstream limits will be pushed. DOCSIS 3.0 allows for a 5-85 upstream, allowing room for growth to handle this change. DOCSIS 3.1 pushes the split to 5-200 which allows for HFC systems to theoretically achieve a GB symmetrical service. 

The Importance of DAA 

DAA architectures such as Remote Phy or Remote Mac/Phy are inseparable from DOCSIS 3.1 when discussing the benefits next-gen DOCSIS platforms. While 3.1 chassis do traditionally offer a higher amount of port density in a chassis, this still becomes a limitation of the box. The next generation of CCAP platforms have more throughput potential than the physical RF output limitations can take advantage of. DAA becomes extremely valuable in that it removes that limitation by providing a digital link to the node itself, eliminating the limitation of physical RF ports. This also provides better link performance which continues to compliment the ability to achieve higher orders of modulation (better throughput performance across the same amount of spectrum). 

Improvement in MB throughput of spectrum by leveraging higher orders of modulation made possible by DOCSIS 3.1 and DAA. 

Perhaps the greatest benefit of DOCSIS 3.1 is that it dramatically extends the life of the HFC network and physical architecture. By extending the life of the physical infrastructure, it extends the life of all the assets of the network—from the video platform, existing CMTS chassis and provisioning systems, optical infrastructure, OSP, and CPE. 

The new urgency of a long-term plan

DOCSIS 3.1 in many ways did swoop in and save the day, but it also brings to light a flaw and errors that cannot be made again. For nearly a decade, many cable operators got trapped in operational mode without a long-term strategy. Had 3.1 not come along, the push to get to FTTH would be exploding at a rate that the supply cannot provide. 3.1 has brought new life to existing infrastructure and has allowed for a more graceful migration to fiber deep, higher bandwidth capacities, system upgrades, service migration and virtualization. All of these solutions need to be executed with an eye on the longer term future, to ensure that the things we do today compliment the needs of tomorrow instead of simply extending the limits of the past.

Tuesday, May 3, 2016

CCI Is Making Investments

CTO Matt Reath explains how CCI is preparing for the future of service providers.

Over the past few years, service providers across America have been challenged with increased costs, increased complexity, changing regulations, security, and most importantly, a changing consumer. Consumers have more choices and more options than ever before when it comes to how they consume content, how they access the Internet and the number of cloud-enabled devices they have. This puts pressure on service providers to increase capacity but not necessarily increase price. 

CCI Systems’ (CCI) unique perspective as both a solutions-focused value-added reseller and a service provider fuels a deep understanding of these challenges. As your clients’ needs change, CCI is taking the steps necessary to ensure those needs are met. We are investing in key technologies and solutions that will enable our clients to create higher customer satisfaction, offer additional services to capture more revenue, and optimize their operations to create a strong bottom-line.

Investment 1: Security

CCI has launched a Cyber Security Practice lead by dual-CCIE Adam Harden. This Practice is unique with its focus on service provider networks. CCI’s Cyber Security Assessment Workshops (CSWAP) program can get a provider on track with a Security Improvement Plan (SIP) that can serve as a personalized guide book of how to secure their network. Security services ranging from DDoS detection and mitigation to clean subscriber access. This will further enable providers to protect network infrastructure, improve their customer experience, and overall improve the performance of their network.

Investment 2: Data Center & Cloud

CCI continues to invest in its Data Center & Cloud Practice to evolve with the changing service provider landscape. The Practice takes two solution approaches. The first approach, internal virtualization, enables providers to reduce operating expenses by consolidating all company servers and storage into a central data center. Providers can gain space, reduce power consumption, and through orchestration—manage it as an integrated solution. The second approach, revenue creation, enables providers to generate new revenue by investing in a data center that includes orchestration and automation. Providers will gain the ability to create and deploy unique value-added managed offerings for their subscribers. A few examples include managed firewalls, virtual CPE services, hosted Wi-Fi, and secure SLA-backed storage.

Investment 3: Hosted Services

Service providers are looking for ways to create additional revenue streams but don’t necessarily have the staff or know-how to launch these services on their own. CCI is investing in infrastructure, talent, and software systems to enable our clients with services ranging from Wi-Fi to virtual CPE solutions. Clients will be able to white-label CCI’s solutions to quickly build a managed service offering in their footprint.

Investment 4: Orchestration and Programmability

Service provider’s networks are becoming increasingly complex and the speed at which new services must be deployed is accelerating. CCI is investing in Cisco Network Services Orchestrator, or NSO, to provide our customers with automated solutions for some of their top business use cases. CCI’s engineers are learning Python and other automation tools. We are investing in new talent and creating industry partnerships to bring the best solutions to our service provider customers.

CCI wants to translate our 60 plus years of expertise into benefits for service providers by addressing their biggest challenges today while preparing them for changes down the road. We work with our customers as a true ‘across the network’ solutions providing partner—enabling them to increase subscriber satisfaction, reduce costs, expand their footprint and grow revenue.

For more information on what CCI can do to help, contact us.

Tuesday, March 15, 2016

Bringing Intelligence to the Cable Plant

Written by: Todd Gingrass, Director of Cable and Media Solutions 

As cable operators, for years we have tried to be proactive about plant maintenance to keep services working at optimal levels and our subscribers happy. However, over the years our perception of proactive plant maintenance has mutated. A decade ago, performing 100% plant sweep and leakage detection was the go-to method for preventing outages and impairments. While that thought process was not incorrect, we all know the reality of it. Not enough time, people and resources. Staying on top of the day-to-day problems had been enough to keep us reactive in our maintenance process, making a full proactive sweep a pipe dream.

We know that a marginally well maintained plant will typically only have issues in about 50% of the actual sweep coverage. What if we could get that 50% of time back?  How would that change the bottom line as an operator? 

Getting out from behind
So how do we break that cycle of always being behind? It is easy to say we are too busy to get out and be proactive, but the reality is much more difficult. 
  • What if we could be proactive and not have to physically “get out there”? 
  • What if we could harness available data from the plant and mix that with decades of good old cable know-how to identify portions of the plant that need the most help? 
  • What if that could help me prioritize my day so when time is spent, it produces the most valuable results?
The answer, it’s all possible today!

Cable modems have been able to tell us a lot about plant performance for quite some time. As an industry, we have been slow to realize and take advantage of these technological advancements. While it is by no means “simple” to make all of this work, it is worth it on the backside.  

The new generation of modern intelligence
During the later stages of DOCSIS 3, modems on the market were able to capture the entire forward bandwidth—not just the original “sliding window” of 40 or 80 MHz, but the entire spectrum from around 50 MHz to 1 GHz. If this “capture” sounds familiar, it should. It’s basically a visualization of the frequency response of the forward bandwidth, otherwise known as… you guessed it, a basic sweep response.
With the new DOCSIS 3.1 standard out and vendors now going through certification, this and many other maintenance tools have been built into both CMTS and cable modems. Imagine being able to see the sweep response of your entire plant from a computer without ever leaving the office. Add on top of that, the fact that before we would typically sweep amp to amp, but now we are basically sweeping modem to modem. You can now see down to the tap where an impairment might present itself.

Actionable intelligence  
The prior example is only one of the potential tools that is or soon will be available to us as operators. Now imagine automating those steps so that information can be acted upon immediately, rather than having to spend hours analyzing and prioritize large amounts of data. We could really start to affect the days of our plant staff, allowing them to increase their productivity and get back to being preventative. This modification of how we view and handle plant maintenance will have a great impact on our real goals: 

As an operator, CCI understands how hard it is to stay ahead and offer the great services that we promise our subscribers. That’s why we have taken our 60+ years of industry experience and changed the game by building these types of tools. We use them ourselves and have seen first-hand how taking a truly proactive approach positively impacts our business.  

If you’re interested in discussing Managed Services for your network, reach out to CCI or follow us on social media.

Wednesday, March 2, 2016

Cisco Recognizes Winners

Each year the CCI team gets amped up for the first week of March. A week known to the select few as Cisco Partner Summit, a premier event attended by over 1,000 top Cisco partners from 75 different countries, to celebrate success and chart the future.

To CCI, the Partner Summit is an opportunity for gathering with friends, elbow rubbing with executives, strategic discussions, technology updates and recognition of top preforming partners. A week to reflect on the past year, and build a plan to accomplish even more the following year; goals are set, commitments are made and achievements are recognized.

Attempting to leave IMT Sunday afternoon
This year, the CCI team consisting of John Jamar, CEO; Joe Smith, CSO; Matt Reath, CTO; Bill Peters, director of strategic sales;  Eric Hiatt, director of sales; and Kae McGuire, director of sales operations, made the journey to San Diego (a two day trip for some due the unforeseen Upper Michigan weather).

Day 1 consisted of a keynote presentation by the newly appointed CEO of Cisco, Chuck Robbins, where he discussed digital transformation and Cisco's strategy in getting there. 

Day 2 was the day we were waiting for, the Global Awards Reception. CCI Systems was recognized with the Architectural Excellence US: SP Architectures award for our innovation, leadership and best practice as a Cisco business partner across the US. CEO, John Jamar and CSO, Joe Smith accepted the award on behalf of CCI which recognized CCI for our commitment and innovation in the SP market space.  

Cisco Partner Summit Theatre awards reflect the top-performing partners within specific technology markets across the US. All award recipients are selected by a group of Cisco Global Partner Organization and regional and theatre executives. 
CCI & Cisco Teams celebrating success
Day 3 of the summit continues today and wraps up Thursday afternoon. As the team heads home from San Diego there will be many takeaways and action items that have developed over the course of a few short days. In the coming weeks CCI will be collaborating to develop strategies on how are we going to not only continue our success together with Cisco, but how we will grow, powered by partnership.

Follow our live summit updates on Twitter @ccisystemsinc or Facebook .

Thursday, January 21, 2016

An Action Packed Year Planned for NCTC WEC 2016

CCI is gearing up to head to sunny Phoenix for the annual NCTC Winter Education Conference (WEC), where over 850+ NCTC members gather for the yearly educational event and vendor showcase. 

This year, CCI is excited! Probably the most excited we've been in a long time and there's a good reason for it. We've got something new to introduce to cable operators; something "state-of-the-art" or "revolutionary" as our engineers are calling it. CCI's Managed Services solutions consist of four components aimed at creating extensive value and cost savings for service providers. The component we're most excited about is CCI's cloud based network monitoring software.

Our cloud based monitoring software has two features we are sure you'll be interested in hearing about, but first, ask yourself these questions:
  •  Are you tired of being behind on plant maintenance? 
  •  Are you tired of not getting the network support you need? 
  • Are you tired of not having monitoring tools that give you actionable info?
If you are feeling any of these pains, keep reading. Heck - stop reading and call us, immediately! We need to talk. We can help.

The first feature, the "Outage Detection" tool, is capable of detecting an outage as it's happening. Big deal, you may say. Correct! It is a big deal. This tool will not only alert you of an outage as it's happening, but it will also pinpoint the location of the outage, saving you time troubleshooting and money by not having to roll excess trucks. Think of how relieved your customers will be to know you are already aware of the issue in their community and you've got technicians onsite looking into the problem. Churn rates will decrease and your outages will be resolved quicker.

The next piece of the super-amazing, ground-breaking, earth-shattering (by now you should be convinced this is a big deal) announcement is that not only can the hosted software detect an outage, it can also alert you BEFORE an outage. The "Plant Impairment" tool is able to identify and correlate modem problems within specific service areas before they result in an outage, and ultimately, a sizable headache for yourself.

The picture below shows snapshots of one of our customer's cable plant. CCI was able to alert them of possible issues within a service area. They then sent their tech to troubleshoot the location that was experiencing signal issues and corrected the problem before the customers were affected by a lapse in service - customer service nightmare averted! 

The third and final piece to this already amazing announcement is that you can try our hosted network monitoring software FREE for 60 days! Stop by and visit Todd and Zach at booth #605 to view a live demo and discuss your free trial. The demo will be fired up and ready to show you all the amazing features. We'll gladly show you what it's capable of and awe you with the flashy screen shots and mobile tech view, but most importantly we'll be able to show you how it will save you money and increase your subscriber satisfaction rates.

As an added bonus, CCI will also have Director of Technical Operations, Chad Kay, on-hand to discuss your TFS and outside plant needs, including:
  • FCC Proofs
  • Fiber Splicing
  • Sweep & Certification 
 Stop by and discuss you plant needs or upgrade thoughts with Chad. 

By now you should be jumping out of your seat searching for our phone number. You want to talk to us, you need to talk to us! Don't fret. We'll be available during the show and anytime before or after to discuss your network needs. Stop by our booth #605 or contact us to discuss your free trial.

CCI is the only partner who can consult, design, build, manage and support your entire RF, IP, Wi-Fi and optical networks. We can help you determine what to do, how to do it and have the expertise to get you there. 

For more information visit our website  or follow us on LinkedIn, Facebook or Twitter.