Bufferbloat – How “Fat” Buffers Are Killing Internet Performance As We Know It!:
With Dr. Jim Gettys, Alcatel-Lucent Bell-Labs

{fastsocialshare}

Conventional wisdom has it that the buffer in a router should be sized as per the bandwidth-delay product (the so-called BDP rule). That is, the buffer must be able to store a number of bits equal to two times the product of the average roundtrip time (RTT) and the bottleneck link capacity. This has persisted, even when this “rule” was devised in the days when links in the Internet had some semblance of “similar” capacity, and when the link capacities themselves were much, much smaller. Over the last decade , however, link capacities have grown exponentially, from 10 Mb/s to 100 Mb/s to 1 Gb/s, 10 Gb/s and beyond! Plus, the variability in link bandwidths in different parts of the network is now so large that the notion of a “bottleneck link capacity” that would make sense in different parts of the network no longer holds. In other words, there is never a single right answer for buffer size in a general purpose communication system.

Yet, as memory has become cheaper, the temptation has been to just provision very large static buffers in all parts of the network (sized, per the BDP rule, using the maximum possible bandwidth that the hardware might ever be used in), irrespective of what the actual bottleneck link capacity is likely to be. This leads to very large or “fat” buffers from the access network segments all the way to the core network.

These “dark buffers” lurk in the shadows, sitting empty most of the time, but on filling-up they destroy TCP’s congestion avoidance mechanism, which requires systematic packet drops to function correctly. In other words, “fat” buffers disrupt TCP’s systematic packet drop process, leading to low TCP throughput, and, consequently, to very poor application performance, particularly for interactive and low-latency applications, and also have implications for application performance in the cloud. This has serious consequences in a world where new and interesting applications are what operators and content providers are banking on to garner new sources of revenue!

To delve into some of the details behind bufferbloat, we invited Dr. Jim Gettys (well-known in the world software community as Mr. X-Man, developer of the X-Windows system, who first brought buffer bloat to the attention of the world in summer 2010) to talk to us in an episode of our signature series “Conversations with Experts,” and explain his discovery of buffer bloat, what that means for the Internet, and his suggestions on how to mitigate and ultimately solve buffer bloat, by a complete redesign exercise.

(Note: Since this episode was recorded when we caught up with Jim at an industry conference, there is hotel background noise in the video. However, the actual discussions with Jim are clear enough to be easily understood by the user.)

{fastsocialshare}

In this Conversations with Experts episode, we focus on the problem of buffer bloat, which is the existence of excessively large buffers at different points in modern communication systems, specifically end-user machines, broadband routers, & metro and core switches and routers.

We begin by asking Jim to explain buffer bloat, and its relationship to packet drops. It turns out that large buffers essentially inhibit timely packet drops, thus disrupting TCP’s congestion control mechanism by disturbing the periodic packet drops that TCP relies on to function normally. Jim then goes on to explain some of his experimentation with teleconferencing systems that lead to the discovery of buffer bloat about a year back, and describes what he observed, and the conclusions he came to.

Thereafter, Jim proposes two solutions to buffer bloat. The first, is a mitigation strategy, which involves reducing significantly the sizes of buffers found in many parts of our communication systems. He explains the myth of the bandwidth-delay product, and the problems with statically sizing buffers using that thumb-rule.

Given that it is impossible today to “right size” a buffer for proper TCP operation, using any statically-sized buffer, the second (more lasting) solution is to have buffers whose depth varies dynamically, achieved using Active Queue Management (AQM), i.e., solutions like Random-Early Discard (RED), and it’s variants. We explore why AQM has hitherto not been actively adopted by the operator community, and discover some interesting facts that have made providers hesitant to deploy it.

After explaining the AQM puzzle, Jim discusses how people can actively contribute to the ongoing efforts to help address buffer bloat, and be a part of its effort to build a good home router.

More details on the Buffer Bloat community’s activities are here, while Jim’s blog is here.

Biography

Dr. Jim Gettys is currently with Alcatel-Lucent Bell Labs. Research, and works on immersive, interactive applications, and their requirements. He is best-known as one of the two original developers of the X-Windows system at MIT, and, later, for his work on the specification of the HTTP v1.1 protocol, and his contributions to the One-Laptop Per Child (OLPC) initiative.

Jim has had a long and distinguished career at a number of marquee companies and institutions over the last 30 years, including DEC, HP, MIT, Princeton, Harvard, and One-Laptop Per Child.

Prior to his current role at Alcatel-Lucent/Bell Labs., Jim was Software Architect and Vice-President of Software at One-Laptop Per Child, where he played a seminal role in reviewing and overhauling much of standard Linux software, in order to make it run faster and consume less memory and power.

He previously served on the GNOME Foundation Board of Directors, and has also worked at the World Wide Web Consortium (W3C). Jim is also the editor of the HTTP/1.1 specification in the Internet Engineering Task Force. Jim also helped establish the handhelds.org community, from which the development of Linux on handheld devices can be traced.

Jim won Bob Metcalfe’s 1997 Internet Plumber of the Year award on behalf of the group who worked on HTTP/1.1., and is one of the keepers of the Flame (USENIX's 1999 Lifetime Achievement Award) on behalf of The X Window System Community at Large.

Jim has BS, MS, and Ph.D. degrees in Earth and Planetary Sciences from MIT.

  Go Back

Flexible Optical Transceivers: Answering the Call for Lower Costs from Network 
Operators!:
With Thomas Wieble, Chief Technology Officer, FlexOptix

{fastsocialshare}

In the present day and age, when minimizing OPEX and CAPEX is the mantra at practically every service provider, operator, ISP, cable provider, and data center and exchange operator anywhere in the world (whether big or small!), one may not think so much about optical transceivers as having the ability to play a big role in the equation. Nothing, however, could be farther from the truth! In fact, the last few years have seen rapid progress in the development of programmable or flexible optical transceivers (and, actually, also flexible transceivers for wireless applications, but that’s another episode!) that provide lower-cost, higher-intelligence modules.

The rapid expansion of IP-based services in the access has increased the demand for optical modules from operators, while making it vital to lower the module prices and the associated operational costs. Indeed, with highly-distributed networks, the network management and operational support system requirements are also becoming more complex, making effective network management a key need for the operator to deliver the highest bandwidths in the shortest possible time. In fact, rapid bandwidth deployment increases the number of upgrades in the network. Such continuous change requires optical module manufacturers to become smarter in the design of their components, and allow for performance monitoring and diagnostics, which increase network reliability, maintain high up-time, and lower operational costs for the network operator.

To discuss these next-generation transceivers, and, most importantly , to understand the numerous benefits they provide to the service provider, we invited Thomas Wieble, CTO of FlexOptix, a leading flexible transceiver provider, to an episode of our signature series “Conversations with Experts.” Thomas enlightened us on the value of programmable transceivers for operators, the benefits they provide, and the cost vs performance trade-offs. Truly fascinating insights!.

 {fastsocialshare}

In this Conversations with Experts episode, we focus on the new breed of flexible optical transceivers, which have emerged in just the last 2-3 years, and have a remarkable impact on the network reliability, troubleshooting and diagnostic capability, and therefore, the network operational cost of a provider’s network!

Thomas starts by discussing the key capabilities of programmable transceivers, and what these mean for a network operator. That is, how the operator may make use of them. We then shift to delving into their benefits for the operator – that is, how they have the potential to streamline operations – from diagnostics to network monitoring.

This leads to an interesting discussion on the differences between carrier architectures in Europe vs the US, and how flexible transceivers play a role in meeting the needs of these different network designs.

Thomas then explains the impact that programmable transceivers have on the ability to efficiently maintain and trouble-shoot the network, versus the costs of these transceivers (when compared to standard passive transceivers). We discover that the cost-performance trade-off works hugely to the advantage of programmability.

We close by asking Thomas to spell out the key lessons and best-practices to take away, based on his and his company’s experience in dealing with operators of all shapes and sizes in different parts of the world.

Biography

Thomas Weible is Chief Technology Officer at FlexOptix, an innovative provider of programmable transceivers, based near Frankfurt, Germany.

Thomas and his team of innovative engineering, software, and optical experts, have, over just the last few years, built FlexOptix into a leading technology holder in the area of reconfigurable transceivers.

At Flexoptix, Thomas has dual responsibility for technology and marketing. He is responsible, on the one hand, for guiding internal technology development, and, on the other, for keeping his finger on the pulse on the industry by interfacing with customers and partners worldwide, including carriers, operators, ISPs, data center and exchange providers, and cable operators.

Thomas received an MS in Computer Science from the University of Applied Sciences Darmstadt, Germany, and also has degrees from the University of Wisconsin, Madison, USA, and the James Cook University, Australia.

  Go Back

The IPv6 Conundrum: The Interplay Between Carriage, Content, and Regulation, and 
the Impact on Economies:
With Geoff Huston, Chief Scientist, APNIC

{fastsocialshare}

Among the many important Internet anniversaries and milestones we reached in 2011, a significant one was in February 2011, when IANA handed out its final blocks of IPv4 address to the RIRs (Regional Internet Registries). Plus, the IPv4 address stocks are rapidly running out in the Asia Pacific, and other regions will soon follow!

Thus, it was timely that on World IPv6 Day, June 8, 2011, Google, Yahoo, Bing and Facebook, converted their main web pages to be reachable over both IPv4 and IPv6 (for a 24-hour test period). This acted to draw the attention of the industry towards preparing their service offerings for IPv6, and was also intended to encourage them to accelerate their deployment plans for IPv6 in the Internet.

The imminent exhaustion of IPv4 addresses and the profusion of mobile devices, and more importantly, of M2M (machine-to-machine communication) over wired and wireless media (creating potential demand for hundreds of millions of IP addresses each year), has brought to the fore very interesting issues relating to the transition to IPv6 – E.g. When will it finally occur? How will that process play out? What are the positions of the various players – service providers, content providers, end-users, regulators, and governments -- involved? What are the pushes and pulls, and even the economic implications of this transition?

Indeed, given that IPv6 development was started as a successor to IPv4, during the original discussions of IPv4 address exhaustion back in 1990, and its specification more or less complete by 1998, even we wondered why is it that IPv6 is not already in deployments, and why the brouhaha about the transition to IPv6?

To delve into these issues, we invited Geoff Huston, the pre-eminent world authority on IPv4 exhaustion (and Internet pioneer and visionary, and “Father of the Internet” in Australia, to those of us who have been around a bit in the industry) to share his perspective on this fascinating and, as will be seen, fairly complex issue. Geoff interacted with us during an episode of our signature series “Conversations with Experts,” to share the essence of his key insights into migration issues, having been quoted in the OECD’s publication “The Internet Technical Community Issues Memorandum on Future of the Internet in a Global Economy,” 2008..

 {fastsocialshare}

In this Conversations with Experts episode, we focus on five key aspects of IPv6 deployment and the issues it raises.

Geoff first provides, in his own words, a brief run down of his work in the Internet, going back to the early 80’s, culminating with his seminal role in bringing the Internet to Australia in 1989.

We then discuss address exhaustion in the Internet, it’s history, and why it’s a problem. From there we segue into IPv6, and what the implications are for the various players – service providers/carriage providers, content providers, and end-users, of such a transition. We focus on the investments needed do so, the cost-benefit argument for doing so (who benefits and who should invest in the infrastructure), and finally, who should pay for this transition and why? All complex and interesting issues!

Geoff then explains the carriage providers' problems, with a beautiful analogy going back to the barge operators of old in England! This explains the perpetual tension between the carriage providers (the barge operator or the service provider) and the content provider (the shipper or merchant of old, and the content providers (Google, Netflix, et al) of today).

Geoff draws upon his extensive work with numerous world bodies such as OCED, ITU-T, APEC, and ICANN, to explain why simple market forces many not, in this case, be sufficient to propel the transition to IPv6. It turns out that the Internet is too economically core an infrastructure (worldwide) for it to be controlled solely by the vagaries of “market forces.” Indeed, we then focus on the impact of such a transition (or not) for the economic well-being of many nations, and it is fascinating to hear Geoff articulating the case for some regulator intervention to nudge activity towards the overall economic good of nations.

Geoff’s insights into IPv6 transition issues can also be found in the March 2011 issue of the Internet Protocol Journal.

Biography

Geoff Huston is currently Chief Scientist at the Asia-Pacific Network Information Centre (APNIC), where he focuses on Internet infrastructure, IP technologies, address distribution, and Internet operation. Geoff is an internationally recognized and sought after expert, and widely regarded as the pre-eminent researcher on IPv4 exhaustion and IPv6 transition issues.

Prior to APNIC, Geoff spent 10 years (1995-2005) at Telstra as Chief Internet Scientist, and earlier as Technical Manager of the pre-eminent Australian Academic and Research Network (AARNET). At Telstra, he took a leading role in the construction and development of Telstra’s Internet service offerings in Australia and internationally, and was instrumental in introducing the Internet in Australia, back in 1989.

Geoff is an internationally recognized authority, active in numerous Internet organizations, such as the IETF, NANOG, RIPE. He has been on the Internet Architecture Board, and the Board of Trustees of the Internet Society. He is an invited speaker, representative, and advisor to numerous international forums and government bodies, advising them on the economic impacts of IP technologies. He is currently Co-Chair of IETF Working Groups on BGP Security and IPv6 Multi-homing.

Geoff is the author/co-author of some fundamental books on IP technology including: The Internet Performance Survival Guide, John Wiley, February 2000; An ISP Survival Guide, John Wiley and Sons, November 1998; Quality of Service: Delivering QoS on the Internet and in Corporate Networks, John Wiley and Sons, February 1998.

Geoff received the BSc (Hons) and MSc (Computer Science) degrees from the Australian National University. His personal website and blog (with insightful observations on all things Internet!) can be found here , and a very interesting bio can be found here.

  Go Back

From Telephone Poles to GPON & Packet-Optical Networking: 100 Years of a Rural 
Carrier’s Evolution!:
With Bob Bender, Network Operations Manager, CTS Telecom

{fastsocialshare}

In this day and age, one hears and reads a lot about operator woes: stiff competition, declining voice revenues, increasing traffic volumes, growing network complexity, convergence to a common IP-network, and the need for sophisticated value added services all keep coming up in conversations! With this thought in mind, we at Metanoia, Inc. set out to find some examples of operators that were bucking the trend, and see if we could glean the essence of how they have been able to accomplish that.

So, with operators concerned about costs and profitability, how does a 100-year old phone company in rural America maintain a profitable data services business? It is really possible to thwart competition from behemoths, grow a loyal customer base, and provide great service, all while still remaining profitable? What are the “secrets” and what lessons might other small to medium operators glean from such an example?

This is a question that intrigued us, as we met and interacted with Bob Bender of CTS Telecom, the CLEC arm of a 100-year old traditional telephone company, Climax Telecom in Climax, Michigan. We learnt that the secret lies in using technology cleverly, making technology transitions at the “right” time, and in scaling operations appropriately.

To help elucidate these concepts further, and to share with us the interesting story of how Climax Telephone evolved from doing traditional phone service to, today, offering packet-based Ethernet EVCs (!) – quite a leap if we may say so :-), we invited Bob Bender of CTS Telecom (who has had an instrumental role in enabling this transition, from in-the-trenches work to high-level design) to an episode of our signature series “Conversations with Experts.” Bob discusses with us this (on-going) transformation.

 {fastsocialshare}

In this Conversations with Experts episode, we talk with Bob Bender about CTS Telecom’s evolution and transition over the years, to draw out some key takeaways of their approach, which we believe would be valuable for other small and medium operators.

Bob starts with telling us a bit about Climax Telephone’s history, and then talks about the range of technologies they’ve had over the years, and their evolution to the present day. The key to the transition was the ability to meet the three main technology challenges – transform their switching for voice from TDM to packets, deploy a high-speed IP core, and a provide Carrier-Ethernet based all-packet access network.

We then seque into discussing CTS Telecom’s differentiators, and talk about what they have meant for their success thus far. Bob then highlights CTS Telecom’s strategy to transform its access network. It’s unique proposition being that, unlike other competitors (large and small), CTS Telecom has invested (and continues to invest) in building/acquiring a last-mile access network, which gives it control of the much-coveted "last mile", while reducing dependence on the incumbents for last-mile access to the end-customer. In addition, CTS Telecom has invested in sbuilding valuable partnerships with its vendors, and expanding its “on-ramp” strategy.

Finally, we conclude with summarizing how all these pieces fit together allowing CTS Telecom to today come full circle – from carrying packets over TDM to carrying TDM over packets!

Biography

Bob Bender is Network Operations Manager at CTS Telecom, and has been there since 1998. (Climax Telephone is a 100-year old ILEC in Climax, MI, with CLEC operations as CTS Telecom since 1996.)

At CTS Telecom, Bob plays a truly multi-faceted role, leading network architecture, design, and technology planning. Additionally, Bob and his team in Operations and Engineering are also responsible for contributing to strategy, and for technical sales, customer meetings, vendor relations, service development, and road-mapping to keep CTS Telecom on the cusp of the technology curve!

Bob and his team have been instrumental in helping CTS Telecom make several significant technology transitions. Recent projects have included the selection, design, and implementation of the Metaswitch softswitch, TDM-to- packet migration planning, NGN packet-optical network planning and implementation, and the launch of MEF-based Ethernet services. Current activity also includes ARRA Broadband Stimulus GPON implementation in the ILEC and a NGN access network project to replace the legacy DCL and Wave7 EPON with the Adtran Total Access 5000 MSAP.

Bob has had to evolve his career, to keep him on the edge of new technology evolutions in computers and networking, amassing the expertise that serves him well to this day!

Indeed, with a background in graphic design, Bob dropped out of art school to work as “compositor” for a publisher, operating phototypesetting equipment. As technology advanced, he became a systems manager, overseeing a mini-computer based text-processing system that output on phototypesetting equipment. With the advent of personal computing and “desktop publishing”, Bob moved to the vendor side, as a network engineer for a publishing systems integrator, delving into LAN and WAN networks and dabbling with routers, just as IP networks started to evolve.

The future of traditional publishing looked bleak, and the constant nationwide travel to publishing sites became exhausting, so an opportunity arose to change careers. An associate of Bob’s who was a manager for Climax Telephone said “How’d you like to learn about the phone business?” Since this meant being home every night, although less lucrative :-), Bob made the official switch from publishing to telecom.

His first assignment was a month at Lucent 5ESS translations school, after which he worked in “translations”. At that point, all of Climax’s revenues were from voice services. As the Internet blossomed, Bob attended a class on Cisco routers. Climax had been providing T1s for several small ISPs, but they proved short-lived. Finally, CTS decided to get into providing Internet access for its customers, initially using IDSL, and then T1 and integrated access (voice and Internet). In 2004, he was named “Data Products Manager” and lead the development of the ISP operations. In 2005, Bob spearheaded the deployment of PON. In 2009/10 he helped design and launch CTS Telecom’s Ethernet and packet optical network, and their Ethernet services business.

  Go Back