There is a growing recognition of the need to deliver interconnection services faster to meet customer demand. This is highlighted by a recent article by Verizon and Equinix announcing a Software Defined Interconnect service [1]. In this article, Verizon describes their Network-as-a-Service strategy that allows their customers to connect with automated, same-day connectivity. With this software defined interconnect capability their customers can now take advantage of “quicker activation of network services, lower access costs and increased networking agility.”
Many other multi-tenant data center (MTDC) companies are discussing similar strategies that offer flexible interconnection services – particularly for connections to cloud providers. These “on-demand” or “exchange” interconnect fabrics are often managed through large router systems and virtual cross connects. Once the enterprise is connected to a router port a virtual connection can be provided to any cloud or service provider that is also connected to the router. The enterprise can also request an increase in bandwidth up to the limit of the physical port speed, providing an option for bandwidth on demand.
While virtual cross connects are effective for some customers, there will always be a segment of the customer base that requires their own physical cross connects, either due to regulations or for security reasons. These include finance, health and government customers and this segment could be up to half of all customers at an MTDC. And even for customers using virtual cross connects, the virtual cross connects only work because of an underlying physical cross connect to the router. To highlight this need for physical cross connects, in the most recent quarter Equinix reported a total of 373,400 physical interconnections with 33,200 virtual interconnections [2]. And of the cross connect growth for the quarter, 73% were physical interconnections. To offer an on-demand, agile network service with automated delivery for all customers requires a technology to automate the physical cross connect. As we will discuss below, with the right choice of technology an automated physical cross connect will also be much cheaper than a high-speed router port.
Based on the above discussion, what are the requirements for an automated physical cross connect system for use in a multi-tenant data center? Some needs include:
Scalability: an MTDC can have thousands of cross connects between enterprises and service providers in the meet-me-room of a data center campus. Any solution should be able to scale in a pay-as-you-grow manner to meet this large number of connections.
Reconfigurable: Along with scale comes the need to handle multiple reconfigurations for customers over time as they migrate their service providers. This means the system should not just automate the initial connection but allow reconfiguration of connections at any point in the future.
Reliability: MTDCs have service level agreements and face penalties when these SLAs are not met. Any system should be tested to meet high standards, should fail in a safe manner (i.e. not disrupting traffic) and should use field replaceable parts without interrupting traffic. In other words, the system “should fail like an escalator, not an elevator.”
Low loss: The system should be future proof to technology improvements. With the development of higher speed optics with lower optical performance margins, low loss is important for the automated system.
Cost: Technology only gets implemented when it offers an economic advantage.
Before examining a solution that meets all the above requirements, it is worth examining a range of technologies that have tried to address the need for optical switching in data centers. These include “free-space” technologies such as micro-electromechanical switches (MEMS) and piezo driven modules as well as multiple robotic approaches.
In the MEMS approach, a micro-mirror is used to steer the signal from an input fiber to the output fiber. This is a free-space approach since the light exits the fiber through a collimation lens to be steered into the desired output fiber. While MEMS systems can provide fast switching speeds, they suffer from high loss, limited scale and concerns about reliability. The loss can be up to 3 dB through a MEMS switch and the scale is limited to several hundred ports per system. With a power outage all connections are lost, and there have been reports that the MEMS switches do not perform well if left in the same position for a long time. Finally, the cost of MEMS optical systems are hundreds of dollars per port. While MEMS optical switches have been around for 20 years or more, they haven’t met the needs of MTDCs.
An alternative free-space approach uses a piezo-crystal to aim the input port to the desired output port. As with the other free-space approaches, this technology also has a problem with scale, high loss and reliability. While it is possible to have lower loss with a smaller port count system, as the scale of the system increase to over a hundred ports, the free space distance between the input and output fibers increases leading to an increase in loss. This is also an active device, so any loss of power leads to a loss of connectivity.
As seen with the approaches above using an active element as the switch, either a MEMS device or piezo crystals, leads to a challenge with reliability and loss. An alternative approach uses robotics to move a fiber to the new connection. Once the robot makes a connection, the connection will act as a passive patch panel with a latched, low-loss connection.
One of the simplest robotic systems attaches a robot to the front of a patch panel. This is described as a semi-automated patch panel since it requires cables to be fed into the system as new connections are made. This approach does offer scale since the robot can be attached across multiple patch panels. However, while the robot can be useful in making an initial connection, it does not solve the problem of how to handle reconfigurations without creating a chaotic tangle of cables that develops just like with traditional patch panels over time. After all, a semi-automated solution is only half the solution.
Other robotic systems do allow reconfiguration by placing the robot inside the system. This allows for the robot to reconfigure the system by disconnecting the fiber from the initial port and reconnecting to the new port. However, some products suffer from a limited scale since the robot must avoid creating an internal tangle after multiple reconfigurations. Limited scale also increases the per port cost since the cost of the robot, electronics and other common parts are amortized over a smaller number of ports.
One solution that does meet all the requirements listed above is the Telescent Network Topology Manager (NTM). The Telescent NTM offers complete cross connect lifecycle management for MTDC operators with the scale, reliability and cost needed for this application. The system monitors, maps and controls the network during installation, reconfiguration and churn of cross connects to enable a Lights-Out Data Center with machine accurate inventory while preserving the value of the installed fiber plant. The NTM consists of 1,008 input and 1,008 output ports, each connected by a short, uninterrupted internal fiber connection. When a reconfiguration is requested, a robot-driven gripper removes an internal fiber from the original port and moves it to the desired new port. The patented algorithm identifies the unique path to route the fiber around the 1,007 other internal fibers in the system to establish a non-blocking, any-to-any connection. The Telescent NTM is strictly non-blocking and completely automated – essentially unlimited configuration and reconfiguration is possible with the Telescent system without any on-site, manual labor and with the original external fiber connections. The Telescent NTM offers the low-loss of FC fiber connections with an average of < 0.3 dB loss per connection. The Telescent NTM has passed the rigorous NEBS Level 3 certification, including making connections even during the 8.3 level earthquake test. For scalability, the NTM can be configured in a leaf-spine network to allow scaling to >10,000 connections while still preserving the ability for any-to-any connectivity. Since the reconfigurations are controlled by software, the NTM can be integrated into an existing SDN framework through the Telescent NTM API. Also all connections are recorded in a database and the state of any connection can be verified at any time providing machine accurate inventory for accurate billing to customers.
Unlike previous optical cross connect technologies, the Telescent Network Topology Manager meets the scalability, reconfigurable, reliability, low loss and cost requirements to bring automation to the MTDC meet-me-room and offer automated delivery of interconnection services demanded by customers today. To learn how to implement the Telescent NTM in your network today, please contact Telescent at info@telescent.com.
[2] Equinix Q2 2021 Earnings Presentation, slide 22, Q2 2021 Earnings Conference Call (equinix.com)