Fiber optics is the plumbing of data centers. While the computation, storage, and switching equipment are the focus in data center design, all this equipment is connected by many thousands of fiber optic cables. While the compute hardware will be replaced several times over the data center's estimated 15-year or longer existence, the fiber optics infrastructure is planned to last the entire time.
Requirements for the fiber optic performance are determined during the design phase, sometimes referred to as “Day 0”. Specifications are typically set to global standards, with the understanding that high performance is required to operate a high-quality network. One key parameter is the insertion loss since excessive loss can indicate poor installation, poorly cleaned connectors or other negligence. Performance against the design specification is determined in “Day 1” or during the deployment phase. In a publication discussing optical networking at cloud scale, Microsoft shared data on the performance of the fiber optic network [1]. They reported that 61% of the fiber met specifications on initial deployment. Of the remaining fiber, 30% met specifications after remediation while the remaining 9% of the fiber did not meet specifications even after remediation. In other words, almost 10% of the fiber did not meet specifications even on day 1 of the data center lifetime.
To meet the continuously expanding need for data, new generations of hardware are necessary and the data center's computing and switching hardware will be upgraded over time. Data center equipment that previously functioned at 10 G data rates has been replaced with technology that now operates at 100 G or even 400 G data rates. Signal modulation formats have likely evolved as well from simple on-off keying (OOK) or non-return to zero (NRZ) to higher order schemes such as multi-level pulse amplitude modulation (e.g. PAM4 or PAM6). All these changes reduce the link budget and place more stringent demands on the performance of the fiber optics. Beyond just the insertion loss, factors such as multi-path interference (MPI) need to be understood to insure performance with the new modulation formats. Multi-path interference occurs due to reflections at interfaces such as connectors and can create a signal penalty depending on the number of links in the connection path. If 10% of the fiber optic network didn’t meet specifications on day 1 of the data center lifetime, how many links will work after years of operation in the data center. And for connections that don’t turn on, how much time can you spend debugging the link or is the link just considered inactive after a limited time for debugging?
During the installation of new equipment, it would be very helpful to understand the current state of the fiber optic plant. How much has performance degraded due to technicians performing reconnections or repairing a fiber with a splice? Since there is always variation in performance of the optical elements, installation could be optimized by pairing transceivers with links. Rather than randomly pairing equipment with fiber optic cables, a “slightly hot” transmitter could be paired with a link with higher loss and vice versa. However, using manual labor to perform this measurement task is almost impossible due to the amount of fiber optics in data centers. Plus, operations tries to treat all components as identical and the process of matching a transmitter to a link would be extremely complex from an operations standpoint.
However, there are systems today that can be designed into the data center network to preserve the value of the fiber optic plant from day 1. Telescent has designed a large-scale robotic patch panel with integrated test equipment. The robotic system allows connections, reconfigurations and disconnections to be handled remotely while offering ultra-low loss and latching performance just like a regular fiber patch panel. And since the Telescent system includes a power monitor as well as optional equipment such as an OTDR, the fiber can be monitored and diagnosed remotely. Upon equipment refresh, the Telescent system could use its power meter and OTDR to test all transmitters as well as fiber optic links and then enhance the connectivity through an optimization algorithm. This would certainly reduce the number of inactive links in a data center and increase the overall efficiency.
Of course, this assessment and optimization during refresh is just one of the many applications for a data center's robotic patch panel. Many manual patch panels in a data center could be replaced with a robotic patch panel, which would eliminate human mistake among other issues. The Telescent system has been certified to NEBS Level 3, has completed many customer trials simulating a 10-year lifetime, and has over 1 billion port hours in live traffic operation, which is important for remote operation in critical functions.
While data centers replace the compute and switch hardware after several years, the fiber optic network is expected to last the life of the data center. Of course the performance is tested on installation, but the ability to measure and monitor the state of the fiber optic network can bring significant value to data center operations. Contact Telescent today to learn more about how the Telescent robotic patch panel with built-in diagnostic equipment can improve your operations.
[1] Mark Filer, et. al., “Low-margin optical networking at cloud scale,” Journal of Optical Communication and Networking, Vol 11, no 10, pp C94-107 (October 2019).