If you’ve watched any TV recently, you have likely seen commercials for 5G wireless from Verizon and other carriers. While your first thought of wireless infrastructure may be the antennas seen along highways, there is a lot of fiber optics to supports the wireless connections. This includes the traditional backhaul networks as well as the extensive fronthaul networks required to support the densification of cell towers for 5G deployment. In articles discussing its 5G deployment plans, Verizon estimated it would purchase 12.5 million miles of fiber per year from Corning for a total of $1 billion [1]. With the added cost of approximately $26,000 per mile for aerial deployment and $173,000 per mile for underground deployment, the scale of the investment in fiber infrastructure becomes very significant [2]. With multi-billion dollars of investment, finding ways to monitor and protect this investment is important.
There are similarly large investments in fiber optics for hyperscale data centers. While, of course, most of the capital spend in a hyperscale data center is on the compute and storage equipment, these data centers also have a large investment in fiber communication equipment to connect the millions of servers and switches. While new hyperscale data centers are constantly being built, older facilities remain in use. Although compute and switch hardware can be refreshed during an upgrade cycle, the existing fiber infrastructure is re-used and, in some instances, can be over a decade old. Links can have from 1 to 6 hops through manual patch panels within data centers, may have been disconnected and reconnected multiple times, and may have degraded performance in terms of increased loss or reflections. Since record-keeping is notoriously poor and subject to human errors, many links have unknown optical performance. The sheer number of links makes manual characterization daunting.
Characterizing the fiber plant in a hyperscale data center is critical as higher bandwidth speeds are used to increase data center performance while maintaining the ability to reuse the installed fiber plant. When 100 G (25 G/lane) was implemented for hyperscale data centers, there were occasional problems. In a published report from Microsoft, almost 40% of links required remediation, and even after remediation, 9% were still out of spec, as shown in fig. 1 [3]. With the deployment of 400 G (100 G/lane) and modulation formats such as PAM4, link budgets will be more stringent. Also, these formats are more susceptible to issues such as multipath interference (MPI). Future deployments of 800 G and beyond will create even more stringent requirements for the link performance and require more knowledge of the installed fiber plant.
While the earlier discussion highlights the need for characterization and monitoring of the fiber plant to protect the significant investment, the number of connections in hyperscale data centers and the number of remote sites for 5G deployment makes manual testing impractical. To address this problem, Telescent has developed the Network Topology Manager (NTM), a 1,008 x 1,008 strictly non-blocking single-rack robotic cross-connect with an integrated fiber cleaning function. This device, shown in figure 2, can connect, disconnect, and reconnect fibers in an arbitrary manner. A key feature of this device enables test equipment to be automatically switched into a given fiber link to enable characterization of fiber performance. Power sources, power monitors, and OTDRs are examples of equipment that may be incorporated into the NTM, permitting in-situ fiber characterization to take place without human intervention. Since the Telescent NTM can insert a range of test equipment, Layer 2 and 3 test equipment can also be remotely controlled to monitor traffic. Additionally, the Telescent system can be used to reconfigure the fiber paths of networks to optimize performance or respond to broken links.
The functionality of the Telescent NTM leads to a few use cases related to 5G networks and hyperscale data center fiber plant. These include:
Using a tunable OTDR with the Telescent NTM to provide complete link testing (BBU to RRH) in the access network remotely. This can reduce network costs by enabling passive-passive WDM links rather than the more expensive active-active WDM design.
Automated characterization during deployment: here the NTM is used during deployment of the structured cabling to characterize the as-installed fiber plant, providing a baseline for future performance.
Automated characterization during fleet refresh: During a major network equipment refresh cycle, the NTM can be installed, either temporarily or permanently, to characterize the fiber plant as the fleet refresh progresses.
Ongoing automated characterization: here the NTM is deployed within the data center network. Fiber links that are not currently in use can be tested to ensure they will support proper performance when used. In-use fiber links can be tested if a particular link starts to degrade, with the NTM being used to identify port or link degradation.
As a specific example, assume that a conduit with 864 fibers is installed and fibers pulled between data centers – either on a hyperscale campus or between data centers in a metro environment. The fiber strands are terminated at a patch panel. These fibers should all be tested for loss. One way to do this is to place Telescent NTMs at each end, connect the fibers, and then run a simple script overnight queuing a job where the NTM measures all the losses. One can also perform this single-ended using the OTDR. If there are splices or connectors in the path, then an OTDR yields more meaningful results. An example of an OTDR trace with high reflectance in the link is shown in fig. 3. After initial performance testing, the NTM + OTDR combination can be used to validate performance before lighting up a dark fiber and to monitor performance over time. One of Telescent’s customers, Mox Networks, used the NTM in this way between data centers in Oregon [4].
As another example, a Telescent NTM may be placed in the leaf-spine layer of the network, connecting switches between those layers. Assume the network is properly operational. Then assume at some point a particular link starts to degrade or have other problems. The NTM can interrogate each side of the link, first to switch 1, then to switch 2, to determine if the fiber layer is OK, using the OTDR function. Then an optical power meter integrated into the NTM can be used to measure the optics. Finally, packet test equipment can also be used inside the NTM to test the port on each switch.
The NTM does not work alone in monitoring the fiber layer. The NTM can be part of an overall Software Defined Networking (SDN) architecture that provides key physical layer information that can be correlated with data from other devices in the network like switches and routers as well as WDM systems. The advantage of having the physical layer monitoring is to protect the value of the installed fiber plant, allow for reconfiguration and avoid installing equipment on links that won’t support the link budget requirements for those specific transceiver optics.
Integration of power monitoring, OTDR, ethernet physical layer test equipment, and then layer 2 test equipment allows for automated, remote testing of any port or link at any layer. The key to the NTM is that the equipment is shared over many links and ports. For example, if a single NTM is used to connect up 1000 ports to another 1000 ports then the equipment would have to stay in that NTM or be removed at some point to be used elsewhere. But as the Telescent NTM usage grows to build a complete network, all links and ports are connected through multiple NTMs. In this case, a single set of test equipment might be able to span an entire data center.
With the current scale of hyperscale data centers and the deployment of higher-speed optics as well as the expansion of access networks to support the deployment of 5G, characterizing the state of the fiber plant will be critical. The Telescent NTM allows this characterization to be performed remotely and can be integrated into other SDN Network functions to bring visibility and control to the physical fiber layer.
1] The Story Behind Verizon’s 5G Secret Weapon | Light Reading
2] How much does it cost to get fiber optic cable installed in a house? (askinglot.com)
3] M. Filer, et. al, “Low-margin optical networking at cloud scale [Invited],” J. Optical Communications and Networking, vol 11, no. 10, pp. C94-C107, 2019