TL:DR – remove electrical spine switches in the Clos network and replace them with optical circuit switches [1,2].
While the above Too Long : Didn’t Read summary is said somewhat tongue-in-cheek knowing that there is a lot of complexity in the replacement of optical for electrical switches, the recent paper from Google discussed their use of an optical circuit switch (OCS) in their data center networking and the tremendous value the received due to this replacement. According to Google, using OCSes allowed them to save 40% on power usage and 30% on capital costs. Google also covered a number of additional advantages, including bit-rate independence in the network which enables the use of multiple transceiver generations, quicker network restriping, and a reduction in manual errors. The Google approach will be detailed in the paragraphs that follow and then an alternative and simpler OCS technology from their approach will be discussed.
The diagram below taken from the Google presentation at SIGGCOMM shows the initial placement of the OCS between the aggregation blocks and the spine layer. The initial value of adding the OCS into the DC fabric was to allow automated restriping for data center expansion. Since the hyperscale data centers can contain tens of thousands of servers, a data center is typically not “turned on” all at once and instead is installed in stages. This also improves the use of capital, allowing the installed capacity to more closely match demand. [3] Since the restriping is only performed periodically (on the order of weeks), the electrical spine layer can be removed and replaced entirely with the optical circuit switches. This creates significant power and cost savings in the DC network fabric as stated above.
There is another benefit of replacing the electric spine layer with the OCS. Due to the gradual buildout of a data center as well as the eventual refresh of the compute and storage equipment, data centers can operate with different generations of equipment. Since optical switches are bit-rate agnostic they can be used with multiple generations of transceivers. This allows both for the latest generation of transceivers to be used as the data center is upgraded and allows efficient use of the mixed bit rate equipment in the network.
To achieve the impressive results described by Google in the papers, they chose to internally develop a MEMS based OCS. As stated in the papers, “due to the difficulties in maintaining reliability and quality of this solution at scale, the decision was made to internally develop an OCS system.” [2] The low port count of 136 ports in the MEMS OCS led Google to use bidirectional optics and cWDM to increase the effective number of ports in the system. The use of bidirectional optics required the use of an optical circulator in the optical path. This increased the effective loss in the system, requiring home-run links between aggregation blocks using APC connections to minimize the effect of multi path interference (MPI) and return loss issues. This also led to the development of improved FEC algorithms as well to deal with the poor return loss value of the optical links.
Telescent has an all-fiber, high-port-count, low-loss optical switch that makes implementing an optical switch layer in the data center fabric much easier. The Telescent system consists of a short fiber link between two LC fiber ports with a robot that moves the selected port to the requested new location. The key element of the Telescent system is the routing algorithm that the robot uses to weave the fiber around over 1,000 other fibers to the new location. With over 1,000 ports in the Telescent system, it has the equivalent port count to 8 of the Google MEMS switches. The Telescent system has passed NEBS Level 3 certification and has been used in production networks. An interesting point of the Telescent system is that today it offers many of the element that the Google paper stated as future development requests of the MEMS approach – including higher port counts, lower loss and better reliability.
Contact Telescent today to learn more about saving $3 billion in data centers networking costs through the use of optical circuit switches.
[1] Google Apollo: The >$3 Billion Game-Changer in Datacenter Networking (semianalysis.com)
[3] Minimal Rewiring: Efficient Live Expansion for Clos Data Center Networks | USENIX