Facebook Focuses On More Efficient Front-End Servers

Facebook has made a series of announcements today at the Open Compute Project (OCP) summit. While these announcements generally focus around innovation and increasing the use or utilization of emerging technology, Facebook is announcing something a little different. Facebook has worked with Intel to develop new processors for their front-end servers, but instead of using servers with more CPU sockets, Facebook and Intel have cut the number down.

Facebook has made a series of announcements today at the Open Compute Project (OCP) summit. While these announcements generally focus around innovation and increasing the use or utilization of emerging technology, Facebook is announcing something a little different. Facebook has worked with Intel to develop new processors for their front-end servers, but instead of using servers with more CPU sockets, Facebook and Intel have cut the number down.

When more compute power is needed the general rule of thumb is to add more CPUs or more powerful CPUs. And Intel has done a wonderful job of creating more and more powerful CPUs each time around. While this is all well and good, each generation of CPU was coming with higher and higher power consumption, up to 120W. With the increase in power usage each generation, Facebook noted that it would cease to be scalable at some point. There is an 11kW limit per rack budget at Facebook and in order to hit this goal they began to work with Intel to create a new processor that would not only meet their needs but would be available to the whole industry.

Facebook began to work with Intel a few years ago to develop a single socket CPU, Broadwell-D, which was minimized to exactly specific needs. While Intel focused on the CPU, Facebook redesigned its server infrastructure. Facebook designed a 4 CPU/sled server that would accommodate the new 65W CPUs. That meant in Facebook’s 30 server rack they could go from 60 120W CPUs to 120 65W CPUs that hit their needs specifically. Twice the CPUs per rack at similar power consumption. It sounds simple stated like that but it meant that Intel had to modify their roadmap, which is anything but simple. Intel’s new line of products is called the Xeon-D line.

Facebook has designed the Mono Lake server, which it calls the server embodiment of Xeon-D and the building block for its SOC-based server designs. It is a highly simplified board that allows for the SOC to be placed with its associated memory and storage for boot and log, currently 32GB RAM and 128GB of storage (boot and log). While this may seem low, particularly with upcoming workloads having higher demands, Facebook states that future designs will have more memory or storage or both if needed.

To incorporate Mono Lake servers into its racks, Facebook is using Yosemite as its sled and associated NIC. Yosemite connects the four Mono Lake servers to the NIC through PCIe lanes, each server has an independent IP address. This design required the use of a multi-host NIC to aggregate the bandwidth from the four servers. The output of the NIC is currently designed to be 2×25 Gbps, which is dynamically shared across the four CPUs. Aggregating ports at the sled level makes for more efficient usage of the top-of-rack (TOR) ports.

Using lower powered CPUs seems counterintuitive to get more performance but Facebook has seen an overall increase using CPUs for specific needs versus running a more powerful CPU all out. This gives Facebook a better overall performance with lower power consumption. While looking at using CPUs in this way is still fairly new, there will be some challenges to overcome in the future such as the limitation of memory channels.

Engineering at Facebook

Open Compute Project

Discuss this story

Sign up for the StorageReview newsletter