NRG: A Network Research Gadget
NRG: A Network Research Gadget
A NetFPGA Project, and a collaboration between Oxford and Cambridge
With contributions by Noa Zilberman, Andrew W Moore, Billy Cooper, Jackson Woodruff, Yuta Tokusashi, Pietro Bressana, Murali Ramanujam, Diana Andreea Popescu, Salvator Galea and Devang Seghal.
What is NRG?
NRG is a toolset that enables reproducible networking experimentation through network emulation and monitoring, recreating realistic data centre network conditions in small scale environments.
NRG is an open-source hardware-software toolset with bandwidth and nanosecond-scale latency and jitter control, providing a black box representation of a data centre network. NRG also provides a programmable, hardware accelerated, line-rate monitoring frameworks that offloads information processing from the end-host to the network and alleviates the need for frequent network probing.
NRG can be either standalone or a core within a network device. It was prototyped on NetFPGA SUME, and ported to several other platforms.
Determining the best provision for a cloud application can be challenging. Even if an application’s network properties are known, it is hard to infer the optimum choice of cloud resources. Once a cloud application is deployed, it is difficult to assess the weight of different factors on the overall application’s performance. If a change in an application’s code unintentionally affects performance, it is hard to find and debug it, even if the effect is large.
NRG provides a realistic local evaluation environment that can provide each and every time the same (network) test conditions. Coupled with that, NRG also provides an emulation of the data center network as a "black box", in terms of latency and bandwidth. NRG provides nanosecond-resolution control over
latency combined with support of line-rate traffic. The data rate supported by NRG is the same as an instantiated network device (NIC, switch or bump-in-the-wire), and seamlessly scales with it.
By changing configuration seeds, NRG enables creating similar-but-not-identical scenarios, yet still repeatable, enabling users to explore a wider range of "what can go wrong" scenarios.
How can NRG be used?
NRG can be used as a bump-in-the-wire, or as part of a NIC, a switch or a different network device. The released NRG project is demonstrating the bump-in-the-wire model, but our paper describes also an experience with switch integration.
The architecture of NRG is built from in-band traffic control (delay and rate control) and out-of-band monitoring. Each of the modules can be instantiated independently of other modules.
What are Network Profiles?
Small network changes may produce large performance changes. Network Profiles refer to a collection of network related characteristics and their relation to an application’s performance. Network profiles are designed for users with an application to deploy and a given system setup who wish to know:
- Is my application bottlenecked by the network?
- Is the network well utilized?
- Can network changes improve applications’ performance?
NRG allows to run on a local setup experiments that actively change network conditions (bandwidth, latency, burst size) and passively collect network statistics (e.g., inter-packet gap, link utilization, flow size). This collection of network conditions and statistics is the Network Profile of an application on a
The figure below shows an example of visualizing part of the network profile of Tensorflow (parameters server mode) as created using NRG in our environment. The figure shows Tensorflow's (normalized) training time dependence on bandwidth and latency.
What is the available data?
Over the time that we have worked with NRG, we have collected a very large dataset. A lot (though not all) of the experiments also included packet capture using Exablaze (now Cisco) ExaNIC, with nanosecond precision.
Due to the size of the dataset, we are still preparing some of the data to be released. Please contact us to request access to specific data that is not included in the release.
We aim to release captured packet traces, scripts, logs, and processed data.
How do you pronounce NRG?
NRG is pronounced en-er-gy.
Please use the following reference to cite NRG:
"NRG: A Network Perspective on Applications' Performance". Noa Zilberman, Andrew W Moore, Billy Cooper, Jackson Woodruff, Yuta Tokusashi, Pietro Bressana, Murali Ramanujam, Diana Andreea Popescu, and Salvator Galea. IFIP TMA Conference, September 2021