Motivation - The need for Experimental Facilities (aka Testbeds)?¶
While the Internet and more generally network technologies improve our lives in many ways, their increasing presence also highlights their various limits. These limits drive the researchers and industries to design and develop new networking technologies. These technologies need to be evaluated and refined before being included in full fledged products.
Simulations allow some rapid first approximations of behaviours, performance and scalability of a new technology. As such, they are valuable (and often inexpensive) initial steps in evaluating a new technology. However, due to design and computational complexities, simulations use simplified models, which fail to capture the dynamics and intricacy of real environments. Emulation is an alternative, which combines the use of implemented prototypes and substitute entities reproducing real-world behaviours. However, these substitutes again often fail to capture the complexity of real environments. As a result, only experimentations using prototype implementations in real and controlled environments (e.g. actual users, wireless radio devices, vehicle movements, building layouts) will provide results close to reality. Such experimental platforms or facilities are referred to as testbeds.
Thus, networking testbeds are a key component in the development of new networking technologies, and this insight led to many large-scale global initiatives aiming at designing, providing and operating such testbeds. The complexity and often high-cost of deploying and operating testbeds make them inaccessible to many experimenters. Moreover, many testbeds are built as part of a specific project and rarely stay operational past the end of that project. This waste of resources has often been quoted as one of the key motivators for dedicated testbed funding and associated systems research. Examples of early successes for this approach are PlanetLab and Orbit. However, both of them are relatively limited in terms of networking experiment capabilities and available technology. The federation of existing and upcoming testbeds under a unified control and management framework would overcome some of these limits, allowing large scale simultaneous resources access and sharing while maintaining administrative control in the hands of the resource owner.
Compared to other scientific research fields, such as physic sciences, the networking field so far has not developed a culture of rigorous peer verification of experimental results. This is mostly due to the fact that even if the same or a similar experimental infrastructure is available (i.e. through a federation of world-wide testbeds), there is currently no unambiguous way to describe and instrument an experiment enabling others to repeat it. Indeed, many high quality publications provide insufficient descriptions of the performed experiments to allow others a fair and valid comparison of different approaches to specific problem in different settings. Thus, to increase the scientific rigour in networking, there is a need for efficient tools and methodologies that provide complete descriptions of experiments, including resources used, and measurements taken.
OMF - A Control and Management Framework for Networking Testbed¶
For the last couple of years, NICTA and Winlab have been co-developing the cOntrol, Management and Measurement Framework (OMF) for networking testbeds.
- From the experimenter's point of view, OMF provides a set of tools to describe and instrument an experiment, execute it and collect its results.
- From the testbed operator's point of view, OMF provides a set of services to efficiently manage and operate the testbed resources (e.g. resetting nodes, retrieving their status information, installing new OS image).
The following figure present a general overview of OMF from the user's point of view. The user/experimenter describes her experiment in a high-level domain-specific language, and passes it on to OMF. The framework will in turn deploy and configure the experiment on the testbed(s) according to the user's description. Then it will initiate and control the execution of this experiment. Finally, during the experiment execution, the framework will measure and collect data according to the user's description. These measurements are sent to a repository available to the user and can also be used to dynamically steer the experiment control.
It is important to node that OMF tools and software are not tied to a specific testbed technology. Indeed, OMF has been deployed and maintained on multiple testbeds with many different types of technologies, at NICTA, Winlab, Intel, SUNY and a US Government Lab. It has been used regularly by many researchers and students for the past two years with some testbeds operating 24/7 catering to experimenters around the world.
The following PDF file contains a short presentation of OMF: OMF-Short-Intro-Web.pdf
Link to old visio diagrams
