Much of the GENI effort in developing network testbeds has been focused on building the control frameworks needed to allocate and initialize the network resources that make up an experiment. We argue that building the instrumentation and measurement system to monitor and capture the behavior of the network is just as important and challenging as setting up the network itself, especially in a virtualized and federated environment where getting information from experimental nodes is too complicated and too much to handle for a typical user. In this paper, we describe the design of an instrumentation and measurement infrastructure that allows users to monitor their experiments. The challenge that virtualization and federation of GENI testbeds bring to instrumentation and monitoring is how to hide the details of instrumentation setup from users so that users do not need to be experts in system administration or network management of virtualized and federated systems, but are still able to "see" what is going on with their experiments. Our instrumentation tool sets up experiment-specific monitoring infrastructure that is tailored to capture, record, and display only information associated with that experiment. Our tools are currently available in GENI, and we present a simple example of how to use them to instrument an experiment.
