From OpenFlow Wiki

Contents 1 Meeting Minutes for SWAI, 24 September 2009 1.1 Present 1.2 Note on HW API 1.3 Testing Framework 1.3.1 Decision on Test Framework Development 1.3.2 Defining the Device/System Under Test 1.3.3 Framework Dependencies 1.3.4 Test Components 1.3.5 Test Framework Requirements 1.3.6 Supported Environments 1.3.7 Targeted Devices to Test 1.3.8 Other Tools for the Framework 1.3.9 Other Notes 1.4 Actions

Meeting Minutes for SWAI, 24 September 2009

Present

Glen, David E, Brandon, Guido, Yiannis, Tatsuya, Bob L, Dan T

Note on HW API

• Dan T: If no complaints, will move ahead with initial implementation

Testing Framework

Decision on Test Framework Development

We reviewed Tatsuya's slides showing three current options. See these PDF docs (below) that he's sent out detailing these and detailing the feedback he's gotten from surveying people.

• Test Framework Options
• Test Survey Feedback

Resolved: The initial test framework will be done in PyUnit.

It should be reasonably easy to extend the work under PyUnit to py.test or nose as they are both built on PyUnit.

Defining the Device/System Under Test

From there we moved to discussing what would be tested. There was some confusion (at least on my part) about what exactly the device under test (DUT) would/could be for this discussion.

Resolved: The DUT is an OpenFlow switch implementation. An independent controller will not be connected to the switch under test.

(Editor's note: It would be reasonable to design things so that a controller could be added and the test framework's controller-like stimulus disabled so that applicable test cases developed for the switch could be applied to a switch/controller combination.)

This being the case, the test/test framework is responsible for generating the control stimulus applied to the DUT.

Framework Dependencies

We discussed the trade off of incorporating existing libraries to leverage others' work, versus the restrictions on the development environment due to the resulting dependencies.

Both directions were argued:

• We should have as few dependencies, and ideally "runs python" as the only one; versus
• We should feel free to leverage other packages when available because it saves work.

The apparent resolution is to allow the inclusion of additional packages, but to identify the environments (see below) that will be supported for the test framework and to ensure that the package dependencies are managed in those environments.

Test Components

We need to identify the components involved in testing. There are two types of interfaces to drive:

• The control interface
• The dataplane switch ports.

Soon there will likely be a third

• The configuration interface

We would like to be able to support a packet generation mechanism that is independent of the rest of the test framework. Using the NetFPGA to do this is already an example of this. The possibility of using commercial packet generators like Ixia or Smartbits shouldn't be excluded. And of course we're getting into wireless as well.

This implies we should define a clean packet generation interface for the test framework to allow different backends to drive that mechanism.

So the test components include:

1. The core test component: What's running the Python test script
2. The control/configuration component: Probably on the same machine as the core test and possibly integrated with core test.
3. The packet generation component: Possibly on a completely different machine than the rest.

Test Framework Requirements

• Python
• Run as root
• Network interface(s)
• libpcap? (See below)

Supported Environments

We need to identify the environments in which the test framework will run. Here's a start at some of the pieces. Please contribute by clarifying and adding.

• Debian Linux platform software switch
• NetFPGA switch
• Independent server running Ubuntu and controlling a hardware switch like the Quanta

Targeted Devices to Test

We need to identify the systems to target. (Would be good to start a "nightly regression" lab.) Please contribute by clarifying and adding.

• Software Switch
  • List
• NetFPGA
• Vendor Platforms
  • Toroki
  • Stanford's LB4G
  • NEC?
  • HP?
• OpenWRT platform(s)

Other Tools for the Framework

• Do we need libpcap? Using raw sockets appears to be another option.
• Currently there's a mechanism to include the C code like openflow.h into the test environment. We'll need something similar. See action below.

Other Notes

• Should the test framework be considered for performance testing as well as conformance testing? Getting latency measurements probably wouldn't be too unreasonable.

Actions

• Tatsuya to start detailing the proposal
• Glen to convey info about using raw sockets
• Open to determine the mechanism for integrating C code header files (like openflow.h) into the framework