From OpenFlow Wiki

Contents 1 Overview 2 External References 3 The oftest Framework 3.1 Requirements for Running the Framework 4 OFTest Framework Internal Architecture 5 Notes on Test Framework Development 5.1 Challenges 5.1.1 Message Types Do Not Map To Structures 5.1.2 Subclassed Messages Are Not Apparent 5.1.3 Variable Length Arrays 5.2 Current Overview 5.3 Current Pointers 5.4 Software Switch Target 5.5 Decoupled Targets 6 Test Inventory 6.1 Tests by Category

Overview

The original test framework for OpenFlow was done in Perl. The structure of the tests was quite static and focused on testing the NetFPGA 4-port design. It was considered difficult to add new tests and to target other platforms.

In late 2009, an initiative was started in the SWAI group to address these issues. This was driven by Tatsuya Yabe. This page is a repository for the work transitioning from the old test infrastructure to the new Python based test framework.

Tentative documentation is available here: OpenFlow Test Framework Documentation. The readme for the current deployment is on the wiki: OFTestReadme. The remainder of this document provides an overview of the architecture and to document the reasons for the decisions made. A tutorial is under construction at OFTestTutorial.

Here is an attempt at a comprehensive Test List Page to be addressed by this framework.

External References

See SWAI Meeting Notes from 9/24/2009 for links to results from a survey on user experience and proposals for the test framework.

The following are some web links that describe test plans and cases for network switches. Although much of the content is not applicable, they provide ideas for ensuring reasonable coverage as we develop test cases for the OpenFlow reference design.

• Ixia switching test plan
• Please add more links here

The oftest Framework

Requirements for Running the Framework

• Python (includes PyUnit); 2.5 is preferred, 2.4 should work.
• make
• gcc
• scapy: Main scapy page.
  • You should be able to apt-get scapy on Ubuntu, etc.
  • For CentOS (which uses Python 2.4), use version 1.2:
  • wget http://hg.secdev.org/scapy/raw-file/v1.2.0.2/scapy.py
  • Move to /usr/lib/python2.4/site-packages
  • See: Installing 1.2 documentation

The oftest README is available in the oftest download; it is also linked here

• Target OpenFlow installation

Documentation requires DoxyPy and doxygen.

OFTest Framework Internal Architecture

See OFTestReadme for an overview of the directory and file setup.

Tools convert the C header file openflow.h to Python classes with the following conventions:

• Fill in

Notes on Test Framework Development

Challenges

This section describes some of the challenges in generating a scriptable interface to the OpenFlow protocol from the C header file, openflow.h. Hopefully enumerating these challenges here will allow future design changes to address them.

Message Types Do Not Map To Structures

OpenFlow message types are indicated by the type member of the ofp_header structure and are listed in enum ofp_type. However, there is no direct map from these types to structures representing the data in the message. Some types have no body (e.g., echo_request) and yet may have data beyond the header when on the wire. ofp_get_config_reply and ofp_set_config both map to the structure ofp_switch_config.

Subclassed Messages Are Not Apparent

Some messages could be well represented with subclasses. Specifically ofp_error_msg, ofp_stats_request and ofp_stats_reply all should be subclassed.

Variable Length Arrays

The most difficult part of converting openflow.h is due to the fact that some messages have variable length data. To make matters worse, there is inconsistency in the way the length of variable length data is represented.

The worst offender is ofp_packet_out which may have a trailing packet after a variable length array (actions) whose members are themselves of variable length.

Current Overview

This Test Framework is targeted at testing switch implementations. It is not meant to test a system with a switch and a controller, although many of the test cases and some of the underlying framework may be adapted to that environment in the future.

Here are the interfaces to the switch that need to be exercised:

1. The switch's controller interface. The switch will send and receive OpenFlow protocol messages across this interface. Full control of what is sent on this interface needs to be provided to the test framework as well as monitoring messages received from the switch.
2. The dataplane Ethernet port interfaces. The test framework must be able to send and receive data plane packets.
3. Coming Soon: The configuration interface. The test framework will need access to send and receive packets on the configuration interface of the switch when that interface is implemented.
4. Possible Extension: The switch command interface. Commands (like "start ofprotocol") may need to be conveyed to the switch. Currently, this interface is not supported in the framework. It is assumed that the switch is running and actively trying to connect to the test framework controller. Full automation will require this interface be exposed with output and error messages accessible to the framework.

It is important to note that access to each of these interfaces is platform specific. In addition, future extensions of the test environment may add complexity to items; the most important example would be providing packet generator control for the data plane ports.

Current Pointers

The current test framework is only available locally at Stanford. yuba.stanford.edu:/usr/local/git/openflow-projects/oftest

The original test framework which used SWIG is at: yuba.stanford.edu:/usr/local/git/openflow-projects/regtest

Software Switch Target

The standard reference implementation runs OpenFlow on Linux hosts with ports being standard Linux network interfaces. This is the initial target for development of the test framework. In this case, the Python scripts are running directly on the host which controls the OpenFlow executables and the connection to those executables can be done via "localhost".

Decoupled Targets

Decoupled targets are those where the testing scripts are running on a different host than the switch control software. Examples include most hardware switch platforms such as the LB4G from Quanta to which OpenFlow has been ported. There are two challenges for these platforms

1. The mechanism to send commands to the OpenFlow switch needs to be abstracted;
2. The plumbing connecting the target ports needs to be specified.

For the first, Brandon suggested paramiko which provides persistent ssh connections to a target.

For the second, TBD. Currently, the physical interfaces are specified per OpenFlow port.

Test Inventory

Most of this section has been removed with the content shifting to OFTestListPage.

Generating long test descriptions and categorizations is likely not worth the effort at this point. The goal is two fold:

1. Provide some high level categories for tests
2. Provide a template for describing a test case

From this, we should then generate test descriptions.

Tests by Category

Here are the proposed high level categories for tests:

1. Protocol: Test the interaction of the switch with a controller by means of protocol packets.
2. Operational: (aka Functional?) Test the functionality of the switch by pushing configuration into the switch and applying packets to data plane ports to verify correct operation.
3. Performance: Tests related to time or capacities such as validating expiration timers, sizing flow tables or measuring timing of specific operations such as flow setup.
4. System: Tests that validate behavior for higher level applications or protocols; these may involve more components than the switch.

Tatsuya put together a large spreadsheet identifying tests: the spread sheet. (Note: The items in the file are based on OpenFlow0.8.9 and need to be updated.)

It has been converted to a wiki table: View as wiki table