From OpenFlow Wiki

Contents 1 Current Test Cases 2 Prioritization of Test Cases to Write 3 Overview of Test Case Space 4 High Level Categories 5 Test Categories 5.1 Packet Variations 5.2 Matching 5.3 Actions 5.3.1 Drop 5.3.2 Direction 5.3.3 Modification 5.3.4 Control plane boundary 5.4 Protocol 5.4.1 Hello 5.4.2 Echo 5.4.3 Packet out 5.4.4 Flow delete 5.4.5 Async flow removed notification 5.4.6 Switch capabilities 5.4.7 Switch description 5.4.8 Switch config 5.4.9 Provoke error messages 5.5 Statistics 5.5.1 Flow statistics 5.5.2 Table statistics 5.5.3 Port statistics 5.6 Table Maintenance 5.6.1 Flow add 5.6.2 Flow modify 5.6.3 Flow delete 5.6.4 Flow expiration 5.6.5 Flow priorities 5.6.6 Buffer ID for pkt in/pkt out processing 5.7 Port status 5.7.1 Validate state flags 6 Deferred Tests

Current Test Cases

Here is the list of available tests by module in oft. Last updated 7/20/2010.

Test List:
  Module flow_stats:  Flow stats test case.
    FlowStats:        Verify flow stats are properly retrieved.

  Module flow_expire:  Flow expire test case.
    FlowExpire:       Verify flow expire messages are properly generated.

  Module basic:       Basic protocol and dataplane test cases
    DataPlaneOnly:    Root class that sets up only the dataplane
    Echo:             Test echo response with no data
    EchoWithData:     Test echo response with short string data
    FlowMod:          Insert a flow
    FlowStatsGet:     Get stats 
    PacketIn:         Test packet in function
    PacketOut:        Test packet out function
    PortConfigMod:    Modify a bit in port config and verify changed
    SimpleDataPlane:  Root class that sets up the controller and dataplane
    SimpleProtocol:   Root class for setting up the controller
    TableStatsGet:    Get table stats 

  Module caps:        Basic capabilities and capacities tests
  * FillTableExact:   Fill the flow table with exact matches; can take a while
    FillTableWC:      Fill the flow table with wildcard matches

  Module pktact:      Test cases for testing actions taken on packets
    AddVLANTag:       Add a VLAN tag to an untagged packet
    All:              Send to OFPP_ALL port
    AllExceptOneWildcardMatch: 
                      Match exactly one field
    AllExceptOneWildcardMatchTagged: 
                      Match one field with tagged packets
    AllPlusIngress:   Send to OFPP_ALL port and ingress port
    AllWildcardMatch: Create Wildcard-all flow and exercise for all ports
    AllWildcardMatchTagged: 
                      AllWildcardMatch with tagged packets
    BaseMatchCase:    No description
    DirectMC:         Multicast to all ports including ingress
    DirectMCNonIngress: 
                      Multicast to all non-ingress ports
    DirectPacket:     Send packet to single egress port
    DirectPacketICMP: Send ICMP packet to single egress port
    DirectTwoPorts:   Send packet to two egress ports
    ExactMatch:       Exercise exact matching for all port pairs
    ExactMatchTagged: Exact match for all port pairs with tagged pkts
  * ExactMatchTaggedMany: 
                      ExactMatchTagged with many VLANS
    Flood:            Flood to all ports except ingress
    FloodMinusPort:   Config port with No_Flood and test Flood action
    FloodPlusIngress: Flood to all ports plus send to ingress port
  * MixedVLAN:        Test mixture of VLAN tag actions
    ModifyL2Dst:      Modify the dest MAC address (TP1)
    ModifyL2Src:      Modify the source MAC address (TP1)
    ModifyL3Dst:      Modify the dest IP address of an IP packet (TP1)
    ModifyL3Src:      Modify the source IP address of an IP packet (TP1)
    ModifyL4Dst:      Modify the dest TCP port of a TCP packet (TP1)
    ModifyL4Src:      Modify the source TCP port of a TCP packet (TP1)
    ModifyTOS:        Modify the IP type of service of an IP packet (TP1)
    ModifyVID:        Modify the VLAN ID in the VLAN tag of a tagged packet
  * PacketOnly:       Just send a packet thru the switch
  * PacketOnlyTagged: Just send a packet thru the switch
    SingleWildcardMatch: 
                      Exercise wildcard matching for all ports
    SingleWildcardMatchTagged: 
                      SingleWildcardMatch with tagged packets
    StripVLANTag:     Strip the VLAN tag from a tagged packet

Tests preceded by * are not run by default
Tests marked (TP1) after name take --test-params including:
    'vid=N;strip_vlan=bool;add_vlan=bool'

Prioritization of Test Cases to Write

See the current BlackBox test list summary. We should be working towards getting that coverage.

Here is a rough prioritization of test cases to add taken from the more complete list below.

• Matching: Using output action, run positive and negative tests to:
  • Exercise all single wildcard Matching cases. [Glen] (Tatsuya. Done)
  • Exercise all "all wildcard except X" Matching cases [Glen] (Tatsuya. Done)
  • Exercise all wildcard Matching case [Glen] (Tatsuya. Done)
• Actions: Using an exact match, run positive and negative tests to:
  • Exercise output to inport Action (Bob L)
  • Exercise multicast to 2 ports Action, not to in port (Dan. Done)
  • Exercise multicast to all ports except in port (Dan. Done)
  • Exercise multicast to all ports including in port (Dan. Done)
  • Exercise flood to all ports except in port (Dan. Done)
  • Exercise flood to all ports and send to in port (Dan. Done)
  • Exercise 'send to all' port Action, not to in port (Dan. Done)
  • Exercise 'send to all' port Action and send to in port (Dan. Done)
  • Set one port to NO_FLOOD; exercise FLOOD and ALL actions
  • Exercise rewrite-actions: (Done)
    • Exercise rewrite L2 fields if supported (Tatsuya. Done)
    • Exercise rewrite VLAN ID if supported (Tatsuya. Done)
    • Exercise rewrite L3 fields if supported (Tatsuya. Done)
    • Exercise rewrite L4 fields if supported (Tatsuya. Done)
  • Exercise some combinations of output and rewrite... TBD
• Protocol:
  • Test switch features get (Dan. Done)
  • Test port modification (Dan. Done)
  • Send some packets and verify
    • Table stats
    • Port stats
    • Flow stats
  • Test MATCH and NO MATCH packet-in notifications
• Table Maintenance
  • Test flow add by querying flow table (rather than packet matching)
  • Test flow modification with existing flow in table
  • Test hard and soft flow expirations with and without notification

Overview of Test Case Space

This is an attempt to identify possible tests for OpenFlow switches.

In the test lists, the following markers are used.

• CHOOSE ONE: This means that one item in the following list should be chosen for a test case. If only one item is given, consider it a boolean.
• OPTION: This means any combination of items in the following list can be chosen for a test case
• COMBINE WITH: Test cases are created in combination with choices from the indicated category. Not all combinations will be tested, but some variation may be required for coverage.
• DEPENDS ON: Test case may depend on some external requirement like support in the particular platform.

High Level Categories

• Matching
• Actions
• Protocol
• Statistics
• Table Maintenance
• Port Status

Test Categories

Packet Variations

Tests involving the dataplane may have variations in the packets that are used. These are not really test cases directly, but parameters that may be used by test cases.

TBD: This needs to be straightened out a bit

• CHOOSE ONE
  • Unknown ethertype
  • OPTION:
    • VLAN tag absent
    • VLAN, VID == 0
    • VLAN, VID != 0
    • OPTION CFI 0 or 1
    • OPTION PCP 0 - 7
  • IPv4
    • OPTION: TOS value
    • OPTION: Include IP Options
  • IPv6
  • TCP
  • ICMP
  • SNAP zero OUI
  • SNAP non-zero OUI
  • ARP

Matching

Note each test generally includes a positive test (send packet matching flow and expect action) and a negative test (send packet not matching flow and verify action did not occur, at least through flow counter).

• COMBINE WITH: Packet Variations
• OPTION: For each of the following CHOOSE wildcard or specific value
  • Ingress port
  • L2 dst
  • L2 src
  • L2 type
  • CHOOSE ONE
    • VLAN absence
    • OPTION
      • VLAN VID
      • VLAN PCP
  • L3 tos
  • L3 proto
  • CHOOSE ONE: L3 src bitmask 0 - 32
  • CHOOSE ONE: L3 dst bitmask 0 - 32
  • L4 sport
  • L4 dport
  • ICMP fields

• CHOOSE ONE
  • Positive match (packet sent matches flow)
  • Negative match (packet sent does not match flow)

Actions

Drop

• COMBINE WITH: Matching

Direction

• DEPENDS ON: Reported switch capabilities
• COMBINE WITH: Matching
• OPTION
  • Output action
  • Enqueue action
• CHOOSE ONE
  • inport as output port
• CHOOSE ONE
  • unicast
  • multicast to 2 ports
  • multicast to more than 2 ports (requires port count greater than 4)
  • flood
  • all

Modification

The following modification actions may be carried out in any combination. Of course we'll start with the single field change operations; do a few multiple field operations and then do as many as possible.

• DEPENDS ON: Reported switch capabilities
• COMBINE WITH: Matching
• COMBINE WITH: Actions
• OPTION
  • Change L2 dst
  • Change L2 src
  • Change L2 type
  • Change VLAN VID
  • Change VLAN PCP
  • Strip VLAN
  • Change L3 src
  • Change L3 dst
  • Change L3 tos
  • Change L4 sport
  • Change L4 pport

Control plane boundary

• Packet in
  • CHOOSE ONE
    • With OFPPC_NO_PACKET_IN for port
  • CHOOSE ONE
    • No match
    • COMBINE WITH: Matching
  • OPTION Send length (at least miss send length bytes sent)

Protocol

Hello

Echo

• CHOOSE ONE
  • With data

Packet out

• OPTION
  • COMBINE WITH: Actions
  • Apply table

Flow delete

• OPTION
  • Request response

Async flow removed notification

Switch capabilities

Switch description

Switch config

Provoke error messages

• TBD: Itemize

Statistics

• OPTION
  • Packet counters
  • Byte counters

Flow statistics

• CHOOSE ONE
  • Single
  • All
  • Aggregate CHOOSE ONE
    • Cookie
    • COMBINE WITH Matching

Table statistics

Port statistics

Table Maintenance

Flow add

OPTION: Check overlap flag

Flow modify

OPTION: cookie?

Flow delete

• OPTION: Strict
• OPTION: out_port specified
• OPTION: cookie

Flow expiration

• OPTION
  • Idle timeout
  • Hard timeout
• Timer precision

Flow priorities

Buffer ID for pkt in/pkt out processing

Port status

Validate state flags

• Link down flag

Deferred Tests

• Emergency flow table related tests/flags
• Tests that require some external configuration management

during the test (such as a port status that requires changing the physical port state).

• Failover
• Capacity related tests
  • Insert flows until failure
    • Exact match
    • Wildcard
• Performance related
  • Flow setup time
  • Flow insert rate
  • Queuing characteristics
  • Bandwidth