IndigoReleaseNotes

From OpenFlow Wiki

Jump to: navigation, search

Contents

Indigo Release Notes

POINTER TO CURRENT NOTES

Please see the OpenFlow Hub Indigo pages for the latest Indigo releases, forums, bug tracking and documentation.

These release notes date from the end of 2010 and apply to old releases that are no longer distributed.

For the latest version of this document, please visit the webpage http://www.openflow.org/wk/index.php/IndigoReleaseNotes. 

   Indigo OpenFlow Switching Software Package
   Stanford Broadcom Binary Code Release
   OpenFlow Version 1.0
   Stanford University
   September 2010

History

indigo-1.0-beta-4

Released: September 5.

Documented and organized configuration/initialization sequence. Enabled DHCP in u-boot. Initial support for Pronto 3290. Fix for port status updates. Fix for 10-gig link issues.

indigo-1.0-beta-3

Released: 8/2/10.

Fix for hard flow removal bug. Dataplane mgmt filtering.

indigo-1.0-beta-2

Released: 7/22/10.

Fixes for VLAN tag processing. Dataplane management.

indigo-1.0-beta

Released: 6/28/10.

Version information is now reported.

indigo-1.0-alpha-5

Released: 6/7/10.

Hardware addresses are now assigned to each OpenFlow Switch port

indigo-1.0-alpha-4

Released: 5/21/10.

Patch version of indigo-1.0-alpha-3. Allow DPID to be set from boot parameters. Fixed crash on port modification. Support no_flood port configuration setting. Watchdog is now "opt-in" rather than "opt-out".

indigo-1.0-alpha-3

Released: 4/19/10.

Patch version of indigo-1.0-alpha-2 addressing flow insertion performance issue. Watchdog feature add.

indigo-1.0-alpha-2

Released: 2/22/10.

Patch version of indigo-1.0-alpha addressing critical memory leak.

indigo-1.0-alpha

Released: 2/17/10.

Initial userspace Stanford Reference Release for LB4G.

Overview

To get the Indigo firmware you need to request access by sending email to dtalayco@stanford.edu. You will get a username and password allowing you to download the firmware images.

This software release includes binary files that can be used to boot the following Broadcom based Ethernet switches. <release> is one of alpha, beta, beta-2, etc.

  • indigo-1.0-<release>.tgz is for BCM 56514 based systems
    • Quanta LB4G
    • Pronto 3240
  • indigo-1.0-<release>-3290.tgz is for BCM 5653x based systems
    • Pronto 3290

Future support for the the Broadcom 956634 reference design is planned with preliminary support for the 956820 reference design to follow. Throughout this document references to the LB4G apply equally to the Pronto 3240.

The distribution supports hardware as an OpenFlow native switch with hardware flow support and works only with OpenFlow version 1.0 as of the latest release.

Materials

The LB4G release is distributed in the file indigo-1.0-<release>.tgz. The Pronto 3290 is distributed in the file indigo-1.0-<release>-3290.tgz.

The releases include the following files:

  • u-boot.bin: The U-Boot boot loader for the LB4G.
  • uImage: A binary image of the Linux kernel
  • uInitrd2m: A binary image of the root file system
  • doc: A directory containing documentation
  • licenses: A directory containing additional license info
  • init-samples: The initialization and configuration files used

Note that the images are not interchangeable between the LB4G and the 3290.

Most Indigo documents referenced on openflow.org are available in the doc directory. However, see the web for the latest updates.

License Information

The Linux kernel code for the Pronto 3290 is based on version 2.6.27 and is released under GPLv2.

The Linux kernel code for the LB4G is based on version 2.6.15 and is released under GPLv2.

The Linux kernel code for the Broadcom reference designs is based on version 2.6.25 and is released under GPLv2.

U-Boot is based on version 1.1.6 (1.3.0 for the Pronto 3290) and is released under the GPLv2. It is available at http://www.denx.de/wiki/U-Boot. Patches from Quanta have been applied and are available from Stanford.

Busybox is version 1.4.2 and is released under the GPLv2. It is available at http://busybox.net/downloads/.

Dropbear is version 0.52; see the accompanying file in licenses/dropbear for additional information.

The OpenFlow code is covered by the OpenFlow license detailed at http://www.openflowswitch.org/wp/legal/. It is available at http://www.openflowswitch.org/.

The linux-bcm-core.ko file is released as a binary-only file under the OpenFlow license (see above).

The of-bcm-cli code is released as a binary-only file under the OpenFlow license (see above). This CLI itself is copyright Broadcom.

Getting Started

To get started with the LB4G, see http://www.openflow.org/wk/index.php/IndigoQuickStartLB4G.

To get started with the Pronto 3290, see http://www.openflow.org/wk/index.php/IndigoQuickStart3290.

Initialization and Configuration

Please see the document http://www.openflow.org/wk/index.php/IndigoConfiguration for details on the initialization and configuration of an Indigo switch.

General Notes and Limitations

This software distribution is intended to make the target hardware into a pure OpenFlow switch. This means that non-OpenFlow configuration is limited.

VLANs are an example of configuration that is often of concern. In order for OpenFlow to control VLANs:

  • One VLAN is reserved as the system's untagged VLAN
  • Incoming untagged packets are mapped to this VLAN; if no other VLAN operations are carried out on the packet, it will exit the switch untagged.
  • All other VLANs are created and all ports are added as "tagged" ports. This is because the VLAN tag could be wildcarded and would then apply to all VLAN ids.

In order to strip a tag, a packet is mapped to the untagged VLAN which will result in the VLAN tag being stripped on egress. Note that an explicit "modify VLAN id to the untagged VLAN" will usually have the same effect. Additionally, see item 21 below regarding Trac issue 104.

Due to the way VLAN actions are currently implemented, the number of flows which act on VLAN tags may be more limited than the number of other types of flows. Currently there is a limit of 256 such flows.

For systems without an external management port, the dataplane management port is reserved for this use and should not have OpenFlow rules associated with it. It is not yet known if this has implications in interoperability with FlowVisor.

The MTU of switch ports defaults to the Ethernet standard of 1518 (1522 for VLAN tagged packets). This is not currently configurable.

Interacting With the OpenFlow/Broadcom CLI

Note that there are two CLIs involved here: The Linux command line with the prompt ~ $ and the OpenFlow/Broadcom CLI with the prompt OF-BCM.0>. Below, CLI refers the the OpenFlow/Broadcom CLI.

If the normal initialization sequence is used (described in http://www.openflow.org/wk/index.php/IndigoConfiguration), the CLI will not be available for interactive use. If the system is configured manually (for example, by setting no_sysconf=1 in extra_boot_args) then the CLI may be accessed to provide limited control and monitoring of the switch. Here is a sample sequence:

  • Bring the system up to ~ $ or # prompt
  • Do 'ps'; if of-bcm-cli is present, kill the process 
  ~ $ ps
  ...
  893 root     27220 S    of-bcm-cli 
  ...
  ~ $ kill 893
  • Connected via the serial port or by ssh (user: root, default password is OpenFlow). You'll need two windows (serial or ssh).
  • Edit config.bcm (see the IndigoConfiguration page mentioned above) and add no_ofd=1
  • In one window, execute /sbin/of-bcm-cli.
    • See below for using the CLI
    • Run the OpenFlow datapath in the background: 
  OF-BCM.0> setenv portlist 1,2,3,4,5,6,7
  OF-BCM.0> bg "ofd --interfaces=$portlist ptcp:"
    • Use ? for help in the of-bcm shell.
    • You can exit to a Linux shell from the BCM prompt with "shell"; exiting from that shell takes you back to the BCM prompt.
  • In another window a the Linux prompt, start the ofprotocol executable to connect the datapath and controller: 
  ~ $ ofprotocol tcp:127.0.0.1 tcp:172.16.12.12:6634
  • You can use '/sbin/dpctl <cmd> tcp:127.0.0.1' at the Linux prompt to examine the status of ofdatapath as normal

Highlights of the CLI:

  • Use ? for help
  • For the 3240, there are two physical switch devices controlled independently. Use 0: and 1: to switch between them. Creating a VLAN on one, for example, will not create the VLAN on the second. The "current" device is indicated in the prompt.
  • Ports are referred to as ge0, ge1, ... on each switch device. The OpenFlow port numbers map ge0 -> 1, ge1 -> 2, etc., for the first device and ge0 -> 25, ge1 -> 26, etc., for the second device.
  • The command 'show' will give a variety of MIB counters for ports.
  • The command 'ofdatapath' takes the same arguments as the user-datapath executable of the same name in the OpenFlow reference software release.
  • The shell command language is similar to csh and provides various control structures and environment features
  • Only a limited number of switch control features are available to control ports and VLANs

Notes on Specific Features

DataPlane Management

For platforms without network connections external to the switch fabric, special support has been added under the name dp-mgmt. This is signaled to the OpenFlow driver by passing the argument --dp-mgmt to ofdatapath. The (OpenFlow) port number may be specified with the additional argument --dp-mgmt-port=<port>.

Although this special port is given an OpenFlow number and this OpenFlow number may be exposed to the controller, no flows should be made that affect the dp-mgmt port. This is currently enforced in the driver where flows that refer to the dp-mgmt port are modified so as to remove it.

Version Update Notes

Changes from indigo-1.0-alpha to indigo-1.0-alpha-2

Addition of dropbear SSH server support.

Set default root password to OpenFlow.

Addressed hardware issue where management ethernet ports would be inoperative after a soft reset.

Limited some debug output to CLI.

Decouple packet receive thread using queue into buffers for main event loop.

Resolved memory leak on broadcast packets transmitted from the switch's CPU.

Changes from indigo-1.0-alpha-2 to indigo-1.0-alpha-3

Added support for watchdog timer. Disable with kernel parameter no_watchdog=1 or set no_watchdog=1 in config.bcm. If enabled and software freezes, the system will reset in about 1 minute.

A separate thread collecting counters has been deployed to address the issue that flow timeout and expirations were taking a long time.

A major bottle neck for flow insertions was addressed by changing how flow priorities are handled in the table. This addresses a major performance issue for exact match flows. High throughput of wildcarded flows will still result in some performance issues.

At the CLI, the command 'ofd' without parameters now prints out various counters and other state information.

Changes from indigo-1.0-alpha-3 to indigo-1.0-alpha-4

The watchdog timer is now "opt-in" rather than "opt-out". That is, specify "watchdog=1" in the arguments to the kernel to enable the software watchdog. (Previously, you specified "no_watchdog=1" to disable it.)

The datapath ID may be specified in the kernel arguments as "dpid=0123456789ab". Note that the DPID MUST BE exactly 12 hex digits.

Previously, the no_flood bit in the port configuration was ignored. This has been updated so that the no_flood bit can be set for the port (which also caused a segfault). However, changing the setting while running will not update existing flows that use the flood action.

Changes from indigo-1.0-alpha-4 to indigo-1.0-alpha-5

Previously the hardware addresses associated with hardware controlled (dataplane) OpenFlow ports were always reported as all 0. This caused obvious problems for controllers using a link layer discovery protocol that used those addresses. The current approach is to use the OpenFlow OUI (00:26:E1), a random number for the next 16 bits and the port number for the bottom 8 bits.

Changes from indigo-1.0-alpha-5 to indigo-1.0-beta

The version, indigo-1.0-beta, is reported in various places including: during the system initialization script rc.sh, via the 'version' command at the Linux prompt (which prints the contents of /etc/indigo-version) and in the of-bcm-cli via the command 'show version'.

Changes from indigo-1.0-beta to indigo-1.0-beta-2

Support for dataplane management added.

This release includes changes to support VLAN tag processing. In previous releases, VLAN modification actions were rejected by the hardware driver and so were not properly applied.

The implemented solution to this configures all VLANs on the switch since it is not known what VLAN tags may appear at ingress and flow modifications with the VLAN ID wildcard may be specified.

See Known Issues 22 and 23 below.

Changes from indigo-1.0-beta-2 to indigo-1.0-beta-3

Fix for hard flow removal bug. If a flow was inserted with a hard time out it would not expire.

Dataplane mgmt filtering. if dataplane management is enabled, the filtering mode is set to "drop" meaning non-management packets on the management port are dropped. Future configuration may allow the selection of "pass" or "packet-in" as options.

Changes from indigo-1.0-beta-3 to indigo-1.0-beta-4

Initial support for Pronto 3290 platform.

Fixes for 10-gig port hardware settings.

Significant changes to initialization sequence. See the new wiki page at: http://www.openflow.org/wk/index.php/IndigoConfiguration

Added variables init_port_state and add_port_state to determine in what state (enable/disable) ports should be in as the system comes up and as ports are added to the OpenFlow datapath.

u-boot now supports DHCP. Added notes about this and NFS in the configuration document mentioned above.

ssh shell path now includes /sbin and /usr/sbin

Added check in system_config to validate dpid format.

Known Issues

1. This release has received only limited validation and many features are untested. Expect bug-fix releases soon.

1A. RESOLVED in release indigo-1.0-alpha-2. (Trac #64) There is a known memory leak in the image. It appears to use memory on a moderately loaded switch (about 160 active flows at a time with a lot of broadcast arp traffic) at the rate of about 2-3 % per minute. On an isolated system with lower flow table turnover (~ten flows/minute) the system has stayed up for 5 days or more.

2, 3: These have been combined into 20, below.

4. (Trac #69) Per-flow counters: For flows, only packet counters or byte counters are supported, not both simultaneously. In the configuration of this release, only packet counters are supported.

5. The switch has only been tested in an untagged environment, although it should pass through tags transparently to the controller and should support matching and rewriting of VLAN tags. You can create a VLAN for the purposes of being untagged by using the CLI discussed above.

6. Not all OpenFlow actions are supported. The following are not currently supported: 

   OFPAT_SET_DL_SRC (Set L2 MAC source address)
   OFPAT_SET_DL_DST (Set L2 MAC destination address)
   OFPAT_SET_NW_SRC (Set L3 IP source address)
   OFPAT_SET_NW_DST (Set L3 IP destination address)
   OFPAT_SET_TP_SRC (Set L4 source port)
   OFPAT_SET_TP_DST (Set L4 destination port)
   OFPAT_STRIP_VLAN (Strip VLAN tag):  This may be supported 
       by setting to the system's untagged VLAN, but that is 
       untested.

7. Actions sequencing is not supported. For example, the action list with 4 actions: "(1) set VID to X, (2) output to port A, (3) set VID to Y, (4) output to port B" has the intention of sending the packet out on different ports with different VLANs. This is not supported. However, sending to multiple ports by specifying each port in a different output action is supported.

8. Actions with a wildcard source port may not specify "flood" nor multicast with the expectation of source port blocking.

9. RESOLVED: There is an issue with the management ports (labelled eth1 and eth2 on the front pane)l. These ports may become inoperable after a reset, particularly from the boot prompt. A power cycle appears to fix the problem. This issue is under investigation by Quanta.

10. There are observed issues where dpctl is used locally when the switch is undergoing heavy use. The switch driver's transmit thread is starved and fails. This eventually results in DMA packet free failures. Use of dpctl is currently discouraged when the switch is connected to a controller and under heavy load.

11. COS control is not currently supported in the CLI. This will be added with more complete queuing support. Queues may be created using dpctl and the enqueue action is supported, though not completely verified.

12. (Trac #68) On certain exit conditions from the CLI, the terminal echo will be disabled, though command processing continues.

13. (Trac #74) The "reason" value for packet_in commands may not accurately reflect the reason the packet was forwarded to the controller.

14. (Trac #71) ADDRESSED in alpha-3 release, though not fully resolved. Flow set up time is still very slow (and has not been fully evaluated). In alpha-3, we have measured a reliable 100 flows/second (including packet forwarding through the controller and flow setups) with hping.

15. (Trac #91) ADDRESSED in alpha-4 release. The NO_FLOOD setting of the port configuration was ignored. So the flood action sent the packet out to all ports independent of this setting.

16. (Trac #94) If the NO_FLOOD setting for a port is changed, the change applies only to new flows. Existing flows with the FLOOD action will not be updated.

17. (Trac #95) Repeated re-connections with the controller have been seen with Nox 0.6. This can result in network instability.

18. (Trac #98) RESOLVED in alpha-5. The dataplane ports MAC addresses were always set to all 0.

19. (Trac #96). The ordering of actions in an action list may not be respected. Normally the switch is supposed to reject a flow mod if it cannot carry out the actions in the order specified in the flow mod's action list. This does not happen.

20. (Trac #102). RESOLVED in beta-1. Switch capabilities were being incorrectly reported. Only the capabilities of the hardware will be reported for actions that can be applied to a flow.

21. (Trac #105). RESOLVED in beta-1. VLAN tag modification actions resulted in error when installed.

22. (Trac #104). Differentiating on VLAN properties only may result in flow collapse. If two flows differ only in VLAN tagging characteristics and the VLAN tags are manipulated (the VLAN tag is stripped or the VID is changed or an untagged packet gets a new VLAN tag) then treatment of the two flows for some operations may be combined. The work around is to ensure that flow modifications have some additional qualification to differentiate flows. No known solution to this problem currently exists.

23. There is currently no support for virtualizing the switch by way of VLAN tags. All VLANs are reserved for use by OpenFlow.

24. (Trac #106). RESOLVED in beta-3. Hard timeouts were ignored.

25. (Trac #116). RESOLVED in beta-4. Path does not include sbin for ssh users.

26. (Trac #114). RESOLVED in beta-4. Support $OFP_OPTIONS env variable for passing arguments to ofprotocol.

27. (Trac #116). RESOLVED in beta-4. Controller is not notified of port link status changes.

Other Notes

The MTU of switch ports defaults to the Ethernet standard of 1518 (1522 for VLAN tagged packets). This is not currently configurable.

Retrieved from "http://www.openflow.org/wk/index.php/IndigoReleaseNotes"