HP SN3000B Troubleshooting Manual
  1. Manuals
  2. Brands
  3. HP Manuals
  4. Software
  5. SN3000B
  6. Troubleshooting manual

HP SN3000B Troubleshooting Manual

Troubleshooting and diagnostics guide
Hide thumbs Also See for SN3000B:
Table of Contents

Advertisement

Quick Links

See also: Administrator's Manual , Reference Manual
53-1002150-02
®
03 June 2011
Fabric OS
Troubleshooting and Diagnostics Guide
Supporting Fabric OS v7.0.0

Advertisement

Table of Contents
loading

Related Manuals for HP SN3000B

Summary of Contents for HP SN3000B

  • Page 1 53-1002150-02 ® 03 June 2011 Fabric OS Troubleshooting and Diagnostics Guide Supporting Fabric OS v7.0.0...
  • Page 2 Copyright © 2008-2011 Brocade Communications Systems, Inc. All Rights Reserved. Brocade, the B-wing symbol, BigIron, DCX, Fabric OS, FastIron, IronPoint, IronShield, IronView, IronWare, JetCore, NetIron, SecureIron, ServerIron, StorageX, and TurboIron are registered trademarks, and Brocade Network Advisor (formerly Data Center Fabric Manager or DCFM), Extraordinary Networks, and SAN Health are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries.
  • Page 3 Document History Title Publication number Summary of changes Date Fabric OS Troubleshooting and 53-0000853-01 First released edition. March 2008 Diagnostics Guide Fabric OS Troubleshooting and 53-1001187-01 Added support for Virtual Fabrics, fcPing, November 2008 Diagnostics Guide pathInfo, and additional troubleshooting tips.
  • Page 4 Fabric OS Troubleshooting and Diagnostics Guide 53-1002150-02...

  • Page 5: Table Of Contents

    Contents About This Document In this chapter ......... . . xv How this document is organized .
  • Page 6 Time ........... 11 Frame Viewer.
  • Page 7 Chapter 4 Configuration In this chapter ......... . . 45 Configuration upload and download issues .
  • Page 8 Chapter 8 ISL Trunking In this chapter ......... . . 73 Link issues.
  • Page 9 Equipment status ........98 Checking the temperature, fan, and power supply .
  • Page 10 Fabric OS Troubleshooting and Diagnostics Guide 53-1002150-02...
  • Page 11 Figures Figure 1 Superping and fcPing paths ......... 18 Figure 2 Simple port initialization process .
  • Page 12 Converged Enhanced Ethernet Administrator’s Guide 53-1002163-01...
  • Page 13 Tables Table 1 Common troubleshooting problems and tools ......2 Table 2 Common symptoms ..........3 Table 3 Environmental changes .
  • Page 14 Converged Enhanced Ethernet Administrator’s Guide 53-1002163-01...

  • Page 15: About This Document

    About This Document In this chapter • How this document is organized ........xv •...

  • Page 16: Supported Hardware And Software

    Supported hardware and software In those instances in which procedures or parts of procedures documented here apply to some switches but not to others, this guide identifies which switches are supported and which are not. Although many different software and hardware configurations are tested and supported by Brocade Communications Systems, Inc.

  • Page 17: Document Conventions

    Document conventions This section describes text formatting conventions and important notice formats used in this document. Text formatting The narrative-text formatting conventions that are used are as follows: bold text Identifies command names Identifies the names of user-manipulated GUI elements Identifies keywords and operands Identifies text to enter at the GUI or CLI italic text...

  • Page 18: Notes, Cautions, And Warnings

    Notes, cautions, and warnings The following notices and statements are used in this manual. They are listed below in order of increasing severity of potential hazards. NOTE A note provides a tip, guidance, or advice, emphasizes important information, or provides a reference to related information.

  • Page 19: Other Industry Resources

    White papers, online demonstrations, and data sheets are available through the Brocade website http://www.brocade.com/products-solutions/products/index.page For additional Brocade documentation, visit the Brocade website: http://www.brocade.com Release notes are available on the MyBrocade website and are also bundled with the Fabric OS firmware. Other industry resources For additional resource information, visit the Technical Committee T11 website.

  • Page 20: Document Feedback

    • Brocade 300, 5100, and 5300 — On the switch ID pull-out tab located on the bottom of the port side of the switch. • Brocade 6510 — On the switch ID pull-out tab located inside the chassis on the port side on the left.

  • Page 21: Introduction

    Chapter Introduction In this chapter • Troubleshooting overview ......... 1 •...

  • Page 22: Most Common Problem Areas

    Most common problem areas Most common problem areas Table 1 identifies the most common problem areas that arise within SANs and identifies tools to use to resolve them. TABLE 1 Common troubleshooting problems and tools Problem area Investigate Tools • •...

  • Page 23: Table 2 Common Symptoms

    Questions for common symptoms TABLE 2 Common symptoms Symptom Areas to check Chapter or Document Blade is faulty Firmware or application download Chapter 2, “General” Hardware connections Chapter 5, “Firmware Download Errors” Chapter 7, “Virtual Fabrics” Blade is stuck in the “LOADING” state Firmware or application download Chapter 5, “Firmware Download Errors”...
  • Page 24 Questions for common symptoms TABLE 2 Common symptoms (Continued) Symptom Areas to check Chapter or Document License issues Licensing Chapter 2, “General” LSAN is slow or times-out LSAN tagging Chapter 2, “General” Marginal link Links Chapter 3, “Connectivity” No connectivity between host and storage Cables Chapter 3, “Connectivity”...

  • Page 25: Gathering Information For Your Switch Support Provider

    Gathering information for your switch support provider TABLE 2 Common symptoms (Continued) Symptom Areas to check Chapter or Document User is unable to change switch settings RBAC settings Chapter 6, “Security” Account settings Virtual Fabric does not form FIDs Chapter 7, “Virtual Fabrics” Zone configuration mismatch Effective configuration Chapter 9, “Zoning”...

  • Page 26: Capturing Output From A Console

    Gathering information for your switch support provider 2. Type the appropriate supportSave command based on your needs: • If you are saving to an FTP or SCP server, use the following syntax: supportSave When invoked without operands, this command goes into interactive mode. The following operands are optional: -n Does not prompt for confirmation.

  • Page 27: Capturing Command Output

    Building a case for your switch support provider 3. Set the utility to capture output from the screen. Some utilities require this step to be performed prior to opening up a session. Check with your utility vendor for instructions. 4. Type the command or start the process to capture the required data on the console. Capturing command output 1.

  • Page 28: Detailed Problem Information

    Building a case for your switch support provider • Do you have IPsec installed on the switch’s Ethernet interface? Yes or no. • Use the ipsecConfig show command to determine the answer. • Do you have Inband Management installed on the switches GigE ports? Yes or no. •...

  • Page 29: Gathering Additional Information

    Building a case for your switch support provider • Host information: OS version and patch level HBA type HBA firmware version HBA driver version Configuration settings • Storage information: Disk/tape type Disk/tape firmware level Controller type Controller firmware level Configuration settings Storage software (such as EMC Control Center, Veritas SPC, etc.) •...
  • Page 30 Building a case for your switch support provider Fabric OS Troubleshooting and Diagnostics Guide 53-1002150-02...

  • Page 31: In This Chapter

    Chapter General In this chapter • Licenses ............11 •...

  • Page 32: Frame Viewer

    Frame Viewer Frame Viewer When a frame is unable to reach its destination due to timeout, it is discarded. You can use Frame Viewer to find out which flows contained the dropped frames, which in turn can help you determine which applications might be impacted.

  • Page 33: Switch Boot

    Switch boot Symptom MQ errors are appearing in the switch log. Probable cause and recommended action An MQ error is a message queue error. Identify an MQ error message by looking for the two letters MQ followed by a number in the error message: 2004/08/24-10:04:42, [MQ-1004], 218,, ERROR, ras007, mqRead, queue = raslog-test- string0123456-raslog, queue I D = 1, type = 2...

  • Page 34: Rolling Reboot Detection

    Switch boot Rolling Reboot Detection A rolling reboot occurs when a switch or enterprise-class platform has continuously experienced unexpected reboots. This behavior is continuous until the rolling reboot is detected by the system. Once the Rolling Reboot Detection (RRD) occurs, the switch is put into a stable state so that a minimal supportSave can be collected and sent to your service support provider for analysis.
  • Page 35 Switch boot This is an HA bootup-related issue and happens when switch is unable to recover to a stable state. HASM log contains more detail and specific information on this type of failure, such as one of the following: Failover recovery failed: This occurs when failover recovery failed and has to reboot the CP. Failover when standby CP unready: Occurs when the active CP has to failover, but the standby CP is not ready to take over mastership.

  • Page 36: Fc-Fc Routing Connectivity

    FC-FC routing connectivity * Fabric OS has detected frequent switch reboot condition. * * Following actions can be taken to recover the switch: * * - take off or replace the bad blades. * * - use supportsave to collect supportsave data. * ************************************************************** Please change passwords for switch default accounts now.
  • Page 37 FC-FC routing connectivity received reply from 0x60f00: 12 bytes time:430 usec received reply from 0x60f00: 12 bytes time:462 usec 5 frames sent, 5 frames received, 0 frames rejected, 0 frames timeout Round-trip min/avg/max = 430/467/506 usec Pinging 0x5f001 with 12 bytes of data: received reply from 0x5f001: 12 bytes time:2803 usec received reply from 0x5f001: 12 bytes time:2701 usec received reply from 0x5f001: 12 bytes time:3193 usec...

  • Page 38: Superping

    FC-FC routing connectivity Superping Superping refers to the fcPing --allpaths command which is a diagnostic tool used to test all least cost ISLs between a source and destination switch. When you run the command you are provided with a list of all available least cost paths from a source domain to a destination device. Superping isolates links with potential failures so that you can investigate these ISLs to determine the exact links.
  • Page 39 FC-FC routing connectivity In the following example, superping is invoked using the fcPing --allpaths command to destination domain 165. The following example displays each hop as (Domain1/Index1-> Domain2/Index2) format. To reach destination domain 165 from source domain 3 there are 2 unique end-to-end paths.

  • Page 40: Route And Statistical Information

    FC-FC routing connectivity Restrictions • Fabric reconfiguration cannot occur while using the superping tool. It is assumed that the fabric is stable before the fcPing --allpaths command is executed. • The control path for interswitch communication should be available, even if the data path for device to device communication may have resource starvation.
  • Page 41 FC-FC routing connectivity Target port is Embedded Hop In Port Domain ID (Name) Out Port BW Cost --------------------------------------------------------- (web226) 1G 1000 (web229) 1G 1000 (web228) 1G 1000 (web225) To display basic and extended statistics in interactive mode: switch:admin> pathinfo Max hops: (1..127) [25] Fabric Id: (1..128) [-1] Domain|Wwn|Name: [] 8 Source port: (0..15) [-1]...

  • Page 42: Performance Issues

    FC-FC routing connectivity For details about the pathInfo command, refer to the Fabric OS Command Reference. Performance issues Symptom General slow-down in FCR performance and scalability. Probable cause and recommended action As LSAN zone databases get bigger, it takes more switch resources to process them. Use the enforce tag feature to prevent a backbone switch from accepting unwanted LSAN zone databases into its local database.

  • Page 43: Port Initialization And Fcp Auto-Discovery Process

    Chapter Connectivity In this chapter • Port initialization and FCP auto-discovery process ....23 • Link issues ........... . 25 •...
  • Page 44 Port initialization and FCP auto-discovery process • VEX_Port—A virtual EX_Port. It connects a Fibre Channel router to an edge fabric. From the point of view of a switch in an edge fabric, a VEX_Port appears as a normal VE_Port. It follows the same Fibre Channel protocol as other VE_Ports.

  • Page 45: Link Issues

    Link issues Link issues Symptom Port LEDs are flashing. Probable cause and recommended action Depending on the rate of the flash and the color of the port LED this could mean several things. To determine what is happening on either your port status LED or power status LED, refer to that switch’s model hardware reference manual.

  • Page 46: Checking The Logical Connection

    Connection problems Checking the logical connection 1. Enter the switchShow command. 2. Review the output from the command and determine if the device successfully logged in to the switch. • A device that is logically connected to the switch is registered as an F_, L_, E_, EX_, VE_, VEX_, or N_Port.
  • Page 47 Connection problems 021a00; 2,3;20:00:00:e0:69:f0:07:c6;10:00:00:e0:69:f0:07:c6; 895 Fabric Port Name: 20:0a:00:60:69:10:8d:fd 051edc; 3;21:00:00:20:37:d9:77:96;20:00:00:20:37:d9:77:96; na FC4s: FCP [SEAGATE ST318304FC 0005] Fabric Port Name: 20:0e:00:60:69:10:9b:5b 051ee0; 3;21:00:00:20:37:d9:73:0f;20:00:00:20:37:d9:73:0f; na FC4s: FCP [SEAGATE ST318304FC 0005] Fabric Port Name: 20:0e:00:60:69:10:9b:5b 051ee1; 3;21:00:00:20:37:d9:76:b3;20:00:00:20:37:d9:76:b3; na FC4s: FCP [SEAGATE ST318304FC 0005] Fabric Port Name: 20:0e:00:60:69:10:9b:5b 051ee2;...

  • Page 48: Link Failures

    Link failures Link failures A link failure occurs when a server, storage, or switch device is connected to a switch, but the link between the devices does not come up. This prevents the devices from communicating to or through the switch. If the switchShow command or LEDs indicate that the link has not come up properly, use one or more of the following procedures.

  • Page 49: Checking For A Loop Initialization Failure

    Link failures Checking for a loop initialization failure 1. Verify the port is an L_Port. a. Enter the switchShow command. b. Check the last field of the output to verify that the switch port indicates an L_Port. If a loop device is connected to the switch, the switch port must be initialized as an L_Port.

  • Page 50: Table 4 Switchshow Output And Suggested Action

    Marginal links After becoming an active port, the port becomes an F_Port or an E_Port depending on the device on the opposite side. If the opposite device is a host or target device, the port becomes an F_Port. If the opposite device is another switch, the port becomes an E_Port. If there is a problem with the host or target device, enter portCfgGPort to force the port to try to come up as point-to-point only.

  • Page 51: Table 5 Loopback Modes

    Marginal links Only one frame is transmitted and received at any given time. An external cable is not required to run this test. The port LEDs flicker green rapidly while the test is running. Table 5 shows the different loopback modes you can use when using portLoopbackTest to test a marginal link.
  • Page 52 Marginal links switch:admin> porterrshow frames enc disc link loss loss frjt fbsy err g_eof shrt long eof out c3 fail sync sig ============================================================================ 665k 7.0k 665k 7.4k (output truncated) 3. If you suspect a marginal link, isolate the areas by moving the suspected marginal port cable to a different port on the switch.

  • Page 53: Device Login Issues

    Device login issues Device login issues A correct login is when the port type matches the device type that is plugged in. In the following example, it shows that the device connected to Port 1 is a fabric point-to-point device and it is correctly logged in an F_Port.
  • Page 54 Device login issues In some cases, you may find that the port has been locked as an L_Port and the device attached is a fabric point-to-point device such as a host or switch. This would be an incorrect configuration for the device and therefore the device cannot log into the switch. To correct this type of problem, remove the Lock L_Port configuration using the portCfgDefault command.

  • Page 55: Table 6 Component Test Descriptions

    Media-related issues ------------------------------------------------- Thu Nov 6 16:52:39 2008 16:52:39.066 PORT 00010004,4302000f,02000000 16:52:39.066 PORT ce3dfab0,d9672800,00000002 16:52:39.066 PORT ce3dfab0,d9672800,00000080 16:52:39.066 PORT 00000000,00000000,00000002 16:52:39.066 PORT 00010004,4302000f,00000002 16:52:39.066 PORT 00010004,4302000f,02000000 16:52:39.071 PORT ioctl 88010004 1,0 * 4 16:52:42.311 SPEE 00000000,00000000,00000000 16:52:42.558 SPEE 00000000,00000000,00000000 16:52:42.558 SPEE 00000000,00000000,00000000 16:52:42.558 SPEE 00000001,00000000,00000000...

  • Page 56: Table 7 Switch Component Tests

    Segmented fabrics Testing a port’s external transmit and receive path 1. Connect to the switch and log in as admin. 2. Connect the port you want to test to any other switch port with the cable you want to test. 3.

  • Page 57: Reconciling Fabric Parameters Individually

    Segmented fabrics • Incompatible zoning configuration (see Chapter 9, “Zoning”). • Domain ID conflict (see “Reconciling fabric parameters individually” on page 37). • Fabric ID conflict (see Chapter 7, “Virtual Fabrics”). • Incompatible security policies. • Incorrect fabric mode. • Incorrect policy distribution.

  • Page 58: Downloading A Correct Configuration

    Segmented fabrics Downloading a correct configuration You can restore a segmented fabric by downloading a previously saved correct backup configuration to the switch. Downloading in this manner reconciles any discrepancy in the fabric parameters and allows the segmented switch to rejoin the main fabric. For details on uploading and downloading configurations, refer to the Fabric OS Administrator’s Guide.

  • Page 59: Reconciling Incompatible Software Features

    Port mirroring Allow XISL Use (yes, y, no, n): [yes] R_A_TOV: (4000..120000) [10000] E_D_TOV: (1000..5000) [2000] WAN_TOV: (0..30000) [0] MAX_HOPS: (7..19) [7] Data field size: (256..2112) [2112] Sequence Level Switching: (0..1) [0] Disable Device Probing: (0..1) [0] Suppress Class F Traffic: (0..1) [0] Per-frame Route Priority: (0..1) [0] Long Distance Fabric: (0..1) [0] BB credit: (1..27) [16]...

  • Page 60: In-Order Delivery

    Port mirroring Port mirroring captures traffic between two devices. It mirrors only the frames containing the SID/DID to the mirror port. Because of the way it handles mirroring, a single mirror port can mirror multiple mirror connections. This also means that the port cannot exceed the maximum bandwidth of the mirror port.

  • Page 61: Table 8 Port Mirroring Platform Supportability

    Port mirroring • If you have NPIV or 10-bit address mode enabled on a Brocade 300, 5300, 5410, 5450, 5460, 5470, 5480, 7800, and the M5424 platforms, all devices from the same NPIV port or 10-bit addressing mode hit the same mirror connection, regardless of different AL_PAs, since the validation is done only for the first 16-bits of the SID/DID.

  • Page 62: Table 9 Maximum Number Of Mirror Connections

    Port mirroring Maximum mirror connections Table 9 shows the maximum number of mirror connections you can add to a mirror port. TABLE 9 Maximum number of mirror connections Model Maximum Number of Mirror Connections (chassis-wide) Brocade 300 Brocade 5100 Brocade 5300 Brocade 5410 Brocade 5450 Brocade 5460...

  • Page 63: Deleting A Port Mirror Connection

    Port mirroring The configuration database keeps information about the number of port mirror connections configured on a switch, the number of chunks of port mirroring data that are stored, and the chunk number. When removing a mirror connection, always use this method to ensure that the data is cleared.
  • Page 64 Port mirroring Fabric OS Troubleshooting and Diagnostics Guide 53-1002150-02...

  • Page 65: Configuration Upload And Download Issues

    Chapter Configuration In this chapter • Configuration upload and download issues ......45 • Brocade configuration form ........48 Configuration upload and download issues It is important to maintain consistent configuration settings on all switches in the same fabric because inconsistent parameters (such as inconsistent PID formats) can cause fabric...
  • Page 66 Configuration upload and download issues • The FTP or SCP server’s IP address cannot be contacted. Verify that you can connect to the FTP server. Use your local PC to connect to the FTP server or ping the FTP server. Example of a successful ping C:\>...
  • Page 67 Configuration upload and download issues • On a Virtual Fabrics-enabled switch, you do not have the chassis role permission set on your user account. Implement one change at a time, then issue the command again. By implementing one change at a time, you are able to determine what works and what does not work.

  • Page 68: Table 10 Brocade Configuration And Connection

    Brocade configuration form Probable cause and recommended action Verify that the switch was rebooted by checking the system log. If you are doing this on a enterprise-class platform, verify that both CPs rebooted by checking the system log. If any error occurs during the download, such as an error about a particular key, it is important to issue the configDefault command and attempt to repeat the configDownload command.
  • Page 69 Brocade configuration form TABLE 10 Brocade configuration and connection (Continued) Brocade configuration settings Value Switch name Ethernet IP address Ethernet subnet mask Total number of local devices (nsShow) Total number of devices in fabric (nsAllShow) Total number of switches in the fabric (fabricShow) Fabric OS Troubleshooting and Diagnostics Guide 53-1002150-02...
  • Page 70 Brocade configuration form Fabric OS Troubleshooting and Diagnostics Guide 53-1002150-02...

  • Page 71: Blade Troubleshooting Tips

    Chapter Firmware Download Errors In this chapter • Blade troubleshooting tips ........51 •...

  • Page 72: Firmware Download Issues

    Firmware download issues Symptom The blade is faulty (issue slotShow to confirm). Probable cause and recommended action If the port or application blade is faulty, enter the slotPowerOff and slotPowerOn commands for the port or application blade. If the port or application blade still appears to be faulty, remove it and re-insert it into the chassis.
  • Page 73 Firmware download issues Symptom Firmware download times out. Probable cause and recommended action This can be caused by an excessively slow network. If it takes more than 30 minutes to download firmware on a switch, or on each CP in a director, the firmware download process times out. If a timeout occurs on a switch, the firmware download process synchronizes the two partitions on the switch by starting a firmware commit operation.
  • Page 74 Firmware download issues Password: <hidden> Server IP: 192.126.168.115, Protocol IPv4 Checking system settings for firmwaredownload... Firmware access timeout. The server is inaccessible or firmware path is invalid. Please make sure the server name or IP address, the user/password and the firmware path are valid. Symptom Cannot download the requested firmware.

  • Page 75: Troubleshooting The Firmwaredownload Command

    Troubleshooting the firmwareDownload command Troubleshooting the firmwareDownload command A network diagnostic script and preinstallation check is a part of the firmwareDownload procedure. The script and preinstallation check performs troubleshooting and automatically checks for any blocking conditions. If the firmware download fails, refer to the Fabric OS Message Reference for details about error messages.

  • Page 76: Table 11 Usb Error Handling

    Considerations for downgrading firmware TABLE 11 USB error handling Scenario under which download fails Error handling Action An access error occurs during Firmwaredownload times out None. firmwaredownload because the and commit isisstarted to repair removal of the USB device, or USB the partitions of the CPUs that device hardware failure, etc.
  • Page 77 Considerations for downgrading firmware File Name: /home/userfoo/v6.3.0 Network Protocol (1-auto-select, 2-FTP, 3-SCP) [1]: Password: <hidden> Checking System Settings... Version compatibility check passed. Checking version compatibility... Version compatibility check passed.. The following items must be addressed before downloading the specified firmware: Downgrade is not allowed because there are more than 2 FX8-24 blades plugged in to the chassis.

  • Page 78: Blade Types

    Considerations for downgrading firmware IPSec on FCIP tunnels requires version v6.3.1 or higher for 7800, and v7.0.0 or higher for FX8-24. Please use "portcfg fciptunnel" to disable IPSec for all FCIP tunnels and try again. Downgrade is not allowed because Persistent PID is enabled. Please use "ag --persistentpidenable 0"...

  • Page 79: Firmware Versions

    Considerations for downgrading firmware Message The FS8-18 (type 43) blade is not supported by the target firmware. Please use slotshow to find out which slot it is in and remove it first. Probable cause and recommended action The firmware download operation was attempting to downgrade a system to Fabric OS v6.1.1_enc or earlier with one or more Brocade FC8-18 AP blades (blade ID 43) in the system.

  • Page 80: Routing

    Considerations for downgrading firmware Message Downgrade is not allowed because VF is enabled. Please run "lscfg --config" and "lscfg --delete" commands to remove the non-default LS first, then run "fosconfig --disable vf" to disable VF before proceeding. Probable cause and recommended action You cannot downgrade because Virtual Fabrics are enabled.

  • Page 81: Passwords

    Chapter Security In this chapter • Passwords ........... . 61 •...

  • Page 82: Table 12 Password Recovery Options

    Device authentication Symptom Unable to log into the boot PROM. Probable cause and recommended action To recover a lost boot PROM password, contact your switch service provider. You must have previously set a recovery string to recover the boot PROM password. This does not work on lost or forgotten passwords in the account database.

  • Page 83: Table 13 Ssl Messages And Actions

    Protocol and certificate management Symptom Switch is unable to form an F_Port. Probable cause and recommended action Regardless of the device authentication policy mode on the switch, the F_Port is disabled if the DH-CHAP protocol fails to authenticate. If the HBA sets the FC-SP bit during FLOGI and the switch sends a FLOGI accept with FC-SP bit set, then the switch expects the HBA to start the AUTH_NEGOTIATE.

  • Page 84: Snmp

    SNMP • If you think it may be related to E_Port authentication then collect a supportSave -n from both switches of the affected E_Port. • If you think this is a policy-related issue, FCS switch or other security server-related issue then use supportSave -n to collect data from the Primary FCS switch and all affected switches.
  • Page 85 FIPS The results of all self-tests, for both power-up and conditional, are recorded in the system log or are output to the local console. This includes logging both passing and failing results. If the tests fail on your switch it constantly reboots. Because boot PROM access is disabled you are not able to exit out of the reboot.
  • Page 86 FIPS Fabric OS Troubleshooting and Diagnostics Guide 53-1002150-02...

  • Page 87: General Virtual Fabrics Troubleshooting

    Chapter Virtual Fabrics In this chapter • General Virtual Fabrics troubleshooting ......67 •...

  • Page 88: Fabric Identification Issues

    Fabric identification issues • VE_Ports cannot exist in a logical switch that has XISL use turned on. Although VE_Ports are allowed in a base switch, Fabric OS v6.2.0 and later do not support the use of VE_Ports to carry traffic for logical fabrics using XISLs. They can be used to carry FCR traffic through EX_ and VEX_Ports.

  • Page 89: Base Switch Issues

    Base switch issues Symptom The switch with domain <domain> with firmware version <fw version> has joined the FID <fid> fabric and may not be compatible with XISL use. Probable cause and recommended action This message indicates the specified switch in the logical fabric using XISLs is running an incompatible firmware version and must be upgraded to Fabric OS v6.2.0.

  • Page 90: Logical Switch Issues

    Logical switch issues Logical switch issues CAUTION When a logical switch is created, all configuration for the logical switch is set to factory defaults. When a logical switch is deleted, all configuration for the logical switch is deleted permanently and is not recoverable. Symptom The indicated slot is empty.

  • Page 91: Switch Configuration Blade Compatibility

    Switch configuration blade compatibility Symptom Unable to remove ports from their current switch. Probable cause and recommended action When moving ports to a switch, they are first removed from the switch in which they reside. This error message is displayed if this step fails. Symptom A non-GE blade is within the slot range.

  • Page 92: Gathering Additional Information

    Gathering additional information Gathering additional information For Virtual Fabrics-related issues, use the following guidelines to gather additional data for your switch support provider: • Perform the supportSave command. • If not sure about the problem area, perform the supportSave command on all chassis and logical switches in the fabric.

  • Page 93: In This Chapter

    Chapter ISL Trunking In this chapter • Link issues ........... . 73 •...

  • Page 94: Buffer Credit Issues

    Buffer credit issues • Trunking is not supported in switch interoperability mode. • Port trunking is disabled. • The port is not an E_Port. • The port is not 2 Gbps, 4 Gbps, or 8 Gbps. • The port connects to a switch other than the one you want it to. To correct this issue, connect additional ISLs to the switch with which you want to communicate.

  • Page 95: Overview Of Corrective Action

    Chapter Zoning In this chapter • Overview of corrective action ........75 •...

  • Page 96: Segmented Fabrics

    Segmented fabrics If you enter the cfgShow command to display information about all zones, the TI zones appear in the defined zone configuration only and do not appear in the effective zone configuration. 1. Connect to the switch and log in as admin. 2.

  • Page 97: Table 14 Commands For Debugging Zoning

    Zone conflicts Symptom Zone conflict appears in logs and fabric is segmented. Probable cause and recommended action This issue is usually caused by incompatible zoning configurations. Verify the following are true: • The effective cfg (zone set) on each end of the segmented ISL is identical. •...

  • Page 98: Correcting A Fabric Merge Problem Quickly

    Zone conflicts TABLE 14 Commands for debugging zoning (Continued) Command Function cfgEnable Use to enable and activate (make effective) the specified configuration. cfgSave Use to save changes to the zone configuration database. cfgTransAbort Use to abort the current zoning transaction without committing it. cfgTransShow Use to display the ID of the current zoning transaction defZone...

  • Page 99: Changing The Default Zone Access

    Zone conflicts Changing the default zone access A switch is not allowed to merge with another switch that has an active effective configuration if the default zone is set to “no access”. Before the switch can join, the default zone setting has to be set to "all access".

  • Page 100: Checking For Fibre Channel Connectivity Problems

    Zone conflicts Checking for Fibre Channel connectivity problems Enter the fcPing command (refer to the Fabric OS Command Reference for more information on this command), which checks the zoning configuration for the two ports specified by: • Generates an Extended Link Service (ELS) frame ECHO request to the source port specified and validates the response.

  • Page 101: Checking For Zoning Problems

    Zone conflicts received reply from 0x020800: 12 bytes time:1006 usec received reply from 0x020800: 12 bytes time:1008 usec received reply from 0x020800: 12 bytes time:1038 usec received reply from 0x020800: 12 bytes time:1010 usec 5 frames sent, 5 frames received, 0 frames rejected, 0 frames timeout Round-trip min/avg/max = 1006/1044/1159 usec Pinging 22:00:00:04:cf:75:63:85 [0x217d9] with 12 bytes of data: Request timed out...

  • Page 102: Gathering Additional Information

    Gathering additional information Gathering additional information Collect the data from a supportSave -n command. Then collect the data from the cfgTransShow command. For the port having the problem, collect the data from the filterPortShow <port> command. Fabric OS Troubleshooting and Diagnostics Guide 53-1002150-02...

  • Page 103: About Fabric Os Diagnostics

    Chapter Diagnostic Features In this chapter • About Fabric OS diagnostics ........83 •...

  • Page 104: Power-On Self Test

    Power-on self test Most information can be captured using the supportSave command and downloaded by FTP off the switch, but when you are collecting information from specialized commands, such as supportShow, this information has to be captured using a Telnet client. To save a set of files that customer support technicians can use to further diagnose the switch condition, enter the supportSave command.
  • Page 105 Power-on self test modprobe: modprobe: Can't open dependencies file /lib/modules/2.4.19/modules.dep (No such file or directory) INIT: version 2.78 booting INIT: Entering runlevel: 3 eth0: Link status change: Link Up. 100 Mbps Full duplex Auto (autonegotiation complete). INITCP: CPLD Vers: 0x95 Image ID: 0x19 uptime: 2008;...

  • Page 106: Disabling Post

    Switch status Disabling POST A reboot is not required for this command to take effect. 1. Connect to the switch and log in with a user account that has admin privileges with the chassis-role permission. 2. Enter the diagDisablePost command. This disables POST1 and POST2.

  • Page 107: Table 15 Switch Summary Information

    Switch status Faulty ports monitor HEALTHY Missing SFPs monitor HEALTHY All ports are healthy For more information on how the overall switch status is determined, refer to the switchStatusPolicySet command in the Fabric OS Command Reference. Displaying switch information Table 15 lists the switch summary information TABLE 15 Switch summary information...

  • Page 108: Table 17 Switchshow Command Output

    Switch status The switchShow command also displays the following information for ports on the specified switch: TABLE 17 switchShow command output Variable Definition Index Index follows Area up to 255. Then it continues to the maximum port of the platform. Index identifies the port number relative to the switch.

  • Page 109: Table 18 Port Type Support

    Using the SpinFab and portTest commands Using the SpinFab and portTest commands The spinFab command is an online diagnostics command to verify the ISL links between switches at the maximum speed. It is done by setting up the routing functionality in the hardware such that the test frames received by E_Port are retransmitted on the same E_Port.
  • Page 110 Using the SpinFab and portTest commands Link errors Once the frame is sent out of the port, the spinFab command monitors the link errors in the ASIC. If any of the error counters are non-zero, spinFab reports ERROR and the test fails on the port. ERROR: DIAG ERRSTAT spinfab, pass 6, Pt0/17(7) Ch0/7 CRC_err Error Counter is 109738997 sb 0, ERROR: DIAG ERRSTAT spinfab, pass 1,...

  • Page 111: Clearing The Error Counters

    Using the SpinFab and portTest commands Tx/Rx errors Following errors are seen when the port fails to transmit or receive the frames. ERROR: DIAG PORTSTOPPED spinfab:spinfab, 0 nMegs, Pt7/2(2) Ch0/2 No Longer Transmitting, FTX Counter Stuck at 116295726, ERROR: DIAG TIMEOUT spinfab:spinfab, pass 2, Pt0/17(7) Ch0/7 Receive Error/Timeout The following are debugging procedures: •...

  • Page 112: Diagnostic Port

    Diagnostic Port • To disable a port that is persistently enabled, enter the command portCfgPersistentDisable portnumber or portCfgPersistentDisable slotnumber/portnumber. Diagnostic Port Fabric OS v7.0.0 allows you to convert a fibre channel port, including ISLs and loopback ports, into a Diagnostic Port (D_Port). This port lets you isolate the inter-switch link (ISL) to diagnose link level faults.
  • Page 113 Diagnostic Port For complete information on the portdporttest and portcfgdport commands, refer to the Fabric OS Command Reference. NOTE These tests can be run on EX_Ports by using the portcfgexport command to enable and disable the EX_Ports. For detailed information on the portcfgexport command, refer to the Fabric OS Command Reference.

  • Page 114: Disabling D_Port

    Diagnostic Port Remote WWNN: 10:00:00:05:33:13:2f:b5 Remote port: Mode: Automatic Start time: Wed Feb 2 01:41:43 2011 End time: Wed Feb 2 01:43:23 2011 Status: RESPONDER ============================================================================= Test Start time Result EST(secs) Comments ============================================================================= Electrical loopback 01:42:08 PASS ---------- Optical loopback 01:42:16 RESPONDER remote port results...

  • Page 115: D_Port Limitations

    Port information D_Port limitations The D_Port state and test data is not synchronized between the ports. If one of the switches has a failover, the D_Port link goes though re-initialization. If the remote switch is in final state after reboot or failover, the D_Port tries to automatically start the test. However, the test cannot proceed as the other port is already in the final state.

  • Page 116: Displaying The Port Statistics

    Port information LE domain: 0 FC Fastwrite: OFF Interrupts: Link_failure: 0 Frjt: Unknown: Loss_of_sync: 3 Fbsy: Lli: Loss_of_sig: Proc_rqrd: Protocol_err: 0 Timed_out: Invalid_word: 563851 Rx_flushed: Invalid_crc: Tx_unavail: Delim_err: Free_buffer: Address_err: Overrun: Lr_in: Suspended: Lr_out: Parity_err: Ols_in: 2_parity_err: Ols_out: CMI_bus_err: Port part of other ADs: No Refer to the Fabric OS Command Reference for additional portShow command information, such as the syntax for slot or port numbering, displaying IP interfaces on a GbE port, or displaying FCIP tunnel connection or configuration information.

  • Page 117: Displaying A Summary Of Port Errors For A Switch

    Port information er_c3_dest_unreach Class 3 frames discarded due to destination unreachable er_other_discard Other discards er_type1_miss frames with FTB type 1 miss er_type2_miss frames with FTB type 2 miss er_type6_miss frames with FTB type 6 miss er_zone_miss frames with hard zoning miss er_lun_zone_miss frames with LUN zoning miss er_crc_good_eof...

  • Page 118: Table 19 Error Summary Description

    Equipment status 664k 6.7k (output truncated) The portErrShow command output provides one output line per port. See Table 19 for a description of the error types. TABLE 19 Error summary description Error type Description frames tx Frames transmitted frames rx Frames received enc in Encoding errors inside frames...

  • Page 119: Checking The Status Of The Fans

    Equipment status 4. Check the fan speed output. If any of the fan speeds display abnormal RPMs, replace the fan FRU. 5. Check the power supply status. If any power supplies show a status other than OK, consider replacing the power supply as soon as possible.

  • Page 120: Checking Temperature Status

    System message log If any of the power supplies show a status other than OK, consider replacing the power supply as soon as possible. For certain switch models, the OEM serial ID data displays after each power supply status line. Checking temperature status 1.

  • Page 121: Displaying The System Message Log, With No Page Breaks

    Port log The FID can be a number from 0 to 128, and the identifier CHASSIS depends on the instance that generates the message and that it was generated by a chassis instance. The identifier FID 128 means the message was generated by the default switch instance. 2008/08/01-00:19:44, [LOG-1003], 1, SLOT 6 | CHASSIS, INFO, Silkworm12000, The log has been cleared.

  • Page 122: Table 20 Commands For Port Log Management

    Port log switch:admin> portlogshow time task event port cmd args ------------------------------------------------- Fri Feb 22 16:48:45 2008 16:48:45.208 SPEE 00000009,00000000,00000000 16:48:46.783 PORT 02fffffd,00fffffd,02e2ffff,14000000 16:48:46.783 PORT c0fffffd,00fffffd,02e201bf,00000001 16:48:46.783 FCPH read 02fffffd,00fffffd,be000000,00000000,02e201bf 16:48:46.783 FCPH 22380000,02e201bf,00000c1e,0000001c,00000000 16:48:46.828 SPEE 00000009,00000000,00000000 16:48:46.853 PORT 02fffffd,00fffffd,02e3ffff,14000000 16:48:46.853 PORT c0fffffd,00fffffd,02e301c1,00000001 16:48:46.853 FCPH read...

  • Page 123: Table 21 Fabric Os To Unix Message Severities

    Syslogd configuration 20:29:20.804 PORT 02fffffd,00fffffd,0306ffff,14000000 20:29:20.805 PORT c0fffffd,00fffffd,030609a7,00000001 20:29:20.805 FCPH read 02fffffd,00fffffd,d1000000,00000000,030609a7 20:29:20.805 FCPH 22380000,030609a7,00000608,0000001c,00000000 20:29:20.805 PORT 02fffffd,00fffffd,02eeffff,14000000 20:29:20.806 PORT c0fffffd,00fffffd,02ee09a8,00000001 20:29:20.806 FCPH read 02fffffd,00fffffd,d2000000,00000000,02ee09a8 20:29:20.806 FCPH 22380000,02ee09a8,00000608,0000001c,00000000 20:29:32.638 FCPH write 00fffffd,00fffffd,00000000,00000000,00000000 20:29:32.638 FCPH 00300000,00000000,000005f4,00020182,00000000 20:29:32.638 PORT 02fffffd,00fffffd,09a9ffff,14000000 20:29:32.638 FCPH write 40 00fffffd,00fffffd,00000000,00000000, 00000000 20:29:32.638 FCPH...

  • Page 124: Configuring The Switch

    Syslogd configuration local7.info /var/adm/swinfo local7.debug /var/adm/debug7 If you prefer to map Fabric OS severities to a different UNIX local7 facility level, see “Setting the facility level” on page 104. Configuring the switch Configuring the switch involves specifying syslogd hosts and, optionally, setting the facility level. You can also remove a host from the list of syslogd hosts.

  • Page 125: Automatic Trace Dump Transfers

    Automatic trace dump transfers 3. Verify the IP address was deleted using the syslogdIpShow command. Automatic trace dump transfers You can set up a switch so that diagnostic information is transferred automatically to a remote server. If a problem occurs, you can then provide your customer support representative with the most detailed information possible.

  • Page 126: Saving Comprehensive Diagnostic Files To The Server

    Automatic trace dump transfers Saving comprehensive diagnostic files to the server 1. Connect to the switch and log in as admin. 2. Enter the supportSave -c command and respond to the prompts. switch:admin> supportsave -c This command will collect RASLOG, TRACE, supportShow, core file, FFDC data and other support information and then transfer them to a FTP/SCP server or a USB device.
  • Page 127 Appendix Switch Type and Blade ID The switchType is a displayed field listed when you run the switchShow command. When you are gathering information to give to your switch support provider, you may be asked the switch model. If you do not know the model, you can use this chart to convert the switchType to a B-Series model number.

  • Page 128: Table 22 Switchtype To B-Series Model Converter

    Switch Type and Blade ID TABLE 22 switchType to B-Series model converter (Continued) switchType B-Series switch model ASIC Base switch speed Brocade Encryption Switch Condor2 8 Gb 16-port encryption switch 5410 GoldenEye2 8 Gb 12-port embedded switch GoldenEye2 8 Gb 16-port switch 5480 GoldenEye2 8 Gb 24-port embedded switch...

  • Page 129: Hexadecimal Overview

    Appendix Hexadecimal Hexadecimal overview Hexadecimal, also known as hex, is a numeral system with a base of 16, usually written using unique symbols 0–9 and A–F, or a–f. Its primary purpose is to represent the binary code that computers interpret in a format easier for humans to remember. It acts as a form of shorthand, in which one hexadecimal digit takes the place of four binary bits.

  • Page 130: Table 24 Decimal To Hex Conversion Table

    Hexadecimal overview TABLE 24 Decimal to Hex conversion table Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Decimal Fabric OS Troubleshooting and Diagnostics Guide 53-1002150-02...
  • Page 131 Hexadecimal overview TABLE 24 Decimal to Hex conversion table (Continued) Decimal Decimal Decimal Decimal Decimal Decimal Decimal Fabric OS Troubleshooting and Diagnostics Guide 53-1002150-02...
  • Page 132 Hexadecimal overview Fabric OS Troubleshooting and Diagnostics Guide 53-1002150-02...
  • Page 133 Index correcting device login issues link failures account management marginal links lost password recovery options crc_err counter recovering forgotten passwords crc_g_eof counter unable to modify switch settings user forgot password user unable to change switch settings D_Port devices blade errors login issues missing AP blade type 24 is inserted...
  • Page 134 EX_Ports firmwareDownload errors AP blade type 24 is inserted blade application firmware failed blade application firrmware failed blade errors blade is faulty F_Port blade is stuck in the ‘LOADING’ state fabric cannot download the requested firmware issues cannot download with interop mode turned on merge fails cannot upgrade directly to v6.0 parameters...
  • Page 135 L_Port passwords, recovering forgotten LEDs pathInfo flashing performance problems no light PLOGI steady License issues enabling ports licensing issues point-to-point initialization failure link port intermittent connectivty bypassed marginal disabled no port LED light enabling port LEDs flashing in wrong mode port LEDs steady initialization logical connection...
  • Page 136 resolving zone conflicts test Rolling Reboot Detection a port a switch TI zone problem time is not in-sync troubleshooting certificates SCSI corrupt certificate retry errors invalid certificate timeout errors trunk security, gathering additional information bounces segmentation failed to form segmented fabrics trunk goes offline and online (bounces) setting up automatic trace dump transfers slow-down in FCR performance...
  • Page 137 zone configuration configuration mismatch content mismatch resolving conflicts troubleshooting type mismatch Fabric OS Troubleshooting and Diagnostics Guide 53-1002150-02...
  • Page 138 Fabric OS Troubleshooting and Diagnostics Guide 53-1002150-02...

This manual is also suitable for:

Brocade bladesystem 4/24Fabric os 7.0.0

Table of Contents