H3C's First General-Purpose Mid-to-High-End Layer 3 Switch: A Breakthrough Story – Sixth Tribulation: Port Hang

Table of Content [Hide]

After struggling through the previous five tribulations, the S6500 switches deployed at the 112 nodes seemed to operate stably. Just as the S6500 trial support team planned to reallocate part of the support resources to other projects, the sixth tribulation emerged. Reports of port hangs on the S6500 came in one after another: the port indicator lights either stayed off permanently or stayed on continuously. Shutting down and re-enabling the faulty ports, or unplugging and reinserting the cables sometimes restored normal operation, while at other times these methods proved completely ineffective.

The support team promptly dispatched engineers to the faulty sites. On-site engineers inspected the operation mode, running status, bit error rate and error packet count of the faulty ports one by one, and confirmed that the ports were fully hung up and permanently damaged. However, the S6500 had been in service for only a few months, which raised the question of why port hangs and permanent port failures occurred so frequently.

Following the instructions from the R&D team, engineers collected runtime information and debug logs of the S6500, and then carried out board replacement.

After analyzing fault data and debug information collected from multiple nodes, the S6500 R&D team confirmed that the failures were caused by abnormal port chips on the service boards. The root cause lay in the low-cost port chips adopted under aggressive cost-cutting measures. Such chips delivered poor stability and were prone to hangs and permanent damage under heavy port traffic or poor line quality.

The S6500 R&D team immediately released a temporary software version equipped with a monitoring program for port chip operating status. The program automatically restarts the port chip as soon as a hang is detected. There was no effective software solution for permanent port damage, and the only remedy was to replace the entire service board.

The S6500 trial support team upgraded the software version on all S6500 devices across the 112 trial nodes. The new version resolved the port hang issue but could not fix permanent port damage.

After a period of observation and statistics, it was found that the port failure rate of the S6500 was noticeably higher than that of other Ethernet switches, and the corresponding board replacement volume also increased significantly.

The removed faulty boards were sent back to the R&D department for further analysis. The conclusion showed that repeated restarts accelerated the aging of port chips and eventually led to complete hardware damage. To mitigate the issue, the team had to prepare sufficient spare boards to get through this sixth tribulation.

Although the S6500 overcame the six major technical difficulties through various solutions, some inherent problems remained unsolved: complicated Interface VLAN configurations, insufficient ASIC performance, poor stability of port chips, and high failure rates of ports and service boards. Due to these persistent defects, the S6500 failed to compete with peer Ethernet switches in the market and was officially discontinued on March 10, 2007.

Despite being an unsuccessful product, as H3C's first general-purpose mid-to-high-end Layer 3 switch, the S6500 accumulated valuable experience during its development and trial deployment. It laid a solid foundation and served as a pioneer for the success of H3C's subsequent generations of mid-to-high-end Layer 3 switches.

By Lanbras

Lanbras specializes in translating cutting-edge optical and Ethernet transmission technologies into clear, valuable insights that help our customers stay ahead in a fast-evolving digital world.

By turning complex technical concepts into practical, business-driven content, we aim to empower decision-makers with the knowledge they need to make confident, future-ready choices.

PREV: H3C's First General-Purpose Mid-to-High-End Layer 3 Switch: A Breakthrough Story – Fifth Tribulation: Excessively High CPU Utilization

NEXT: Telenor Myanmar New Network Planning - Project Background

References