H3C's First General-Purpose Mid-to-High-End Layer 3 Switch: A Breakthrough Story – First Tribulation: Process Hang

Table of Content [Hide]

In late 2004, H3C's newly developed S6500 general-purpose mid-to-high-end Layer 3 switch was delivered for trial deployment at 112 broadband MAN nodes of friendly user XX Telecom. As user services were gradually cut over to the new network, the performance and stability of the S6500 were put to the test.

To support the large-scale trial across 112 MAN nodes, the S6500 R&D team assigned 2 R&D engineers on a long-term rotating basis to be stationed at the local office. Together with the office's IP engineers and partner engineers, they formed the S6500 trial support team to respond and resolve technical issues in a timely manner.

Now came the first tribulation of the S6500! As services went live, some nodes reported occasional total service outages on their S6500 switches. Engineers from the support team arrived on-site promptly to collect fault information and troubleshoot, only to find that the upstream link of the S6500 became unreachable via ping.

After standard troubleshooting measures—such as shutting down and re-enabling the upstream physical port, deleting and reconfiguring the upstream Interface VLAN interface—failed to restore service, on-site engineers collected runtime information from the faulty S6500 under R&D guidance and then rebooted the switch. Service was restored after reboot.

layer-3-switch-a-breakthrough-story-first-tribulation-process-hang-1.jpg

Upon receiving the fault information collected by on-site engineers, the S6500 R&D team immediately conducted analysis and built a test environment to reproduce the issue. After examining fault logs from multiple nodes, the root cause was finally identified: the ARP process had hung.

layer-3-switch-a-breakthrough-story-first-tribulation-process-hang-2.jpg

The ARP process is one of the fundamental processes of IP devices. Its unexpected hang dealt a considerable psychological impact on XX Telecom. Fortunately, XX Telecom was a friendly user with an excellent cooperative relationship. Furthermore, since this was a trial deployment, both parties had reasonable expectations that some unexpected issues might occur. Therefore, the ARP process hang did not halt the trial.

After pinpointing the ARP process hang issue, the S6500 R&D team released a corresponding temporary software version. It added a monitoring program for the running status of key processes, which automatically restarts the affected process immediately upon detecting a hang. The root cause of why the ARP process hung required further in-depth investigation.

layer-3-switch-a-breakthrough-story-first-tribulation-process-hang-3.jpg

The S6500 trial support team then upgraded the software version on all S6500 switches at the 112 trial nodes, overcoming the first tribulation.

By Lanbras

Lanbras specializes in translating cutting-edge optical and Ethernet transmission technologies into clear, valuable insights that help our customers stay ahead in a fast-evolving digital world.

By turning complex technical concepts into practical, business-driven content, we aim to empower decision-makers with the knowledge they need to make confident, future-ready choices.

PREV: H3C's First General-Purpose Mid-to-High-End Layer 3 Switch: A Breakthrough Story – Background

NEXT: H3C's First General-Purpose Mid-to-High-End Layer 3 Switch: A Breakthrough Story – Second Tribulation: Specification Limitation

References

Lastest News & Blog about Lanbras

H3C's First General-Purpose Mid-to-High-End Layer 3 Switch: A Breakthrough Story – Fifth Tribulation: Excessively High CPU Utilization

Before the routing abnormality issue was fully resolved, the fifth tribulation struck! Following the recommendation of the S6500 R&D team, the routine inspection tasks of the S6500 support team we...

 May 09, 2026

Learn More

H3C's First General-Purpose Mid-to-High-End Layer 3 Switch: A Breakthrough Story – Fourth Tribulation: Routing Abnormality

As more and more internet cafe users and enterprise users were cut over and activated on the S6500, the fourth tribulation emerged! From time to time, nodes reported that internet cafe users connected...

 May 08, 2026

Learn More

H3C's First General-Purpose Mid-to-High-End Layer 3 Switch: A Breakthrough Story – Third Tribulation: Forwarding Abnormality

After a large number of internet cafe users were cut over to the S6500, the third tribulation struck! Nodes reported occasional service outages affecting one or several internet cafes connected under ...

 May 07, 2026

Learn More