H3C's First General-Purpose Mid-to-High-End Layer 3 Switch: A Breakthrough Story – First Tribulation: Process Hang

Table of Content [Hide]

Large-scale S650 trial deployment and the first service-outage incident

In late 2004, H3C's newly developed S6500 general-purpose mid-to-high-end Layer 3 switch was delivered for trial deployment at 112 broadband MAN nodes of friendly user XX Telecom. As user services were gradually cut over to the new network, the performance and stability of the S6500 were put to the test.

To support the large-scale trial across 112 MAN nodes, the S6500 R&D team assigned 2 R&D engineers on a long-term rotating basis to be stationed at the local office. Together with the office's IP engineers and partner engineers, they formed the S6500 trial support team to respond and resolve technical issues in a timely manner.

Now came the first tribulation of the S6500! As services went live, some nodes reported occasional total service outages on their S6500 switches. Engineers from the support team arrived on-site promptly to collect fault information and troubleshoot, only to find that the upstream link of the S6500 became unreachable via ping.

After standard troubleshooting measures—such as shutting down and re-enabling the upstream physical port, deleting and reconfiguring the upstream Interface VLAN interface—failed to restore service, on-site engineers collected runtime information from the faulty S6500 under R&D guidance and then rebooted the switch. Service was restored after reboot.

layer-3-switch-a-breakthrough-story-first-tribulation-process-hang-1.jpg

Upon receiving the fault information collected by on-site engineers, the S6500 R&D team immediately conducted analysis and built a test environment to reproduce the issue. After examining fault logs from multiple nodes, the root cause was finally identified: the ARP process had hung.

layer-3-switch-a-breakthrough-story-first-tribulation-process-hang-2.jpg

ARP process hang in the S650 trial and the temporary software workaround

The ARP process is one of the fundamental processes of IP devices. Its unexpected hang dealt a considerable psychological impact on XX Telecom. Fortunately, XX Telecom was a friendly user with an excellent cooperative relationship. Furthermore, since this was a trial deployment, both parties had reasonable expectations that some unexpected issues might occur. Therefore, the ARP process hang did not halt the trial.

After pinpointing the ARP process hang issue, the S6500 R&D team released a corresponding temporary software version. It added a monitoring program for the running status of key processes, which automatically restarts the affected process immediately upon detecting a hang. The root cause of why the ARP process hung required further in-depth investigation.

layer-3-switch-a-breakthrough-story-first-tribulation-process-hang-3.jpg

The S6500 trial support team then upgraded the software version on all S6500 switches at the 112 trial nodes, overcoming the first tribulation.

By Lanbras

Lanbras specializes in translating cutting-edge optical and Ethernet transmission technologies into clear, valuable insights that help our customers stay ahead in a fast-evolving digital world.

By turning complex technical concepts into practical, business-driven content, we aim to empower decision-makers with the knowledge they need to make confident, future-ready choices.

PREV: H3C's First General-Purpose Mid-to-High-End Layer 3 Switch: A Breakthrough Story – Background

NEXT: H3C's First General-Purpose Mid-to-High-End Layer 3 Switch: A Breakthrough Story – Second Tribulation: Specification Limitation

References

Lastest News & Blog about Lanbras

New Campus Network Construction Project of a University – Switch Deployment Planning

After determining AP installation locations and wired/fiber access points on each floor of all functional buildings, the port counts (electrical ports and optical ports) and deployment positions of ac...

 Jul 22, 2026

Learn More

New Campus Network Construction Project of a University – AP Coverage Planning

Different functional buildings vary in internal space layout, as well as the type, quantity and density of connected end-user terminals, thus requiring differentiated access solutions.Canteens, audito...

 Jul 20, 2026

Learn More

New Campus Network Construction Project of a University – Project Background

To accommodate the expanding student enrollment scale, a university launched the construction of a new campus in 2017. As civil engineering works of the new campus were gradually completed, the constr...

 Jul 15, 2026

Learn More