Chengdu Shuwei Communication Technology Co., Ltd.

In the intricate world of network management and security, having clear visibility into network traffic is paramount. NetTAP, a leader in the field, specializes in Network Traffic Visibility, Network Data Visibility, and Network Packet Visibility. In this technical blog, we'll explore how NetTAP's solutions excel in capturing, replicating, and aggregating network data traffic, ensuring seamless delivery to the right tools for effective network monitoring, analysis, and security.   Understanding NetTAP's Specializations NetTAP's expertise lies in three core areas:   1. Network Traffic Visibility: NetTAP provides unparalleled visibility into network traffic, allowing organizations to monitor and analyze data flows in real-time. By capturing every packet traversing the network, NetTAP ensures that no valuable information is missed. 2. Network Data Visibility: NetTAP goes beyond traditional traffic monitoring by offering deep insights into network data. Through advanced analysis techniques, NetTAP enables organizations to extract meaningful information from network traffic, empowering informed decision-making and troubleshooting. 3. Network Packet Visibility: NetTAP's packet-level visibility ensures that organizations have full control over their network data. By capturing, replicating, and aggregating network packets, NetTAP facilitates efficient data distribution to monitoring tools, such as IDS (Intrusion Detection Systems), APM (Application Performance Monitoring), and NPM (Network Performance Monitoring).   NetTAP's Solutions at Work To better understand how NetTAP's solutions operate, let's delve into their key functionalities: Feature Description Capture Capabilities NetTAP's advanced capture technology ensures the seamless acquisition of network traffic data. Replication NetTAP replicates network traffic to distribute data to multiple monitoring and analysis tools. Aggregation Aggregating network traffic simplifies the monitoring process, providing a consolidated view of data. Network Tapping NetTAP utilizes Network Taps to access network traffic without disrupting network operations. Packet Brokering NetTAP's Packet Brokers intelligently route packets to their intended destinations, minimizing latency and packet loss.   Application Scenarios NetTAP's solutions find applications across various scenarios: 1. Network Monitoring: By providing comprehensive visibility into network traffic, NetTAP enables organizations to monitor network performance in real-time, facilitating proactive issue resolution. 2. Network Analysis: NetTAP's deep packet inspection capabilities allow organizations to conduct granular analysis of network traffic, uncovering insights, trends, and anomalies. 3. Network Security: With cyber threats on the rise, NetTAP enhances network security by detecting and mitigating malicious activity in real-time, ensuring data integrity and confidentiality.   NetTAP's specialization in Network Traffic Visibility, Network Data Visibility, and Network Packet Visibility is unparalleled in the industry. By capturing, replicating, and aggregating network traffic seamlessly, NetTAP empowers organizations to make informed decisions, optimize network performance, and fortify their security posture. With NetTAP's solutions, organizations can navigate the complexities of modern network management with confidence, knowing that their data is in safe hands.

Since its establishment in 2007, NetTAP® (Chengdu Shuwei Communication Technology Co., Ltd.) has been at the forefront of research and development in the field of network communication technology. Specializing in Network TAPs/NPBs (Network Packet Brokers), Data Security Analysis Equipment, and related solutions, NetTAP® plays a crucial role in providing cutting-edge products and services for a wide range of industries including Telecom, TV Broadcasting, Government, Education, IT, Finance, Healthcare, Transportation, Energy, and more. With a firm commitment to innovation and excellence, NetTAP® has continually pushed the boundaries of what is possible in the realm of network communication equipment. Their product offerings span across Big Data Acquisition, Data Storage, Data Monitoring, Data Processing, and Data Analysis, catering to the diverse needs of organizations operating in various sectors.   NetTAP®'s portfolio boasts a diverse range of high-density devices designed to meet the evolving demands of modern networks. From Fast Gigabit to 10 Gigabit Ethernet Network Traffic Replication, Aggregation, and Distribution Network TAPs/NPBs, to intelligent Hybrid Traffic Network TAPs, and even high-performance solutions for 25G, 40G, and 100G networks, NetTAP® has a solution for every network infrastructure requirement.   Central to NetTAP®'s success is its unwavering focus on quality and innovation. The company boasts a state-of-the-art hardware production plant where all communication products are meticulously crafted to meet the highest standards. Furthermore, NetTAP® prides itself on holding complete Software and Hardware Independent Intellectual Property Rights for all its products, with comprehensive Copyright Registration and Product Certification ensuring the integrity and authenticity of their offerings.   In an era where data security and network performance are of paramount importance, organizations across industries rely on NetTAP®'s expertise and solutions to stay ahead of the curve. By leveraging NetTAP®'s advanced network communication technology, businesses can enhance their data acquisition, monitoring, processing, and analysis capabilities, thereby driving efficiency, security, and overall performance.   As technology continues to advance at a rapid pace, NetTAP® remains committed to innovation, ensuring that its products and services evolve in tandem with the changing needs of its customers. With a proven track record of excellence and a dedication to pushing the boundaries of what is possible, NetTAP® stands as a trusted partner for organizations seeking to unlock the full potential of their network infrastructure.

Netflow (Network Data Flow Detection Protocol) With the software system upgrade and the maturity of the vulnerability repair scheme, the virus attack mode that directly invades the host for damage is gradually reduced, and then turns to malicious consumption of limited network resources, causing network congestion, thus destroying the ability of the system to provide external services. In response to such attacks, the industry has proposed a method of detecting network data Flow to judge network anomalies and attacks. By detecting network data flow information in real time, network managers can check the status of the whole network in real time by matching the historical pattern (to judge whether it is normal) or the abnormal pattern (to judge whether it is attacked). Detect possible bottlenecks in network performance, and automatically handle or alarm display to ensure efficient and reliable network operation. Netflow technology was first invented by Darren Kerr and Barry Bruins of Cisco in 1996 and registered as a US patent in May of the same year. Netflow technology is first used in network equipment to accelerate data exchange, and can realize the measurement and statistics of high-speed forwarded IP data flow. After years of technological evolution, the original function of Netflow for data exchange acceleration has been gradually replaced by dedicated ASIC chips in network devices, while the function of measuring and statistics of IP data flow through network devices is still retained. It has become the most recognized industry standard for IP/MPLS traffic analysis, statistics and billing in the Internet field. Netflow technology can analyze and measure the detailed behavior pattern of IP/MPLS network traffic, and provide detailed statistics of network operation. The Netflow system consists of three main parts: the Exporter, the Collector, and the Analysis reporting system. Exporter: monitors network data Collector: Used to collect network data exported from Exporter Analysis: Used to analyze the network data collected from the Collector and generate reports   By analyzing the information collected by Netflow, network administrators can know the source, destination, network service type of packets, and the cause of network congestion. It may not provide a complete record of network traffic like tcpdump does, but when put together it is much easier to manage and read. NetFlow network data output from routers and switches consists of expired data flows and detailed traffic statistics. These data flows contain the IP address associated with the source and destination of the packet, as well as the Protocol and Port used by the end-to-end session. The traffic statistics include the data flow timestamp, source and destination IP addresses, source and destination port numbers, input and output interface numbers, next hop IP addresses, total bytes in the flow, number of packets in the flow, and time stamps of the first and last packets in the flow. And front mask, packet number, etc. Netflow V9 is a new flexible and extensible Netflow data output format with template-based statistics output. Easy to add data fields that need to be output and support a variety of new functions, such as: Multicase Netflow, MPLS Aware Netflow, BGP Next Hop V9, Netflow for IPv6, and so on. In 2003, Netflow V9 was also selected as the IPFIX (IP Flow Information Export) standard by the IETF from five candidates.   IPFIX (Network Traffic Monitoring) Flow-based technology is widely used in network field, it has great value in QoS policy setting, application deployment and capacity planning. However, network administrators lack a standard format for output data streams. IPFIX (IP Flow Information Export, IP data flow information output) is a standard protocol for measuring flow information in networks published by IETF. The format defined by IPFIX is based on the Cisco Netflow V9 data output format, which standardizes the statistics and output standards of IP data flows. It is a protocol for analyzing data flow characteristics and output data in a template-based format. Therefore, it has strong scalability. If traffic monitoring requirements change, network administrators can modify the corresponding configurations without upgrading network device software or management tools. Network administrators can easily extract and view important traffic statistics stored in these network devices. For a more complete output, IPFIX uses seven key domains of network devices by default to represent network traffic per share: 1. Source IP address 2. Destination IP address 3. TCP/UDP source port 4. TCP/UDP destination port 5. Layer 3 protocol type 6. The Type of service (Type of service) byte 7. Enter a logical interface If all seven key domains in different IP packets match, the IP packets are regarded as belonging to the same traffic. By recording the characteristics of the traffic on the network, such as the traffic duration and average packet length, you can learn about the current network application, optimize the network, detect the security, and charge the traffic.   IPFIX networking architecture To sum up, IPFIX is based on the concept of Flow. A Flow refers to packets from the same subinterface with the same source and destination IP address, protocol type, source and destination port number, and ToS. The packets are usually 5-tuples. IPFIX records statistics about the stream, including the timestamp, the number of packets, and the total number of bytes. IPFIX consists of three devices: Exporter, Collector, and Analyzer. The relationships among the three devices are as follows: Export analyzes network flows, extracts qualified flow statistics, and sends the statistics to Collector. The Collector parses Export data packets and collects statistics in the database for analysis by the Analyser. The Analyser extracts statistics from the Collector, performs subsequent processing, and displays the statistics as a GUI for various services.   IPFIX application scenarios Usage-based Accounting Traffic billing in network operators is generally based on the upload and download traffic of each user. Because IPFIX can be accurate to the destination IP address, protocol port and other fields, the future traffic charging can be segmented based on the characteristics of the application service. Of course, the protocol also explains that IPFIX packet statistics are "sampled". In many applications (such as the backbone layer), the more detailed the data flow statistics are, the better. Due to the performance of network devices, the sampling rate cannot be too small, so it is not necessary to provide completely accurate and reliable traffic billing. However, at the network operator level, the billing unit is generally more than 100 megabits, and the sampling accuracy of IPFIX can meet the relevant needs. Traffic Profiling, Traffic Engineering The record output of IPFIX Exporter, IPFIX Collector can output very rich Traffic record information in the form of various charts, this is the concept of Traffic Profiling. However, just the record of information, can not take advantage of the powerful function of IPFIX, IETF also launched the concept of Traffic Engineering: in the actual operation of the network, often planned load balancing and redundant backup, but the various protocols are generally according to the predetermined route of the network planning, or protocol principles are adjusted. If IPFIX is used to monitor the traffic on the network and a large amount of data is found in a certain period of time, the network administrator can be reported to adjust the traffic, so that more network bandwidth can be allocated to related applications to reduce the uneven load. In addition, you can bind configuration rules, such as route adjustment, bandwidth allocation, and security policies, to the operations on the IPFIX Collector to automatically adjust network traffic. Attack/Intrusion Detection Attack/Intrusion detection IPFIX can detect network attacks based on traffic characteristics. For example, typical IP scanning, port scanning, DDOS attacks. The sampling standard IPFIX protocol can also use a "signature database" upgrade to block the latest network attacks, just like the general host-side virus protection. QoS Monitoring (Network Quality of Service Monitoring) Typical QoS parameters are: Packet loss condition: loss [RFC2680] one-way delay: One-way delay [RFC2679] round-trip delay: round-trip delay [RFC2681] delay variation [RFC3393] Previous technologies are difficult to monitor the above information in real time, but IPFIX's various custom fields and monitoring intervals can easily monitor the above values of various messages.   Here's an expanded table that provides more details about the differences between NetFlow and IPFIX:  

In the typical NPB application scenario, the most troublesome problem for administrators is packet loss caused by the congestion of mirrored packets and NPB networks. Packet loss in NPB can cause the following typical symptoms in back-end analysis tools: An alarm is generated when the APM service performance monitoring indicator decreases, and the transaction success rate decreases The NPM network performance monitoring indicator exception alarm is generated The security monitoring system fails to detect network attacks due to event omission Loss of service behavior audit events generated by the service audit system ... As a centralized capture and distribution system for Bypass monitoring, the importance of NPB is self-evident. At the same time, the way it processes data packet traffic is quite different from the traditional live network switch, and the traffic congestion control technology of many service live networks is not applicable to NPB. How to solve NPB packet loss, let's start from the root cause analysis of packet loss to see it!   NPB/TAP Packet Loss Congestion Root Cause Analysis First of all, we analyze the actual traffic path and the mapping relationship between the system and the incoming and outgoing of the level 1 or level NPB network. No matter what kind of network topology NPB forms, as a collection system, there is a many-to-many traffic input and output relationship between "access" and "output" of the whole system.   Then we look at the business model of NPB from the perspective of ASIC chips on a single device:   Feature 1: The "traffic" and "physical interface rate" of the input and output interfaces are asymmetrical, resulting in a large number of micro-bursts is an inevitable result. In typical many-to-one or many-to-many traffic aggregation scenarios, the physical rate of the output interface is usually smaller than the total physical rate of the input interface. For example, 10 channels of 10G collection and 1 channel of 10G output; In a multilevel deployment scenario, all NPBBS can be viewed as a whole. Feature 2: ASIC chip cache resources are very limited. In terms of the currently commonly used ASIC chip, the chip with 640Gbps exchange capacity has a cache of 3-10Mbytes; A 3.2Tbps capacity chip has a cache of 20-50 mbytes. Including BroadCom, Barefoot, CTC, Marvell and other manufacturers of ASIC chips. Feature 3: The conventional end-to-end PFC flow control mechanism is not applicable to NPB services. The core of the PFC flow control mechanism is to achieve end-to-end traffic suppression feedback, and ultimately reduce the sending of packets to the protocol stack of the communication endpoint to alleviate congestion. However, the packet source of NPB services is mirrored packets, so the congestion processing strategy can only be discarded or cached.   The following is the appearance of a typical micro-burst on the flow curve:                                                                                                 Taking 10G interface as an example, in the second level traffic trend analysis diagram, the traffic rate is maintained at about 3Gbps for a long time. On the micro millisecond trend analysis chart, the traffic spike (MicroBurst) has greatly exceeded the 10G interface physical rate.   Key Techniques for Mitigating NPB Microburst Reduce the impact of asymmetric physical interface rate mismatch - When designing a network, reduce asymmetric input and output physical interface rates as much as possible. A typical method is to use a higher rate uplink interface link, and avoid asymmetric physical interface rates (for example, copying 1 Gbit/s and 10 Gbit/s traffic at the same time). Optimize the cache management policy of the NPB service - The common cache management policy applicable to the switching service is not applicable to the forwarding service of the NPB service. The cache management policy of static guarantee + Dynamic sharing should be implemented based on the features of the NPB service. In order to minimize the impact of NPB microburst under the current chip hardware environment limitation. Implement classified traffic engineering management - Implement priority traffic engineering service classification management based on traffic classification. Ensure service quality of different priority queues based on category queue bandwidths, and ensure that user sensitive service traffic packets can be forwarded without packet loss. A reasonable system solution enhances the packet caching capability and traffic shaping capability - Integrates the solution through various technical means to expand the packet caching capability of the ASIC chip. By shaping the flow at different locations, the micro-burst becomes micro-uniform flow curve after shaping.   NetTAP Micro Burst Traffic Management Solution Scheme 1 - Network-optimized cache management strategy + network-wide classified service quality priority management Cache management strategy optimized for the whole network Based on the in-depth understanding of NPB service characteristics and practical business scenarios of a large number of customers, NetTAP traffic collection products implement a set of "static assurance + dynamic sharing" NPB cache management strategy for the whole network, which has a good effect on traffic cache management in the case of a large number of asymmetric input and output interfaces. The microburst tolerance is realized to the maximum extent when the current ASIC chip cache is fixed.   Microburst processing technology - Management based on business priorities     When the traffic capturing unit is deployed independently, it can also be prioritized according to the importance of the back-end analysis tool or the importance of the service data itself. For example, among many analysis tools, APM/BPC has a higher priority than security analysis/security monitoring tools because it involves the monitoring and analysis of various indicator data of important business systems. Therefore, for this scenario, the data required by APM/BPC can be defined as high priority, the data required by security monitoring/security analysis tools can be defined as medium priority, and the data required by other analysis tools can be defined as low priority. When the collected data packets enter the input port, the priorities are defined according to the importance of the packets. Packets of higher priorities are preferentially forwarded after the packets of higher priorities are forwarded, and packets of other priorities are forwarded after the packets of higher priorities are forwarded. If packets of higher priorities continue to arrive, packets of higher priorities are preferentially forwarded. If the input data exceeds the forwarding capability of the output port for a long period of time, the excess data is stored in the cache of the device. If the cache is full, the device preferentially discards the packets of the lower order. This prioritized management mechanism ensures that key analysis tools can efficiently obtain the original traffic data required for analysis in real time.   Microburst Processing Technology - classification guarantee mechanism of the whole network service quality   As shown in the above figure, traffic classification technology is used to distinguish different services on all devices at the access layer, aggregation/core layer, and output layer, and the priorities of captured packets are re-marked. The SDN controller delivers the traffic priority policy in a centralized manner and applies it to the forwarding devices. All devices participating in the networking are mapped to different priority queues according to the priorities carried by packets. In this way, the small-traffic advanced priority packets can achieve zero packet loss. Effectively solve the packet loss problem of APM monitoring and special service audit bypass traffic services.   Solution 2 - GB-level Expansion System Cache + Traffic Shaping Scheme GB Level System Extended Cache When the device of our traffic acquisition unit has advanced functional processing capabilities, it can open up a certain amount of space in the memory (RAM) of the device as the global Buffer of the device, which greatly improves the Buffer capacity of the device. For a single acquisition device, at least GB capacity can be provided as the cache space of the acquisition device. This technology makes the Buffer capacity of our traffic acquisition unit device hundreds of times higher than that of the traditional acquisition device. Under the same forwarding rate, the maximum micro burst duration of our traffic acquisition unit device becomes longer. The millisecond level supported by traditional acquisition equipment has been upgraded to the second level, and the micro-burst time that can be withstand has been increased by thousands of times.   Multi-queue traffic shaping capability Microburst processing technology - a solution based on large Buffer caching + traffic shaping With a super-large Buffer capacity, the traffic data generated by micro-burst is cached, and the traffic shaping technology is used in the outgoing interface to achieve smooth output of packets to the analysis tool. Through the application of this technology, the packet loss phenomenon caused by micro-burst is fundamentally solved.