Ethernet Rates, Times and Statistics

Bit Rate, Packet Rate and Loading

Some of the most common statistics to see for ethernet are the bit rate for a connection (e.g. 1G, 10G), the packet rate (e.g. 1.8Mpps) and the loading (e.g. 37%). But how are these calculated, and what do they mean in practice?

The bit rate is the logical speed or bandwidth of the link, and indicates the maximum data rate that can be transmitted, including overheads (see below).

The maximum packet rate isn't fixed, but depends on the average packet size, and this in turn depends on the traffic type. For example, VoIP packets tend to be very small, whilst web browsing packets are usually much larger.

Loading describes what proportion of the link's bandwidth is being used, and takes into account the amount required by the packet itself plus ethernet overheads such as preamble and inter-frame gap.

What’s the maximum possible packet rate on a 10G link?

One of the smallest packets commonly seen on networks is a TCP ACK. This has a 20 byte IP header and a 20 byte TCP header, adding up to 40 bytes. Because this is smaller than ethernet's minimum payload size of 46 bytes, it is automatically padded prior to transmission to bring it up to size. It is then wrapped with a 14 byte header and 4 byte CRC, to give the minimum ethernet frame size of 64 bytes.

When transmitted, each packet must also be preceded by a 7-byte preamble and 1-byte start-of-frame delimiter, and must be followed by an inter-frame gap of at least 12 bytes. This makes the smallest transmission in ethernet effectively 84 bytes.

We now have sufficient information to calculate the maximum packet rate on a 10G link:

10Gbps / (84 bytes * 8 bits) = 14.88Mpps

But the data rate that will be reported by most tools is significantly less, due to not counting the overheads:

14.88Mpps * 64 bytes * 8 bits = 7.62Gbps

Finally, we can work out the IP transmission efficiency (as opposed to ethernet efficiency) - it's pretty poor with such small packets:

14.88Mpps * 40 bytes * 8 bits / 10Gbps = 47.6%

Calculating the Loading

We can calculate the loading for a given 10G link quite simply. If the average packet rate is 1.5Mpps, and the average packet size is 350 bytes, the ethernet frame size will be 350+18 = 368, and with overheads the average transmission size will be 368 + 20 = 388 bytes. So the loading will be:

1.5Mpps * 388 bytes * 8 bits / 10Gbps = 46.56%

On The Wire

At the physical level, ethernet traffic is broken into blocks and encoded before transmission, with the reverse happening on receipt. This process is invisible to both the user and the operating system, being handled in the network adapters at each end of the link. Standard 10G ethernet is sent serially over fibre, using on-off keying and 66b/64b encoding. So the actual bit rate transmitted is:

10G x (66 / 64) = 10.3125Gbps

40G ethernet is sent as 4 lanes of 10G data. Over multi-mode fibre, each of these lanes runs over a separate fibre pair, typically bundled into a MPO cable. Over single-mode fibre, the lanes are transmitted with different frequency lasers multiplexed onto the same fibre pair - known as CWDM (coarse wave-division multiplexing).

Latency Calculations

As data rates increase, the latency introduced by even fairly short lengths of fibre can become quite significant. They can be calculated as follows:

the speed of light in the fibre = the speed of light in a vacuum / the refractive index of the fibre

The speed of light in a vacuum is 3 x 10^8 m/s, and the refractive index of fibre is typically 1.5. So:

the speed of light in fibre = 3 x 10^8 / 1.5 = 2 x 10^8 m/s

And the delay per metre is:

delay = 1 / 2 x 10^8 m/s = 5ns per metre.

So two 10m lengths of fibre will introduce a 100ns latency, in addition to the delay through the NICs and switch as a packet moves from one server to another.

Volumetrics

When taking packet captures for analysis, it is helpful to have an estimate of how much data will be seen in a particular period, and how rapidly it will need to be written to disk. The table below shows the maximum amount of data that can be transmitted over different link types in varying periods of time, and the maximum per-second data rate that could be seen.