Bandwidth vs latency: what each one limits, and how to measure both on a server

Short answer: bandwidth is how much data a connection can carry per second, measured in Mbit/s or Gbit/s. Latency is how long one piece of data takes to cross it, measured in milliseconds, usually as a round trip. Bandwidth is the width of the road; latency is its length. A wider road moves more cars at once, but it does not bring any single car to its destination sooner — which is why a 10 Gbit/s server can still feel slow to a user on the other side of the world.

The two are independent, and most performance problems come from treating one as the other. This guide defines both precisely, shows where each comes from, works through the arithmetic that links them and ends with the commands that measure them on a Linux server.

What do bandwidth, throughput and latency mean?

Bandwidth is a property of the link — a 1 Gbit/s port can put at most one gigabit on the wire per second. Throughput is what you get through it end to end, and it is always lower: protocol overhead, the slowest link on the path, the other side’s limits and, above all, latency and loss eat into it. Note the units, too: network speeds are quoted in bits, file sizes in bytes, so a 1 Gbit/s link tops out at roughly 118 MB/s of payload, not 1,000.

Latency is normally quoted as round-trip time (RTT), the time from sending a packet to receiving the answer — what ping shows. One-way latency is harder to measure because it needs synchronised clocks on both ends, so in practice “latency” means RTT.

Where does latency come from?

Four delays add up on every hop of the path:

• Propagation: the signal’s travel time. Light in optical fibre moves at about 200,000 km/s, two-thirds of its speed in a vacuum, which works out to roughly 5 microseconds per kilometre. A 6,000 km fibre path therefore costs about 30 ms one way and 60 ms per round trip before any equipment does anything — and real cable routes are longer than the straight line on a map.
• Serialisation: the time to clock a packet onto the wire. A 1,500-byte packet is 12,000 bits: 12 µs on a 1 Gbit/s link, 1.2 ms on a 10 Mbit/s one. This is the only part of latency that more bandwidth reduces, and on fast links it is already negligible.
• Queuing: time spent waiting in router and switch buffers when a link is busy. Oversized buffers that fill up under load — bufferbloat — can add hundreds of milliseconds while a big upload or download runs.
• Processing: routers, firewalls and your own server inspecting and forwarding packets. Usually microseconds, but a saturated CPU or a long firewall ruleset adds more.

Propagation sets a floor no hardware can beat. The only ways to lower it are a shorter path — a server closer to its users — or fewer round trips per request.

Why can a fast connection still be slow?

Because a single TCP connection cannot send more than its window before it has to wait for an acknowledgement, and that wait is one round trip. The ceiling for one connection is therefore:

max throughput = window size / RTT

64 KiB window, 100 ms RTT:  65,536 B x 8 / 0.1 s  =  about 5.2 Mbit/s
64 KiB window,  10 ms RTT:  65,536 B x 8 / 0.01 s =  about 52 Mbit/s

Same window, same 1 Gbit/s port, ten times the distance, a tenth of the speed. Modern stacks use TCP window scaling (RFC 7323) and Linux grows the window automatically, so the 64 KiB case is now mostly found in old clients, middleboxes that strip the option and badly tuned applications — but the rule still holds for whatever window is in use.

Turned around, it gives the bandwidth-delay product (BDP): how much data must be in flight to fill a link. A 1 Gbit/s link with a 50 ms RTT needs 1,000,000,000 × 0.05 / 8 = 6.25 MB in flight. If either end’s buffers are smaller than that, the link is never full, however much bandwidth you buy. On Linux you can see the receive buffer limits with:

sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem

Packet loss makes it worse, and worse again the longer the path. Loss-based TCP congestion control cuts its sending rate on every loss and needs round trips to climb back, so its throughput falls roughly with RTT × √loss (the Mathis formula). Half a percent of loss barely matters across a city and cripples a transfer across an ocean. Loss-tolerant congestion control such as BBR, available in Linux since 4.9, helps on long paths:

sysctl -w net.core.default_qdisc=fq
sysctl -w net.ipv4.tcp_congestion_control=bbr
# make it permanent:
printf 'net.core.default_qdisc=fq\nnet.ipv4.tcp_congestion_control=bbr\n' > /etc/sysctl.d/90-bbr.conf

Which one matters for your workload?

A page load shows how quickly round trips add up. Before the first byte of HTML arrives, the browser needs a DNS lookup, a TCP handshake (one RTT) and a TLS handshake (one RTT with TLS 1.3, two with TLS 1.2), then the request itself — four or five round trips. At 100 ms RTT that is close to half a second spent waiting, on any bandwidth.

How do you measure each on a server?

Latency: ping gives the round trip; mtr shows it per hop together with loss, which tells you where delay or loss starts.

ping -c 20 example.com
mtr -rwzc 100 example.com     # report mode, 100 probes, AS numbers

Throughput: iperf3 between two machines you control measures what the path delivers, without disks or web servers in the way. Run it with several parallel streams to see whether one connection is window-limited, and in reverse to test the other direction.

# on the server
iperf3 -s
# on the client: 4 streams for 30 s, then the reverse direction
iperf3 -c SERVER_IP -P 4 -t 30
iperf3 -c SERVER_IP -P 4 -t 30 -R

Where an HTTP request spends its time: curl’s timing variables split a request into DNS, connect, TLS and server time.

curl -o /dev/null -s -w 'dns %{time_namelookup}  connect %{time_connect}  tls %{time_appconnect}  first byte %{time_starttransfer}  total %{time_total}\n' https://example.com/

If connect minus dns is large, the path is long. If first byte is far behind TLS, the server or the application is slow, not the network.

How do you reduce latency?

• Put the server near its users. Distance is the floor. For a mostly European audience a server in central Europe beats one on another continent, whatever its port speed; Frankfurt or Amsterdam is a question of which networks your users sit on, so measure from their side.
• Cut round trips. TLS 1.3, HTTP/2 or HTTP/3, keep-alive and connection reuse remove handshakes; fewer redirects and fewer third-party hosts remove whole lookups.
• Cache close to the user. A CDN answers static files from an edge location near the visitor, so only uncached requests travel to your server.
• Keep queues short. A fair-queuing qdisc (fq or fq_codel, the default on most current distributions) stops one bulk transfer from adding delay to every interactive connection on the same server.

How do you get more throughput?

• Check the plan’s port speed first — no tuning goes past it.
• Use parallel streams (rsync in several jobs, aria2c -x, multipart uploads) on long paths.
• Make sure buffers cover the bandwidth-delay product, and try BBR on lossy long-distance links.
• Compress what compresses: text, logs and database dumps shrink several times over.
• Rule out the disk and the CPU — a transfer from a slow disk or into a single-threaded encryption step is not a network problem.

Bandwidth and latency on a VMHeaven server

Each plan lists its port speed, from 200 Mbit/s on the smallest Standard KVM plan up to 10 Gbit/s on the largest, and traffic is fair use — never billed per gigabyte. Servers run in Amsterdam and Frankfurt. Latency depends on where your users are and which networks they use, so the honest test is your own: rent the smallest plan in each city for a month and run ping, mtr and curl between the servers and the places your users connect from, then keep the one that answers faster. The Frankfurt vs Amsterdam comparison shows how to read the output. Game servers and other latency-sensitive, single-thread workloads usually belong on Hi-CPU KVM.

Frequently asked