---

   nfs    ( 5 )

---

TRANSPORT METHODS

NFS clients send requests to NFS servers via Remote Procedure Calls, or RPCs. The RPC client discovers remote service endpoints automatically, handles per-request authentication, adjusts request parameters for different byte endianness on client and server, and retransmits requests that may have been lost by the network or server. RPC requests and replies flow over a network transport.

In most cases, the mount(8) command, NFS client, and NFS server can automatically negotiate proper transport and data transfer size settings for a mount point. In some cases, however, it pays to specify these settings explicitly using mount options.

Traditionally, NFS clients used the UDP transport exclusively for transmitting requests to servers. Though its implementation is simple, NFS over UDP has many limitations that prevent smooth operation and good performance in some common deployment environments. Even an insignificant packet loss rate results in the loss of whole NFS requests; as such, retransmit timeouts are usually in the subsecond range to allow clients to recover quickly from dropped requests, but this can result in extraneous network traffic and server load.

However, UDP can be quite effective in specialized settings where the networks MTU is large relative to NFSs data transfer size (such as network environments that enable jumbo Ethernet frames). In such environments, trimming the rsize and wsize settings so that each NFS read or write request fits in just a few network frames (or even in a single frame) is advised. This reduces the probability that the loss of a single MTU-sized network frame results in the loss of an entire large read or write request.

TCP is the default transport protocol used for all modern NFS implementations. It performs well in almost every conceivable network environment and provides excellent guarantees against data corruption caused by network unreliability. TCP is often a requirement for mounting a server through a network firewall.

Under normal circumstances, networks drop packets much more frequently than NFS servers drop requests. As such, an aggressive retransmit timeout setting for NFS over TCP is unnecessary. Typical timeout settings for NFS over TCP are between one and ten minutes. After the client exhausts its retransmits (the value of the retrans mount option), it assumes a network partition has occurred, and attempts to reconnect to the server on a fresh socket. Since TCP itself makes network data transfer reliable, rsize and wsize can safely be allowed to default to the largest values supported by both client and server, independent of the network's MTU size.

Using the mountproto mount option This section applies only to NFS version 2 and version 3 mounts since NFS version 4 does not use a separate protocol for mount requests.

The Linux NFS client can use a different transport for contacting an NFS server's rpcbind service, its mountd service, its Network Lock Manager (NLM) service, and its NFS service. The exact transports employed by the Linux NFS client for each mount point depends on the settings of the transport mount options, which include proto, mountproto, udp, and tcp.

The client sends Network Status Manager (NSM) notifications via UDP no matter what transport options are specified, but listens for server NSM notifications on both UDP and TCP. The NFS Access Control List (NFSACL) protocol shares the same transport as the main NFS service.

If no transport options are specified, the Linux NFS client uses UDP to contact the server's mountd service, and TCP to contact its NLM and NFS services by default.

If the server does not support these transports for these services, the mount(8) command attempts to discover what the server supports, and then retries the mount request once using the discovered transports. If the server does not advertise any transport supported by the client or is misconfigured, the mount request fails. If the bg option is in effect, the mount command backgrounds itself and continues to attempt the specified mount request.

When the proto option, the udp option, or the tcp option is specified but the mountproto option is not, the specified transport is used to contact both the server's mountd service and for the NLM and NFS services.

If the mountproto option is specified but none of the proto, udp or tcp options are specified, then the specified transport is used for the initial mountd request, but the mount command attempts to discover what the server supports for the NFS protocol, preferring TCP if both transports are supported.

If both the mountproto and proto (or udp or tcp) options are specified, then the transport specified by the mountproto option is used for the initial mountd request, and the transport specified by the proto option (or the udp or tcp options) is used for NFS, no matter what order these options appear. No automatic service discovery is performed if these options are specified.

If any of the proto, udp, tcp, or mountproto options are specified more than once on the same mount command line, then the value of the rightmost instance of each of these options takes effect.

Using NFS over UDP on high-speed links Using NFS over UDP on high-speed links such as Gigabit can cause silent data corruption.

The problem can be triggered at high loads, and is caused by problems in IP fragment reassembly. NFS read and writes typically transmit UDP packets of 4 Kilobytes or more, which have to be broken up into several fragments in order to be sent over the Ethernet link, which limits packets to 1500 bytes by default. This process happens at the IP network layer and is called fragmentation.

In order to identify fragments that belong together, IP assigns a 16bit IP ID value to each packet; fragments generated from the same UDP packet will have the same IP ID. The receiving system will collect these fragments and combine them to form the original UDP packet. This process is called reassembly. The default timeout for packet reassembly is 30 seconds; if the network stack does not receive all fragments of a given packet within this interval, it assumes the missing fragment(s) got lost and discards those it already received.

The problem this creates over high-speed links is that it is possible to send more than 65536 packets within 30 seconds. In fact, with heavy NFS traffic one can observe that the IP IDs repeat after about 5 seconds.

This has serious effects on reassembly: if one fragment gets lost, another fragment from a different packet but with the same IP ID will arrive within the 30 second timeout, and the network stack will combine these fragments to form a new packet. Most of the time, network layers above IP will detect this mismatched reassembly - in the case of UDP, the UDP checksum, which is a 16 bit checksum over the entire packet payload, will usually not match, and UDP will discard the bad packet.

However, the UDP checksum is 16 bit only, so there is a chance of 1 in 65536 that it will match even if the packet payload is completely random (which very often isn't the case). If that is the case, silent data corruption will occur.

This potential should be taken seriously, at least on Gigabit Ethernet. Network speeds of 100Mbit/s should be considered less problematic, because with most traffic patterns IP ID wrap around will take much longer than 30 seconds.

It is therefore strongly recommended to use NFS over TCP where possible, since TCP does not perform fragmentation.

If you absolutely have to use NFS over UDP over Gigabit Ethernet, some steps can be taken to mitigate the problem and reduce the probability of corruption:

Jumbo frames: Many Gigabit network cards are capable of transmitting frames bigger than the 1500 byte limit of traditional Ethernet, typically 9000 bytes. Using jumbo frames of 9000 bytes will allow you to run NFS over UDP at a page size of 8K without fragmentation. Of course, this is only feasible if all involved stations support jumbo frames.

To enable a machine to send jumbo frames on cards that support it, it is sufficient to configure the interface for a MTU value of 9000.

Lower reassembly timeout: By lowering this timeout below the time it takes the IP ID counter to wrap around, incorrect reassembly of fragments can be prevented as well. To do so, simply write the new timeout value (in seconds) to the file /proc/sys/net/ipv4/ipfrag_time.

A value of 2 seconds will greatly reduce the probability of IPID clashes on a single Gigabit link, while still allowing for a reasonable timeout when receiving fragmented traffic from distant peers.