Путеводитель по Руководству Linux

  User  |  Syst  |  Libr  |  Device  |  Files  |  Other  |  Admin  |  Head  |



   rdma_cm    ( 7 )

коммуникационный менеджер RDMA (RDMA communication manager.)

Имя (Name)

rdma_cm - RDMA communication manager.

Синопсис (Synopsis)

#include <rdma/rdma_cma.h>

Описание (Description)

Used to establish communication over RDMA transports.

Примечание (Note)

The RDMA CM is a communication manager used to setup reliable,
       connected and unreliable datagram data transfers.  It provides an
       RDMA transport neutral interface for establishing connections.
       The API concepts are based on sockets, but adapted for queue pair
       (QP) based semantics: communication must be over a specific RDMA
       device, and data transfers are message based.

The RDMA CM can control both the QP and communication management (connection setup / teardown) portions of an RDMA API, or only the communication management piece. It works in conjunction with the verbs API defined by the libibverbs library. The libibverbs library provides the underlying interfaces needed to send and receive data.

The RDMA CM can operate asynchronously or synchronously. The mode of operation is controlled by the user through the use of the rdma_cm event channel parameter in specific calls. If an event channel is provided, an rdma_cm identifier will report its event data (results of connecting, for example), on that channel. If a channel is not provided, then all rdma_cm operations for the selected rdma_cm identifier will block until they complete.

The RDMA CM gives an option to different libibverbs providers to advertise and use various specific to that provider QP configuration options. This functionality is called ECE (enhanced connection establishment).


RDMA VERBS

The rdma_cm supports the full range of verbs available through
       the libibverbs library and interfaces.  However, it also provides
       wrapper functions for some of the more commonly used verbs
       funcationality.  The full set of abstracted verb calls are:

rdma_reg_msgs - register an array of buffers for sending and receiving

rdma_reg_read - registers a buffer for RDMA read operations

rdma_reg_write - registers a buffer for RDMA write operations

rdma_dereg_mr - deregisters a memory region

rdma_post_recv - post a buffer to receive a message

rdma_post_send - post a buffer to send a message

rdma_post_read - post an RDMA to read data into a buffer

rdma_post_write - post an RDMA to send data from a buffer

rdma_post_recvv - post a vector of buffers to receive a message

rdma_post_sendv - post a vector of buffers to send a message

rdma_post_readv - post a vector of buffers to receive an RDMA read

rdma_post_writev - post a vector of buffers to send an RDMA write

rdma_post_ud_send - post a buffer to send a message on a UD QP

rdma_get_send_comp - get completion status for a send or RDMA operation

rdma_get_recv_comp - get information about a completed receive


CLIENT OPERATION

This section provides a general overview of the basic operation
       for the active, or client, side of communication.  This flow
       assume asynchronous operation with low level call details shown.
       For synchronous operation, calls to rdma_create_event_channel,
       rdma_get_cm_event, rdma_ack_cm_event, and
       rdma_destroy_event_channel would be eliminated.  Abstracted
       calls, such as rdma_create_ep encapsulate several of these calls
       under a single API.  Users may also refer to the example
       applications for code samples.  A general connection flow would
       be:

rdma_getaddrinfo retrieve address information of the destination

rdma_create_event_channel create channel to receive events

rdma_create_id allocate an rdma_cm_id, this is conceptually similar to a socket

rdma_resolve_addr obtain a local RDMA device to reach the remote address

rdma_get_cm_event wait for RDMA_CM_EVENT_ADDR_RESOLVED event

rdma_ack_cm_event ack event

rdma_create_qp allocate a QP for the communication

rdma_resolve_route determine the route to the remote address

rdma_get_cm_event wait for RDMA_CM_EVENT_ROUTE_RESOLVED event

rdma_ack_cm_event ack event

rdma_connect connect to the remote server

rdma_get_cm_event wait for RDMA_CM_EVENT_ESTABLISHED event

rdma_ack_cm_event ack event

Perform data transfers over connection

rdma_disconnect tear-down connection

rdma_get_cm_event wait for RDMA_CM_EVENT_DISCONNECTED event

rdma_ack_cm_event ack event

rdma_destroy_qp destroy the QP

rdma_destroy_id release the rdma_cm_id

rdma_destroy_event_channel release the event channel

rdma_set_local_ece set desired ECE options

An almost identical process is used to setup unreliable datagram (UD) communication between nodes. No actual connection is formed between QPs however, so disconnection is not needed.

Although this example shows the client initiating the disconnect, either side of a connection may initiate the disconnect.


SERVER OPERATION

This section provides a general overview of the basic operation
       for the passive, or server, side of communication.  A general
       connection flow would be:

rdma_create_event_channel create channel to receive events

rdma_create_id allocate an rdma_cm_id, this is conceptually similar to a socket

rdma_bind_addr set the local port number to listen on

rdma_listen begin listening for connection requests

rdma_get_cm_event wait for RDMA_CM_EVENT_CONNECT_REQUEST event with a new rdma_cm_id

rdma_create_qp allocate a QP for the communication on the new rdma_cm_id

rdma_accept accept the connection request

rdma_ack_cm_event ack event

rdma_get_cm_event wait for RDMA_CM_EVENT_ESTABLISHED event

rdma_ack_cm_event ack event

Perform data transfers over connection

rdma_get_cm_event wait for RDMA_CM_EVENT_DISCONNECTED event

rdma_ack_cm_event ack event

rdma_disconnect tear-down connection

rdma_destroy_qp destroy the QP

rdma_destroy_id release the connected rdma_cm_id

rdma_destroy_id release the listening rdma_cm_id

rdma_destroy_event_channel release the event channel

rdma_get_remote_ece get ECe options sent by the client

rdma_set_local_ece set desired ECE options


Возвращаемое значение (Return value)

=  0   success

= -1 error - see errno for more details

Most librdmacm functions return 0 to indicate success, and a -1 return value to indicate failure. If a function operates asynchronously, a return value of 0 means that the operation was successfully started. The operation could still complete in error; users should check the status of the related event. If the return value is -1, then errno will contain additional information regarding the reason for the failure.

Prior versions of the library would return -errno and not set errno for some cases related to ENOMEM, ENODEV, ENODATA, EINVAL, and EADDRNOTAVAIL codes. Applications that want to check these codes and have compatibility with prior library versions must manually set errno to the negative of the return code if it is < -1.


Смотри также (See also)

rdma_accept(3), rdma_ack_cm_event(3), rdma_bind_addr(3),
       rdma_connect(3), rdma_create_ep(3), rdma_create_event_channel(3),
       rdma_create_id(3), rdma_create_qp(3), rdma_dereg_mr(3),
       rdma_destroy_ep(3), rdma_destroy_event_channel(3),
       rdma_destroy_id(3), rdma_destroy_qp(3), rdma_disconnect(3),
       rdma_event_str(3), rdma_free_devices(3), rdma_getaddrinfo(3),
       rdma_get_cm_event(3), rdma_get_devices(3), rdma_get_dst_port(3),
       rdma_get_local_addr(3), rdma_get_peer_addr(3),
       rdma_get_recv_comp(3), rdma_get_remote_ece(3),
       rdma_get_request(3), rdma_get_send_comp(3), rdma_get_src_port(3),
       rdma_join_multicast(3), rdma_leave_multicast(3), rdma_listen(3),
       rdma_migrate_id(3), rdma_notify(3), rdma_post_read(3)
       rdma_post_readv(3), rdma_post_recv(3), rdma_post_recvv(3),
       rdma_post_send(3), rdma_post_sendv(3), rdma_post_ud_send(3),
       rdma_post_write(3), rdma_post_writev(3), rdma_reg_msgs(3),
       rdma_reg_read(3), rdma_reg_write(3), rdma_reject(3),
       rdma_resolve_addr(3), rdma_resolve_route(3),
       rdma_get_remote_ece(3), rdma_set_option(3), mckey(1),
       rdma_client(1), rdma_server(1), rping(1), ucmatose(1), udaddy(1)