QCP Asynchronous Connection Pool

The asynchronous connection pool (Quantum Connection Pool, QCP) provides connection and communication support to remote clients.

Operation Framework

QCP accepts incoming connections from remote clients at the address specified in listening_address (default is 0.0.0.0:8080) and forwards their traffic to the database servers listed in the servers section. Connections to the servers are established (and terminated) automatically, as needed.

Depending on the operating mode specified in relay_mode parameter, redirection of data traffic from clients to servers is performed in one of the following ways:

relay_mode: Session: when the client first accesses the server, a unique connection to the database is allocated, and it is returned to the pool only when the client disconnects.
relay_mode: Smart: when accessing the server, a unique connection to the database is allocated for the client, and it is returned to the pool only if the server's response contains the Idle flag (see section Message formats, ReadyForQuery message).

A full description of the settings can be found in the example configuration file qcp/config-example.yaml.

Limitations on QCP Usage

Using QCP in Smart mode may cause problems when using the following features:

set/reset;
listen;
with hold cursor;
prepare/deallocate;
preserve/delete rows temp tables;
load statement;
Session-level advisory locks.

Start and Operation

Starting is done using qcp (qcp --help for the list of parameters), stopping is done using qcp-ctrl (qcp-ctrl --help for the list of parameters). For example, to stop a running QCP instance, you need to execute qcp-ctrl quit command.

Output of log entries during QCP operation is controlled by the log_output parameter (see qcp/config-example.yaml), while the logging level is set by the log_level parameter (see the LevelFilter help page on the corresponding site for more information on logging levels). For example, to print log entries of level Info and higher to the file /tmp/qcp.log, you must set the following parameters in the configuration file:

log_level: Info
log_output:
    file: /tmp/qcp.log

Memory Consumption

When the program starts, the amount of memory specified in the arena section of the configuration file is allocated at once:

# Memory configuration (optional)
arena:
    chunk_size: 65 KB # Size of one memory chunk; supported suffixes are B, KB, MB, GB

    # The total amount of memory consumed can be specified either by using the parameter
    # "memory chunk count":
    chunks_count: 3150 # The number of such chunks
    # or by specifying the total size directly:
    total_size: 3.1 GB # Supported suffixes are B, KB, MB, GB

Connecting to Database System

QCP can connect to either a single DBMS server or a DBMS cluster managed by Patroni.

To connect to a DBMS server, you must specify its address in the server section, for example:

server:
    # Single server
    address: # Connection type and address (TCP or Unix domain socket)
      Tcp: "127.0.0.1:1234"
      # Unix: "/home/mexus/devel/optim/qhb-with-rust/build/dbsockets/.s.PGSQL.5432"

To connect to a DBMS cluster managed by Patroni:

server:
    # Patroni cluster
    patroni:
      # Servers (nodes) of the cluster
      nodes:
        - dbms_address: # DBMS address
            Tcp: "127.0.0.1:1234"
          rest_url: "http://127.0.0.1:8090" # URL REST API patroni
        - dbms_address: # DBMS address
            Unix: "/путь/к/сокету/unix"
          rest_url: "http://127.0.0.2:8090" # URL REST API patroni
      instance_name: my_cluster  # Cluster name (it is used as prefix for metrics).
      single_try_timeout: 5 sec # Optional parameter: maximum time limit for a single attempt
                                # to find out which server in the cluster is the master.
      master_check_interval: 1500 ms # Optional parameter: how often to poll the cluster.
                                     # Default is 1500 milliseconds.

You need to list all cluster nodes, specifying the DBMS address and the REST API Patroni address for each node. You also need to specify the cluster name using instance_name.

When connected in this way, QCP will regularly poll Patroni on all nodes in order to determine the current master. If the master changes, QCP will terminate all current queries from connected clients and connect to the new master.

You can also specify additional parameters:

single_try_timeout: time limit for each REST query,
master_check_interval: how often to poll the cluster to determine the current master.

Regardless of whether it is connection to a cluster or to a single DBMS, the following optional parameters are used to describe connections:

inactive_timeout: amount of time after which an inactive connection to the server will be closed, provided that there is a minimum number of connections.
lifetime: approximate maximum lifetime of a connection to the server, after which it will be closed.
heart_beat_interval: validity checking interval for a connection to the server during the period when the connection is not associated with any client.

Cluster Mode with QHB Agent

QCP supports cluster management for the QHB cluster via the qhb-agent agent. In this mode, QCP monitors the state of cluster nodes and issues commands to qhb-agent to switch QHB nodes to replica mode. Thus, on each node, QCP acts as the cluster coordinator, and qhb-agent as the command executor on each node.

qhb-agent is installed on each QHB cluster node from the Quantom repository. After installation, you need to create the configuration file /etc/qhb-agent/config.yaml:

listen: <host-ip-address>:<port>    # The address at which the agent receives commands from QCP
qhb:
    path: /usr/local/qhb          # The path to the QHB binaries
    port: 5432                    # The QHB port
    pgdata: /qhb/data             # The QHB data directory
    replica-user: <user>          # User for replication
    replica-pwd: <password>
    admin-user: <user>            # Administrative user
    db-name: <database-name>
    application-name: <node-name> # Unique node name in a cluster

Running and management are performed via systemd:

systemctl enable --now qhb-agent

Checking agent functionality are performed via HTTP API:

curl http://<address>:<port>/health

To connect QCP to a cluster via qhb-agent, use the qluster section in the server section:

server:
    qluster:
        cluster-name: qluster          # Cluster name
        name: <this-qcp-node-name>     # QCP ID in the cluster
        readonly-statements-poll-period: 15s
        api-listen: 0.0.0.0:<api-port> # The API port of QCP itself
        qhb:
            db-user: <user>
            db-name: <database-name>
            qhb-timeout: 2s            # Timeout of the operations with QHB 
            connection-lifetime: 1m    # Connection lifetime
            agent-timeout: 30s         # Command timeout for qhb-agent
            promote-wait: 1s           # Waiting after promote
        raft:
            vote-timeout: 12s
            accident-mode-timeout: 20s
            max-delay: 30s
            healthcheck-interval: 5s
        servers:
            - address:
                  Tcp: "<node-address-1>:<port>"
            - address:
                  Tcp: "<node-address-2>:<port>"

The servers section lists all nodes in the QHB cluster. QCP itself determines the current master and directs client connections to it. When changing the master, QCP terminates current connections, switches the affected nodes to replica mode via qhb-agent, and connects to the new master.

The raft section specifies the parameters of the consensus protocol used for coordination between cluster nodes:

vote-timeout — voting timeout when choosing a leader;
accident-mode-timeout — timeout before entering emergency mode;
max-delay — maximum delay of replication operations;
healthcheck-interval — node status check interval.

When using qluster, the connections section is set in the same way as in other modes (see the Section "Multi-user mode" below).

Procedure for Periodic Checking of Cluster State

Purpose

qcp_healthcheck is a stored procedure on the qhb side that is periodically called by the qcp node.

The procedure performs four tasks:

Participates in the election of the current qcp leader.
Manages the emergency mode lifetime.
Collects and returns gossip data from qcp nodes.
Returns the local state of qhb node: lsn and the replication source.

The implementation is divided into independent blocks. All state is stored only in the process's memory and protected by a shared mutual exclusion (mutex).

API

Table 1. Input Parameters

Parameter	Type	Description
qcp_name	text	The name of the qcp node that calls the procedure.
term	bigint	Election number / logical time.
timeout	int	Timeout in milliseconds. Used for leadership and gossip entry lifetime.
participate_in_elections	bool	Controls the participation of the calling qcp node in elections.
accident_mode_timeout	int	Accident mode timeout in milliseconds. A value greater than 0 enables or extends the mode.
gossip	bytea	Binary data that is stored for qcp_name and returned to other callers.

Table 2. Output Fields

The procedure returns one row.

Field	Type	Description
qcp_leader	text not null	The current qcp leader according to this qhb node. An empty string means there is no leader.
qcp_leader_expired_at	int not null	Unix timestamp (in milliseconds) when the current leader's term expires.
term	bigint not null	Current stored election number.
accident_mode_expired_at	int not null	Unix timestamp (in milliseconds) when the accident mode expired.
lsn	bigint not null	Current database lsn.
qhb_replication_source	text not null	host:port of the master from which the current qhb node is replicating. An empty string is returned for the master.
gossip_names	text[]	Names of qcp nodes for which there are relevant gossip entries.
gossip_values	bytea[]	The gossip values. The index corresponds to the same index in gossip_names.

Internal State

Table 3. qcp Leader Elections

Field	Description
qcp_leader	The current qcp leader. The default is an empty string.
qcp_leader_expired_at	Unix timestamp (in milliseconds) when the current leader's term expires.
current_term	Election number. The default is 0.

Table 4. The Accident Mode

Field	Description
accident_mode_expired_at	Unix timestamp (in milliseconds) when the accident mode expired.

Gossip

The state is stored in a hash map:

qcp_name -> (gossip, expired_at)

where:

qcp_name — the name of the qcp node;
gossip — binary data from the input parameter;
expired_at — Unix timestamp (in milliseconds) after which the entry is considered expired.

Behavior

qcp Leader Elections

If participate_in_elections = true, then the calling qcp node participates in the elections.

Algorithm:

If qcp_leader is an empty string, the procedure remembers a new leader:

qcp_leader = qcp_name
current_term = term
qcp_leader_expired_at = now() + timeout

If qcp_leader == qcp_name, the procedure extends the leadership period:
```
qcp_leader_expired_at = now() + timeout
```
If term > current_term, the procedure accepts the new term and remembers the new leader:
```
qcp_leader = qcp_name
current_term = term
qcp_leader_expired_at = now() + timeout
```

If qcp_leader_expired_at has expired, the procedure remembers a new leader:

qcp_leader = qcp_name
current_term = term
qcp_leader_expired_at = now() + timeout

If participate_in_elections = false, then the calling qcp node does not attempt to become the leader.

Algorithm:

If the current leader is the same as qcp_name, the leadership is reset:
```
qcp_leader = ""
current_term = 0
qcp_leader_expired_at = 0
```
If qcp_leader_expired_at has expired, leadership is also reset:
```
qcp_leader = ""
current_term = 0
qcp_leader_expired_at = 0
```

The Accident Mode

State:

accident_mode_expired_at

If accident_mode_timeout > 0 is passed as input, the procedure sets:

accident_mode_expired_at = now() + accident_mode_timeout

gossip Data Collection by qcp Nodes

If qcp_name != "", the procedure stores the passed gossip in a hash map:

gossip_map[qcp_name] = (gossip, now() + timeout)

Then the procedure iterates through all the values in the hash map and deletes entries whose expired_at has expired.

The current entries are returned as two arrays:

gossip_names  = [qcp_name_1, qcp_name_2, ...]
gossip_values = [gossip_1,   gossip_2,   ...]

gossip_names[i] matches gossip_values[i].

Returning the lsn and Replication Source

This procedure returns the current database lsn.

If the current qhb node is a replica, qhb_replication_source returns the host:port of the master from which it is replicating.

If the current qhb node is not a replica, qhb_replication_source returns an empty string.

Multi-User Mode

QCP supports the ability to connect to a DBMS cluster using various user-DB pairs.

A separate connection pool is created for each of these pairs. When a client connects to QCP, it specifies the user login and the database name, and, according to these parameters, QCP places it in the correct pool.

When attempting to connect to QCP with a user-database pair that is not appear in the configuration, the client will receive an error:

FATAL:  Unknown user name, database name, or invalid combination of user and database

In this case, the following message will appear in the QCP logs:

[qcp::relay_helper] [WARN] Client ... startup sequence failed: Unknown user-database pair: user "user", db "db"

User-DB pairs are listed in the connections section of the server section, for example:

      # First connection pool.
      - username: user # Username
        password: something # Optional parameter: password for authentication
        database: utilities # Optional parameter: database name.
                            # If not specified, the username will be used as the database name.
        backends: # Number of connections with given username and database
            initial: 1 # .. at pool start
            min: 1     # .. minimum
            max: 3     # .. maximum
        # Optional section. Describes how the client should be authenticated in QCP.
        # If this section is not specified, the same password as the one set for
        # the server, if any, will be used to authorize the user with given
        # username for given database in QCP.
        client_auth:
            # Authentication method: trust or md5.
            method: md5
            # Password for authorization. If not specified, the same password
            # as for connecting QCP to the database will be used.
            password: lol2
        options: # Optional parameter: additional connection parameters, passed to the database
            option_name: option_value
      # Second connection pool.
      - username: qhb # Username
        backends: # Number of connections with given username and database
            initial: 2 # .. at pool start
            min: 2     # .. minimum
            max: 5     # .. maximum

To describe each connection pool for a user-BD pair, you must specify username and database. The database field can be omitted, in which case it will be set to the same value as username.

In addition, you need to specify the limits on the number of connections to the DBMS in this pool, in the backends section:

initial specifies how many connections will be opened to the DBMS with this user-DB pair when QCP starts.
min — limit on the minimum number of connections to the DBMS, may be less than initial.
max — the maximum number of connections to the DBMS. Must be no less than initial and min.

If a password is required to connect to the DBMS with given user and database, it can be specified in the password field.

There is also an options section available that allows you to pass arbitrary command line arguments to the server in the form name: value.

Client Authorization in QCP

By default, for each user-DB pair, QCP uses the same settings for authorizing users as for authorizing QCP in the DBMS: if the DBMS requires a password, then QCP will also require a password from the user; if the DBMS does not require a password, then QCP will not require it too.

You can override this behavior using the client_auth section:

the method field allows you to override the authorization method,
the password field allows you to specify a password different from the password used to connect to the DBMS.

Applying Settings Without QCP Restarting

Some configuration parameters can be changed without restarting QCP. This method is called "reloading the configuration file".

To do this, you need to change the required parameters in the configuration file, then run

sudo systemctl reload qcp

If QCP is not running under systemd, you need to determine the QCP PID (contact your system administrator to find out how to do this on your OS), and then run the following command, substituting <pid> for the PID found:

sudo kill -USR1 <pid>

In order for the updated configuration to be applied, the configuration file must be valid. Otherwise, the QCP logs will show that the updated configuration cannot be applied, and QCP will continue to operate in the previous mode. In this case partial configuration application is impossible.

If the application is possible, a corresponding entry will appear in the log:

[INFO] Reloaded configuration file /путь/к/config.yaml

Note that applying the updated configuration takes some time, which can reach several seconds under high load.

Applying the connection parameters for each user-database pair is accompanied by log entries:

[INFO] Updating connection configuration for user "qhb", db "qhb"
[INFO] Applied new connection parameters for user "qhb", db "qhb"

Applying the incoming connection parameters is accompanied by log entry:

[INFO] Incoming parameters updated

The following parameters cannot be changed without restarting QCP, so they will be ignored by QCP when reloading the configuration file:

arena
log_level
log_output
log_colouring
log_timestamps
worker_threads

The following parameters will be updated the next time you use them:

accept_clients_no_backends: the parameter will be updated when the next client connects, or when a connection to the DBMS becomes available if accepting of new connections from clients has been suspended.
in the servers_retry_back_off section, the min_wait, max_wait and max_wait_at parameters will be updated at the next attempt to establish a connection to the DBMS.
in the server section, the inactive_timeout, lifetime, check_interval and heart_beat_interval parameters will be applied to connections to the DBMS established after reloading the configuration file.

Changing the type of DBMS connection (single server, cluster managed by Patroni) will disconnect all current clients and break all current connections to the DBMS.

Removing a connection from the connections section of the server section will disconnect all clients using such connections and break all such connections to the DBMS.

Changing username, database, password or options for a connection will result in all clients using such connections being disconnected, and all such connections to the DBMS being broken and then reconnected to the DBMS with the updated parameters.

Changing the client_auth field for a connection will be applied to all subsequent connections to QCP.