Installing VerneMQ

VerneMQ is a high-performance, distributed MQTT message broker. It scales horizontally and vertically on commodity hardware to support a high number of concurrent publishers and consumers while maintaining low latency and fault tolerance. To use it, all you need to do is install the VerneMQ package.

Choose your OS and follow the instructions:

• CentOS/RHEL

• Debian/Ubuntu

It is also possible to run VerneMQ using our Docker image:

• Docker

Starting VerneMQ

To start a VerneMQ broker, use the vernemq start command in your Shell:

vernemq start

A successful start will return no output. If there is a problem starting the broker, an error message is printed to STDERR.

To run VerneMQ with an attached interactive Erlang console:

vernemq console

A VerneMQ broker is typically started in console mode for debugging or troubleshooting purposes. Note that if you start VerneMQ in this manner, it is running as a foreground process that will exit when the console is closed.

You can close the console by issuing this command at the Erlang prompt:

q().

Once your broker has started, you can initially check that it is running with the vernemq ping command:

vernemq ping

The command will respond with pong if the broker is running or Node <NodeName> not responding to pings in case it’s not.

Start a VerneMQ cluster node

docker run --name vernemq1 -d erlio/docker-vernemq

Somtimes you need to configure a forwarding for ports (on a Mac for example):

docker run -p 1883:1883 --name vernemq1 -d erlio/docker-vernemq

This starts a new node that listens on 1883 for MQTT connections and on 8080 for MQTT over websocket connections. However, at this moment the broker won't be able to authenticate the connecting clients. To allow anonymous clients use the DOCKER_VERNEMQ_ALLOW_ANONYMOUS=on environment variable.

docker run -e "DOCKER_VERNEMQ_ALLOW_ANONYMOUS=on" --name vernemq1 -d erlio/docker-vernemq

Autojoining a VerneMQ cluster

This allows a newly started container to automatically join a VerneMQ cluster. Assuming you started your first node like the example above you could autojoin the cluster (which currently consists of a single container 'vernemq1') like the following:

docker run -e "DOCKER_VERNEMQ_DISCOVERY_NODE=<IP-OF-VERNEMQ1>" --name vernemq2 -d erlio/docker-vernemq

(Note, you can find the IP of a docker container using docker inspect <CONTAINER_NAME> | grep \"IPAddress\").

Checking cluster status

To check if the above containers have successfully clustered you can issue the vmq-admin command:

docker exec vernemq1 vmq-admin cluster show
+--------------------+-------+
|        Node        |Running|
+--------------------+-------+
|VerneMQ@172.17.0.151| true  |
|VerneMQ@172.17.0.152| true  |
+--------------------+-------+

Authentication

VerneMQ comes with a simple file-based password authentication mechanism which is enabled by default. If you don't need this it can be disabled by setting:

plugins.vmq_passwd = off

Per default VerneMQ doesn't accept any client that hasn't been configured using vmq-passwd. If you want to change this and accept any client connection you can set:

allow_anonymous = on

In a production setup we recommend to use the provided password based authentication mechanism or implement your own authentication plugins.

VerneMQ periodically checks the specified password file.

vmq_passwd.password_file = /etc/vernemq/vmq.passwd

The check interval defaults to 10 seconds and can also be defined in the vernemq.conf.

vmq_passwd.password_reload_interval = 10

Setting the password_reload_interval = 0 disables automatic reloading.

Manage Password Files for VerneMQ

vmq-passwd is a tool for managing password files for the VerneMQ broker. Usernames must not contain ":", passwords are stored in similar format to crypt(3).

How to use vmq-passwd

vmq-passwd [-c | -D] passwordfile username

vmq-passwd -U passwordfile

Options

-c

Creates a new password file. If the file already exists, it will be overwritten.

-D

Deletes the specified user from the password file.

-U

This option can be used to upgrade/convert a password file with plain text passwords into one using hashed passwords. It will modify the specified file. It does not detect whether passwords are already hashed, so using it on a password file that already contains hashed passwords will generate new hashes based on the old hashes and render the password file unusable. Note, with this option neither usernames or passwords may contain ":".

passwordfile

The password file to modify.

username

The username to add/update/delete.

Examples

Add a user to a new password file: (you can choose an arbitrary name for the password file, it only has to match the configuration in the VerneMQ configuration file).

vmq-passwd -c /etc/vernemq/vmq.passwd henry

Delete a user from a password file

vmq-passwd -D /etc/vernemq/vmq.passwd henry

Acknowledgements

The original version of vmq-passwd was developed by Roger Light (roger@atchoo.org).

vmq-passwd includes :

• software developed by the [OpenSSL

  Project](http://www.openssl.org/) for use in the OpenSSL Toolkit.

• cryptographic software written by Eric Young

  (eay@cryptsoft.com)

• software written by Tim Hudson (tjh@cryptsoft.com)

Authorization

VerneMQ comes with a simple ACL based authorization mechanism which is enabled by default. If you don't need this it can be disabled by setting:

plugins.vmq_acl = off

VerneMQ periodically checks the specified ACL file.

vmq_acl.acl_file = /etc/vernemq/vmq.acl

The check interval defaults to 10 seconds and can also be defined in the vernemq.conf.

vmq_acl.acl_reload_interval = 10

Setting the acl_reload_interval = 0 disables automatic reloading.

Managing the ACL entries

Topic access is added with lines of the format:

topic [read|write] <topic>

The access type is controlled using read or write. If not provided then read an write access is granted for the topic. The topic can use the MQTT subscription wildcards + or #.

The first set of topics are applied to all anonymous clients (assuming allow_anonymous = on). User specific ACLs are added after a user line as follows (this is the username not the client id):

user <username>

It is also possible to define ACLs based on pattern substitution within the the topic. The form is the same as for the topic keyword, but using pattern as the keyword.

pattern [read|write] <topic>

The patterns available for substitution are:

• %c to match the client id of the client

• %u to match the username of the client

The substitution pattern must be the only text for that level of hierarchy. Pattern ACLs apply to all users even if the user keyword has previously been given.

Example:

pattern write sensor/%u/data

Simple ACL Example

# ACL for anonymous clients
topic bar
topic write foo
topic read all


# ACL for user 'john'
user john
topic foo
topic read baz
topic write all

Anonymous users are allowed to

• publish & subscribe to topic bar.

• publish to topic foo.

• subscribe to topic all.

User john is allowed to

• publish & subscribe to topic foo.

• subscribe to topic baz.

• publish to topic all.

Install VerneMQ

Once you have downloaded the binary package, execute the following command to install VerneMQ:

sudo yum install vernemq-<VERSION>.centos7.x86_64.rpm

or:

sudo rpm -Uvh vernemq-<VERSION>.centos7.x86_64.rpm

Activate VerneMQ node

Once you've installed VerneMQ, start it on your node:

service vernemq start

Verify your installation

You can verify that VerneMQ is successfully installed by running:

rpm -qa | grep vernemq

If VerneMQ has been installed successfully vernemq is returned.

Next Steps

Now that you've installed VerneMQ, check out How to configure VerneMQ.

Retry Interval

Set the time in seconds after a QoS=1 or QoS=2 message has been sent that VerneMQ will wait before retrying when no response is received.

retry_interval = 20

This option default to 20 seconds.

Inflight Messages

This option defines the maximum number of QoS 1 or 2 messages that can be in the process of being transmitted simultaneously.

max_inflight_messages = 20

Defaults to 20 messages, use 0 for no limit. The inflight window serves as a protection for sessions, on the incoming side.

Load Shedding

The maximum number of messages to hold in the queue above those messages that are currently in flight. Defaults to 1000. Set to -1 for no limit. This option protects a client session from overload by dropping messages (of any QoS).

max_online_messages = 1000

Defaults to 1000 messages, use -1 for no limit. This parameter was named max_queued_messages in 0.10.*. Note that 0 will totally block message delivery from any queue!

This option specifies the maximum number of QoS 1 and 2 messages to hold in the offline queue.

max_offline_messages = 1000

Defaults to 1000 messages, use -1 for no limit, use 0 if no messages should be stored.

In contrast to the session based inflight window, max_online_messages and max_offline_messages serves as a protection of queues, on the outgoing side.

Listeners specify on which IP address and port VerneMQ should accept new incoming connections. Depending on the chosen transport (TCP, SSL, WebSocket) different configuration parameters have to be provided. VerneMQ allows to write the listener configurations in a hierarchical manner, enabling very flexible setups. VerneMQ applies reasonable defaults on the top level, which can be of course overridden if needed.

# defines the default nr of allowed concurrent 
# connections per listener
listener.max_connections = 10000

# defines the nr. of acceptor processes waiting
# to concurrently accept new connections
listener.nr_of_acceptors = 10

# used when clients of a particular listener should
# be isolated from clients connected to another 
# listener.
listener.mountpoint = off

These are the only default parameters that are applied for all transports, and the only one that are of interest for plain TCP and WebSocket listeners.

These global defaults can be overridden for a specific transport protocol listener.tcp.CONFIG = VAL, or even for a specific listener listener.tcp.LISTENER.CONFIG = VAL. The placeholder LISTENER is freely chosen and is only used as a reference for further configuring this particular listener.

Mountpoints

Normally, an MQTT broker hosts one single topic tree. This means that all topics are accessible to all publishers and subscribers (limited by the ACLs you configured, of course). Mountpoints are a way to host multiple topic trees in a single broker. They are completely separated and clients with different topic trees cannot publish messages to each other. This could be useful if you provide MQTT services to multiple separated use cases/verticals or clients, with a single broker. Note that mountpoints are configured via different listeners. As a consequence, the MQTT clients will have to connect to a specific port to connect to a specific topic space (mountpoint).

The mountpoints can be configured on the protocol level or configurred or overridden on the specific listener level.

listener.ssl.mountpoint = ssl-mountpoint

listener.tcp.listener1.mountpoint = tcp-listener1
listener.tcp.listener2.mountpoint = tcp-listener2

Allowed protocol versions

Since VerneMQ 1.5.0 it is possible to configure which MQTT protocol versions as listener will accept.

VerneMQ supports MQTT 3.1, 3.1.1, and 5.0 (since VerneMQ 1.6.0). To allow these protocol versions, set:

listener.tcp.allowed_protocol_versions = 3,4,5

Here 3,4,5 are the protocol level versions corresponding to MQTT 3.1, 3.1.1 and 5.0 respectively. The default value is 3,4 thus allowing MQTT 3.1 and 3.1.1, while MQTT 5.0 is disabled.

Sample Config

Listen on TCP port 1883 and for WebSocket Connections on port 8888:

listener.tcp.default = 127.0.0.1:1883
listener.ws.default = 127.0.0.1:8888

An additional listener can be added by using a different name. In the example above the name equals to default and can be used for further configuring this particular listener. The following example demonstrates how an additional listener is defined as well as how the maximum number of connections can be limited for this listener:

listener.tcp.my_other = 127.0.0.1:18884
listener.tcp.my_other.max_connections = 100

PROXY protocol

VerneMQ listeners can be configured to accept connections from a proxy server that supports the PROXY protocol. This enables VerneMQ to retrieve peer information such as source IP/Port but also PROXY Version 2 protocol TLS client certificate details if the proxy was used to terminate TLS.

To enable the PROXY protocol for tcp listeners use listener.tcp.proxy_protocol = on or for a specific listener use listener.tcp.LISTENER.proxy_protocol = on.

If client certificates are used you can set listener.tcp.proxy_protocol_use_cn_as_username = on which will overwrite the MQTT username set by the client with the common name from the client certificate before authentication and authorization is performed.

Sample SSL Config

Accepting SSL connections on port 8883:

listener.ssl.cafile = /etc/ssl/cacerts.pem
listener.ssl.certfile = /etc/ssl/cert.pem
listener.ssl.keyfile = /etc/ssl/key.pem

listener.ssl.default = 127.0.0.1:8883

If you want to use client certificates to authenticate your clients you have to set the following option:

listener.ssl.require_certificate = on

If you use client certificates and want to use the certificates CN value as a username you can set:

listener.ssl.use_identity_as_username = on

Both options require_certificate and use_identity_as_username default to off.

The same configuration options can be used for securing WebSocket connections, just use wss as the protocol identifier e.g. listener.wss.require_certificate.

Maximum Client Id Size

Set the maximum size for client ids, MQTT v3.1 specifies a limit of 23 characters.

max_client_id_size = 23

This option default to 23.

Persistent Client Expiration

This option allows persistent clients (those with clean_session set to false) to be removed if they do not reconnect within a certain time frame.

The expiration period should be an integer followed by one of h, d, w, m, y for hour, day, week, month, and year; or never:

persistent_client_expiration = 1w

This option defaults to never.

Message Size Limit

Limit the maximum publish payload size in bytes that VerneMQ allows. Messages that exceed this size won't be accepted.

max_message_size = 0

Defaults to 0, which means that all valid messages are accepted. MQTT specification imposes a maximum payload size of 268435455 bytes.

VerneMQ supports the WebSocket protocol out of the box. To be able to open a WebSocket connection to VerneMQ, you have to configure a WebSocket listener or Secure WebSocket listener in the vernemq.conf file first:

listener.ws.default = 127.0.0.1:9001

listener.wss.default = 127.0.0.1:9002

Keep in mind that you'll use MQTT-over-WebSocket, so you will need a Javascript library that implements the MQTT client behaviour. We have used the as well as

You won't be able to open WebSocket connections on a base URL, always add the /mqtt path.

There are a couple of hidden options you can set in the vernemq.conf file. Hidden means that you have to add and set the value explicitly. Hidden options still have default values. Changing them should be considered advanced, possibly with the exception of setting a max_message_rate.

Queue Deliver mode

Specify how the queue should deliver messages when multiple sessions are allowed. In case of fanout all the attached sessions will receive the message, in case of balance an attached session is choosen randomly.

queue_deliver_mode = balance

Queue Type

Specify how queues should process messages, either the fifo or lifo way. Default is fifo.

queue_type = fifo

Max Message Rate

Specifies the maximum incoming publish rate per session per second. Depending on the underlying network buffers this rate isn't enforced. Defaults to 0, which means no rate limits apply. Setting to a value of 2 limits any publisher to 2 messages per second, for instance.

max_message_rate = 2

Max Drain Time

Due to the eventual consistent nature of the subscriber store it is possible that during queue migration messages still arrive on the old cluster node. This parameter enables to compensate this by keeping the queue around for some time (in seconds) after it was migrated to the other cluster node.

max_drain_time = 20

Max Msgs per Drain Step

Specifies the number of messages that are delivered to the remote node per drain step. A large value will provide a faster migration of a queue, but increases the waste of bandwidth in case the migration fails.

max_msgs_per_drain_step = 1000

Default Reg View

Allows to select a new default reg_view. A reg_view is a pre-defined way to route messages. Multiple views can be loaded and used, but one has to be selected as a default. The default routing is vmq_reg_trie, i.e. routing via the built-in trie data structure.

vmq_reg_view = "vmq_reg_trie"

Reg Views

A list of views that are started during startup. It's only used in plugins that want to choose dynamically between routing reg_views.

reg_views = "[vmq_reg_trie]"

Outgoing Clustering Buffer Size

An integer specifying how many bytes are buffered in case the remote node is not available. Default is 10000

outgoing_clustering_buffer_size = 15000

Sometimes consumers get overwhelmed by the number of messages they receive. VerneMQ can load balance between multiple consumer instances subscribed to the same topic with the same ClientId.

Enabling Session Balancing

To enable session balancing, activate the following two settings in vernemq.conf

 allow_multiple_sessions = on
 queue_deliver_mode = balance

Console Logging

Where should VerneMQ emit the default console log messages (which are typically at info severity):

log.console = off | file | console | both

VerneMQ defaults to log the console messages to a file, which can specified by:

log.console.file = /path/to/log/file

This option defaults to /var/log/vernemq/console.log for Ubuntu, Debian, RHEL and Docker installs.

The default console logging level info could be setting one of the following:

log.console.level = debug | info | warning | error

Error Logging

VerneMQ log error messages by default. One can change the default behaviour by setting:

log.error = on | off

VerneMQ defaults to log the error messages to a file, which can specified by:

log.error.file = /path/to/log/file

This option defaults to /var/log/vernemq/error.log for Ubuntu, Debian, RHEL and Docker installs.

Crash Logging

VerneMQ log crash messages by default. One can change the default behaviour by setting:

log.crash = on | off

VerneMQ defaults to log the crash messages to a file, which can specified by:

log.crash.file = /path/to/log/file

This option defaults to /var/log/vernemq/crash.log for Ubuntu, Debian, RHEL and Docker installs.

The maximum sizes in bytes of inidividual messages in the crash log defaults to 64KB but can be specified by:

log.crash.maximum_message_size = 64KB

VerneMQ rotate crash logs. By default, the crash log file is rotated at midnight or when the size exceeds 10MGB. This behaviour can be changed by setting:

## Acceptable values:
##   - a byte size with units, e.g. 10GB
log.crash.size = 10MB

## For acceptable values see https://github.com/basho/lager/blob/master/README.md#internal-log-rotation
log.crash.rotation = $D0

The default number of rotated log files is 5 and can be set with the option:

log.crash.rotation.keep = 5

SysLog

VerneMQ supports logging to SysLog, enable it by setting:

log.syslog = on

Logging to SysLog is disabled by default.

VerneMQ uses Google's LevelDB as a fast storage backend for messages and subscriber information. Each VerneMQ node runs its own embedded LevelDB store.

Configuration of LevelDB memory

There's not much you need to know about LevelDB and VerneMQ. One really important thing to note is that LevelDB manages its own memory. This means that VerneMQ will not allocate and free memory for LevelDB. Instead you'll have to configure a configuration value in vernemq.conf that tells LevelDB how much memory it can use up.

Configuring LevelDB memory:

leveldb.maximum_memory.percent = 20

Many aspects of VerneMQ can be extended using plugins. The standard VerneMQ package comes with several official plugins. You can show the enabled & running plugins via:

vmq-admin plugin show

The command above displays all the enabled plugins together with the hooks they implement:

+-----------+-----------+-----------------+-----------------------------+
|  Plugin   |   Type    |     Hook(s)     |            M:F/A            |
+-----------+-----------+-----------------+-----------------------------+
|vmq_passwd |application|auth_on_register |vmq_passwd:auth_on_register/5|
|  vmq_acl  |application| auth_on_publish |  vmq_acl:auth_on_publish/6  |
|           |           |auth_on_subscribe| vmq_acl:auth_on_subscribe/3 |
+-----------+-----------+-----------------+-----------------------------+

Enable a plugin

vmq-admin plugin enable --name=vmq_acl

This enables the ACL plugin. Because the vmq_acl plugin is already started the above command won't succeed. In case the plugin sits in an external directory you must also to provide the --path=PathToPlugin.

Disable a plugin

vmq-admin plugin disable --name=vmq_acl

Persisting plugins

To make a plugin start when VerneMQ starts they need to be configured in the main vernemq.conf file.

The general syntax to enable a plugin is to add a line like plugins.pluginname = on, using the vmq_passwd plugin as an example:

plugins.vmq_passwd = on

And if the plugin is external the path can be specified like this:

plugins.myplugin = on
plugins.myplugin.path = /path/to/plugin

Plugin specific settings can be configured via myplugin.somesetting = value, like:

vmq_passwd.password_file = ./etc/vmq.passwd

See the vernemq.conf file for details.

A shared subscription is a mechanism for distributing messages to a set of subscribers to shared subscription topic, such that each message is received by only one subscriber. This contrasts with normal subscriptions where each subscriber will receive a copy of the published message.

A shared subscription is on the form $share/sharename/topic and subscribers to this topic will receive messages published to the topic topic. The messages will be distributed according to the defined distribution policy.

Configuration

Currently three message distribution policies for shared subscriptions are supported: prefer_local, random and local_only. Under the random policy messages will be published to a random member of the shared subscription, if any exist. Under the prefer_local policy messages will be delivered to a random node-local member of the shared subscription, if none exist, the message will be delivered to a random member of the shared subscription on a remote cluster node. Under the local_only policy message will be delivered to a random node-local member of the shared subscription.

shared_subscription_policy = prefer_local

When a messages is being delivered to subscribers of a shared subscription, the message will be delivered to an online subscriber if possible, otherwise the message will be delivered to an offline subscriber.

Examples

Subscriptions Note: When subscribing to a shared topic, make sure to escape the $

So, for dash or bash shells

mosquitto_sub -h mqtt.example.io -p 1883 -q 2 -t \$share/group/topicname
mosquitto_sub -h mqtt.example.io -p 1883 -q 2 -t \$share/group/topicname/#

Publishing Note: When publishing to a shared topic, do not include the prefix $share/group/ as part of the publish topic name

mosquitto_pub -h mqtt.example.io -p 1883 -t topicname -m "This is a test message"
mosquitto_pub -h mqtt.example.io -p 1883 -t topicname/group1 -m "This is a test message"

VerneMQ can be easily clustered. Clients can then connect to any cluster node and receive messages from any other cluster nodes. However, the MQTT specification gives certain guarantees that are hard to fulfill in a distributed environment, especially when network partitions occur. We'll discuss the way VerneMQ deals with network partitions in its

A note on statefulness

Before you go ahead and experience the full power of clustering VerneMQ, be aware of its stateful character. An MQTT broker is a stateful application and a VerneMQ cluster is a stateful cluster.

What does this mean in detail? It means that clustered VerneMQ nodes will share information about connected clients and sessions but also meta-information about the cluster itself.

For instance, if you stop a cluster node, the VerneMQ cluster will not just forget about it. It will know that there's a node missing and it will keep looking for it. It will know there's a netsplit situation and it will heal the partition when the node comes back up. But if the missing nodes never comes back there's an eternal netsplit. (still resolvable by making the missing nodes explicitly leave).

This doesn't mean that a VerneMQ cluster cannot dynamically grow and shrink. But it means you have to tell the cluster what you intend to do, by using join and leave commands.

If you want a cluster node to leave the cluster, well... use the vmq-admin cluster leave command. If you want a node to join a cluster, well... use the vmq-admin cluster join command.

Makes sense? Go ahead and create your first VerneMQ cluster!

Joining a Cluster

vmq-admin cluster join discovery-node=<OtherClusterNode>

Leaving a Cluster

vmq-admin cluster leave node=<NodeThatShouldGo> (only the first step!)

Detailed Cluster Leave, Case A: Make a live node leave

A cluster leave will actually do a lot more work, and gives you some options to choose. The node leaving the cluster will go to great length trying to migrate its existing queues to other nodes. As queues (online or offline) are live processes in a VerneMQ node, it will only exit after it has migrated them.

Let's look at the steps in detail:

1. vmq-admin cluster leave node=<NodeThatShouldGo>

This first step will only stop the MQTT Listeners of the node to ensure that no new connections are accepted. It will not interrupt the existing connections, and behind the scenes the node will not leave the cluster yet. Existing clients are still able to publish and receive messages at this point.

The idea is to give a grace period with the hope that existing clients might re-connect (to another node). If you have decided that this period is over (after 5 minutes or 1 day is up to you), you proceed with step 2: disconnecting the rest of the clients.

1. vmq-admin cluster leave node=<NodeThatShouldGo> -k

The -k flag will delete the MQTT Listeners of the leaving node, taking down all live connections. If this is what you want from the beginning, you can do this right away as a first step.

Now, queue migration is triggered by clients re-connecting to other nodes. They will claim their queue and it will get migrated. Still, there might be some offline queues remaining on the leaving node, because they were pre-existing or because some clients do not re-connect and do not reclaim their queues.

VerneMQ will throw an exception if there are remaining offline queues after a configurable timeout. The default is 60 seconds, but you can set it as an option to the cluster leave command. As soon as the exception shows in console or console.log, you can actually retry the cluster leave command (including setting a migration timeout (-t), and an interval in seconds (-i) indicating how often information on the migration progress should be printed to the console.log):

1. vmq-admin cluster leave node=<NodeThatShouldGo> -k -i 5 -t 120

After this timeout VerneMQ will forcefully migrate the remaining offline queues to other cluster nodes in a round robin manner. After doing that, it will stop the leaving VerneMQ node.

Detailed Cluster Leave, Case B: Make a stopped node leave

So, case A was the happy case. You left the cluster with your node in a controlled manner, and everything worked, including a complete queue (and message) transfer to other nodes.

Let's look at the second possibility where the node is already down. Your cluster is still counting on it though and possibly blocking new subscription for that reason, so you want to make the node leave.

To do this, use the same command(s) as in the first case. There is one important consequence to note: by making a stopped node leave, you basically throw away persistant queue content, as VerneMQ won't be able to migrate or deliver it.

Let's repeat that to make sure:

Getting Cluster Status Information

vmq-admin cluster show

This section elaborates how a VerneMQ cluster deals with network partitions (aka. netsplit or split brain situation). A netsplit is mostly the result of a failure of one or more network devices resulting in a cluster where nodes can no longer reach each other.

VerneMQ is able to detect a network partition, and by default it will stop serving CONNECT, PUBLISH, SUBSCRIBE, and UNSUBSCRIBE requests. A properly implemented client will always resend unacked commands and messages are therefore not lost (QoS 0 publishes will be lost). However, the time window between the network partition and the time VerneMQ detects the partition much can happen. Moreover, this time frame will be different on every participating cluster node. In this guide we're referring to this time frame as the Window of Uncertainty.

Possible Scenario for Message Loss:

VerneMQ follows an eventually consistent model for storing and replicating the subscription data. This also includes retained messages.

Due to the eventually consistent data model it is possible that during the Window of Uncertainty a publish won't take into account a subscription made on a remote node (in another partition). Obviously, VerneMQ can't deliver the message in this case. The same holds for delivering retained messages to remote subscribers.

last will messages that are triggered during the Window of Uncertainty will be delivered to the reachable subscribers. Currently during a netsplit, but after the Window of Uncertainty last will messages will be lost.

Possible Scenario for Duplicate Clients:

Normally, client registration is synchronized using an elected leader node for the given client id. Such a synchronization removes the race condition between multiple clients trying to connect with the same client id on different nodes. However, during the Window of Uncertainty it is currently possible that VerneMQ fails to disconnect a client connected to a different node. Although this scenario sounds like artificially crafted it is possible to end up with duplicate clients connected to the cluster.

Recovering from a Netsplit

As soon as the partition is healed, and connectivity reestablished, the VerneMQ nodes replicate the latest changes made to the subscription data. This includes all the changes 'accidentally' made during the Window of Uncertainty. Using Dotted Version Vectors VerneMQ ensures that convergence regarding subscription data and retained messages is eventually reached.

VerneMQ uses the Erlang distribution mechanism for most inter-node communication. VerneMQ identifies other machines in the cluster using Erlang identifiers (e.g. VerneMQ@10.9.8.7). Erlang resolves these node identifiers to a TCP port on a given machine via the Erlang Port Mapper daemon (epmd) running on each cluster node.

By default, epmd binds to TCP port 4369 and listens on the wildcard interface. For inter-node communication, Erlang uses an unpredictable port by default; it binds to port 0, which means the first available port.

For ease of firewall configuration, VerneMQ can be configured to instruct the Erlang interpreter to use a limited range of ports. For example, to restrict the range of ports that Erlang will use for inter-Erlang node communication to 6000-7999, add the following lines to vernemq.conf on each VerneMQ node:

erlang.distribution.port_range.minimum = 6000
erlang.distribution.port_range.maximum = 7999

The settings above are only used for distributing subscription updates and maintenance messages. For distributing the 'real' MQTT messages the proper vmq listener must be configured in the vernemq.conf.

listener.vmq.clustering = 0.0.0.0:44053

Attributions:

This section, "VerneMQ Inter-node Communication", is a derivative of Security and Firewalls by Riak, used under Creative Commons Attribution 3.0 Unported License.

On every VerneMQ node you'll find the vmq-admin command line tool in the release's bin directory. It has different sub-commands that let you check for status, start and stop listeners, re-configure values and a couple of other administrative tasks.

VerneMQ can be monitored in several ways. We implemented native support for Graphite, MQTT $SYS tree, and Prometheus.

The metrics are also available via the command line tool:

vmq-admin metrics show

Or with:

vmq-admin metrics show -d

Which will output the metrics together with a short description describing what the metric is about. An example looks like:

# The number of AUTH packets received.
counter.mqtt_auth_received = 0

# The number of times a MQTT queue process has been initialized from offline storage.
counter.queue_initialized_from_storage = 0

# The number of PUBLISH packets sent.
counter.mqtt_publish_sent = 10

# The number of bytes used for storing retained messages.
gauge.retain_memory = 21184

Notice that the metrics:

mqtt_connack_not_authorized_sent
mqtt_connack_bad_credentials_sent
mqtt_connack_server_unavailable_sent
mqtt_connack_identifier_rejected_sent
mqtt_connack_unacceptable_protocol_sent
mqtt_connack_accepted_sent

Are no longer used (always 0) and will be removed in the future. They were replaced with mqtt_connack_sent using the return_code label. For MQTT 5.0 the reason_code label is used instead.

The output on the command line are aggregated by default, but details for a label can be shown as well, for example all metrics with the not_authorized label:

vmq-admin metrics show --return_code=not_authorized
counter.mqtt_connack_sent = 0

All available labels can be show using vmq-admin metrics show --help.

You can configure as many listeners as you wish in the vernemq.conf file. In addition to this, the vmq-admin listener command let's you configure, start, stop and delete listeners on the fly. Those can be MQTT, WebSocket or Cluster listeners, in the command line output they will be tagged mqtt, ws or vmq accordingly.

Status of all listeners

vmq-admin listener show

    +----+-------+------------+-----+----------+---------+
    |type|status |     ip     |port |mountpoint|max_conns|
    +----+-------+------------+-----+----------+---------+
    |vmq |running|192.168.1.50|44053|          |  30000  |
    |mqtt|running|192.168.1.50|1883 |          |  30000  |
    +----+-------+------------+-----+----------+---------+
`

Starting a new listener

vmq-admin listener start address=192.168.1.50 port=1884 --mountpoint /test --nr_of_acceptors=10 --max_connections=1000

This will start an MQTT listener on port 1884 and IP address 192.168.1.50. If you want to start a WebSocket listener, just tell VerneMQ by adding the --websocket flag. There are more options, mainly for configuring SSL (use vmq-admin listener start --help).

You can isolate client connections accepted by a certain listener from other clients by setting a mountpoint.

To start an MQTT listener using defaults, just set the port and IP address as a minimum.

Stopping a listener

vmq-admin listener stop address=192.168.1.50 port=1884

You can add the -k or --kill_sessions switch to that command. This will disconnect all client connections setup by that listener. In combination with a mountpoint, this can be useful for terminating clients for a specific application, or to force re-connects to another cluster node (to prepare for a cluster leave for your node).

Restarting a stopped listener

vmq-admin listener restart address=192.168.1.50 port=1884

Deleting a stopped listener

vmq-admin listener delete address=192.168.1.50 port=1884

You can dynamically re-configure most of VerneMQ's settings on a running node by using the vmq-admin set command.

The following config values can be handled dynamically:

allow_anonymous
topic_alias_max_broker
receive_max_broker
vmq_acl.acl_file
graphite_host
vmq_acl.acl_reload_interval
graphite_enabled
queue_type
suppress_lwt_on_session_takeover
max_message_size
vmq_passwd.password_file
graphite_port
max_client_id_size
upgrade_outgoing_qos
max_message_rate
graphite_interval
allow_multiple_sessions
systree_enabled
max_last_will_delay
retry_interval
receive_max_client
max_offline_messages
max_online_messages
max_inflight_messages
allow_register_during_netsplit
vmq_passwd.password_reload_interval
topic_alias_max_client
systree_interval
allow_publish_during_netsplit
coordinate_registrations
remote_enqueue_timeout
persistent_client_expiration
allow_unsubscribe_during_netsplit
graphite_include_labels
shared_subscription_policy
queue_deliver_mode
allow_subscribe_during_netsplit

Setting a value for the local node

Let's change the max_client_id_size as an example. (We might have noticed that some clients can't login because their client ID is too long, but we don't want to restart the broker for that). Note that you can also set multiple values with the same command.

vmq-admin set max_client_id_size=45

Setting a value for an arbitrary cluster node

vmq-admin set max_client_id_size=45 --node=VerneMQ20@192.168.1.20

Setting a value for all cluster nodes

vmq-admin set max_client_id_size=45 --all

Show current VerneMQ config values

For the local node

You can show one or multiple values in a simple table:

vmq-admin show max_client_id_size retry_interval

+----------------------+------------------+--------------+
|         node         |max_client_id_size|retry_interval|
+----------------------+------------------+--------------+
|VerneMQ20@192.168.1.50|        28        |      20      |
+----------------------+------------------+--------------+

`

For an arbitrary node

vmq-admin show max_client_id_size retry_interval --node VerneMQ20@192.168.1.20

For all cluster nodes

vmq-admin show max_client_id_size retry_interval --all

+----------------------+------------------+--------------+
|         node         |max_client_id_size|retry_interval|
+----------------------+------------------+--------------+
|VerneMQ30@192.168.1.30|        33        |      20      |
|VerneMQ40@192.168.1.40|        33        |      20      |
|VerneMQ10@192.168.1.10|        33        |      20      |
|VerneMQ50@192.168.1.50|        33        |      20      |
|VerneMQ20@192.168.1.20|        28        |      20      |
+----------------------+------------------+--------------+