Welcome to the VerneMQ documentation! This is a reference guide for most of the available features and options of VerneMQ. The might be a good entry point.

For a more general overview on VerneMQ and MQTT, you might want to start with the .

For downloading VerneMQ see .

On every VerneMQ node you'll find the vmq-admin command line tool in the release's bin directory. It has different sub-commands that let you check for status, start and stop listeners, re-configure values and a couple of other administrative tasks.

A simple way to gauge the health of a VerneMQ cluster is to query the /health path on the HTTP listener.

The health check will return 200 when VerneMQ is accepting connections and is joined with the cluster (for clustered setups). 503 will be returned in case any of those two conditions are not met.

Install VerneMQ

Once you have downloaded the binary package, execute the following command to install VerneMQ:

or:

Activate VerneMQ node

Once you've installed VerneMQ, start it on your node:

Verify your installation

You can verify that VerneMQ is successfully installed by running:

If VerneMQ has been installed successfully vernemq is returned.

Next Steps

Now that you've installed VerneMQ, check out How to configure VerneMQ.

VerneMQ supports the WebSocket protocol out of the box. To be able to open a WebSocket connection to VerneMQ, you have to configure a WebSocket listener or Secure WebSocket listener in the vernemq.conf file first:

listener.ws.default = 127.0.0.1:9001

listener.wss.default = 127.0.0.1:9002

Keep in mind that you'll use MQTT-over-WebSocket, so you will need a Javascript library that implements the MQTT client behaviour. We have used the Eclipse Paho client as well as MQTT.js

You won't be able to open WebSocket connections on a base URL, always add the /mqtt path.

VerneMQ uses Google's LevelDB as a fast storage backend for messages and subscriber information. Each VerneMQ node runs its own embedded LevelDB store.

Configuration of LevelDB memory

There's not much you need to know about LevelDB and VerneMQ. One really important thing to note is that LevelDB manages its own memory. This means that VerneMQ will not allocate and free memory for LevelDB. Instead you'll have to configure a configuration value in vernemq.conf that tells LevelDB how much memory it can use up.

Configuring LevelDB memory:

leveldb.maximum_memory.percent = 20

Installing VerneMQ

VerneMQ is a high-performance, distributed MQTT message broker. It scales horizontally and vertically on commodity hardware to support a high number of concurrent publishers and consumers while maintaining low latency and fault tolerance. To use it, all you need to do is install the VerneMQ package.

Choose your OS and follow the instructions:

• CentOS/RHEL

It is also possible to run VerneMQ using our Docker image:

To start a VerneMQ broker, use the vernemq start command in your Shell:

A successful start will return no output. If there is a problem starting the broker, an error message is printed to STDERR.

To run VerneMQ with an attached interactive Erlang console:

A VerneMQ broker is typically started in console mode for debugging or troubleshooting purposes. Note that if you start VerneMQ in this manner, it is running as a foreground process that will exit when the console is closed.

You can close the console by issuing this command at the Erlang prompt:

Once your broker has started, you can initially check that it is running with the vernemq ping command:

The command will respond with pong if the broker is running or Node <NodeName> not responding to pings in case it’s not.

Every VerneMQ node has to be configured. Depending on the installation method and chosen platform the configuration file vernemq.conf resides at different locations. If VerneMQ was installed through a Linux package the default location for the configuration file is /etc/vernemq/vernemq.conf.

File Format

• A single setting is handled on one line.

• Lines are structured Key = Value

• Any line starting with # is a comment, and will be ignored

You certainly want to try out VerneMQ right away. For that you could disable authentication like so:

• Set allow_anonymous = on

By default the vmq_acl authorization plugin is enabled and configured to allow publishing and subscribing to any topic, see for more information.

Retry Interval

Set the time in seconds after a QoS=1 or QoS=2 message has been sent that VerneMQ will wait before retrying when no response is received.

retry_interval = 20

This option default to 20 seconds.

Inflight Messages

This option defines the maximum number of QoS 1 or 2 messages that can be in the process of being transmitted simultaneously.

max_inflight_messages = 20

Defaults to 20 messages, use 0 for no limit. The inflight window serves as a protection for sessions, on the incoming side.

Load Shedding

The maximum number of messages to hold in the queue above those messages that are currently in flight. Defaults to 1000. Set to -1 for no limit. This option protects a client session from overload by dropping messages (of any QoS).

Defaults to 1000 messages, use -1 for no limit. This parameter was named max_queued_messages in 0.10.*. Note that 0 will totally block message delivery from any queue!

This option specifies the maximum number of QoS 1 and 2 messages to hold in the offline queue.

Defaults to 1000 messages, use -1 for no limit, use 0 if no messages should be stored.

In contrast to the session based inflight window, max_online_messages and max_offline_messages serves as a protection of queues, on the outgoing side.

Sometimes consumers get overwhelmed by the number of messages they receive. VerneMQ can load balance between multiple consumer instances subscribed to the same topic with the same ClientId.

Enabling Session Balancing

To enable session balancing, activate the following two settings in vernemq.conf

 allow_multiple_sessions = on
 queue_deliver_mode = balance

VerneMQ comes with a built-in status page which by default is enabled and is available on http://localhost:8888/status, see HTTP listeners.

The status page is a simple overview of the cluster and the individual nodes in the cluster as seen below:

VerneMQ supports enhanced authentication flows or SASL style authentication for MQTT 5.0 sessions. The enhanced authentication mechanism can be used for initial authentication when the client connects or to re-authenticate clients at a later point.

The on_auth_m5 hook allows the plugin to implement SASL style authentication flows by either accepting, rejecting (disconnecting the client) or continue the flow. The on_auth_m5 hook is specified in the Erlang behaviour on_auth_m5_hook in the vernemq_dev repo.

Once you have downloaded the binary package, execute the following command to install VerneMQ:

You can verify that VerneMQ is successfully installed by running:

If VerneMQ has been installed successfully Status: install ok installed is returned.

Once you've installed VerneMQ, start it on your node:

The whereis vernemq command will give you a couple of directories:

The VerneMQ HTTP listener is used to serve various VerneMQ subsystems such as:

Set the maximum size for client ids, MQTT v3.1 specifies a limit of 23 characters.

This option default to 23.

This option allows persistent clients (those with clean_session set to false) to be removed if they do not reconnect within a certain time frame.

The expiration period should be an integer followed by one of h, d,

Somtimes you need to configure a forwarding for ports (on a Mac for example):

This starts a new node that listens on 1883 for MQTT connections and on 8080 for MQTT over websocket connections. However, at this moment the broker won't be able to authenticate the connecting clients. To allow anonymous clients use the DOCKER_VERNEMQ_ALLOW_ANONYMOUS=on environment variable.

This allows a newly started container to automatically join a VerneMQ cluster. Assuming you started your first node like the example above you could autojoin the cluster (which currently consists of a single container 'vernemq1') like the following:

(Note, you can find the IP of a docker container using docker inspect <CONTAINER_NAME> | grep \"IPAddress\").

VerneMQ uses the Erlang distribution mechanism for most inter-node communication. VerneMQ identifies other machines in the cluster using Erlang identifiers (e.g. VerneMQ@10.9.8.7). Erlang resolves these node identifiers to a TCP port on a given machine via the Erlang Port Mapper daemon (epmd) running on each cluster node.

By default, epmd binds to TCP port 4369 and listens on the wildcard interface. For inter-node communication, Erlang uses an unpredictable port by default; it binds to port 0, which means the first available port.

For ease of firewall configuration, VerneMQ can be configured to instruct the Erlang interpreter to use a limited range of ports. For example, to restrict the range of ports that Erlang will use for inter-Erlang node communication to 6000-7999, add the following lines to vernemq.conf on each VerneMQ node:

The settings above are only used for distributing subscription updates and maintenance messages. For distributing the 'real' MQTT messages the proper vmq listener must be configured in the vernemq.conf.

The systree functionality is enabled by default and reports the broker metrics at a fixed interval defined in the vernemq.conf. The metrics defined are transformed to MQTT topics e.g. mqtt_publish_received is transformed to $SYS/<nodename>/mqtt/publish/received. <nodename> is your node's name, as configured in the vernemq.conf. To find it, you can grep the file for it: grep nodename vernemq.conf

The complete list of metrics can be found

This section elaborates how a VerneMQ cluster deals with network partitions (aka. netsplit or split brain situation). A netsplit is mostly the result of a failure of one or more network devices resulting in a cluster where nodes can no longer reach each other.

VerneMQ is able to detect a network partition, and by default it will stop serving CONNECT, PUBLISH, SUBSCRIBE, and UNSUBSCRIBE requests. A properly implemented client will always resend unacked commands and messages are therefore not lost (QoS 0 publishes will be lost). However, the time window between the network partition and the time VerneMQ detects the partition much can happen. Moreover, this time frame will be different on every participating cluster node. In this guide we're referring to this time frame as the Window of Uncertainty.

The Prometheus exporter is enabled by default and installs an HTTP handler on http://localhost:8888/metrics. To read more about configuring the HTTP listener, see .

Add the following configuration to the scrape_configs section inside prometheus.yml of your Prometheus server.

This tells Prometheus to scrape the VerneMQ metrics endpoint every 5 seconds.

Please follow the documentation on the website to properly configure the metrics scraping as well as how to access those metrics and configure alarms and graphs.

In this section the subscription flow is described. VerneMQ provides several hooks to intercept the subscription flow. The most important ones are the auth_on_subscribe and auth_on_subscribe_m5 hooks which act as an application level firewall granting or rejecting subscribe requests.

The auth_on_subscribe and auth_on_subscribe_m5 hooks allow your plugin to grant or reject subscribe requests sent by a client. They also makes it possible to rewrite the subscribe topic and qos. The auth_on_subscribe hook is specified in the Erlang behaviour and the auth_on_subscribe

To list the retained messages simply invoke vmq-admin retain show:

Besides listing the retained messages it is also possible to filter them:

In the above example we list only the payload for the topic some/topic.

Another example where all topics are list with retained messages with a specific payload:

See the full set of options and documentation by invoking vmq-admin retain show --help

The graphite exporter reports the broker metrics at a fixed interval (defined in milliseconds) to a graphite server. The necessary configuration is done inside the vernemq.conf.

You can further tune the connection to the Graphite server:

Sometimes you want to have a quick way to test a cluster on your development machine as a VerneMQ developer.

You need to take care of a couple things if you want to run multiple VerneMQ instances on the same machine. There is a make option that let's you build multiple releases, as a commodity, taking care of all the configuration.

First, build a normal release (this is just needed the first time) with:

➜ default git:(master) ✗ make rel

The following command will then prepare 3 correctly configured vernemq.conf files, with different ports for the MQTT listeners etc. It will also build 3 full VerneMQ releases.

➜ default git:(master) ✗ make dev1 dev2 dev3

Check if you have the 3 new releases in the _build directory of your VerneMQ code repo.

You can then start the respective broker instances in 3 terminal windows, by using the respective commands and directory paths. Example:

➜ (_build/dev2/rel/vernemq/bin) ✗ vernemq console

The MQTT listeners will of course be configured differently for each node (the default 1883 port is not used, so that you can still run a default MQTT broker besides your dev nodes). A couple of other ports are also adapted (HTTP status page, cluster communication). The MQTT ports are automically configured in increasing steps of 50: (if in doubt, consult the respective vernemq.conf files)

Note that the dev nodes are not automatically clustered. You still need to manually cluster them with commands like the following:

➜ (_build/dev2/rel/vernemq/bin) ✗ vmq-admin cluster join discovery-node=dev1@127.0.0.1

VerneMQ provides multiple hooks throughout the lifetime of a session. The most important ones are the auth_on_register and auth_on_register_m5 hooks which act as an application level firewall granting or rejecting new clients.

auth_on_register and auth_on_register_m5

The auth_on_register and auth_on_register_m5 hooks allow your plugin to grant or reject new client connections. Moreover it lets you exert fine grained control over the configuration of the client session. The auth_on_register hook is specified in the Erlang behaviour auth_on_register_hook and the auth_on_register_m5 hook in the behaviour available in the repo.

Every plugin that implements the auth_on_register or auth_on_register_m5 hooks are part of a conditional plugin chain. For this reason we allow the hook to return different values depending on how the plugin grants or rejects this client. In case the plugin doesn't know the client it is best to return next as this would allow subsequent plugins in the chain to validate this client. If no plugin is able to validate the client it gets automatically rejected.

The on_auth_m5 hook allows your plugin to implement MQTT enhanced authentication, see .

The on_register and on_register_m5 hooks allow your plugin to get informed about a newly authenticated client. The hook is specified in the Erlang behaviour and the behaviour available in the repo.

Once a new client was successfully authenticated and the above described hooks have been called, the client attaches to its queue. If it is a returning client using clean_session=false or if the client had previous sessions in the cluster, this process could take a while. (As offline messages are migrated to a new node, existing sessions are disconnected). The hook is called at the point where a queue has been successfully instantiated, possible offline messages migrated, and potential duplicate sessions have been disconnected. In other words: when the client has reached a completely initialized, normal state for accepting messages. The hook is specified in the Erlang behaviour on_client_wakeup_hook available in the repo.

This hook is called if an MQTT 3.1/3.1.1 client using clean_session=false or an MQTT 5.0 client with a non-zero session_expiry_interval closes the connection or gets disconnected by a duplicate client. The hook is specified in the Erlang behaviour available in the repo.

This hook is called if an MQTT 3.1/3.1.1 client using clean_session=true or an MQTT 5.0 client with the session_expiry_interval set to zero closes the connection or gets disconnected by a duplicate client. The hook is specified in the Erlang behaviour available in the repo.

A shared subscription is a mechanism for distributing messages to a set of subscribers to shared subscription topic, such that each message is received by only one subscriber. This contrasts with normal subscriptions where each subscriber will receive a copy of the published message.

A shared subscription is on the form $share/sharename/topic and subscribers to this topic will receive messages published to the topic topic. The messages will be distributed according to the defined distribution policy.

Configuration

Currently three message distribution policies for shared subscriptions are supported: prefer_local, random and local_only. Under the random policy messages will be published to a random member of the shared subscription, if any exist. Under the prefer_local policy messages will be delivered to a random node-local member of the shared subscription, if none exist, the message will be delivered to a random member of the shared subscription on a remote cluster node. Under the local_only policy message will be delivered to a random node-local member of the shared subscription.

When a messages is being delivered to subscribers of a shared subscription, the message will be delivered to an online subscriber if possible, otherwise the message will be delivered to an offline subscriber.

Subscriptions Note: When subscribing to a shared topic, make sure to escape the $

So, for dash or bash shells

Publishing Note: When publishing to a shared topic, do not include the prefix $share/group/ as part of the publish topic name

There are a couple of hidden options you can set in the vernemq.conf file. Hidden means that you have to add and set the value explicitly. Hidden options still have default values. Changing them should be considered advanced, possibly with the exception of setting a max_message_rate.

Queue Deliver mode

Specify how the queue should deliver messages when multiple sessions are allowed. In case of fanout all the attached sessions will receive the message, in case of balance an attached session is choosen randomly.

queue_deliver_mode = balance

Queue Type

Specify how queues should process messages, either the fifo or lifo way. Default is fifo.

Specifies the maximum incoming publish rate per session per second. Depending on the underlying network buffers this rate isn't enforced. Defaults to 0, which means no rate limits apply. Setting to a value of 2 limits any publisher to 2 messages per second, for instance.

Due to the eventual consistent nature of the subscriber store it is possible that during queue migration messages still arrive on the old cluster node. This parameter enables to compensate this by keeping the queue around for some time (in seconds) after it was migrated to the other cluster node.

Specifies the number of messages that are delivered to the remote node per drain step. A large value will provide a faster migration of a queue, but increases the waste of bandwidth in case the migration fails.

Allows to select a new default reg_view. A reg_view is a pre-defined way to route messages. Multiple views can be loaded and used, but one has to be selected as a default. The default routing is vmq_reg_trie, i.e. routing via the built-in trie data structure.

A list of views that are started during startup. It's only used in plugins that want to choose dynamically between routing reg_views.

An integer specifying how many bytes are buffered in case the remote node is not available. Default is 10000

Console Logging

Where should VerneMQ emit the default console log messages (which are typically at info severity):

log.console = off | file | console | both

VerneMQ defaults to log the console messages to a file, which can specified by:

log.console.file = /path/to/log/file

This option defaults to /var/log/vernemq/console.log for Ubuntu, Debian, RHEL and Docker installs.

The default console logging level info could be setting one of the following:

log.console.level = debug | info | warning | error

Error Logging

VerneMQ log error messages by default. One can change the default behaviour by setting:

log.error = on | off

VerneMQ defaults to log the error messages to a file, which can specified by:

This option defaults to /var/log/vernemq/error.log for Ubuntu, Debian, RHEL and Docker installs.

VerneMQ log crash messages by default. One can change the default behaviour by setting:

VerneMQ defaults to log the crash messages to a file, which can specified by:

This option defaults to /var/log/vernemq/crash.log for Ubuntu, Debian, RHEL and Docker installs.

The maximum sizes in bytes of inidividual messages in the crash log defaults to 64KB but can be specified by:

VerneMQ rotate crash logs. By default, the crash log file is rotated at midnight or when the size exceeds 10MGB. This behaviour can be changed by setting:

The default number of rotated log files is 5 and can be set with the option:

VerneMQ supports logging to SysLog, enable it by setting:

Logging to SysLog is disabled by default.

VerneMQ can be monitored in several ways. We implemented native support for Graphite, MQTT $SYS tree, and Prometheus.

The metrics are also available via the command line tool:

vmq-admin metrics show

Or with:

vmq-admin metrics show -d

Which will output the metrics together with a short description describing what the metric is about. An example looks like:

# The number of AUTH packets received.
counter.mqtt_auth_received = 0

# The number of times a MQTT queue process has been initialized from offline storage.
counter.queue_initialized_from_storage = 0

# The number of PUBLISH packets sent.
counter.mqtt_publish_sent = 10

# The number of bytes used for storing retained messages.
gauge.retain_memory = 21184

Notice that the metrics:

mqtt_connack_not_authorized_sent
mqtt_connack_bad_credentials_sent
mqtt_connack_server_unavailable_sent
mqtt_connack_identifier_rejected_sent
mqtt_connack_unacceptable_protocol_sent
mqtt_connack_accepted_sent

Are no longer used (always 0) and will be removed in the future. They were replaced with mqtt_connack_sent using the return_code label. For MQTT 5.0 the reason_code label is used instead.

The output on the command line are aggregated by default, but details for a label can be shown as well, for example all metrics with the not_authorized label:

All available labels can be show using vmq-admin metrics show --help.

You can configure as many listeners as you wish in the vernemq.conf file. In addition to this, the vmq-admin listener command let's you configure, start, stop and delete listeners on the fly. Those can be MQTT, WebSocket or Cluster listeners, in the command line output they will be tagged mqtt, ws or vmq accordingly.

Status of all listeners

vmq-admin listener show

    +----+-------+------------+-----+----------+---------+
    |type|status |     ip     |port |mountpoint|max_conns|
    +----+-------+------------+-----+----------+---------+
    |vmq |running|192.168.1.50|44053|          |  30000  |
    |mqtt|running|192.168.1.50|1883 |          |  30000  |
    +----+-------+------------+-----+----------+---------+
`

Starting a new listener

vmq-admin listener start address=192.168.1.50 port=1884 --mountpoint /test --nr_of_acceptors=10 --max_connections=1000

This will start an MQTT listener on port 1884 and IP address 192.168.1.50. If you want to start a WebSocket listener, just tell VerneMQ by adding the --websocket flag. There are more options, mainly for configuring SSL (use vmq-admin listener start --help).

You can isolate client connections accepted by a certain listener from other clients by setting a mountpoint.

To start an MQTT listener using defaults, just set the port and IP address as a minimum.

You can add the -k or --kill_sessions switch to that command. This will disconnect all client connections setup by that listener. In combination with a mountpoint, this can be useful for terminating clients for a specific application, or to force re-connects to another cluster node (to prepare for a cluster leave for your node).

In this section the publish flow is described. VerneMQ provides multiple hooks throughout the flow of a message. The most important ones are the auth_on_publish and auth_on_publish_m5 hooks which acts as an application level firewall granting or rejecting a publish message.

auth_on_publish and auth_on_publish_m5

The auth_on_publish and auth_on_publish_m5 hooks allow your plugin to grant or reject publish requests sent by a client. It also enables to rewrite the publish topic, payload, qos, or retain flag and in the case of auth_on_publish_m5 properties. The auth_on_publish hook is specified in the Erlang behaviour auth_on_publish_hook and the auth_on_publish_m5 hook in the behaviour available in the repo.

Every plugin that implements the auth_on_publish or auth_on_publish_m5 hooks are part of a conditional plugin chain. For this reason we allow the hook to return different values. In case the plugin can't validate the publish message it is best to return next as this would allow subsequent plugins in the chain to validate the request. If no plugin is able to validate the request it gets automatically rejected.

The on_publish and on_publish_m5 hooks allow your plugin to get informed about an authorized publish message. The hook is specified in the Erlang behaviour and the on_publish_m5 hook in the behaviour available in the repo.

The on_offline_message hook allows your plugin to get notified about a new a queued message for a client that is currently offline. The hook is specified in the Erlang behaviour available in the repo.

The on_deliver and on_deliver_m5 hooks allow your plugin to get informed about outgoing publish messages, but also allows you to rewrite topic and payload of the outgoing message. The hook is specified in the Erlang behaviour and the on_deliver_m5 hook in the behaviour available in the repo.

Every plugin that implements the on_deliver or on_deliver_m5 hooks are part of a conditional plugin chain, although NO verdict is required in this case. The message gets delivered in any case. If your plugin uses this hook to rewrite the message the plugin system stops evaluating subsequent plugins in the chain.

We recommend to use the rebar3 toolchain to generate the basic Erlang OTP application boilerplate and start from there.

rebar3 new app name="myplugin" desc="this is my first VerneMQ plugin"
===> Writing myplugin/src/myplugin_app.erl
===> Writing myplugin/src/myplugin_sup.erl
===> Writing myplugin/src/myplugin.app.src
===> Writing myplugin/rebar.config
===> Writing myplugin/.gitignore
===> Writing myplugin/LICENSE
===> Writing myplugin/README.md

Change the rebar.config file to include the vernemq_dev dependency:

{erl_opts, [debug_info]}.
{deps, [{vernemq_dev,
    {git, "git://github.com/vernemq/vernemq_dev.git", {branch, "master"}}}
]}.

Compile the application, this will automatically fetch vernemq_dev.

rebar3 compile                             
===> Verifying dependencies...
===> Fetching vmq_commons ({git,
                                      "git://github.com/vernemq/vernemq_dev.git",
                                      {branch,"master"}})
===> Compiling vernemq_dev
===> Compiling myplugin

Now you're ready to implement the hooks. Don't forget to add the proper vmq_plugin_hooks entries to your src/myplugin.app.src file.

For a complete example, see the .

You can loadtest VerneMQ with our . It is based on Machinezone's very powerful and lets you narrow down what hardware specs are needed to meet your performance goals. You can state your requirements for latency percentiles (and much more) in a formal way, and let vmq_mzbench automatically fail, if it can't meet the requirements.

If you have an AWS account, vmq_mzbench can automagically provision worker nodes for you. You can also run it locally, of course.

Please follow the

Actually, you don't even have to install vmq_mzbench, if you don't want to. Your scenario file will automatically fetch vmq_mzbench for any test you do. vmq_mzbench runs every test independently, so it has a provisioning step for any test, even if you only run it on a local worker.

To install vmq_mzbench on your computer, go through the following steps:

When working with a system like VerneMQ sometimes when troubleshooting it would be nice to know what a client is actually sending and receiving and what VerneMQ is doing with this information. For this purpose VerneMQ has a built-in tracing mechanism which is safe to use in production settings as there is very little overhead in running the tracer and has built-in protection mechanisms to stop traces that produce too much information.

To trace a client the following command is available:

See the available flags by calling vmq-admin trace client --help.

A typical trace could look like the following:

In this particular trace a trace was started for the client with client-id client. At first no clients are connected to the node where the trace has been started, but a little later the client connects and we see the trace come alive. The strange identifier

You can dynamically re-configure most of VerneMQ's settings on a running node by using the vmq-admin set command.

The following config values can be handled dynamically:

Setting a value for the local node

Let's change the max_client_id_size as an example. (We might have noticed that some clients can't login because their client ID is too long, but we don't want to restart the broker for that). Note that you can also set multiple values with the same command.

Setting a value for an arbitrary cluster node

Setting a value for all cluster nodes

Show current VerneMQ config values

For the local node

You can show one or multiple values in a simple table:

For an arbitrary node

For all cluster nodes

Listeners specify on which IP address and port VerneMQ should accept new incoming connections. Depending on the chosen transport (TCP, SSL, WebSocket) different configuration parameters have to be provided. VerneMQ allows to write the listener configurations in a hierarchical manner, enabling very flexible setups. VerneMQ applies reasonable defaults on the top level, which can be of course overridden if needed.

# defines the default nr of allowed concurrent 
# connections per listener
listener.max_connections = 10000

# defines the nr. of acceptor processes waiting
# to concurrently accept new connections
listener.nr_of_acceptors = 10

# used when clients of a particular listener should
# be isolated from clients connected to another 
# listener.
listener.mountpoint = off

These are the only default parameters that are applied for all transports, and the only one that are of interest for plain TCP and WebSocket listeners.

These global defaults can be overridden for a specific transport protocol listener.tcp.CONFIG = VAL, or even for a specific listener listener.tcp.LISTENER.CONFIG = VAL. The placeholder LISTENER is freely chosen and is only used as a reference for further configuring this particular listener.

Mountpoints

Normally, an MQTT broker hosts one single topic tree. This means that all topics are accessible to all publishers and subscribers (limited by the ACLs you configured, of course). Mountpoints are a way to host multiple topic trees in a single broker. They are completely separated and clients with different topic trees cannot publish messages to each other. This could be useful if you provide MQTT services to multiple separated use cases/verticals or clients, with a single broker. Note that mountpoints are configured via different listeners. As a consequence, the MQTT clients will have to connect to a specific port to connect to a specific topic space (mountpoint).

The mountpoints can be configured on the protocol level or configurred or overridden on the specific listener level.

Since VerneMQ 1.5.0 it is possible to configure which MQTT protocol versions as listener will accept.

VerneMQ supports MQTT 3.1, 3.1.1, and 5.0 (since VerneMQ 1.6.0). To allow these protocol versions, set:

Here 3,4,5 are the protocol level versions corresponding to MQTT 3.1, 3.1.1 and 5.0 respectively. The default value is 3,4 thus allowing MQTT 3.1 and 3.1.1, while MQTT 5.0 is disabled.

Listen on TCP port 1883 and for WebSocket Connections on port 8888:

An additional listener can be added by using a different name. In the example above the name equals to default and can be used for further configuring this particular listener. The following example demonstrates how an additional listener is defined as well as how the maximum number of connections can be limited for this listener:

VerneMQ listeners can be configured to accept connections from a proxy server that supports the PROXY protocol. This enables VerneMQ to retrieve peer information such as source IP/Port but also PROXY Version 2 protocol TLS client certificate details if the proxy was used to terminate TLS.

To enable the PROXY protocol for tcp listeners use listener.tcp.proxy_protocol = on or for a specific listener use listener.tcp.LISTENER.proxy_protocol = on.

If client certificates are used you can set listener.tcp.proxy_protocol_use_cn_as_username = on which will overwrite the MQTT username set by the client with the common name from the client certificate before authentication and authorization is performed.

Accepting SSL connections on port 8883:

If you want to use client certificates to authenticate your clients you have to set the following option:

If you use client certificates and want to use the certificates CN value as a username you can set:

Both options require_certificate and use_identity_as_username default to off.

The same configuration options can be used for securing WebSocket connections, just use wss as the protocol identifier e.g. listener.wss.require_certificate.

VerneMQ can be easily clustered. Clients can then connect to any cluster node and receive messages from any other cluster nodes. However, the MQTT specification gives certain guarantees that are hard to fulfill in a distributed environment, especially when network partitions occur. We'll discuss the way VerneMQ deals with network partitions in its own subsection

A note on statefulness

Before you go ahead and experience the full power of clustering VerneMQ, be aware of its stateful character. An MQTT broker is a stateful application and a VerneMQ cluster is a stateful cluster.

What does this mean in detail? It means that clustered VerneMQ nodes will share information about connected clients and sessions but also meta-information about the cluster itself.

For instance, if you stop a cluster node, the VerneMQ cluster will not just forget about it. It will know that there's a node missing and it will keep looking for it. It will know there's a netsplit situation and it will heal the partition when the node comes back up. But if the missing nodes never comes back there's an eternal netsplit. (still resolvable by making the missing nodes explicitly leave).

This doesn't mean that a VerneMQ cluster cannot dynamically grow and shrink. But it means you have to tell the cluster what you intend to do, by using join and leave commands.

If you want a cluster node to leave the cluster, well... use the vmq-admin cluster leave command. If you want a node to join a cluster, well... use the vmq-admin cluster join command.

Makes sense? Go ahead and create your first VerneMQ cluster!

A cluster leave will actually do a lot more work, and gives you some options to choose. The node leaving the cluster will go to great length trying to migrate its existing queues to other nodes. As queues (online or offline) are live processes in a VerneMQ node, it will only exit after it has migrated them.

Let's look at the steps in detail:

1. vmq-admin cluster leave node=<NodeThatShouldGo>

This first step will only stop the MQTT Listeners of the node to ensure that no new connections are accepted. It will not interrupt the existing connections, and behind the scenes the node will not leave the cluster yet. Existing clients are still able to publish and receive messages at this point.

The idea is to give a grace period with the hope that existing clients might re-connect (to another node). If you have decided that this period is over (after 5 minutes or 1 day is up to you), you proceed with step 2: disconnecting the rest of the clients.

1. vmq-admin cluster leave node=<NodeThatShouldGo> -k

The -k flag will delete the MQTT Listeners of the leaving node, taking down all live connections. If this is what you want from the beginning, you can do this right away as a first step.

Now, queue migration is triggered by clients re-connecting to other nodes. They will claim their queue and it will get migrated. Still, there might be some offline queues remaining on the leaving node, because they were pre-existing or because some clients do not re-connect and do not reclaim their queues.

VerneMQ will throw an exception if there are remaining offline queues after a configurable timeout. The default is 60 seconds, but you can set it as an option to the cluster leave command. As soon as the exception shows in console or console.log, you can actually retry the cluster leave command (including setting a migration timeout (-t), and an interval in seconds (-i) indicating how often information on the migration progress should be printed to the console.log):

1. vmq-admin cluster leave node=<NodeThatShouldGo> -k -i 5 -t 120

After this timeout VerneMQ will forcefully migrate the remaining offline queues to other cluster nodes in a round robin manner. After doing that, it will stop the leaving VerneMQ node.

So, case A was the happy case. You left the cluster with your node in a controlled manner, and everything worked, including a complete queue (and message) transfer to other nodes.

Let's look at the second possibility where the node is already down. Your cluster is still counting on it though and possibly blocking new subscription for that reason, so you want to make the node leave.

To do this, use the same command(s) as in the first case. There is one important consequence to note: by making a stopped node leave, you basically throw away persistant queue content, as VerneMQ won't be able to migrate or deliver it.

Let's repeat that to make sure:

Inspecting sessions

VerneMQ comes with powerful tools for inspecting the state of MQTT sessions. To list current MQTT sessions simply invoke vmq-admin session show:

$ vmq-admin session show
+---------+---------+----------+---------+---------+---------+
|client_id|is_online|mountpoint|peer_host|peer_port|  user   |
+---------+---------+----------+---------+---------+---------+
| client2 |  true   |          |127.0.0.1|  37098  |undefined|
| client1 |  true   |          |127.0.0.1|  37094  |undefined|
+---------+---------+----------+---------+---------+---------+

To see detailed information about the command see vmq-admin session show --help.

The command is able to show a lot of different information about a client, for example the client id, the peer host and port if the client is online or offline and much more, see vmq-admin session show --help for details. Further the information can also be used to filter information which is very helpful when wanting to narrow down the information to a single client.

A sample query which lists only the node where the client session exists and if the client is online would look like the following:

Listing the clients and the subscriptions one can do the following:

And to list only the clients subscribed to the topic some/topic:

To figure out when the queue for a persisted session (clean_session=false) was created and when the client last connected one can use the --queue_started_at and --session_started_at to list the POSIX timestamps (in microseconds):

Besides the examples above it is also possible to inspect the number of online or offline messages as well as their payloads and much more. See vmq-admin session show --help for an exhaustive list of all the available options.

VerneMQ also supports disconnecting clients and reauthorizing client subscriptions. To disconnect a client and cleanup store messages and remove subscriptions one can invoke:

See vmq-admin session disconnect --help for more options and details.

To reauthorize subscriptions for a client issue the following command:

This works by reapplying the logic in any installed auth_on_subscribe or auth_on_subscribe_m5 plugin hooks to check the validity of the existing topics and removing those that are no longer allowed. In the example above the reauthorization of the client subscriptions resulted in no changes.

Bridges are a non-standard way, although kind of a de-facto standard among MQTT broker implementations, to connect two different MQTT brokers to eachother. This allows for example that a topic tree of a remote broker becomes part of the topic tree on the local broker. VerneMQ supports plain TCP connections as well as SSL connections.

in VerneMQ the bridge is distributed with VerneMQ as a plugin and is not enabled by default. After configuring the bridge as described below, make sure to enable the plugin by setting:

See for more information on working with plugins.

When the plugin is enabled a simple status interface is available:

Setup a bridge to a remote broker:

Different connection parameters can be set:

VerneMQ is implemented in Erlang OTP and therefore runs on top of the Erlang VM. For this reason plugins have to be developed in a programming language that runs on the Erlang VM. The most popular choice is obviously the Erlang programming language itself, but Elixir or Lisp flavoured Erlang LFE could be used too.

This guide explains the different flows that expose different hooks to be used for custom plugins. It also describes the code structure a plugin must comply to in order to be successfully loaded and started by the VerneMQ plugin mechanism.

All the hooks that are currently exposed fall into one of three categories.

1. Hooks that allow you to change the protocol flow. An example could be to authenticate a client using the

General relation to OS configuration values

You need to know about and configure a couple of Operating System and Erlang VM configs to operate VerneMQ efficiently. First, make sure you have set appropriate OS file limits according to our guide here. Second, when you run into performance problems, don't forget to check the settings in the vernemq.conf file. (Can't open more than 10k connections? Well, is the listener configured to open more than 10k?)

TCP buffer sizes

This is the number one topic to look at, if you need to keep an eye on RAM usage.

Context: All network I/O in Erlang uses an internal driver. This driver will allocate and handle an internal application side buffer for every TCP connection. The default size of these buffers will determine your overall RAM use in VerneMQ. The sndbuf and recbuf of the TCP socket will not count towards VerneMQ RAM, but will be used by the Linux Kernel.

VerneMQ calculates the buffer size from the OS level TCP send and receive buffers:

val(buffer) >= max(val(sndbuf),val(recbuf))

Those values correspond to net.ipv4.tcp_wmem and net.ipv4.tcp_rmem in your OS's sysctl configuration. One way to minimize RAM usage is therefore to configure those settings (Debian example):

This would result in a 32KB application buffer for every connection.

If your VerneMQ use case requires the use of different TCP buffer optimisations (per groups of clients for instance) you will have to make sure the that the Linux OS buffer configuration, namely net.ipv4.tcp_wmem and net.ipv4.tcp_rmemallows for this kind of flexibility, allowing for small TCP buffers and big TCP buffers at the same time.

Example 1 above would allow VerneMQ to allocate minimal TCP read and write buffers of 4KB in the Linux Kernel, a max read buffer of 32KB in the kernel, and a max write buffer of 65KB in the kernel. VerneMQ itself would set its own internal per connection buffer to 65KB in addition.

What we just described is VerneMQ automatically configuring TCP read and write buffers and internal buffers, deriving their values from OS settings.

There are multiple additional ways to configure TCP buffers described below:

If VerneMQ finds an advanced.config file, it will use the buffer sizes you have configured there for all it’s TCP listeners (and the TCP connections accepted by those listeners), except the Erlang distribution listeners within the cluster.

(You'll find an example in the section below on the advanced.config )

If VerneMQ finds a per protocol configuration (listener.tcp.buffer_sizes) in the vernemq.conf file, it will use those buffer sizes for the specific protocol. (currently only MQTT or MQTTS. Support for WS/WSS/HTTP/VMQ listeners is on the roadmap).

For listener.tcp.buffer_sizes you’ll always have to state 3 values in bytes: the TCP receive buffer (recbuf), the TCP send buffer (sndbuf), and the internal application side buffer (buffer). You should set “buffer” (the 3rd value) toval(buffer) >= max(val(sndbuf),val(recbuf))

If VerneMQ finds per listener config values (listener.tcp.my_listener.buffer_sizes), it will use those buffer sizes for all connections setup by that specific listener. This is the most useful approach if you want to set specific different buffer sizes, like huge send buffers for listeners that accept massive consumers. (consumers with high expected message throughput).

You would then configure a different listener for those massive consumers, and by that have the option to fine tune the TCP buffer sizes.

For listener.tcp.my_listener.buffer_sizes you’ll always have to state 3 values in bytes: the TCP receive buffer (recbuf), the TCP send buffer (sndbuf), and an internal application side buffer (buffer). You should set “buffer” (the 3rd value) toval(buffer) >= max(val(sndbuf),val(recbuf))

This scenario would be possible with a plugin.

The advanced.config file is a supplementary configuration file that sits in the same directory as the vernemq.conf. You can set additional config values for any of the OTP applications that are part of a VerneMQ release. To just configure the TCP buffer size manually, you can create an advanced.config file:

For very advanced & custom configurations, you can add a vm.args file to the same directory where the vernemq.conf file is located. Its purpose is to configure parameters for the Erlang Virtual Machine. This will override any Erlang specific parameters your might have configured via the vernemq.conf. Normally, VerneMQ auto-generates a vm.args file for every boot in /var/lib/vernemq/generated.configs/ (Debian package example) from vernemq.conf and other potential configuration sources.

This is how a vm.args might look like:

Using TLS will of course increase the CPU load during connection setup. Latencies in message delivery will be increased, and your overall message throughput per second will be lower.

TLS will require considerably more RAM. Instead of 2 Erlang processes per connection, TLS will use 3. You'll have a session process, a queue process, and a TLS handler process that can encapsulate quite a big state (> 30KB).

Erlang/OTP uses it's own TLS implementation, only using OpenSSL for crypto, but not connection handling. For situations with high connection setup rate or overall high connection churn rate, the Erlang TLS implementation might be too slow. On the other hand, Erlang TLS gives you great concurrency & fault isolation for long-lived connections.

Some Erlang deployments terminate SSL/TLS with an external component or with a load balancer component. Do some testing & try to find out what works best for you.

VerneMQ can consume a large number of open file handles when thousands of clients are connected as every connection requires at least one file handle.

Most operating systems can change the open-files limit using the ulimit -n command. Example:

ulimit -n 65536

However, this only changes the limit for the current shell session. Changing the limit on a system-wide, permanent basis varies more between systems.

Linux

On most Linux distributions, the total limit for open files is controlled by sysctl.

sysctl fs.file-max
fs.file-max = 50384

As seen above, it is generally set high enough for VerneMQ. If you have other things running on the system, you might want to consult the manpage for how to change that setting. However, what most needs to be changed is the per-user open files limit. This requires editing /etc/security/limits.conf, for which you'll need superuser access. If you installed VerneMQ from a binary package, add lines for the vernemq user like so, substituting your desired hard and soft limits:

On Ubuntu, if you’re always relying on the init scripts to start VerneMQ, you can create the file /etc/default/vernemq and specify a manual limit like so:

This file is automatically sourced from the init script, and the VerneMQ process started by it will properly inherit this setting. As init scripts are always run as the root user, there’s no need to specifically set limits in /etc/security/limits.conf if you’re solely relying on init scripts.

On CentOS/RedHat systems, make sure to set a proper limit for the user you’re usually logging in with to do any kind of work on the machine, including managing VerneMQ. On CentOS, sudo properly inherits the values from the executing user.

It can be helpful to enable PAM user limits so that non-root users, such as the vernemq user, may specify a higher value for maximum open files. For example, follow these steps to enable PAM user limits and set the soft and hard values for all users of the system to allow for up to 65536 open files.

Edit /etc/pam.d/common-session and append the following line:

If /etc/pam.d/common-session-noninteractive exists, append the same line as above.

Save and close the file.

Edit /etc/security/limits.conf and append the following lines to the file:

1. Save and close the file.

2. (optional) If you will be accessing the VerneMQ nodes via secure shell (ssh), you should also edit /etc/ssh/sshd_config and uncomment the following line:

and set its value to yes as shown here:

1. Restart the machine so that the limits to take effect and verify

   that the new limits are set with the following command:

1. Edit /etc/security/limits.conf and append the following lines to

   the file:

1. Save and close the file.

2. Restart the machine so that the limits to take effect and verify that the new limits are set with the following command:

In Solaris 8, there is a default limit of 1024 file descriptors per process. In Solaris 9, the default limit was raised to 65536. To increase the per-process limit on Solaris, add the following line to /etc/system:

Reference:

To check the current limits on your Mac OS X system, run:

The last two columns are the soft and hard limits, respectively.

To adjust the maximum open file limits in OS X 10.7 (Lion) or newer, edit /etc/launchd.conf and increase the limits for both values as appropriate.

For example, to set the soft limit to 16384 files, and the hard limit to 32768 files, perform the following steps:

1. Verify current limits:

   The response output should look something like this:

2. Edit (or create) /etc/launchd.conf and increase the limits. Add lines that look like the following (using values appropriate to your environment):

Attributions

This work, "Open File Limits", is a derivative of Open File Limits by Riak, used under Creative Commons Attribution 3.0 Unported License. "Open File Limits" is licensed under Creative Commons Attribution 3.0 Unported License by Erlio GmbH.

A typical VerneMQ deployment could from a high level look like the following:

In this scenario MQTT clients connect from the internet and are authenticated and authorized against the Authentication Management Service and publish and receive messages, either with each other or with the Backend-Services which might be responsible for sending control messages to the clients or storing and forwarding messages to other systems or databases for later processing.

To build and deploy a system such as the above a lot of decisions has to be made. These can concern how to do authentication and authorization, where to do TLS termination, how the load balancer should be configured (if one is needed at all), what the MQTT architecture and topic trees should look and how and to what level the system can/should scale. To simplify the following discussion we'll set a few requirements:

• Clients connecting from the internet are using TLS client certificates

• The messaging pattern is largely fan-in: The clients continuously publish a lot of messages to a set of topics which have to be handled by the Backend-Services.

• The client sessions are persistent, which means the broker will store QoS 1 & 2 messages routed to the clients while the clients are offline.

In the following we'll cover some of these options and concerns.

Often a load balancer is deployed between MQTT clients and the VerneMQ cluster. One of the main purposes of the load balancer is to ensure that client connections are distributed between the VerneMQ nodes so each node has the same amount of connections. Usually a load balancer provides different load balancing strategies for deciding how to select the node where it should route an incoming connection. Examples of these are random, source hashing (based on source IP) or even protocol-aware balancing based on for example the MQTT client-id. The last two are examples of sticky balancing or session affine strategies where a client will always be routed to the same cluster node as long as the source IP or client-id remains the same.

When using a load balancer the client is no longer directly connected to the VerneMQ nodes and therefore the peer port and IP-address VerneMQ sees is therefore not that of the client, but of the load balancer. The peer information is often important for logging reasons or if a plugin checks it up against a white/black list.

To solve this problem VerneMQ supports the v1 and v2 which is designed to transport connection information across proxies. See how to enable the proxy protocol for an MQTT listener. In case TLS is terminated at the load balancer and client certificates are used PROXY Protocol (v2) will also take care of forwarding TLS client certificate details.

Often if client certificates are used to verify and authenticate the clients. VerneMQ makes it possible to make the client certificate common name (CN) available for the authentication plugin system by overriding the MQTT username with the CN, before authentication is performed. If TLS is terminated at the load balancer then the PROXY Protocol would be used This works for both if TLS is terminated in a load balancer or if TLS is terminated directly in VerneMQ. In case TLS is terminated at the load balancer then the listener can be configured as follows to achieve this effect:

If TLS is terminated directly in VerneMQ the PROXY protocol isn't needed as the TLS client certificate is directly available in VerneMQ and the CN can be used to instead of the username by setting:

See the details in the section.

The actual authentication can then be handled by an authentication and authorization plugin like which supports , , , and as backends for storing credentials and ACL rules.

Another important aspect of running a VerneMQ is having proper monitoring and alerting in place. All the usual things should be monitored at the OS level such as memory and cpu usage and alerts should be put in place to actions can be taken if a disk is filling up or a VerneMQ node is starting to use too much CPU. VerneMQ exports a large number of metrics and depending on the use case these can be used as important indicators that the system is running

When designing a system like the one described here, there are a number of things to consider in order to get the best performance out of the available resources.

As mentioned earlier clients in this scenario are using persistent sessions. In VerneMQ a persistent session exists only on the VerneMQ node where the client connected. This implies that if the client using a persistent session later reconnects to another node, then the session, including any offline messages, will be moved to the new node. This has a certain overhead and can be avoided if the load balancer in front of VerneMQ is using a session affine load balancing strategy such as IP source hashing to assign the client connecting to a node. Of course this strategy isn't perfect if clients often change their IP addresses, but for most cases it is a huge improvement over a random load balancing strategy.

In many systems the MQTT clients provide a lot of data by periodically broadcasting data to the MQTT cluster. The amount of published messages can very easily become hard to manage for a single MQTT client. Further using normal MQTT subscriptions all subscribers would receive the same messages, so adding more subscribers to a topic doesn't help handling the amount of messages. To solve this VerneMQ implements a concept called which makes it possible to distribute MQTT messages published to a topic over several MQTT clients. In this specific scenario this would mean the Backend-Services would consist of a set of clients subscribing to cluster nodes using shared subscriptions.

To avoid expensive intra-node communication, VerneMQ shared subscriptions support a policy called local_only which means that messages being will be delivered to shared subscribers on the local node only and not forwarded to shared subscribers on other nodes in the cluster. With this policy messages for the backend-services can be delivered in the fastest and most expedient manner with the lowest overhead. See the documentation for more information.

Controlling TCP buffer sizes is important in ensuring optimal memory usage. The rule is that the more bandwith or the lower latency required, the larger the TCP buffer sizes should be. Many IoT communicate with a very low bandwith and as such the server side TCP buffer sizes for these does not need to be very large. On the other hand, in this scenario the consumers handling the fan-ins in the Bacend-Services will have many (thousands or tens of thousands of messages per second) and they can benefit from larger TCP buffer sizes. Read more about tuning TCP buffer sizes .

An important guideline in protecting a VerneMQ cluster from overload is to allow only what is necessary. This means having and enforcing sensible authentication and authorization rules as well as configuring conservatively so resources cannot be exhausted due to human error or MQTT clients that have turned mailicious. For example in VerneMQ it is possible to specify how many offline messages a persistent session can maximally hold via the max_offline_messages setting - and it should then be set to the lowest acceptable value which works for all clients and/or use a plugin which is able to override such settings on a per-client basis. The load balancer can also play an important role in protecting the system in that it can control the connect rates as well as imposing bandwith restrictions on clients.

Somehow a system like this has to be deployed. How to do this will not be covered here, bit it is certainly possible to deploy VerneMQ using tools like, or or use container solutions such as Kubernetes. For more information on how to deploy VerneMQ on Kubernetes check out our guide: .

Kubernetes (K8s) is possibly the most mature technology for deploying Docker containers at scale. While running a single Docker container is supposed to be easy, running a Kubernetes cluster definitely isn't. That's why we recommended to work with a certified Kubernetes partner such as , , , or .

If your applications already live in Docker containers and are deployed on Kubernetes it can be beneficial to also run VerneMQ on Kubernetes. This guide covers how to successfully deploy a VerneMQ cluster on Kubernetes. Multiple options exist to deploy a VerneMQ cluster at this point. This guide describes how to use the official Helm chart as well as the still experimental Kubernetes Operator.

For the sake of clarity, this guide defines the following terms:

VerneMQ comes with a simple file-based password authentication mechanism which is enabled by default. If you don't need this it can be disabled by setting:

Per default VerneMQ doesn't accept any client that hasn't been configured using vmq-passwd. If you want to change this and accept any client connection you can set:

In a production setup we recommend to use the provided password based authentication mechanism or implement your own authentication plugins.

VerneMQ periodically checks the specified password file.

The check interval defaults to 10 seconds and can also be defined in the vernemq.conf.

The VerneMQ HTTP API is enabled by default and installs an HTTP handler on http://localhost:8888/api/v1. To read more about configuring the HTTP listener, see HTTP Listener Configuration. You can configure a HTTP listener, or a HTTPS listener to serve the HTTP API v1.

Managing API keys

The VerneMQ HTTP API uses basic authentication where an API key is passed as the username and the password is left empty. So the first step to us the HTTP API is to create an API key:

$ vmq-admin api-key create
JxctXkZ1OTVnlwvguSCE9KtujacMkOLF

The key is persisted and available on all cluster nodes.

To list existing keys do:

$ vmq-admin api-key show
+--------------------------------+
|              Key               |
+--------------------------------+
|JxctXkZ1OTVnlwvguSCE9KtujacMkOLF|
+--------------------------------+

To add an API key of your own choosing, do:

vmq-admin api-key add key=mykey

To delete an API key do:

The VerneMQ HTTP API is a wrapper over the CLI tool, and anything that can be done using vmq-admin can be done using the HTTP API. Note that the HTTP API is therefore subject to any changes made to the vmq-admin tools and their flags & options structure. All requests are performed doing a HTTP GET and if no errors occurred an HTTP 200 OK code is returned with a possible non-empty JSON payload.

The API is using basic auth where the API key is passed as the username. An example using curl would look like this:

The mapping between vmq-admin and the HTTP API is straightforward, and if one is already familiar with how the vmq-admin tool works, working with the API should be easy. The mapping works such that the command part of a vmq-admin invocation is turned into a path, and the options and flags are turned into the query string.

A mandatory parameter like the client-id in the vmq-admin session disconnect client-id=myclient command should be translated as: ?client-id=myclient.

An optional flag like --cleanup in the vmq-admin session disconnect client-id=myclient --cleanup command should be translated as: &--cleanup

Let's look at the cluster join command as an example, which looks like this:

This turns into a GET request:

To test, run it with curl:

And the returned response would look like:

Below are some other examples.

Request:

Curl:

Response:

Request:

Curl:

Response:

Request:

Curl:

Response:

Request:

Curl:

Response:

Request:

Curl:

Response:

Request:

Curl:

Response:

Introduction and general setup

VerneMQ supports authentication and authorization using a number of popular databases and the below sections describe how to configure the different databases.

The database drivers are handled using the vmq_diversity plugin and it therefore needs to be enabled:

plugins.vmq_diversity = on

When using database based authentication/authorization the enabled-by-default file based authentication and authorization are most likely not needed and should be disabled:

plugins.vmq_passwd = off
plugins.vmq_acl = off

In order to use a database for authentication and authorization the database must be properly configured and the auth-data (username, clientid, password, acls) to be present. The following sections show some sample requests that can be used to insert such data.

While the handling of authentication differs among the different databases, the handling of ACLs is roughly identical and make use of a JSON array containing one or many ACL objects per configured client.

A minimal publish & subscribe ACL JSON object takes the following form:

General ACL

The pattern is a MQTT topic string that can contain MQTT wildcards, but also the template variables %m (mountpoint), %u (username), and %c (client id) which are automatically substituted with the auth data provided.

Publish ACL

The publish ACL makes it possible to control the maximum QoS and payload size that is allowed, and if the message is allowed to be retained.

Moreover, the publish ACL makes it possible to modify the properties of a published message through specifying one or multiple modifiers. Please note that the modified message isn't re-validated by the ACL.

Subscribe ACL

The subscribe ACL makes it possible to control the maxium QoS a client is allowed to subscribe to.

Like the publish ACL, the subscribe ACL makes it possible to change the current subscription request by returning a custom set of topic/qos pairs. Please note that the modified subscription isn't re-validated by the ACL.

When deciding on which database to use one has to consider which kind of password hashing and key derivation functions are available and required. Different databases provide different mechanisms, for example PostgreSQL provides the pgcrypto module which supports verifying hashed and salted passwords, while Redis has no such features. VerneMQ therefore also provides client-side password verification mechanisms such as bcrypt.

There is a trade-off between verifying passwords on the client-side versus on the server-side. Verifying passwords client-side of course means doing the computations on the VerneMQ broker and this takes away resources from other tasks such as routing messages. With hashing functions such as bcrypt which are designed specifically to be slow (proportional to the number of rounds) in order to make brute-force attacks infeasible, this can become a problem. For example, if verifying a password with bcrypt takes 0.5 seconds then on a single threaded core 2 verifications/second are possible and using 4 single threaded cores 8 verifications/second. So, the number of rounds/security paramenters have a direct impact on the max number of verifications/second and hence also the maximum arrival rate of new clients per second.

For each database it is specified which password verification mechanisms are available and if they are client-side or server-side.

To enable PostgreSQL authentication and authorization the following need to be configured in the vernemq.conf file:

PostgreSQL hashing methods:

The following SQL DDL must be applied, the pgcrypto extension is required if using the server-side crypt hashing method:

To enter new ACL entries use a query similar to the following:

To enable PostgreSQL authentication and authorization the following need to be configured in the vernemq.conf file:

Notice that if the CockroachDB installation is secure, then TLS is required. If using an insecure installation without TLS, then vmq_diversity.cockroachdb.ssl can be set to off.

CockroachDB hashing methods:

The following SQL DDL must be applied:

To enter new ACL entries use a query similar to the following, the example is for the bcrypt hashing method:

For MySQL authentication and authorization configure the following in vernemq.conf:

MySQL hashing methods:

It should be noted that all the above options stores unsalted passwords which are vulnerable to rainbow table attacks, so the threat-model should be considered carefully when using these. Also note the methods marked with * are no longer considered secure hashes.

The following SQL DDL must be applied:

To enter new ACL entries use a query similar to the following, the example uses PASSWWORD to for password hashing:

For MongoDB authentication and authorization configure the following in vernemq.conf:

MongoDB hashing methods:

Insert the ACL using the mongo shell or any software library. The passhash property contains the bcrypt hash of the clients password.

For Redis authentication and authorization configure the following in vernemq.conf:

Redis hashing methods:

Insert the ACL using the redis-cli shell or any software library. The passhash property contains the bcrypt hash of the clients password. The key is an encoded JSON array containing the mountpoint, username, and client id. Note that no spaces are allowed between the array items.

Note, currently bcrypt version 2a (prefix $2a$) is supported.

Developing VerneMQ plugins in Erlang is the most powerful way to extend the functionality of a VerneMQ broker but is a barrier developers not familiar with Erlang. For this reason we've implemented a VerneMQ extension that allows you to develop plugins using the Lua scripting language. This extension is called vmq_diversity and is shipped as part of VerneMQ.

Moreover vmq_diversity provides simple Lua libraries to communicate with MySQL, PostgreSQL, MongoDB, and Redis within your Lua VerneMQ plugins. An additional Json encoding/decoding library as well as a generic HTTP client library provide your Lua scripts a great way to talk to external services.

Configuration

To enable vmq_diversity make sure to set:

plugins.vmq_diversity = on

To specify a script to load when VerneMQ starts can be done like this:

vmq_diversity.myscript1.file = Path/to/Script.lua

It is also possible to dynamically load a Lua script using vmq-admin:

To reload a script after a change:

A VerneMQ plugin typically consists of one or more implemented VerneMQ hooks. We tried to keep the differences between the traditional Erlang based and Lua based plugins as small as possible. Please check out the for more information about the different flows and a description of the different hooks.

Let's start with a first very basic example that implements a basic authentication and authorization scheme.

This subsection describes the data providers currently available to a Lua script. Every data provider is backed by a connection pool that has to be configured by your script.

ensure_pool

Ensures that the connection pool named config.pool_id is setup in the system. The config argument is a Lua table holding the following keys:

• pool_id: Name of the connection pool (mandatory).

• size: Size of the connection pool (default is 5).

• user

This call throws a badarg error in case it cannot setup the pool otherwise it returns true.

execute

Executes the provided SQL statement using a connection from the connection pool.

• pool_id: Name of the connection pool to use for this statement.

• stmt: A valid MySQL statement.

• args...

Depending on the statement this call returns true or false or a Lua array containing the resulting rows (as Lua tables). In case the statement cannot be executed a badarg error is thrown.

ensure_pool

Ensures that the connection pool named config.pool_id is setup in the system. The config argument is a Lua table holding the following keys:

• pool_id: Name of the connection pool (mandatory).

• size: Size of the connection pool (default is 5).

• user

This call throws a badarg error in case it cannot setup the pool otherwise it returns true.

execute

Executes the provided SQL statement using a connection from the connection pool.

• pool_id: Name of the connection pool to use for this statement.

• stmt: A valid MySQL statement.

• args...

Depending on the statement this call returns true or false or a Lua array containing the resulting rows (as Lua tables). In case the statement cannot be executed a badarg error is thrown.

ensure_pool

Ensures that the connection pool named config.pool_id is setup in the system. The config argument is a Lua table holding the following keys:

• pool_id: Name of the connection pool (mandatory).

• size: Size of the connection pool (default is 5).

• login

This call throws a badarg error in case it cannot setup the pool otherwise it returns true.

insert

Insert the provided document (or list of documents) into the collection.

• pool_id: Name of the connection pool to use for this statement.

• collection: Name of a MongoDB collection.

• doc_or_docs

The provided document can set the document id using the _id key. If the id isn't provided one gets autogenerated. The call returns the inserted document(s) or throws a badarg error if it cannot insert the document(s).

update

Updates all documents in the collection that match the given selector.

• pool_id: Name of the connection pool to use for this statement.

• collection: Name of a MongoDB collection.

• selector

The call returns true or throws a badarg error if it cannot update the document(s).

delete

Deletes all documents in the collection that match the given selector.

• pool_id: Name of the connection pool to use for this statement.

• collection: Name of a MongoDB collection.

• selector

The call returns true or throws a badarg error if it cannot delete the document(s).

find

Finds all documents in the collection that match the given selector.

• pool_id: Name of the connection pool to use for this statement.

• collection: Name of a MongoDB collection.

• selector

The call returns a MongoDB cursor or throws a badarg error if it cannot setup the iterator.

next

Fetches next available document given a cursor object obtained via find.

The call returns the next available document or false if all documents have been fetched.

take

Fetches the next nr_of_docs documents given a cursor object obtained via find.

The call returns a Lua array containing the documents or false if all documents have been fetched.

close

Closes and cleans up a cursor object obtained via find.

The call returns true.

find_one

Finds the first document in the collection that matches the given selector.

• pool_id: Name of the connection pool to use for this statement.

• collection: Name of a MongoDB collection.

• selector

The call returns the matched document or false in case no document was found.

ensure_pool

Ensures that the connection pool named config.pool_id is setup in the system. The config argument is a Lua table holding the following keys:

• pool_id: Name of the connection pool (mandatory).

• size: Size of the connection pool (default is 5).

• password

This call throws a badarg error in case it cannot setup the pool otherwise it returns true.

cmd

Executes the given Redis command.

• pool_id: Name of the connection pool

• command: Redis command string.

• args...