Install VerneMQ

Once you have downloaded the binary package, execute the following command to install VerneMQ:

Note: Replace bionic with appropriate OS version such as focal/trusty/xenial.

Verify your installation

You can verify that VerneMQ is successfully installed by running:

If VerneMQ has been installed successfully Status: install ok installed is returned.

Activate VerneMQ node

Once you've installed VerneMQ, start it on your node:

Default Directories and Paths

The whereis vernemq command will give you a couple of directories:

Next Steps

Now that you've installed VerneMQ, check out .

Installing VerneMQ

VerneMQ is a high-performance, distributed MQTT message broker. It scales horizontally and vertically on commodity hardware to support a high number of concurrent publishers and consumers while maintaining low latency and fault tolerance. To use it, all you need to do is install the VerneMQ package.

Choose your OS and follow the instructions:

Install VerneMQ

Once you have downloaded the binary package, execute the following command to install VerneMQ:

or:

Every VerneMQ node has to be configured as the default configuration probably does not match your needs. Depending on the installation method and chosen platform the configuration file vernemq.conf resides at different locations. If VerneMQ was installed through a Linux package the default location for the configuration file is /etc/vernemq/vernemq.conf.

General Format of the vernemq.conf file

Maximum Client Id Size

Set the maximum size for client ids, MQTT v3.1 specifies a limit of 23 characters.

This option default to 23.

VerneMQ supports the WebSocket protocol out of the box. To be able to open a WebSocket connection to VerneMQ, you have to configure a WebSocket listener or Secure WebSocket listener in the vernemq.conf file first:

listener.ws.default = 127.0.0.1:9001

listener.wss.wss_default = 127.0.0.1:9002
# To use WSS, you'll have to configure additional options for your WSS listener (called `wss_default` here):
listener.wss.wss_default.cafile = ./etc/cacerts.pem
listener.wss.wss_default.certfile = ./etc/cert.pem
listener.wss.wss_default.keyfile = ./etc/key.pem

Keep in mind that you'll use MQTT-over-WebSocket, so you will need a Javascript library that implements the MQTT client behaviour. We have used the Eclipse Paho client as well as MQTT.js

You won't be able to open WebSocket connections on a base URL, always add the /mqtt path.

When establishing a WebSocket connection to the VerneMQ MQTT broker, the process begins with an HTTP connection that is then upgraded to WebSocket. This upgrade mechanism means the broker's ability to accept connections can be influenced by HTTP listener settings.

In certain scenarios, such as when connecting from a frontend application, the size of HTTP request headers (including cookies) can exceed the default maximum allowed by VerneMQ. This can lead to a 'HTTP 431 Request Header Fields Too Large' error, preventing the connection from being established.

This behavior is configurable in the vernemq.conf file to accommodate larger headers:

Console Logging

Where should VerneMQ emit the default console log messages (which are typically at info severity):

log.console = off | file | console | both

VerneMQ defaults to log the console messages to a file, which can specified by:

This option defaults to /var/log/vernemq/console.log for Ubuntu, Debian, RHEL and Docker installs.

The default console logging level info could be setting one of the following:

Error Logging

VerneMQ log error messages by default. One can change the default behaviour by setting:

VerneMQ defaults to log the error messages to a file, which can specified by:

This option defaults to /var/log/vernemq/error.log for Ubuntu, Debian, RHEL and Docker installs.

Crash Logging

VerneMQ log crash messages by default. One can change the default behaviour by setting:

VerneMQ defaults to log the crash messages to a file, which can specified by:

This option defaults to /var/log/vernemq/crash.log for Ubuntu, Debian, RHEL and Docker installs.

The maximum sizes in bytes of individual messages in the crash log defaults to 64KB but can be specified by:

VerneMQ rotate crash logs. By default, the crash log file is rotated at midnight or when the size exceeds 10MB. This behaviour can be changed by setting:

The default number of rotated log files is 5 and can be set with the option:

SysLog

VerneMQ supports logging to SysLog, enable it by setting:

Logging to SysLog is disabled by default.

The VerneMQ HTTP listener is used to serve various VerneMQ subsystems such as:

• Status page

• Prometheus metrics

• management API

By default listener runs on port 8888. To disable the HTTP listener, use a HTTPS listener instead or change the port, adapt the configuration in vernemq.conf:

You can have multiple HTTP(s) listener listening to different port and running different modules:

This configuration snippet defines two HTTPS listeners with different modules. One for default traffic and one for management traffic. It specifies which HTTP modules will be enabled on each listener, allowing for status, health, and metrics information to be retrieved from the default listener and providing a web-based interface for managing and monitoring VerneMQ through the management listener.

Retry Interval

Set the time in seconds after a QoS=1 or QoS=2 message has been sent that VerneMQ will wait before retrying when no response is received.

retry_interval = 20

This option default to 20 seconds.

Inflight Messages

This option defines the maximum number of QoS 1 or 2 messages that can be in the process of being transmitted simultaneously.

Defaults to 20 messages, use 0 for no limit. The inflight window serves as a protection for sessions, on the incoming side.

Load Shedding

The maximum number of messages to hold in the queue above those messages that are currently in flight. Defaults to 1000. Set to -1 for no limit. This option protects a client session from overload by dropping messages (of any QoS).

Defaults to 1000 messages, use -1 for no limit. This parameter was named max_queued_messages in 0.10.*. Note that 0 will totally block message delivery from any queue!

This option specifies the maximum number of QoS 1 and 2 messages to hold in the offline queue.

Defaults to 1000 messages, use -1 for no limit, use 0 if no messages should be stored.

In contrast to the session based inflight window, max_online_messages and max_offline_messages serves as a protection of queues, on the outgoing side.

When an offline session transits to online, by default VerneMQ will adhere to the queue sizes also for moving data from the offline queue to the online queue. Therefore, if max_offline_messages > max_online_message VerneMQ will start dropping messages. It is possible to override this behaviour and allow VerneMQ to move all messages from the offline queue to the online queue. The queue will then batched (or streamed) to the subscribers, and the messages are read from disk in batches as well. The additional memory needed thus is just the amount needed to store references to those messages and not the messages themselves.

Welcome to the VerneMQ documentation! This is a reference guide for most of the available features and options of VerneMQ. The Getting Started guide might be a good entry point.

The VerneMQ documentation is based on the VerneMQ Documentation project. Any changes on Github are automatically deployed to the VerneMQ online Documentation.

For a more general overview on VerneMQ and MQTT, you might want to start with the introduction.

For downloading the subscription-based binary VerneMQ packages and/or a quick description on how to compile VerneMQ from sources, see Downloads.

How to help improve this documentation

The is an open-source effort, and your contributions are very welcome and appreciated. You can contribute on all levels:

• Language, style and typos

• Fixing obvious documentation errors and gaps

• Providing more details and/or examples for specific topics

• Extending the documentation where you find this useful to do

Note that the documentation is versioned according to the VerneMQ releases. You can click the "Edit on Github" button in the upper right corner of every page to check what branch and document you are on. You can then create a Pull Request (PR) against that branch from your fork of the VerneMQ documentation repository. (Direct edits on Github are possible for members of the documentation repository).

During every boot up, VerneMQ will run your vernemq.conf file against the Schema files of the VerneMQ release. This serves as a validation and as a mechanism to create the timestamped internal config files that you'll find in the generated.configs directory.

In general, every application of the VerneMQ release has its own schema file in the priv subdirectory (the only exception is the vmq.schema file in the file directory). A my_app.schema defines all the configuration settings you can use for that application.

And that's almost the only reason to know a bit about schema files: you can browse them for possible settings if you suspect a minor settings is not yet fully documented. Most of the time you'll also find at least a short snippet documenting the setting in the schema.file.

An example from the vmq_server.schema:

This is a relatively minor feature where you can set a default expiry for API keys. You can determine from the mapping schema that a default is not set. To set the value in the vernemq.conf file, always use the left-hand name from the mapping in the schema:

You can also see the keyword hidden in the mapping. This means that the setting will not show up automatically in the vernemq.conf file and you'll have to add the it manually.

VerneMQ supports enhanced authentication flows or SASL style authentication for MQTT 5.0 sessions. The enhanced authentication mechanism can be used for initial authentication when the client connects or to re-authenticate clients at a later point.

The on_auth_m5 hook allows the plugin to implement SASL style authentication flows by either accepting, rejecting (disconnecting the client) or continue the flow. The on_auth_m5 hook is specified in the Erlang behaviour on_auth_m5_hook in the vernemq_dev repo.

Listeners specify on which IP address and port VerneMQ should accept new incoming connections. Depending on the chosen transport (TCP, SSL, WebSocket) different configuration parameters have to be provided. VerneMQ allows to write the listener configurations in a hierarchical manner, enabling very flexible setups. VerneMQ applies reasonable defaults on the top level, which can be of course overridden if needed.

These are the only default parameters that are applied for all transports, and the only one that are of interest for plain TCP and WebSocket listeners.

These global defaults can be overridden for a specific transport protocol listener.tcp.CONFIG = VAL, or even for a specific listener listener.tcp.LISTENER.CONFIG = VAL. The placeholder LISTENER is freely chosen and is only used as a reference for further configuring this particular listener.

You can configure as many listeners as you wish in the vernemq.conf file. In addition to this, the vmq-admin listener command let's you configure, start, stop and delete listeners on the fly. Those can be MQTT, WebSocket or Cluster listeners, in the command line output they will be tagged mqtt, ws or vmq accordingly.

On every VerneMQ node you'll find the vmq-admin command line tool in the release's bin directory (in case you use the binary VerneMQ packages, vmq-admin should already be callable in your path, without changing directories). It has different sub-commands that let you check for status, start and stop listeners, re-configure values and a couple of other administrative tasks.

vmq-admin has different sub-commands with a lot of respective options. You can familiarize yourself by using the --help option on the different levels of vmq-admin. You might see additional sub-commands in case integrated plugins are running (vmq-admin bridge is an example).

VerneMQ can be monitored in several ways. We implemented native support for , , and .

The metrics are also available via the command line tool:

Or with:

Which will output the metrics together with a short description describing what the metric is about. An example looks like:

Notice that the metrics:

Are no longer used (always 0) and will be removed in the future. They were replaced with mqtt_connack_sent using the return_code label. For MQTT 5.0 the reason_code label is used instead.

The output on the command line are aggregated by default, but details for a label can be shown as well, for example all metrics with the not_authorized

VerneMQ supports different Transport Layer Security (TLS) options, which allow for secure communication between MQTT clients and VerneMQ. Certificates typically have only a limited validity (for example one year) after which they have to be replaced. VerneMQ allows to replace a certificate without interrupting active connections.

Replace a certificate

Replacing a certificate straightforward. One just need to replace (overwrite) the corresponding PEM files. VerneMQ will pickup the new certificates.

For example, if you have the following configuration

the files cacerts.pem, cert.pem and key.pem can be overwritten (on the filesystem!) with new certificates. VerneMQ will pick-up the certificate after some time (by default around 2min). It is possible to invalidate the certificate immedialtly by issuing the following command

The graphite exporter reports the broker metrics at a fixed interval (defined in milliseconds) to a graphite server. The necessary configuration is done inside the vernemq.conf.

You can further tune the connection to the Graphite server:

VerneMQ comes with a built-in Status Page that is enabled by default and is available on http://localhost:8888/status, see .

The Status Page is a simple overview of the cluster and the individual nodes in the cluster as seen below. Note that while the Status Page is running on each node of the cluster, it's enough to look at one of them to get a quick status of your cluster.

The Status Page has the following sections:

• Issues (Warnings on netsplits, etc)

A simple way to gauge the health of a VerneMQ cluster is to query the /health path on the .

The health check will return 200 when VerneMQ is accepting connections and is joined with the cluster (for clustered setups). 503 will be returned in case any of those two conditions are not met. In addition to the simple /health path, the following options are available as well

• /health/ping: Cowboy (ie. Verne) is up.

The systree functionality is enabled by default and reports the broker metrics at a fixed interval defined in the vernemq.conf. The metrics defined are transformed to MQTT topics e.g. mqtt_publish_received is transformed to $SYS/<nodename>/mqtt/publish/received. <nodename> is your node's name, as configured in the vernemq.conf. To find it, you can grep the file for it: grep nodename vernemq.conf

The complete list of metrics can be found

This option defaults to 20000 milliseconds.

If the systree feature is not required it can be disabled in vernemq.conf

VerneMQ uses the Erlang distribution mechanism for most inter-node communication. VerneMQ identifies other machines in the cluster using Erlang identifiers (e.g. [email protected]). Erlang resolves these node identifiers to a TCP port on a given machine via the Erlang Port Mapper daemon (epmd) running on each cluster node.

By default, epmd binds to TCP port 4369 and listens on the wildcard interface. For inter-node communication, Erlang uses an unpredictable port by default; it binds to port 0, which means the first available port.

For ease of firewall configuration, VerneMQ can be configured to instruct the Erlang interpreter to use a limited range of ports. For example, to restrict the range of ports that Erlang will use for inter-Erlang node communication to 6000-7999, add the following lines to vernemq.conf on each VerneMQ node:

erlang.distribution.port_range.minimum = 6000
erlang.distribution.port_range.maximum = 7999

The settings above are only used for distributing subscription updates and maintenance messages. For distributing the 'real' MQTT messages the proper vmq listener must be configured in the vernemq.conf.

listener.vmq.clustering = 0.0.0.0:44053

Attributions:

This section, "VerneMQ Inter-node Communication", is a derivative of Security and Firewalls by Riak, used under Creative Commons Attribution 3.0 Unported License.

We recommend to use the rebar3 toolchain to generate the basic Erlang OTP application boilerplate and start from there.

rebar3 new app name="myplugin" desc="this is my first VerneMQ plugin"
===> Writing myplugin/src/myplugin_app.erl
===> Writing myplugin/src/myplugin_sup.erl
===> Writing myplugin/src/myplugin.app.src
===> Writing myplugin/rebar.config
===> Writing myplugin/.gitignore
===> Writing myplugin/LICENSE
===> Writing myplugin/README.md

Change the rebar.config file to include the vernemq_dev dependency:

{erl_opts, [debug_info]}.
{deps, [{vernemq_dev,
    {git, "git://github.com/vernemq/vernemq_dev.git", {branch, "master"}}}
]}.

Compile the application, this will automatically fetch vernemq_dev.

rebar3 compile                             
===> Verifying dependencies...
===> Fetching vmq_commons ({git,
                                      "git://github.com/vernemq/vernemq_dev.git",
                                      {branch,"master"}})
===> Compiling vernemq_dev
===> Compiling myplugin

Now you're ready to implement the hooks. Don't forget to add the proper vmq_plugin_hooks entries to your src/myplugin.app.src file.

For a complete example, see the vernemq_demo_plugin.

Sometimes consumers get overwhelmed by the number of messages they receive. VerneMQ can load balance between multiple consumer instances subscribed to the same topic with the same ClientId.

Enabling Session Balancing

To enable session balancing, activate the following two settings in vernemq.conf

This section elaborates how a VerneMQ cluster deals with network partitions (aka. netsplit or split brain situation). A netsplit is mostly the result of a failure of one or more network devices resulting in a cluster where nodes can no longer reach each other.

VerneMQ is able to detect a network partition, and by default it will stop serving CONNECT, PUBLISH, SUBSCRIBE, and UNSUBSCRIBE requests. A properly implemented client will always resend unacked commands and messages are therefore not lost (QoS 0 publishes will be lost). However, the time window between the network partition and the time VerneMQ detects the partition much can happen. Moreover, this time frame will be different on every participating cluster node. In this guide we're referring to this time frame as the Window of Uncertainty.

Possible Scenario for Message Loss:

VerneMQ follows an eventually consistent model for storing and replicating the subscription data. This also includes retained messages.

Due to the eventually consistent data model it is possible that during the Window of Uncertainty a publish won't take into account a subscription made on a remote node (in another partition). Obviously, VerneMQ can't deliver the message in this case. The same holds for delivering retained messages to remote subscribers.

last will messages that are triggered during the Window of Uncertainty will be delivered to the reachable subscribers. Currently during a netsplit, but after the Window of Uncertainty last will messages will be lost.

Possible Scenario for Duplicate Clients:

Normally, client registration is synchronized using an elected leader node for the given client id. Such a synchronization removes the race condition between multiple clients trying to connect with the same client id on different nodes. However, during the Window of Uncertainty it is currently possible that VerneMQ fails to disconnect a client connected to a different node. Although this scenario sounds like artificially crafted it is possible to end up with duplicate clients connected to the cluster.

Recovering from a Netsplit

As soon as the partition is healed, and connectivity reestablished, the VerneMQ nodes replicate the latest changes made to the subscription data. This includes all the changes 'accidentally' made during the Window of Uncertainty. Using VerneMQ ensures that convergence regarding subscription data and retained messages is eventually reached.

Default Output Format

The default output format is called human-readable. It will print tables or text answers in response to your CLI commands.

JSON Output Format

The only alternative format is JSON. You can request it by adding the --format=json key to a command.

To pretty-print your JSON or extract the table object, use the jq command. Currently, not all responses give you a nice table and attributes format. Namely, vmq-admin metrics show will only give the metrics as text.

A shared subscription is a mechanism for distributing messages to a set of subscribers to shared subscription topic, such that each message is received by only one subscriber. This contrasts with normal subscriptions where each subscriber will receive a copy of the published message.

A shared subscription is on the form $share/sharename/topic and subscribers to this topic will receive messages published to the topic topic. The messages will be distributed according to the defined distribution policy.

The MQTT spec only defines shared subscriptions for protocol version 5. VerneMQ supports shared subscription for v5 (as per the specification) and for v3.1.1 (backported feature).

Configuration

Currently four message distribution policies for shared subscriptions are supported: prefer_local, random, local_only and prefer_online_before_local. Under the random policy messages will be published to a random member of the shared subscription, if any exist. Under the prefer_local policy messages will be delivered to a random node-local member of the shared subscription, if none exist, the message will be delivered to a random member of the shared subscription on a remote cluster node. The prefer_online_before_local policy works similar to prefer_local, but will look for an online subscriber on a non-local node, if there are only offline subscribers on the local one. Under the local_only policy message will be delivered to a random node-local member of the shared subscription.

When a messages is being delivered to subscribers of a shared subscription, the message will be delivered to an online subscriber if possible, otherwise the message will be delivered to an offline subscriber.

Examples

Subscriptions Note: When subscribing to a shared topic, make sure to escape the $

So, for dash or bash shells

Publishing Note: When publishing to a shared topic, do not include the prefix $share/group/ as part of the publish topic name

VerneMQ uses Google's LevelDB as a fast storage backend for messages and subscriber information. Each VerneMQ node runs its own embedded LevelDB store.

Configuration of LevelDB memory

There's not much you need to know about LevelDB and VerneMQ. One really important thing to note is that LevelDB manages its own memory. This means that VerneMQ will not allocate and free memory for LevelDB. Instead, you'll have to tell LevelDB how much memory it can use up by setting leveldb.maximum_memory.percent.

Configuring LevelDB memory:

leveldb.maximum_memory.percent = 20

Advanced options

(e)LevelDB exposes a couple of additional configuration values that we link here for the sake of completeness. You can change all the values mentioned in the. VerneMQ mostly uses the configured defaults, and for most use cases it should not be necessary to change those.

To list the retained messages simply invoke vmq-admin retain show:

$ vmq-admin retain show
+------------------+----------------+
|     payload      |     topic      |
+------------------+----------------+
| a-third-message  | a/third/topic  |
|some-other-message|some/other/topic|
|    a-message     |   some/topic   |
|    a-message     | another/topic  |
+------------------+----------------+

Besides listing the retained messages it is also possible to filter them:

$ vmq-admin retain show --payload --topic=some/topic
+---------+
| payload |
+---------+
|a-message|
+---------+

In the above example we list only the payload for the topic some/topic.

Another example where all topics are list with retained messages with a specific payload:

See the full set of options and documentation by invoking vmq-admin retain show --help:

Sometimes you want to have a quick way to test a cluster on your development machine as a VerneMQ developer.

You need to take care of a couple things if you want to run multiple VerneMQ instances on the same machine. There is a make option that let's you build multiple releases, as a commodity, taking care of all the configuration.

First, build a normal release (this is just needed the first time) with:

➜ default git:(master) ✗ make rel

The following command will then prepare 3 correctly configured vernemq.conf files, with different ports for the MQTT listeners etc. It will also build 3 full VerneMQ releases.

➜ default git:(master) ✗ make dev1 dev2 dev3

Check if you have the 3 new releases in the _build directory of your VerneMQ code repo.

You can then start the respective broker instances in 3 terminal windows, by using the respective commands and directory paths. Example:

➜ (_build/dev2/rel/vernemq/bin) ✗ vernemq console

The MQTT listeners will of course be configured differently for each node (the default 1883 port is not used, so that you can still run a default MQTT broker besides your dev nodes). A couple of other ports are also adapted (HTTP status page, cluster communication). The MQTT ports are automically configured in increasing steps of 50: (if in doubt, consult the respective vernemq.conf files)

Note that the dev nodes are not automatically clustered. You still need to manually cluster them with commands like the following:

➜ (_build/dev2/rel/vernemq/bin) ✗ vmq-admin cluster join [email protected]

The Prometheus exporter is enabled by default and installs an HTTP handler on http://localhost:8888/metrics. To read more about configuring the HTTP listener, see HTTP Listener Configuration.

Example Scrape Config

Add the following configuration to the scrape_configs section inside prometheus.yml of your Prometheus server.

# A scrape configuration containing exactly one endpoint to scrape: 
# Here it's Prometheus itself.
scrape_configs:
  - job_name: 'vernemq'
    scrape_interval: 5s
    scrape_timeout: 5s
    static_configs:
      - targets: ['localhost:8888']

This tells Prometheus to scrape the VerneMQ metrics endpoint every 5 seconds.

Please follow the documentation on the website to properly configure the metrics scraping as well as how to access those metrics and configure alarms and graphs.

In this section the publish flow is described. VerneMQ provides multiple hooks throughout the flow of a message. The most important ones are the auth_on_publish and auth_on_publish_m5 hooks which acts as an application level firewall granting or rejecting a publish message.

auth_on_publish and auth_on_publish_m5

The auth_on_publish and auth_on_publish_m5 hooks allow your plugin to grant or reject publish requests sent by a client. It also enables to rewrite the publish topic, payload, qos, or retain flag and in the case of auth_on_publish_m5 properties. The auth_on_publish hook is specified in the Erlang behaviour and the auth_on_publish_m5 hook in the behaviour available in the repo.

Every plugin that implements the auth_on_publish or auth_on_publish_m5 hooks are part of a conditional plugin chain. For this reason we allow the hook to return different values. In case the plugin can't validate the publish message it is best to return next as this would allow subsequent plugins in the chain to validate the request. If no plugin is able to validate the request it gets automatically rejected.

on_publish and on_publish_m5

The on_publish and on_publish_m5 hooks allow your plugin to get informed about an authorized publish message. The hook is specified in the Erlang behaviour and the on_publish_m5 hook in the behaviour available in the repo.

on_offline_message

The on_offline_message hook allows your plugin to get notified about a new a queued message for a client that is currently offline. The hook is specified in the Erlang behaviour available in the repo.

on_deliver and on_deliver_m5

The on_deliver and on_deliver_m5 hooks allow your plugin to get informed about outgoing publish messages, but also allows you to rewrite topic and payload of the outgoing message. The hook is specified in the Erlang behaviour and the on_deliver_m5 hook in the behaviour available in the repo.

Every plugin that implements the on_deliver or on_deliver_m5 hooks are part of a conditional plugin chain, although NO verdict is required in this case. The message gets delivered in any case. If your plugin uses this hook to rewrite the message the plugin system stops evaluating subsequent plugins in the chain.

In this section the subscription flow is described. VerneMQ provides several hooks to intercept the subscription flow. The most important ones are the auth_on_subscribe and auth_on_subscribe_m5 hooks which act as an application level firewall granting or rejecting subscribe requests.

auth_on_subscribe and auth_on_subscribe_m5

The auth_on_subscribe and auth_on_subscribe_m5 hooks allow your plugin to grant or reject subscribe requests sent by a client. They also makes it possible to rewrite the subscribe topic and qos. The auth_on_subscribe hook is specified in the Erlang behaviour and the auth_on_subscribe hook in the behaviour available in the repo.

on_subscribe and on_subscribe_m5

The on_subscribe and on_subscribe_m5 hooks allow your plugin to get informed about an authorized subscribe request. The on_subscribe hook is specified in the Erlang behaviour and the on_subscribe_m5 hook in the behaviour available in the repo.

on_unsubscribe and on_unsubscribe_m5

The on_unsubscribe and on_unsubscribe_m5 hooks allow your plugin to get informed about an unsubscribe request. They also allow you to rewrite the unsubscribe topic if required. The on_subscribe hook is specified in the Erlang behaviour and the on_unsubscribe_m5 hook in the behaviour available in the repo.

Authentication

VerneMQ comes with a simple file-based password authentication mechanism which is enabled by default. If you don't need this it can be disabled by setting:

Per default VerneMQ doesn't accept any client that hasn't been configured using vmq-passwd. If you want to change this and accept any client connection you can set:

There are a couple of hidden options you can set in the vernemq.conf file. Hidden means that you have to add and set the value explicitly. Hidden options still have default values. Changing them should be considered advanced, possibly with the exception of setting a max_message_rate.

Queue Deliver mode

Specify how the queue should deliver messages when multiple sessions are allowed. In case of fanout all the attached sessions will receive the message, in case of balance

Inspecting sessions

VerneMQ comes with powerful tools for inspecting the state of MQTT sessions. To list current MQTT sessions simply invoke vmq-admin session show:

To see detailed information about the command see vmq-admin session show --help.

The command is able to show a lot of different information about a client, for example the client id, the peer host and port if the client is online or offline and much more, see vmq-admin session show --help for details. Further the information can also be used to filter information which is very helpful when wanting to narrow down the information to a single client.

VerneMQ provides multiple hooks throughout the lifetime of a session. The most important ones are the auth_on_register and auth_on_register_m5 hooks which act as an application level firewall granting or rejecting new clients.

auth_on_register and auth_on_register_m5

The auth_on_register and

General relation to OS configuration values

You need to know about and configure a couple of Operating System and Erlang VM configs to operate VerneMQ efficiently. First, make sure you have set appropriate OS file limits according to our guide here. Second, when you run into performance problems, don't forget to check the settings in the vernemq.conf file. (Can't open more than 10k connections? Well, is the listener configured to open more than 10k?)

TCP buffer sizes

This is the number one topic to look at, if you need to keep an eye on RAM usage.

Context: All network I/O in Erlang uses an internal driver. This driver will allocate and handle an internal application side buffer for every TCP connection. The default size of these buffers will determine your overall RAM use.

VerneMQ calculates the buffer size from the OS level TCP send and receive buffers:

val(buffer) >= max(val(sndbuf),val(recbuf))

Those values correspond to net.ipv4.tcp_wmem and net.ipv4.tcp_rmem in your OS's sysctl configuration. One way to minimize RAM usage is therefore to configure those settings (Debian example):

This would result in a 32KB application buffer for every connection. On a multi-purpose server where you install VerneMQ as a test, you might not want to change your OS's TCP settings, of course. In that case, you can still configure the buffer sizes manually for VerneMQ by using the advanced.config file.

The advanced.config file

The advanced.config file is a supplementary configuration file that sits in the same directory as the vernemq.conf. You can set additional config values for any of the OTP applications that are part of a VerneMQ release. To just configure the TCP buffer size manually, you can create an advanced.config file:

The vm.args file

For very advanced & custom configurations, you can add a vm.args file to the same directory where the vernemq.conf file is located. Its purpose is to configure parameters for the Erlang Virtual Machine. This will override any Erlang specific parameters your might have configured via the vernemq.conf. Normally, VerneMQ auto-generates a vm.args file for every boot in /var/lib/vernemq/generated.configs/ (Debian package example) from vernemq.conf and other potential configuration sources.

This is how a vm.args might look like:

A note on TLS

Using TLS will of course increase the CPU load during connection setup. Latencies in message delivery will be increased, and your overall message throughput per second will be lower.

TLS will require considerably more RAM. Instead of 2 Erlang processes per connection, TLS will use 3. You'll have a session process, a queue process, and a TLS handler process that can encapsulate quite a big state (> 30KB).

Erlang/OTP uses its own TLS implementation, only using OpenSSL for crypto, but not connection handling. For situations with high connection setup rate or overall high connection churn rate, the Erlang TLS implementation might be too slow. On the other hand, Erlang TLS gives you great concurrency & fault isolation for long-lived connections.

Some Erlang deployments terminate SSL/TLS with an external component or with a load balancer component. Do some testing & try to find out what works best for you.

You can loadtest VerneMQ with any MQTT-capable loadtesting framework. Our recommendation is to use a framework you are familiar with, with MQTT plugins or scenarios that suit your needs.

While MZBench is currently not actively developed, it is still one of the options you can use vmq_mzbench tool. It is based on Machinezone's very powerful original MZBench system, currently available in a community repository here: MZBench system. MZBench lets you narrow down what hardware specs are needed to meet your performance goals. You can state your requirements for latency percentiles (and much more) in a formal way, and let vmq_mzbench automatically fail, if it can't meet the requirements.

If you have an AWS account, vmq_mzbench can automagically provision worker nodes for you. You can also run it locally, of course.

1. Install MZBench

Please follow the

2. Install vmq_mzbench

Actually, you don't even have to install vmq_mzbench, if you don't want to. Your scenario file will automatically fetch vmq_mzbench for any test you do. vmq_mzbench runs every test independently, so it has a provisioning step for any test, even if you only run it on a local worker.

To install vmq_mzbench on your computer, go through the following steps:

To provision your tests from this local repository, you'll have to tell the scenario scripts to use rsync. Add this to the scenario file:

If you'd just like the script itself fetch vmq_mzbench, then you can direct it to github:

3. Write vmq_mzbench scenario files

You can familiarize yourself quickly with on writing loadtest scenarios.

There's not much to learn, just make sure you understand how pools and loops work. Then you can add the vmq_mzbench statement functions to the mix and define actual loadtest scenarios.

Currently vmq_mzbench exposes the following statement functions for use in MQTT scenario files:

• random_client_id(State, Meta, I): Create a random client Id of length I

• fixed_client_id(State, Meta, Name, Id): Create a deterministic client Id with schema Name ++ "-" ++ Id

• worker_id(State, Meta): Get the internal, sequential worker Id

It's easy to add more statement functions to the MQTT worker if needed, get in touch with us.

Start a VerneMQ cluster node

Sometimes you need to configure a forwarding for ports (on a Mac for example):

This starts a new node that listens on 1883 for MQTT connections and on 8080 for MQTT over websocket connections. However, at this moment the broker won't be able to authenticate the connecting clients. To allow anonymous clients use the DOCKER_VERNEMQ_ALLOW_ANONYMOUS=on environment variable.

Autojoining a VerneMQ cluster

This allows a newly started container to automatically join a VerneMQ cluster. Assuming you started your first node like the example above you could autojoin the cluster (which currently consists of a single container 'vernemq1') like the following:

(Note, you can find the IP of a docker container using docker inspect <CONTAINER_NAME> | grep \"IPAddress\").

Checking cluster status

To check if the above containers have successfully clustered you can issue the vmq-admin command:

You can loadtest VerneMQ with our . It is based on Machinezone's very powerful and lets you narrow down what hardware specs are needed to meet your performance goals. You can state your requirements for latency percentiles (and much more) in a formal way, and let vmq_mzbench automatically fail, if it can't meet the requirements.

If you have an AWS account, vmq_mzbench can automagically provision worker nodes for you. You can also run it locally, of course.

1. Install MZBench

Please follow the

VerneMQ can consume a large number of open file handles when thousands of clients are connected as every connection requires at least one file handle.

Most operating systems can change the open-files limit using the ulimit -n command. Example:

However, this only changes the limit for the current shell session. Changing the limit on a system-wide, permanent basis varies more between systems.

Linux

On most Linux distributions, the total limit for open files is controlled by sysctl

Introduction

When working with a system like VerneMQ sometimes when troubleshooting it would be nice to know what a client is actually sending and receiving and what VerneMQ is doing with this information. For this purpose VerneMQ has a built-in tracing mechanism which is safe to use in production settings as there is very little overhead in running the tracer and has built-in protection mechanisms to stop traces that produce too much information.

A great way to monitor VerneMQ is to use Netdata or Netdata Cloud. Netdata uses VerneMQ in its Netdata Cloud service, and has developed full integration with VerneMQ.

This means that you have one of the best monitoring tools ready for VerneMQ. Netdata will show you all the VerneMQ metrics in a realtime dashboard.

When Netdata runs on the same node as VerneMQ it will automatically discover the VerneMQ node.

Learn how to setup Netdata for VerneMQ with the following guide:

https://learn.netdata.cloud/docs/agent/collectors/go.d.plugin/modules/vernemq

Many aspects of VerneMQ can be extended using plugins. The standard VerneMQ package comes with several official plugins. You can show the enabled & running plugins via:

The command above displays all the enabled plugins together with the hooks they implement:

The table will show the following information:

• name of the plugin

• type (application or single module)

• all the hooks implemented in the plugin

• the exact module and function names (M:F/A) implementing those hooks.

As an example on how to read the table: the vmq_passwd:auth_on_register/5 function is the actual implementation of the auth_on_register hook in the vmq_passwd application plugin.

In addition, you can conclude that the plugin is currently running, as it shows up in the table.

To display information on internal plugins, add the --internal flag. The table below shows you that the generic metadata application and the generic message store are actually internal plugins.

Enable a plugin

This enables the ACL plugin. Because the vmq_acl plugin is already started the above command won't succeed. In case the plugin sits in an external directory you must also to provide the --path=PathToPlugin.

Disable a plugin

Persisting Plugin Configurations and Starts

To make a plugin start when VerneMQ boots, you need to tell VerneMQ in the main vernemq.conf file.

The general syntax to enable a plugin is to add a line like plugins.pluginname = on. Using the vmq_passwd plugin as an example:

If the plugin is external (all your own VerneMQ plugin will be of this category), the path can be specified like this:

Plugin specific settings can be configured via myplugin.somesetting = value, like:

Check the vernemq.conf file for additional details and examples.

VerneMQ can be easily clustered. Clients can then connect to any cluster node and receive messages from any other cluster nodes. However, the MQTT specification gives certain guarantees that are hard to fulfill in a distributed environment, especially when network partitions occur. We'll discuss the way VerneMQ deals with network partitions in its own subsection

A note on statefulness

Before you go ahead and experience the full power of clustering VerneMQ, be aware of its stateful character. An MQTT broker is a stateful application and a VerneMQ cluster is a stateful cluster.

What does this mean in detail? It means that clustered VerneMQ nodes will share information about connected clients and sessions but also meta-information about the cluster itself.

For instance, if you stop a cluster node, the VerneMQ cluster will not just forget about it. It will know that there's a node missing and it will keep looking for it. It will know there's a netsplit situation and it will heal the partition when the node comes back up. But if the missing node never comes back there's an eternal netsplit. (still resolvable by making the missing node explicitly leave).

This doesn't mean that a VerneMQ cluster cannot dynamically grow and shrink. But it means you have to tell the cluster what you intend to do, by using join and leave commands.

If you want a cluster node to leave the cluster, well... use the vmq-admin cluster leave command. If you want a node to join a cluster use the vmq-admin cluster join command.

Makes sense? Go ahead and create your first VerneMQ cluster!

Joining a Cluster

The discovery-node can be any other node. It is not necessary to always choose the same node as discovery node. It is important that only a node with an empty history joins a cluster. One should not try to add a node, that had already traffic on it, to a cluster.

Leaving a Cluster

Detailed Cluster Leave, Case A: Make a live node leave

A cluster leave will actually do a lot more work, and gives you some options to choose. The node leaving the cluster will go to great length trying to migrate its existing queues to other nodes. As queues (online or offline) are live processes in a VerneMQ node, it will only exit after it has migrated them.

Let's look at the steps in detail:

1. vmq-admin cluster leave node=<NodeThatShouldGo>

This first step will only stop the MQTT Listeners of the node to ensure that no new connections are accepted. It will not interrupt the existing connections, and behind the scenes the node will not leave the cluster yet. Existing clients are still able to publish and receive messages at this point.

The idea is to give a grace period with the hope that existing clients might re-connect (to another node). If you have decided that this period is over (after 5 minutes or 1 day is up to you), you proceed with step 2: disconnecting the rest of the clients.

1. vmq-admin cluster leave node=<NodeThatShouldGo> -k

The -k flag will delete the MQTT Listeners of the leaving node, taking down all live connections. If this is what you want from the beginning, you can do this right away as a first step.

Now, queue migration is triggered by clients re-connecting to other nodes. They will claim their queue and it will get migrated. Still, there might be some offline queues remaining on the leaving node, because they were pre-existing or because some clients do not re-connect and do not reclaim their queues.

VerneMQ will throw an exception if there are remaining offline queues after a configurable timeout. The default is 60 seconds, but you can set it as an option to the cluster leave command. As soon as the exception shows in console or console.log, you can actually retry the cluster leave command (including setting a migration timeout (-t), and an interval in seconds (-i) indicating how often information on the migration progress should be printed to the console.log):

1. vmq-admin cluster leave node=<NodeThatShouldGo> -k -i 5 -t 120

After this timeout VerneMQ will forcefully migrate the remaining offline queues to other cluster nodes in a round robin manner. After doing that, it will stop the leaving VerneMQ node.

Detailed Cluster Leave, Case B: Make a stopped node leave

So, case A was the happy case. You left the cluster with your node in a controlled manner, and everything worked, including a complete queue (and message) transfer to other nodes.

Let's look at the second possibility where the node is already down. Your cluster is still counting on it though and possibly blocking new subscription for that reason, so you want to make the node leave.

To do this, use the same command(s) as in the first case. There is one important consequence to note: by making a stopped node leave, you basically throw away persistant queue content, as VerneMQ won't be able to migrate or deliver it.

Let's repeat that to make sure:

Getting Cluster Status Information

VerneMQ is implemented in Erlang OTP and therefore runs on top of the Erlang VM. For this reason native plugins have to be developed in a programming language that runs on the Erlang VM. The most popular choice is obviously the Erlang programming language itself, but Elixir or Lisp flavoured Erlang LFE could be used too. That said, all the plugin hooks are also exposed over (a subset of) Lua, and over WebHooks. This allows you to implement a VerneMQ plugin, by simply implementing a WebHook endpoint, using any programming language you like. You can also implement a VerneMQ plugin as a Lua script.

This guide explains the different flows that expose different hooks to be used for custom plugins. It also describes the code structure a plugin must comply to in order to be successfully loaded and started by the VerneMQ plugin mechanism.

All the hooks that are currently exposed fall into one of three categories.

1. Hooks that allow you to change the protocol flow. An example could be to authenticate a client using the auth_on_register hook.

2. Hooks that inform you about a certain action, that could be used for example to implement a custom logging or audit plugin.

3. Hooks that are called given a certain condition

Notice that some hooks come in two variants, for example the auth_on_register and then auth_on_register_m5 hooks. The _m5 postfix refers to the fact that this hook is only invoked in an MQTT 5.0 session context whereas the other is invoked in a MQTT 3.1/3.1.1 session context.

Before going into the details, let's give a quick intro to the VerneMQ plugin system.

Plugin System

The VerneMQ plugin system allows you to load, unload, start and stop plugins during runtime, and you can even upgrade a plugin during runtime. To make this work it is required that the plugin is an OTP application and strictly follows the rules of implementing the Erlang OTP application behaviour. It is recommended to use the rebar3 toolchain to compile the plugin. VerneMQ comes with built-in support for the directory structure used by rebar3.

Every plugin has to describe the hooks it is implementing as part of its application environment file. The vmq_acl plugin for instance comes with the application environment file below:

Lines 6 to 10 instruct the plugin system to ensure that those dependent applications are loaded and started. If you're using third party dependencies make sure that they are available in compiled form and part of the plugin load path. Lines 16 to 20 allow the plugin system to compile the plugin rules. Yes, you've heard correctly. The rules are compiled into Erlang VM code to make sure the lookup and execution of plugin code is as fast as possible. Some hooks exist which are used internally such as the change_config/1, we'll describe those at some other point.

The environment value for vmq_plugin_hooks is a list of hooks. A hook is specified by {Module, Function, Arity, Options}.

To streamline the plugin development we provide a different Erlang behaviour for every hook a plugin implements. Those behaviours are part of the vernemq_dev library application, which you should add as a dependency to your plugin. vernemq_dev also comes with a header file that contains all the type definitions used by the hooks.

Chaining

It is possible to have multiple plugins serving the same hook. Depending on the hook the plugin chain is used differently. The most elaborate chains can be found for the hooks that deal with authentication and authorization flows. We also call them conditional chains as a plugin can give control away to the next plugin in the chain. The image show a sample plugin chain for the auth_on_register hook.

Most hooks don't require conditions and are mainly used as event handlers. In this case all plugins in a chain are called. An example for such a hook would be the on_register hook.

A rather specific case is the need to call only one plugin instead of iterating through the whole chain. VerneMQ uses such hooks for it's pluggable message storage system.

Unless you're implementing your custom message storage backend, you probably won't need this style of hook.

Startup

The plugin mechanism uses the application environment file to infer the applications that it has to load and start prior to starting the plugin itself. It internally uses the application:ensure_all_started/1 function call to start the plugin. If your setup is more complex you could override this behaviour by implementing a custom start/0 function inside a module that's named after your plugin.

Teardown

The plugin mechanism uses application:stop/1 to stop and unload the plugin. This won't stop the dependent application started at startup. If you rely on third party applications that aren't started as part of the VerneMQ release, e.g. a database driver, you can implement a custom stop/0 function inside a module that's named after your plugin and properly stop the driver there.

Public Type Specs

The vmq_types.hrl exposes all the type specs used by the hooks. The following types are used by the plugin system:

You can dynamically re-configure most of VerneMQ's settings on a running node by using the vmq-admin set command.

The following config values can be handled dynamically:

Setting a value for the local node

Let's change the max_client_id_size as an example. (We might have noticed that some clients can't login because their client ID is too long, but we don't want to restart the broker for that). Note that you can also set multiple values with the same command.

Setting a value for an arbitrary cluster node

Setting a value for all cluster nodes

Show current VerneMQ config values

For the local node

You can show one or multiple values in a simple table:

For an arbitrary node

For all cluster nodes

Bridges are a non-standard way (but de-facto standard) among MQTT broker implementations to connect two different MQTT brokers. Over a bridge, the topic tree of a remote broker becomes part of the topic tree on the local broker. VerneMQ bridges support plain TCP connections as well as SSL connections.

A bridge will be a point-to-point connection between 2 brokers, but can still forward all the messages from all cluster nodes to another cluster.

Enabling the bridge functionality

The MQTT bridge plugin (vmq_bridge) is distributed with VerneMQ as an integrated plugin but is not enabled by default. After configuring the bridge as described below, make sure to enable the plugin by setting (vernemq.conf):

See for more information on working with plugins.

Basic information on the configured bridges can be displayed on the admin CLI:

Sample MQTT Bridge

To configure vmq_bridge you need to edit the bridge section of the vernemq.conf file to set endpoints and mapping topics. A bridge can push or pull messages, as defined in the topic pattern list.

Setup a bridge to a remote broker:

Different connection parameters can be set:

Define the topics the bridge should incorporate in its local topic tree (by subscribing to the remote), or the topics it should export to the remote broker (by publishing to the remote). We share a similar configuration syntax to that used by the Mosquitto broker:

topic defines a topic pattern that is shared between the two brokers. Any topics matching the pattern (which may include wildcards) are shared. The second parameter defines the direction that the messages will be shared in, so it is possible to import messages from a remote broker using in, export messages to a remote broker using out or share messages in both directions. If this parameter is not defined, VerneMQ defaults to out. The QoS level defines the publish/subscribe QoS level used for this topic and defaults to 0. (Source: mosquitto.conf)

The local-prefix and remote-prefix can be used to prefix incoming or outgoing publish messages.

A simple example:

Sample MQTT Bridge that uses SSL/TLS

SSL bridges support the same configuration parameters as TCP bridges (change .tcp to .ssl), but need further instructions for handling the SSL specifics:

Restarting MQTT Bridges

MQTT Bridges that are initiated from the source broker (push bridges) are started when VerneMQ boots and finds a bridge configuration in the vernemq.conf file. Sometimes it's useful to restart MQTT bridges without restarting a broker. This can be done by disabling, then enabling the vmq_bridge plugin and manually calling the bridge start command:

A typical VerneMQ deployment could from a high level look like the following:

In this scenario MQTT clients connect from the internet and are authenticated and authorized against the Authentication Management Service and publish and receive messages, either with each other or with the Backend-Services which might be responsible for sending control messages to the clients or storing and forwarding messages to other systems or databases for later processing.

To build and deploy a system such as the above a lot of decisions has to be made. These can concern how to do authentication and authorization, where to do TLS termination, how the load balancer should be configured (if one is needed at all), what the MQTT architecture and topic trees should look and how and to what level the system can/should scale. To simplify the following discussion we'll set a few requirements:

Release 2.0.0 has a small number of minor incompatibilities:

Error Logger

VerneMQ now uses the internal logger library instead of the lager library. It's best for your custom VerneMQ plugins to do the same and replace the lager log calls with internal log statements. Instead of using lager:error/2, you can use the following format:

To use the Logger Macros, add this include line to your module: -include_lib("kernel/include/logger.hrl").

VerneMQ provides a HTTP REST pub plugin for publishing messages using HTTP/REST. The http_pub plugin accepts HTTP POST requests containing message payloads, and then forwards those messages to the appropriate MQTT subscribers.

The HTTP REST plugin can be used to publish messages from a wider range of devices and platforms, that may not support MQTT natively. Please note, while the plugin can handle a decent amount of requests, the primary protocol of VerneMQ is MQTT. Whenever possible, it is recommended to use MQTT natively to communicate with VerneMQ.

Enabling the plugin

The MQTTplugin (vmq_http_pub) is distributed with VerneMQ as an integrated plugin, but is not enabled by default. After configuring the plugin as described below, make sure to enable the plugin by setting (vernemq.conf):

Configuration

Bind plugin to HTTP(s) listener

By default the plugin is not bound to any listener. It is recommended to use a dedicated HTTPS listener. For security, reasons the use of HTTPS instead of HTTP is preferred. It is possible to have more than one listener.

This configuration defines an HTTPS listener for an application running on the server at IP address 127.0.0.1 and using port 3001. The listener is used to forward HTTP requests to vmq_http_pub.

Additionally, this configuration sets the authentication method for the vmq_http_pub instance to API key (which is the default). This means that a valid API key is required to access this instance. The API key needs to have the scope httppub. You can create a new API key as follows:

It is important to note that this configuration is only a part of a larger configuration file, and that other settings such as SSL certificates, encryption, protocol versions, etc. may also be defined to improve the security and performance of the HTTPS listener.

Authentication and Authorization

The plugin currently supports two authentication and authorization modes: "on-behalf-of" and "predefined". "On-behalf-of" means, that the client_id, user and password used for authentication and authorization is part of request (payload). Afterwards, the regular VerneMQ authentication and authorization flows are used. When using "predefined" the client, user, and password is bound to the plugin instance. It is recommended to use "on-behalf-of" and use a separate client_id, user and password for REST-based clients. For testing purposes, the plugin also supports the global allow_anonymous flag.

For on-behalf-of authentication use:

For predefined, please use a configuration similar to:

If you configure a listener with "predefined" authorization, but provide authorization information (username, password, client_id) those will be ignored.

MQTT Payload

The plugin currently supports three different payload encodings:

• JSON (regular and base64) in body

• Header parameters, and payload in body

• Query String parameters, and payload in body

Which one to choose is depends on your application.

JSON

In order to allow more complex payload to be encoded as part of the json, the payload itself can be also be base64 encoded. The query string "encoding=base64" has to be used to indicate that the payload is base64 encoded. The encoding query string parameter can either be "base64" or "plain". Plain is the default.

Header parameters

Topic, user, password, qos, retain and user_properties can also be part of the HTTP header. The HTTP body is used for the actual message payload. The payload then does not need to be base64 encoded.

The following header options are supported:

Query String

Topic, user, password, qos and retain flag can also be uurlencoded as part of the query string. The HTTP body is used for the actual message payload. There is no need to specify the encoding in the query string. Query String currently does not support user_properties.

Examples

All required information encoded in the payload

All required information encoded in the payload (base64payload)

MQTT information encoded in header parameters

MQTT information encoded in query string

Metrics

The plugin exposes three metrics:

• The number of messages sent through the REST Publish API

• Number of errors reported by the REST Publish API

• Number of Auth errors reported by the REST Publish API

Misc Notes

• The plugin allows the authentication and authorization flows to override mountpoint, max_message_size, qos and topic.

• Currently, the regular (non m5) authentication and authorization flow is used.

• The query string payload does not allow to set user parameters.

• The plugin currently checks the maximum payload size before base64 decoding.

General relation to OS configuration values

You need to know about and configure a couple of Operating System and Erlang VM configs to operate VerneMQ efficiently. First, make sure you have set appropriate OS file limits according to our guide here. Second, when you run into performance problems, don't forget to check the settings in the vernemq.conf file. (Can't open more than 10k connections? Well, is the listener configured to open more than 10k?)

TCP buffer sizes

This is the number one topic to look at, if you need to keep an eye on RAM usage.

Context: All network I/O in Erlang uses an internal driver. This driver will allocate and handle an internal application side buffer for every TCP connection. The default size of these buffers will determine your overall RAM use in VerneMQ. The sndbuf and recbuf of the TCP socket will not count towards VerneMQ RAM, but will be used by the Linux Kernel.

VerneMQ calculates the buffer size from the OS level TCP send and receive buffers:

val(buffer) >= max(val(sndbuf),val(recbuf))

Those values correspond to net.ipv4.tcp_wmem and net.ipv4.tcp_rmem in your OS's sysctl configuration. One way to minimize RAM usage is therefore to configure those settings (Debian example):

This would result in a 32KB application buffer for every connection.

If your VerneMQ use case requires the use of different TCP buffer optimisations (per groups of clients for instance) you will have to make sure the that the Linux OS buffer configuration, namely net.ipv4.tcp_wmem and net.ipv4.tcp_rmemallows for this kind of flexibility, allowing for small TCP buffers and big TCP buffers at the same time.

Example 1 above would allow VerneMQ to allocate minimal TCP read and write buffers of 4KB in the Linux Kernel, a max read buffer of 32KB in the kernel, and a max write buffer of 65KB in the kernel. VerneMQ itself would set its own internal per connection buffer to 65KB in addition.

What we just described is VerneMQ automatically configuring TCP read and write buffers and internal buffers, deriving their values from OS settings.

There are multiple additional ways to configure TCP buffers described below:

Setting TCP buffer sizes globally within VerneMQ:

If VerneMQ finds an advanced.config file, it will use the buffer sizes you have configured there for all it’s TCP listeners (and the TCP connections accepted by those listeners), except the Erlang distribution listeners within the cluster.

(You'll find an example in the section below on the advanced.config )

Per protocol (since 1.8.0):

If VerneMQ finds a per protocol configuration (listener.tcp.buffer_sizes) in the vernemq.conf file, it will use those buffer sizes for the specific protocol. (currently only MQTT or MQTTS. Support for WS/WSS/HTTP/VMQ listeners is on the roadmap).

For listener.tcp.buffer_sizes you’ll always have to state 3 values in bytes: the TCP receive buffer (recbuf), the TCP send buffer (sndbuf), and the internal application side buffer (buffer). You should set “buffer” (the 3rd value) toval(buffer) >= max(val(sndbuf),val(recbuf))

Per listener (since 1.8.0):

If VerneMQ finds per listener config values (listener.tcp.my_listener.buffer_sizes), it will use those buffer sizes for all connections setup by that specific listener. This is the most useful approach if you want to set specific different buffer sizes, like huge send buffers for listeners that accept massive consumers. (consumers with high expected message throughput).

You would then configure a different listener for those massive consumers, and by that have the option to fine tune the TCP buffer sizes.

For listener.tcp.my_listener.buffer_sizes you’ll always have to state 3 values in bytes: the TCP receive buffer (recbuf), the TCP send buffer (sndbuf), and an internal application side buffer (buffer). You should set “buffer” (the 3rd value) toval(buffer) >= max(val(sndbuf),val(recbuf))

VerneMQ per single ClientID/or TCP connection:

This scenario would be possible with a plugin.�

The advanced.config file

The advanced.config file is a supplementary configuration file that sits in the same directory as the vernemq.conf. You can set additional config values for any of the OTP applications that are part of a VerneMQ release. To just configure the TCP buffer size manually, you can create an advanced.config file:

The vm.args file

For very advanced & custom configurations, you can add a vm.args file to the same directory where the vernemq.conf file is located. Its purpose is to configure parameters for the Erlang Virtual Machine. This will override any Erlang specific parameters your might have configured via the vernemq.conf. Normally, VerneMQ auto-generates a vm.args file for every boot in /var/lib/vernemq/generated.configs/ (Debian package example) from vernemq.conf and other potential configuration sources.

This is how a vm.args might look like:

A note on TLS

Using TLS will of course increase the CPU load during connection setup. Latencies in message delivery will be increased, and your overall message throughput per second will be lower.

TLS will require considerably more RAM. Instead of 2 Erlang processes per connection, TLS will use 3. You'll have a session process, a queue process, and a TLS handler process that can encapsulate quite a big state (> 30KB).

Erlang/OTP uses its own TLS implementation, only using OpenSSL for crypto, but not connection handling. For situations with high connection setup rate or overall high connection churn rate, the Erlang TLS implementation might be too slow. On the other hand, Erlang TLS gives you great concurrency & fault isolation for long-lived connections.

Some Erlang deployments terminate SSL/TLS with an external component or with a load balancer component. Do some testing & try to find out what works best for you.

Intro

Kubernetes (K8s) is possibly the most mature technology for deploying Docker containers at scale. While running a single Docker container is supposed to be easy, running a Kubernetes cluster definitely isn't. That's why we recommended to work with a certified Kubernetes partner such as , , , or .

If your applications already live in Docker containers and are deployed on Kubernetes it can be beneficial to also run VerneMQ on Kubernetes. This guide covers how to successfully deploy a VerneMQ cluster on Kubernetes. Multiple options exist to deploy a VerneMQ cluster at this point. This guide describes how to use the official Helm chart as well as the still experimental Kubernetes Operator.

The VerneMQ HTTP API is enabled by default and installs an HTTP handler on http://localhost:8888/api/v1. To read more about configuring the HTTP listener, see HTTP Listener Configuration. You can configure a HTTP listener, or a HTTPS listener to serve the HTTP API v1.

Managing API keys

The VerneMQ HTTP API uses basic authentication where an API key is passed as the username and the password is left empty, as an alternative the x-api-key header option can be used. API keys have a scope and (optional) can have an expiry date. So the first step to us the HTTP API is to create an API key.

Scopes

Each HTTP Module can be protected by an API key. An API key can be limited to a certain http module or further restrict some functionality within the http module. The scope used by the management API is "mgmt". Currently, the following scopes are supported "status", "mgmt", "metrics", "health".

Create API key

or with scope and an expiry date (in local time)

The keys are persisted and available on all cluster nodes.

List API keys

To list existing keys do:

Add API key

To add an API key of your own choosing, do:

Delete API key

To delete an API key do:

Advanced Settings (key rotation, key complexity)

You can specifiy the minimal length of an API key (default: 0) in vernemq.conf

or a set a max duration of an API key before it expires (default: undefined)

Please note that changing those settings after some api keys have already been created has no influence on already created keys.

You can enable or disable API key authentication per module, or per module per listener.

Possible modules are vmq_metrics_http,vmq_http_mgmt_api, vmq_status_http, vmq_health_http. Possible values for auth.mode are noauth or apikey.

API usage

The VerneMQ HTTP API is a wrapper over the CLI tool, and anything that can be done using vmq-admin can be done using the HTTP API. Note that the HTTP API is therefore subject to any changes made to the vmq-admin tools and their flags & options structure. All requests are performed doing a HTTP GET and if no errors occurred an HTTP 200 OK code is returned with a possible non-empty JSON payload.

The API is using basic auth where the API key is passed as the username. An example using curl would look like this:

The mapping between vmq-admin and the HTTP API is straightforward, and if one is already familiar with how the vmq-admin tool works, working with the API should be easy. The mapping works such that the command part of a vmq-admin invocation is turned into a path, and the options and flags are turned into the query string.

A mandatory parameter like the client-id in the vmq-admin session disconnect client-id=myclient command should be translated as: ?client-id=myclient.

An optional flag like --cleanup in the vmq-admin session disconnect client-id=myclient --cleanup command should be translated as: &--cleanup

Let's look at the cluster join command as an example, which looks like this:

This turns into a GET request:

To test, run it with curl:

And the returned response would look like:

Below are some other examples.

Get cluster status information

Request:

Curl:

Response:

Retrieve session information

Request:

Curl:

Response:

List all installed listeners

Request:

Curl:

Response:

Retrieve plugin information

Request:

Curl:

Response:

Set configuration values

Request:

Curl:

Response:

Disconnect a client

Request:

Curl:

Response:

VerneMQ can consume a large number of open file handles when thousands of clients are connected as every connection requires at least one file handle.

Most operating systems can change the open-files limit using the ulimit -n command. Example:

However, this only changes the limit for the current shell session. Changing the limit on a system-wide, permanent basis varies more between systems.

What will actually happen when VerneMQ runs out of OS-side file descriptors?

In short, VerneMQ will be unable to function properly, because it can't open database files or accept incoming connections. In case you see exceptions with {error,emfile} in the VerneMQ log files, you now know what to do, though: increase the OS settings as described below.

Linux

On most Linux distributions, the total limit for open files is controlled by sysctl.

An alternative way to read the file-max settings is:

This might be high enough for your VerneMQ deployment, or not - we cannot know that. You will need at least 1 file descriptor per TCP connection, and VerneMQ needs additional file descriptors for file access etc. Also, if you have other components running on the system, you might want to consult the manpage for how to change that setting. The fs.file-max setting represents the global maximum of file handlers a Linux kernel will allocate. Make sure this is high enough for your system.

Once you're good regarding file-max, you still need to configure the per-process open files limit. You'll set the number of file descriptors a single process or application like VerneMQ is allowed to grab. As every process belongs to a user, you need to bind the setting to a Linux user (here, the vernemq user). To do this, edit /etc/security/limits.conf, for which you'll need superuser access. If you installed VerneMQ from a binary package, add lines for the vernemq user, substituting your desired hard and soft limits:

On Ubuntu, if you’re always relying on the init scripts to start VerneMQ, you can create the file /etc/default/vernemq and specify a manual limit:

This file is automatically sourced from the init script, and the VerneMQ process started by it will properly inherit this setting. As init scripts are always run as the root user, there’s no need to specifically set limits in /etc/security/limits.conf if you’re solely relying on init scripts.

On CentOS/RedHat systems, make sure to set a proper limit for the user you’re usually logging in with to do any kind of work on the machine, including managing VerneMQ. On CentOS, sudo properly inherits the values from the executing user.

Linux and Systemd service files

Newer VerneMQ packages use a systemd service file. You can adapt the LimitNOFILE setting in the vernemq.service file to the value you need. It is set to infinity by default already, so you only need to adapt it in case you want a lower value. The reason we need to enforce the setting is that systemd doesn't automatically take over the nofile settings from the OS.

Enable PAM-Based Limits for Debian & Ubuntu

It can be helpful to enable PAM user limits so that non-root users, such as the vernemq user, may specify a higher value for maximum open files. For example, follow these steps to enable PAM user limits and set the soft and hard values for all users of the system to allow for up to 65536 open files.

Edit /etc/pam.d/common-session and append the following line:

If /etc/pam.d/common-session-noninteractive exists, append the same line as above.

Save and close the file.

Edit /etc/security/limits.conf and append the following lines to the file:

1. Save and close the file.

2. (optional) If you will be accessing the VerneMQ nodes via secure shell (ssh), you should also edit /etc/ssh/sshd_config and uncomment the following line:

and set its value to yes as shown here:

1. Restart the machine so that the limits to take effect and verify

   that the new limits are set with the following command:

Enable PAM-Based Limits for CentOS and Red Hat

1. Edit /etc/security/limits.conf and append the following lines to

   the file:

1. Save and close the file.

2. Restart the machine so that the limits to take effect and verify that the new limits are set with the following command:

Solaris

In Solaris 8, there is a default limit of 1024 file descriptors per process. In Solaris 9, the default limit was raised to 65536. To increase the per-process limit on Solaris, add the following line to /etc/system:

Reference:

Mac OS X

To check the current limits on your Mac OS X system, run:

The last two columns are the soft and hard limits, respectively.

To adjust the maximum open file limits in OS X 10.7 (Lion) or newer, edit /etc/launchd.conf and increase the limits for both values as appropriate.

For example, to set the soft limit to 16384 files, and the hard limit to 32768 files, perform the following steps:

1. Verify current limits:

   The response output should look something like this:

2. Edit (or create) /etc/launchd.conf and increase the limits. Add lines that look like the following (using values appropriate to your environment):

Attributions

This work, "Open File Limits", is a derivative of Open File Limits by Riak, used under Creative Commons Attribution 3.0 Unported License. "Open File Limits" is licensed under Creative Commons Attribution 3.0 Unported License by Erlio GmbH.