You need to know about and configure a couple of Operating System and Erlang VM configs to operate VerneMQ efficiently. First, make sure you have set appropriate OS file limits according to our guide here. Second, when you run into performance problems, don't forget to check the settings in the
vernemq.conf file. (Can't open more than 10k connections? Well, is the listener configured to open more than 10k?)
This is the number one topic to look at, if you need to keep an eye on RAM usage.
Context: All network I/O in Erlang uses an internal driver. This driver will allocate and handle an internal application side buffer for every TCP connection. The default size of these buffers will determine your overall RAM use in VerneMQ. The sndbuf and recbuf of the TCP socket will not count towards VerneMQ RAM, but will be used by the Linux Kernel.
VerneMQ calculates the buffer size from the OS level TCP send and receive buffers:
val(buffer) >= max(val(sndbuf),val(recbuf))
Those values correspond to
net.ipv4.tcp_rmem in your OS's sysctl configuration. One way to minimize RAM usage is therefore to configure those settings (Debian example):
sudo sysctl -w net.ipv4.tcp_rmem="4096 16384 32768"sudo sysctl -w net.ipv4.tcp_wmem="4096 16384 32768"# Nope, these values are not recommendations!# You really need to decide yourself.
This would result in a 32KB application buffer for every connection.
If your VerneMQ use case requires the use of different TCP buffer optimisations (per groups of clients for instance) you will have to make sure the that the Linux OS buffer configuration, namely
net.ipv4.tcp_rmemallows for this kind of flexibility, allowing for small TCP buffers and big TCP buffers at the same time.
Example 1 (from Linux OS config):net.ipv4.tcp_rmem="4096 16384 32768"net.ipv4.tcp_wmem="4096 16384 65536"
Example 1 above would allow VerneMQ to allocate minimal TCP read and write buffers of 4KB in the Linux Kernel, a max read buffer of 32KB in the kernel, and a max write buffer of 65KB in the kernel. VerneMQ itself would set its own internal per connection buffer to 65KB in addition.