MariaDB HandlerSocket Setup Options

I'm trying to use HandlerSocket with MariaDB. I'm done installing it but setting the host and port options doesn't seem to work.

Checking the variables shows me this:

MariaDB [(none)]> SHOW GLOBAL VARIABLES LIKE 'handlersocket%';
+-------------------------------+-------+
| Variable_name                 | Value |
+-------------------------------+-------+
| handlersocket_accept_balance  | 0     |
| handlersocket_address         |       |
| handlersocket_backlog         | 32768 |
| handlersocket_epoll           | 1     |
| handlersocket_plain_secret    |       |
| handlersocket_plain_secret_wr |       |
| handlersocket_port            |       |
| handlersocket_port_wr         |       |
| handlersocket_rcvbuf          | 0     |
| handlersocket_readsize        | 0     |
| handlersocket_sndbuf          | 0     |
| handlersocket_threads         | 16    |
| handlersocket_threads_wr      | 1     |
| handlersocket_timeout         | 300   |
| handlersocket_verbose         | 10    |
| handlersocket_wrlock_timeout  | 12    |
+-------------------------------+-------+

I tried to add the options

//> /etc/mysql/mariadb.conf.d/50-server.cnf
[mysqld]

#
# * Basic Settings
#
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
datadir         = /var/lib/mysql
tmpdir          = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking

handlersocket_address="127.0.0.1"
handlersocket_port="9998"
handlersocket_port_wr="9999"

And it still doesn't work, I've also tried adding it here:

//> /etc/mysql/my.cnf
handlersocket_address="127.0.0.1"
handlersocket_port="9998"
handlersocket_port_wr="9999"

#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]

# Import all .cnf files from configuration directory
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mariadb.conf.d/

And it just errors out, what am I missing? Am I changing the wrong file?