DEV Community: Martin Hynar

Reason: user 'xxx' does not meet 'require'ments for user to be allowed access

Martin Hynar — Tue, 09 Jun 2020 08:45:52 +0000

This post is meant as lessons learned from setting up Apache HTTPD server with authentication for some backend resources. The goal was to secure the resources with simple password based authentication and allow given list of users to access them. Accounts have been assembed in dbm file on local filesystem.

Configuration that works, but...

<VirtualHost *:80>
    ErrorLog /var/log/httpd/http-error.log
    CustomLog /var/log/httpd/http-access.log

    <Location /myresources>
        AuthType basic
        AuthName "Authenticate using username and password"
        AuthBasicProvider dbm

        AuthDBMUserFile "/etc/httpd/authentication.dbm"
        AuthGroupFile "/etc/httpd/groups"
        <RequireAny>
            Require user adam
            Require user bob
            Require user cecil
        </RequireAny>
    </Location>
</VirtualHost>

So, what we have here

Credentials for user accounts are stored in /etc/httpd/authentication.dbm
Access to myresources is allowed only to used adam, bob, cecil. This is defined using RequireAny meaning that if any of these requirements is matched, user is granted with access.

This will work, but you will see disturbing messages in error log. They are disturbing, because they will be naming those users that are allowed! However, at the same time, you will not see any error response in access log. (Log messages are shortened)

/var/log/httpd/http-error.log

[authz_user:error] [client 10.10.10.10:10000] AH01663: access to /myresources failed, reason: user 'bob' does not meet 'require'ments for user to be allowed access

/var/log/httpd/http-access.log

10.10.10.10 - bob "POST /myresources HTTP/1.1" 200 102

Why is that?

The reason for this is that error log will contain error message for each unsatisfied Require directive. For adam there won't be any error message as he is first in list. For bob there will be 1 and for cecil there will be 2 for each request.

Configuration that works, but...

<VirtualHost *:80>
    ErrorLog /var/log/httpd/http-error.log
    CustomLog /var/log/httpd/http-access.log

    <Location /myresources>
        AuthType basic
        AuthName "Authenticate using username and password"
        AuthBasicProvider dbm

        AuthDBMUserFile "/etc/httpd/authentication-myresources.dbm"
        AuthGroupFile "/etc/httpd/groups"
        Require valid-user
    </Location>
</VirtualHost>

The difference here is that there is no Require user list. Instead, any valid user is allowed. But, the authentication database can have more users that only those allowed to access myresources. This is why there is changed authentication file and you have to keep there correct list of users.

The latter configuration won't be generating false alarms in error log, but I don't consider it perfect too.

Monitoring Kafka brokers using Jolokia, Metricbeat and ElasticSearch

Martin Hynar — Fri, 29 May 2020 13:43:32 +0000

If you don't have commercial installation of Kafka that comes packed with monitoring solution, you have to build it on your own. You always need to have at least some tools to find what is going on in your service. Visualizing available metrics is indeed valuable option.

In this text, I will describe how to get these important metrics from Kafka and ship them to ElasticSearch where they can be visualized using Kibana. I will focus mainly on Kafka and Metricbeat configuration (how to get the metrics) rather than on visualization (make figures to your own taste).

Component list

Data pipeline

Kafka advertises runtime metrics using dedicated MBeans that can be made available using JMX interface. This is exactly the channel we will use here. One of the tools that is developed to collect metrics from various systems is Metricbeat that comes with prepared Kafka module that simply "knows what metrics Kafka provides". Behind the scenes, Metricbeat makes use of Jolokia that serves as bridge for JMX and provides metrics using HTTP/JSON.

Setting parts up

Kafka with Jolokia

Jolokia is a JMX-HTTP bridge giving an alternative to JSR-160 connectors. It is an agent based approach with support for many platforms. In addition to basic JMX operations it enhances JMX remoting with unique features like bulk requests and fine grained security policies.

In this article, I will not describe installation of Kafka. There is plenty of up-to-date documentation available. I am using rpm packaged Kafka distribution from Confluent.io. Let's focus on Jolokia.

We will be running Jolokia in agent mode. This is, we'll put Jolokia agent library on Kafka's class path, configure Jolokia in Kafka's configuration files and run it. This way, Jolokia will connect to Kafka monitoring MBeans and will provide metrics on HTTP interface.

Download Jolokia JVM Agent - https://jolokia.org/download.html
Save it into Kafka's lib folder /usr/share/java/kafka
For easier configuration, either rename or make symlink /usr/share/java/kafka/jolokia-jvm-agent.jar

Now, we have the agent library in place and need to configure it

Open /usr/bin/kafka-server-start and add KAFKA_JMX_OPTS



export KAFKA_JMX_OPTS="
-javaagent:/usr/share/java/kafka/jolokia-jvm-agent.jar=port=8778,host=localhost \
-Dcom.sun.management.jmxremote=true \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Djava.rmi.server.hostname=localhost \
-Dcom.sun.management.jmxremote.host=localhost \
-Dcom.sun.management.jmxremote.port=9999 \
-Dcom.sun.management.jmxremote.rmi.port=9999 \
-Djava.net.preferIPv4Stack=true"

Start Kafka

At this point, you shall be able to see listening process on port 8778. You can try to get some numbers from it.



sudo netstat -ltnp
curl -s http://localhost:8778/jolokia/version | jq

Metricbeat

After installing Metricbeat, you find default configuration in /etc/metricbeat with Kafka module disabled. In Metricbeat, we need to do 2 things:

Configure Kafka module for sampling, and
Configure Metricbeat to send data to ElasticSearch and create Kibana artifacts.

To configure Kafka sampling

Rename /etc/metricbeat/modules.d/kafka.yml.disabled to /etc/metricbeat/modules.d/kafka.yml to activate it.
Open /etc/metricbeat/modules.d/kafka.yml. To read all available metrics, the configuration shall have this contents



# Module: kafka
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.6/metricbeat-module-kafka.html

# Kafka metrics collected using the Kafka protocol
- module: kafka
  metricsets:
    - partition
    - consumergroup
  period: 10s
  hosts: ["localhost:9092"]

  client_id: metricbeat

  # List of Topics to query metadata for. If empty, all topics will be queried.
  #topics: []

# Metrics collected from a Kafka broker using Jolokia
- module: kafka
  metricsets:
    - broker
  period: 10s
  hosts: ["localhost:8778"]

For detailed description of Metricbeat's Kafka module see documentation.

To configure proper forwarding of metrics to ElasticSearch

Open /etc/metricbeat/metricbeat.yml, and configure elasticsearch output



output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "metricbeat-kafka-%{[agent.version]}-%{+yyyy-MM-dd}"

Start Metricbeat service.



sudo systemctl start metricbeat

At this moment, Metricbeat shall start to sample Kafka metrics with 10 seconds interval and send them to ElasticSearch.

If you have Kibana installed, you can command Metricbeat to create visualizations and dashboard for you. For this, you need to configure Kibana endpoint parameters - where is your Kibana - and Kibana dashboard parameters - where in Kibana you want the artifacts to be created.

To create Kibana artifacts, run metricbeat setup command.

Version information

For this post, I used following component versions:

Kafka - Confluent Community 5.4.1
Jolokia 1.6.2
Metricbeat 7.6.2
ElasticSearch 7.6.0
Kibana 7.6.0