We are using SonarQube started from Jenkins jobs.
See more at SonarQube: running tests from Jenkins Pipeline in Docker.
So, the SonarQube is started from the Jenkins – all good here, but inside SonarQube we can see errors:
java.lang.IllegalStateException: Unrecoverable indexation failures: 1 errors among 1 requests
at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:39)
at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:122)
…
And host logs for SonarQube:
Aug 29 12:30:26 jenkins-production docker-compose[12591]: sonarqube_1 | 2019.08.29 09:30:26 ERROR web[][o.s.s.es.BulkIndexer] index [components], type [auth], id [AWzcfdpYaLFDDA6l6km3], message [ClusterBlockException[blocked by: [FORBIDD
EN/12/index read-only / allow delete (api)];]]
Aug 29 12:30:26 jenkins-production docker-compose[12591]: sonarqube_1 | 2019.08.29 09:30:26 ERROR web[][o.s.s.e.RecoveryIndexer] Elasticsearch recovery – too many failures [167/167 documents], waiting for next run
Aug 29 12:30:26 jenkins-production docker-compose[12591]: sonarqube_1 | 2019.08.29 09:30:26 INFO web[][o.s.s.e.RecoveryIndexer] Elasticsearch recovery – 167 documents processed [167 failures] | time=51ms
Aug 29 12:30:48 jenkins-production docker-compose[12591]: sonarqube_1 | 2019.08.29 09:30:48 INFO es[][o.e.c.r.a.DiskThresholdMonitor] low disk watermark [85%] exceeded on [mjJvGbfOSZyVqbKWKCGwxw][sonarqube][/opt/sonarqube/data/es6/nodes
/0] free: 11.6gb[11.8%], replicas will not be assigned to this node
Check the documentation about Disk-based shard allocation – need to set cluster.routing.allocation.disk.watermark.low or disable those checks at all by setting the "cluster.routing.allocation.disk.threshold_enabled" : false.
Although we have 12 GB free from the 100 GB total on the Jenkins host – but ElasticSearch by default count in percents so it’s not enough for him.
Because Elasticsearch is running inside (!) of the SonarQube’s container – let’s find where is the ElasticSearch config is stored there:
root@jenkins-production:/opt/jenkins# docker exec -ti jenkins_sonarqube_1 find / -name elasticsearch.yml
find: ‘/proc/tty/driver’: Permission denied
/opt/sonarqube/temp/conf/es/elasticsearch.yml
/opt/sonarqube/elasticsearch/config/elasticsearch.yml
Here we are – /opt/sonarqube/elasticsearch/config/elasticsearch.yml.
Sonar is started from a Docker Compose file with some files/dirs mapping:
version: '3.5'
networks:
sonar:
external:
name: jenkins
services:
sonarqube:
user: 1004:1004
image: sonarqube:7.9.1-community
ports:
- "9000:9000"
networks:
- sonar
environment:
- sonar.jdbc.url=jdbc:postgresql://db:5432/sonar
volumes:
- /data/sonarqube/conf:/opt/sonarqube/conf
- /data/sonarqube/logs:/opt/sonarqube/logs
- /data/sonarqube/temp:/opt/sonarqube/temp
- /data/sonarqube/data:/opt/sonarqube/data
- /data/sonarqube/extensions:/opt/sonarqube/extensions
- /data/sonarqube/bundled_plugins:/opt/sonarqube/lib/bundled-plugins
logging:
driver: "journald"
...
Available options for the disk management are:
-
cluster.routing.allocation.disk.threshold_enabled: будет ли Elasticsearch вообще проверять состояние диска -
cluster.routing.allocation.disk.watermark.low: default 85%, Elasticsearch не будет размещать шарды на этой ноде -
cluster.routing.allocation.disk.watermark.high: default 90%, Elasticsearch попробует перенести шарды с этой ноды на другие -
cluster.routing.allocation.disk.watermark.flood_stage: default 95%, запрещает обновление индексов, шарды которых на этой ноде
Create a new config on the host – /data/sonarqube/conf/elasticsearch.yml:
cluster.routing.allocation.disk.watermark.flood_stage: 95%
cluster.routing.allocation.disk.watermark.high: 90%
Map it to the SonarQube’s container SonarQube:
...
volumes:
- /data/sonarqube/conf:/opt/sonarqube/conf
- /data/sonarqube/conf/elasticsearch.yml:/opt/sonarqube/elasticsearch/config/elasticsearch.yml
...
To access the Elasticsearch – need to set sonar.search.httpPort for the SonarQube, see the How to Monitor ElasticSearch.
Enable this connector 9100:
...
environment:
- sonar.jdbc.url=jdbc:postgresql://db:5432/sonar
- sonar.search.httpPort=9100
...
Restart the service:
root@jenkins-production:/home/admin# systemctl restart sonarqube
And a bit logs says this option will be deprecated soon:
2019.08.29 10:56:15 INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2019.08.29 10:56:15 WARN app[][o.s.a.es.EsSettings] Elasticsearch HTTP connector is enabled on port 9100. MUST NOT BE USED FOR PRODUCTION
2019.08.29 10:55:10 WARN es[][o.e.d.c.s.Settings] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
Check parameters after the restart to make sure Elasticsearch sees them.
As Elasticsearch listen on the localhost only inside (again!) the SonarQube’s container – use curl to check from the SonarQube itself:
root@jenkins-production:/home/admin# docker exec -ti jenkins_sonarqube_1 curl localhost:9100/_cluster/settings?include_defaults=true | jq '.[].cluster.routing.allocation.disk'
null
null
{
"threshold_enabled": "true",
"watermark": {
"low": "85%",
"flood_stage": "95%",
"high": "90%"
},
"include_relocations": "true",
"reroute_interval": "60s"
}
Okay – new settings applied, can proceed with builds now.
Used disc space per node can be checked by calling the next API:
root@jenkins-production:/home/admin# docker exec -ti jenkins_sonarqube_1 curl 'http://localhost:9100/_cat/allocation?v&pretty'
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
24 613.4kb 86.3gb 11.5gb 97.9gb 88 127.0.0.1 127.0.0.1 sonarqube
All available parameters for the Elasticsearch – https://nickcanzoneri.com/elasticsearch-settings.
Done.
Latest comments (0)