Azure DevOps Server ships with a pretty old version of Elastic Search and by default installs a non-auto updating JVM. I just checked the version of Elastic Search that ships with my version of Azure DevOps Server and it's 6.2.4. That version ships with log4j 2.9.1, which is unfortunately too old for the simple fix described by Elastic to do the trick:
Elasticsearch 6 and 7 are not susceptible to remote code execution with this vulnerability due to our use of the Java Security Manager. Elasticsearch running on JDK8 or below is susceptible to an information leak via DNS which is fixable by the JVM option identified below. This option is effective for Elasticsearch versions 5.6.11+, 6.4+, and 7.0+.
Elasticsearch 5 is susceptible to both remote code execution and an information leak via DNS. For versions 5.6.11 - 5.6.16, this can be mitigated by setting the JVM option. Users on an earlier version of 5.x, are recommended to upgrade to 5.6.16. We are exploring other options for cases where upgrading is not possible. Please note that while we provide these remediations, Elasticsearch 5 is not a supported version, and we always recommend updating to the latest release.
- For TFS 2017 RTM, Elastic Search 1.7.1, which ships with log4j 1.2.17, which is not vulnerable.
- For TFS 2017 Update 1 - TFS 2018 Update 1, Elastic Search 2.4.1, which ships with log4j 1.2.17, which is not vulnerable.
- For TFS 2018 Update 2+, Elastic Search 5.4.1, which ships with log4j 2.8.2, which is vulnerable and can not be resolved by setting the JVM option.
- For Azure DevOps Server 2019, Elastic Search 6.2.4, which ships with log4j 2.9.1, which is vulnerable and can not be resolved by setting the JVM option.
- For Azure DevOps Server 2020, Elastic Search 6.2.4, which ships with log4j 2.9.1, which is vulnerable and can not be resolved by setting the JVM option.
Luckily the default installation mode only binds Elastic Search to the local loopback network interface limiting access from the outside world, but if you've setup Search on a separate VM and didn't follow the Securing Search guidance to limit access you may want to look into that right now.
You currently have 3 options to deal with this issue:
- Remove Search from your installation
- Patch the log4j-core-*.jar files on disk
- Patch the log4j vulnerability at runtime
I also highly recommend you update the JVM that was installed when you configured Search. The default option does not auto-update and chances are the one you're running is already quite old.
Remove Search
The simplest solution, and most secure solution at the moment, is to uninstall Azure DevOps Search from your Application Tiers and disabling/uninstalling the Elastic Search service:
Disable Elastic Search
Or remove the service altogether:
Remove the Elastic Search Service
And remove the Search feature from the Azure DevOps Administration Console:
Remove Azure DevOps Search Service from the Azure DevOps Server Admin Console
Patch Elastic Search files Instead
If you don't want to turn Search off, you'll need to patch log4j-core to strip the vulnerable classes from the jar files. You can remove the vulnerable classes from the jar file with a single command. All of the docs I found give you this command which will work on Linux:
zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
On Windows that command won't work, since the zip command isn't available. If you've got 7-zip installed, use the following the command for Azure DevOps Server 2020:
cd "C:\Program Files\Azure DevOps Server 2020\Search\ES\elasticsearchv6.2\lib"
..\bin\elasticsearch-service stop
"c:\Program Files\7-Zip\7z.exe" d log4j-core-2.9.1.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
..\bin\elasticsearch-service start
To resolve the issue for Azure DevOps Server 2019+:
cd "C:\Program Files\Azure DevOps Server 2019\Search\ES\elasticsearchv6.2\lib"
..\bin\elasticsearch-service stop
"c:\Program Files\7-Zip\7z.exe" d log4j-core-2.9.1.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
..\bin\elasticsearch-service start
To resolve the issue for Team Foundation Server 2018 Update 2+:
# cd "C:\Program Files\Microsoft Team Foundation Server 15.0\Search\ES\elasticsearch-5.4.1\lib"
..\bin\elasticsearch-service stop
"c:\Program Files\7-Zip\7z.exe" d log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
..\bin\elasticsearch-service start
Alternatively, patch log4j at runtime
You can also install log4j-jndi-be-gone and instruct Elastic Search to load it to remove the vulnerability at runtime. This will also work for older versions of Team Foundation Server and Azure DevOps Server.
Download the log4j-jndi-be-gone-1.0.0-standalone.jar from their GitHub repository and save it to your server.
Save the jar file where Elastic Search can find it
Run elasticsearce-service.bat manager to open the configuration panel for Elastic Search and add the jar file to the configuration:
-javaagent:C:\AzureDevOpsData\log4j-jndi-be-gone\log4j-jndi-be-gone-1.0.0-standalone.jar
Add log4j-jndi-be-gone agent to the Elastic Search configuration
Then restart the Elastic Search service:
Restart Elastic Search after adding log4j-jndi-be-gone.
Update the Java Runtime Environment for Search
While you're at it, update the JVM to the most recent version. You can find the latest Java 8 Runtime here.
Note
If you choose to use Azul Zulu OpenJDK, ensure that you download version 8.
Extract the zip file into the AzureDevOpsData\Java\ folder on your Application Tier:
Extract the downloaded JDK to your AzureDevOpsData\Java folder
Top comments (0)