DEV Community

Building Hadoop native libraries on Mac in 2019

Saša Zejnilović on May 20, 2019

TL;DR to be found at the end Recently I came into a situation that I "needed" Hadoop native libraries. Well, when I say "needed", I mean I was jus...
Read full post
Collapse
 
cameronhudson8 profile image
Cameron Hudson •

Depending on which Hadoop version you want to install, you may need to use an earlier Java version to package it. This can be done by temporarily changing the JAVA_HOME environment variable before running mvn package.

In my case, instead of

mvn package -Pdist,native -DskipTests -Dtar

I had to run

JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_221.jdk/Contents/Home mvn package -Pdist,native -DskipTests -Dtar
Collapse
 
lanshunfang profile image
001027261_Shunfang Lan •

JDK11 is still in progress cwiki.apache.org/confluence/displa...

Collapse
 
zejnilovic profile image
Saša Zejnilović •

Yes, you are right. I didn't think of that use case, I just assumed people would have a compliant Java version already installed or set as the main one.

Collapse
 
ajndesai profile image
ajndesai • • Edited

Thanks for this article. I get this error when compiling to generate package using this below command:

mvn package -Pdist,native -DskipTests -Dtar

branch: branch-3.2

[INFO] Apache Hadoop MapReduce NativeTask ................. FAILURE [ 1.766 s]

[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.2.2-SNAPSHOT:cmake-compile (cmake-compile) on project hadoop-mapreduce-client-nativetask: make failed with error code 2 -> [Help 1]

Collapse
 
lanshunfang profile image
001027261_Shunfang Lan •

It's because hadoop doesn't support macOS native building now.

See github.com/apache/hadoop/blob/trun...

Note that building Hadoop 3.1.1/3.1.2/3.2.0 native code from source is broken
on macOS. For 3.1.1/3.1.2, you need to manually backport YARN-8622. For 3.2.0,
you need to backport both YARN-8622 and YARN-9487 in order to build native code.
Collapse
 
zejnilovic profile image
Saša Zejnilović •

Hello, I don't see enough of the message. Do you have java 8? Or maybe more of the error (line with cause by)?

I have tried building this now and it works for me.

Collapse
 
ajndesai profile image
ajndesai • • Edited

Yes I have java 8 and that's my java home. Hadoop, hive setup and working though. I read some of the compression codecs works only with native libraries and jobs will fail with java libraries.

I tried this on mac Mojave OS 10.14.6

Collapse
 
imasli profile image
imasli •

What version of OpenSSL did you use?

I used OpenSSL 1.1 from Brew and I have this error

error: variable has incomplete type 'HMAC_CTX' (aka 'hmac_ctx_st')
Collapse
 
zejnilovic profile image
Saša Zejnilović •

This seems more like an issue with the OpenSSL installation than the version you are using. Anyway, this is my current PC (not sure with which I have built it, but libs work)

╰─$ openssl version
LibreSSL 2.8.3
╰─$ brew list --versions | grep ssl
openssl 1.0.2t
openssl@1.1 1.1.1d
Collapse
 
imasli profile image
imasli • • Edited

According to
issues.apache.org/jira/browse/HADO...

OpenSSL 1.1 broke the compilation. They patched it but didn't include it to the version 2 build. Your tutorial used OpenSSL 1.0 (Open SSL 1.1 will have OpenSSL@1.1 on the path)

Too bad that Homebrew already deprecated OpenSSL 1.0

Thread Thread
 
zejnilovic profile image
Saša Zejnilović •

Thank you very much! Will update the post.

Thread Thread
 
imasli profile image
imasli •

This issue discusses how to forcefully install OpenSSL 1.0 using Homebrew
github.com/Homebrew/homebrew-core/...

Collapse
 
krikoon73 profile image
krikoon73 • • Edited

Hi Sasa,
Thank you for this ! I'm a bit new on hadoop and I'm trying to fix the "native library" thing based on your article. Everything is ok until "Apache Hadoop Common" build :

Mac OSX highSierra
Hadoop version 2.9.2

~/hadoop/tmp/hadoop   branch-2.9.2  brew list
autoconf cmake gettext go libidn2 libunistring openjdk pcre2 pyenv-virtualenv sshpass wget
automake direnv git gzip libmpc maven openssl@1.1 pkg-config readline telnet zlib
bzip2 gcc gmp isl libtool mpfr pandoc pyenv snappy tree

openssl : ok
protoc : 2.5.0 ok

Error message :

[INFO] Apache Hadoop Common ............................... FAILURE [02:08 min]

I have :

[WARNING] /Users/ccompain/hadoop/tmp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:256:14: error: incomplete definition of type 'struct evp_cipher_ctx_st'
[WARNING] if (context->flags & EVP_CIPH_NO_PADDING) {
[WARNING] ~~~~~~~^
[WARNING] /usr/local/include/openssl/ossl_typ.h:90:16: note: forward declaration of 'struct evp_cipher_ctx_st'
[WARNING] typedef struct evp_cipher_ctx_st EVP_CIPHER_CTX;
[WARNING] ^
[WARNING] /Users/ccompain/hadoop/tmp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:262:20: error: incomplete definition of type 'struct evp_cipher_ctx_st'
[WARNING] int b = context->cipher->block_size;
[WARNING] ~~~~~~~^
[WARNING] /usr/local/include/openssl/ossl_typ.h:90:16: note: forward declaration of 'struct evp_cipher_ctx_st'
[WARNING] typedef struct evp_cipher_ctx_st EVP_CIPHER_CTX;
[WARNING] ^
[WARNING] /Users/ccompain/hadoop/tmp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:263:16: error: incomplete definition of type 'struct evp_cipher_ctx_st'
[WARNING] if (context->encrypt) {
[WARNING] ~~~~~~~^
[WARNING] /usr/local/include/openssl/ossl_typ.h:90:16: note: forward declaration of 'struct evp_cipher_ctx_st'
[WARNING] typedef struct evp_cipher_ctx_st EVP_CIPHER_CTX;
[WARNING] ^
[WARNING] /Users/ccompain/hadoop/tmp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:310:14: error: incomplete definition of type 'struct evp_cipher_ctx_st'
[WARNING] if (context->flags & EVP_CIPH_NO_PADDING) {
[WARNING] ~~~~~~~^
[WARNING] /usr/local/include/openssl/ossl_typ.h:90:16: note: forward declaration of 'struct evp_cipher_ctx_st'
[WARNING] typedef struct evp_cipher_ctx_st EVP_CIPHER_CTX;
[WARNING] ^
[WARNING] /Users/ccompain/hadoop/tmp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:313:20: error: incomplete definition of type 'struct evp_cipher_ctx_st'
[WARNING] int b = context->cipher->block_size;
[WARNING] ~~~~~~~^
[WARNING] /usr/local/include/openssl/ossl_typ.h:90:16: note: forward declaration of 'struct evp_cipher_ctx_st'
[WARNING] typedef struct evp_cipher_ctx_st EVP_CIPHER_CTX;
[WARNING] ^
[WARNING] 5 errors generated.

[WARNING] 5 errors generated.
[WARNING] make[2]: *** [CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c.o] Error 1
[WARNING] make[2]: *** Waiting for unfinished jobs....
[WARNING] make[1]: *** [CMakeFiles/hadoop_static.dir/all] Error 2
[WARNING] make[1]: *** Waiting for unfinished jobs....
[WARNING] make[1]: *** [CMakeFiles/hadoop.dir/all] Error 2
[WARNING] make: *** [all] Error 2

Any idea ?
Thks,
Christophe

Collapse
 
zejnilovic profile image
Saša Zejnilović •

Hello, in all honesty, I don't remember having this problem, but seems like you aren't alone. Apparently it is an OpenSSL feature. This apache.org Jira seems to be talking precisely about your problem. Seems like they even provide places you need to fix in the C code.

Collapse
 
dales profile image
Dale Schaffer •

When is $HADOOP_HOME defined?

when i run line 29 in script it fails and prints out the usage text for cp

Collapse
 
zejnilovic profile image
Saša Zejnilović •

$HADOOP_HOME is not defined by me. It is part of proper Hadoop installation and I am not doing that in this article.

Collapse
 
azeddin1717 profile image
Sahir Azeddin •

Hello, I am trying to install hadoop native library on mac but I have some problems here I tried to solve it but I can’t. Please if u have any idea.
My problem looks like:

Image description

Collapse
 
arulselvanmadhavan profile image
Arulselvan Madhavan •

Thanks for writing this. Helped me a lot