Ndtyjky

I am compiling hadoop-yarn-nodemanager.

Compiling environment: MacOS-10.14, java-1.7.0_80, cmake3.13.0-rc3 with clang-1000.10.44.4, Maven 3.6.0, protocbuf 2.5.0.

I'm trying to install Hadoop-2.2.0 on MacOS, but as its document shows,

The native hadoop library is supported on *nix platforms only. The library does not to work with Cygwin or the Mac OS X platform.

So I have to re-compile Hadoop's source code. In the downloaded hadoop-2.2.0-src folder, running mvn package -Pdist,native -DskipTests -Dtar, and the new native library should have been at hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0/lib/native after minutes' compiling. However, I kept getting back error messages. Some already fixed with modification to source codes, but now I'm trapped compiling hadoop-yarn-server-nodemanager.

Compiling Process Now

Here's the error message:

 [exec] [ 57%] Linking C executable target/usr/local/bin/test-container-executor

 [exec] /Applications/CMake.app/Contents/bin/cmake -E cmake_link_script CMakeFiles/test-container-executor.dir/link.txt --verbose=1

 [exec] /Library/Developer/CommandLineTools/usr/bin/cc  -g -Wall -O2 -D_GNU_SOURCE -D_REENTRANT -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names  CMakeFiles/test-container-executor.dir/main/native/container-executor/test/test-container-executor.c.o  -o target/usr/local/bin/test-container-executor libcontainer.a 

 [exec] Undefined symbols for architecture x86_64:

 [exec]   "_fcloseall", referenced from:

 [exec]       _launch_container_as_user in libcontainer.a(container-executor.c.o)

 [exec] ld: symbol(s) not found for architecture x86_64

 [exec] clang: error: linker command failed with exit code 1 (use -v to see invocation)

 [exec] make[2]: *** [target/usr/local/bin/test-container-executor] Error 1

 [exec] make[1]: *** [CMakeFiles/test-container-executor.dir/all] Error 2

 [exec] make: *** [all] Error 2

I've tried to switch cmake's compiler from clang to gcc, but of no use.

Relavant to error message, I find following codes.

In hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt:

add_executable(test-container-executor

    main/native/container-executor/test/test-container-executor.c

)

target_link_libraries(test-container-executor

    container

)

output_directory(test-container-executor target/usr/local/bin)

In hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c:

int launch_container_as_user(const char *user, const char *app_id, 

               const char *container_id, const char *work_dir,

               const char *script_name, const char *cred_file,

               const char* pid_file, char* const* local_dirs,

               char* const* log_dirs, const char *resources_key,

               char* const* resources_values) {...}

In hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/test-container-executor.dir/link.txt:

/Library/Developer/CommandLineTools/usr/bin/cc  -g -Wall -O2 -D_GNU_SOURCE -D_REENTRANT -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names  CMakeFiles/test-container-executor.dir/main/native/container-executor/test/test-container-executor.c.o  -o target/usr/local/bin/test-container-executor libcontainer.a 

As for the compressed file hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/libcontainer.a, I found container-executor.c.o after decompressing, but failed to open it with encoding problem.

Furthermore, an error raised previously when compiling this project:

 [exec] /Users/markdana/Downloads/hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:1252:48: error: too many arguments to function call, expected 4, have 5

 [exec]     if (mount("none", mount_path, "cgroup", 0, controller) == 0) {

 [exec]         ~~~~~                                  ^~~~~~~~~~

 [exec] /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/sys/mount.h:399:1: note: 'mount' declared here

 [exec] int     mount(const char *, const char *, int, void *);

To fix it I modified declaration of the function mount() in mount.h temperately to:

int mount(const char *, const char *, const char *,int, const char *);

It's a bit stupid and I know it, but it works at least. And then encountered the new problem showed in the question. I'm wondering whether they are concerned, or some bugs with linking library.

Having debugged for a whole day and feeling trackless about what to be done. Would appreciate it if you could point out the key, or sharing some similar experience handling cmake linking problems.

It seems that function fcloseall doesn't exist on OS X. From the Porting UNIX/Linux Applications to OS X:

fcloseall

This function is an extension to fclose. Although OS X supports fclose, fcloseall is not supported. You can use fclose to implement fcloseall by storing the file pointers in an array and iterating through the array.

You need to redesign the application and store every file which is supposed to be closed with fcloseall. After that, you may use simple close for every such file, as noted in the citation.