RocksDB temp directory should be set to local APM directory out of the box to prevent errors

Idea created by MatthewConnolly62353783 on Mar 8, 2018
    New
    Score6
    • Yanna
    • zhoyi03
    • Lynn_Williams
    • Aryne
    • Hallett_German
    • annma01

    When the EM is initializing, RocksDB creates a temporary lib.so file within the temp dir of the JVM, given by the java.io.tmpdir system property by default. For example this file looks similar to (the numbers in the name will vary):

    /tmp/librocksdbjni5626861023512855361.so

     

    java.io.tmpdir defaults to /tmp on Linux and Unix systems in general. The .so file is then memory-mapped by the JVM process so code from it can be executed. Memory-mapping requires that the file reside on a filesystem that is executable (the file does not necessarily need execute permission on itself, but it must be readable). Note the distinction here: the FILESYSTEM must be executable, meaning that it must NOT be mounted with a "noexec" flag. It is common security practice to mount such temporary locations with "noexec", however. In such a situation the .so file will fail to memory-map and the EM will fail to start.

     

    The workaround is to define the java.io.tmpdir to point to a temp location on a filesystem that *IS* executable (i.e. does NOT have noexec specified in mount options).

     

    Ideally the APM .lax file used to start the EM process should make sure that the tmp directory is defined in the existing APM home directory which will prevent this issue from occuring :

    EM failed to start - /tmp/librocksdbjni8883977260053861907.so: failed to map segment from shared object: Operation not permitted  

     

    Alternatively, the product documentation should clearly state that it is a requirement that that /tmp is on an executable filesystem. For example, if /tmp is mounted as an individual filesystem, ensure it does not have the noexec flag set. (mount -o remount,exec /tmp)