Caffe on OS X El Capitan

There's a natural barrier to entry for anything that's interesting to work with. If you want something cool, you have to work for it.

Caffe is no exception. Getting the libraries up and running is by now known to be a little bit of a headache. The good news is: so many people have tried it out (sparked by the desire to create hypnotic and viral DeepDream images) that there's plenty of forums dedicated to debugging every possible installation and compilation error (from noob problems to more expert customizations).

To add to this, I include here a few notes for installation and compilation of caffe on my Mac (in case it captures similar problems others may have).

Apparently the ‘make runtest’ (final set of tests to be run to check for correct compilation and library linking) is actually supposed to fail on El Capitan because the new OS strips DYLD_LIBRARY_PATH from the environment, so a number of libraries (that are usually pointed to in the path) are not visible during compile time and cause errors (http://www.megastormsystems.com/news/how-to-install-caffe-on-mac-os-x-10-11). Interestingly, this can cause similar errors for other code bases being compiled on El Capitan (e.g. https://www.postgresql.org/message-id/561E73AB.8060800@gmx.net)

I believe that following the original caffe instructions (http://caffe.berkeleyvision.org/install_osx.html) properly should work ok. I used all the recommended defaults (including anaconda python), except I needed to install OpenBLAS instead of the default.

Then, one can either just skip the make runtest command (since it will fail) and hope for the best, or one has to disable the new OS security feature (http://www.macworld.com/article/2986118/security/how-to-modify-system-integrity-protection-in-el-capitan.html). I disabled SIP (using recovery mode), ran the tests to make sure I didn’t get any errors, and then re-enabled SIP.

After all that, there might still be library linking issues requiring patches (https://github.com/BVLC/caffe/issues/3227#issuecomment-167540070). I also needed to add a symbolic link from my anaconda hdf5 to /usr/local/lib.  

I added all the recommended paths to ~/.bashrc. Here's what it looks like:

export PATH=$HOME/anaconda/bin:/usr/local/cuda/bin:$PATH

export DYLD_FALLBACK_LIBRARY_PATH=/usr/local/cuda/lib:$HOME/anaconda/lib:/usr/local/lib:/usr/lib:$DYLD_FALLBACK_LIBRARY_PATH

export CPLUS_INCLUDE_PATH=$HOME/anaconda/include/python2.7/:

export DYLD_LIBRARY_PATH=/usr/local/cuda/lib:$DYLD_LIBRARY_PATH

export PYTHONPATH=~/Documents/caffe/python:$PYTHONPATH

export LD_LIBRARY_PATH=$HOME/anaconda/lib

I also wanted to get iPython notebook running on my local machine. However, I was having errors linking to libraries in iPython notebook but not in python. Turns out there’s a hack where instead of running ipython from the command line, you run python –m IPython

And respectively, for ipython notebook: python –m IPython notebook

(this imports ipython explicitely) (http://stackoverflow.com/questions/22241420/execution-of-python-code-with-m-option-or-not)

Then everything works as expected with caffe.

In general, I’ve seen mixed opinions about using the anaconda python installation or not, which solves some issues during caffe compilation but comes also with its own

(http://stackoverflow.com/questions/31460102/caffe-import-working-with-python2-but-not-python2-7-ipython-notebook-despite).

On my laptop, my set-up is to use the Pycaffe interface and train using CPU only. Here is what my final Makefile.config looks like:

## Refer to http://caffe.berkeleyvision.org/installation.html

# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).

# USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).

CPU_ONLY := 1

# uncomment to disable IO dependencies and corresponding data layers

# USE_OPENCV := 0

# USE_LEVELDB := 0

# USE_LMDB := 0

# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)

# You should not set this flag if you will be reading LMDBs with any

# possibility of simultaneous read and write

# ALLOW_LMDB_NOLOCK := 1

# Uncomment if you're using OpenCV 3

# OPENCV_VERSION := 3

# To customize your choice of compiler, uncomment and set the following.

# N.B. the default for Linux is g++ and the default for OSX is clang++

# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.

CUDA_DIR := /usr/local/cuda

# On Ubuntu 14.04, if cuda tools are installed via

# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:

# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.

# For CUDA < 6.0, comment the *_50 lines for compatibility.

CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \

-gencode arch=compute_20,code=sm_21 \

-gencode arch=compute_30,code=sm_30 \

-gencode arch=compute_35,code=sm_35 \

-gencode arch=compute_50,code=sm_50 \

-gencode arch=compute_50,code=compute_50

# BLAS choice:

# atlas for ATLAS (default)

# mkl for MKL

# open for OpenBlas

BLAS := open

# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.

# Leave commented to accept the defaults for your choice of BLAS

# (which should work)!

 BLAS_INCLUDE := /usr/local/Cellar/openblas/0.2.18/include/

 BLAS_LIB := /usr/local/Cellar/openblas/0.2.18/lib/

# This is required only if you will compile the matlab interface.

# MATLAB directory should contain the mex binary in /bin.

# MATLAB_DIR := /usr/local

# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.

# We need to be able to find Python.h and numpy/arrayobject.h.

# Anaconda Python distribution is quite popular. Include path:

# Verify anaconda location, sometimes it's in root.

 ANACONDA_HOME := $(HOME)/anaconda

 PYTHON_INCLUDE := $(ANACONDA_HOME)/include \

$(ANACONDA_HOME)/include/python2.7 \

$(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \

# We need to be able to find libpythonX.X.so or .dylib.

PYTHON_LIB := $(ANACONDA_HOME)/lib

# Uncomment to support layers written in Python (will link against Python libs)

 WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.

INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include

LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib

# Uncomment to use `pkg-config` to specify OpenCV library paths.

# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)

# USE_PKG_CONFIG := 1

# N.B. both build and distribute dirs are cleared on `make clean`

BUILD_DIR := build

DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171

# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.

TEST_GPUID := 0

# enable pretty build (comment to see full commands)

Q ?= @