SoftWhat?

Software Engineering tidbits

Using Couchbase Python SDK on Cloud Foundry

Posted at — Jun 9, 2016

I wanted to use Couchbase as a session store for my app instances, and specifically Couchbase Python SDK. The problem I faced was that Couchbase Python SDK (and some of Couchbase other non-C SDKs) requires Couchbase C SDK to be installed a priori – it is not installed by pip automatically.

Sadly, Conda package manager doesn’t have Couchbase C SDK in its repository, so I could not use Cloud Foundry Python Conda buildpack to install it (it can be very useful to install SciPy for example, which is supported by Conda). I wanted to avoid bundling of Couchbase C SDK in my Git repository (binary files ❌), so set out to look for a way of installing Couchbase C SDK dynamically during application push to Cloud Foundry. Cloud Foundry documentation and Google were not so kind to me this time 😢 – it took a lot of poking around, but eventually I found a nifty feature in Cloud Foundry Python buildpack source code – pre and post compile hooks!

The eventual solution was combined of several things (code snippets below):

Note: If you don’t feel like compiling the Couchbase C SDK dynamically each time, you can compile it locally (make sure it’s binary compatible with Cloud Foundry container). Then local installation can be compressed and stored somewhere, for example in Amazon Simple Storage Service (Amazon S3), and retrieved dynamically during the deployment.

bin/pre_compile

#!/usr/bin/env bash
 
_LIBCOUCHBASE_VER=2.5.8
_PROJECT_ROOT=/app
 
# Build and install Couchbase C SDK (core library only).
# Couchbase make code runs some Git commands, so requires making a clone.
git clone --depth 1 --branch $_LIBCOUCHBASE_VER \
    https://github.com/couchbase/libcouchbase.git /tmp/libcouchbase_$$
cd /tmp/libcouchbase_$$
./configure.pl \
    --prefix $_PROJECT_ROOT \
    --disable-tools \
    --disable-tests \
    --disable-plugins \
    --disable-couchbasemock
make
make install

requirements.txt

couchbase==2.0.8 --global-option='build_ext' \
                 --global-option='-L/app/lib' \
                 --global-option='-I/app/include'

manifest.yml

env:
  LD_LIBRARY_PATH: /app/lib