How to install Bitcoin Classic Full Node on Amazon Linux EC2 (also valid for RHEL and CentOS)

These are installation instructions (with compilation) for Bitcoin Classic (Full Node) under Amazon Linux 2016.03.0
They should also work on Red Hat RHEL and CentOS.


- Make sure your box is up to date:

__|  __|_  )        _|  (     /   Amazon Linux AMI       ___|\___|___| https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/ sudo yum update

- Install all this:

(This command is a single line)

sudo yum install git boost boost-devel autoconf automake gcc-c++ libtool openssl openssl-devel libevent libevent-devel

- Add PUIAS repository (needed for Berkeley DB when compiling the Bitcoin Wallet):

* Create this file:

sudo vim /etc/yum.repos.d/puias-computational.repo

* And add this content to it and save:

[puias-computational]

name=PUIAS Computational

baseurl=http://puias.math.ias.edu/data/puias/computational/6/x86_64/

enabled=1

gpgcheck=0

- Install DB48

sudo yum install db48 db48-devel

- Download Bitcoin Classic latest version from GitHub:

cd /usr/src

sudo git clone -b v0.12.0cl1 https://github.com/bitcoinclassic/bitcoinclassic.git

cd bitcoinclassic/

Note: Using label v0.12.0cl1 to avoid issues with lib-boost-1.55 (a 0.12.1 requisite)

- Configure:

sudo ./autogen.sh

sudo ./configure

Expected output:

$ sudo ./configure checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking whether to enable maintainer-specific portions of Makefiles... yes checking whether make supports nested variables... (cached) yes checking for g++... g++ checking whether the C++ compiler works... yes checking for C++ compiler default output file name... a.out checking for suffix of executables...  checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C++ compiler... yes checking whether g++ accepts -g... yes checking for style of include used by make... GNU checking dependency style of g++... gcc3 checking for g++... g++ checking whether we are using the GNU Objective C++ compiler... no checking whether g++ accepts -g... no checking dependency style of g++... gcc3 checking how to print strings... printf checking for gcc... gcc checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking dependency style of gcc... gcc3 checking for a sed that does not truncate output... /bin/sed checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for fgrep... /bin/grep -F checking for ld used by gcc... /usr/bin/ld checking if the linker (/usr/bin/ld) is GNU ld... yes checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B checking the name lister (/usr/bin/nm -B) interface... BSD nm checking whether ln -s works... yes checking the maximum length of command line arguments... 1572864 checking whether the shell understands some XSI constructs... yes checking whether the shell understands "+="... yes checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop checking for /usr/bin/ld option to reload object files... -r checking for objdump... objdump checking how to recognize dependent libraries... pass_all checking for dlltool... no checking how to associate runtime and link libraries... printf %s\n checking for ar... ar checking for archiver @FILE support... @ checking for strip... strip checking for ranlib... ranlib checking command to parse /usr/bin/nm -B output from gcc object... ok checking for sysroot... no checking for mt... no checking if : is a manifest tool... no checking how to run the C preprocessor... gcc -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking for dlfcn.h... yes checking for objdir... .libs checking if gcc supports -fno-rtti -fno-exceptions... no checking for gcc option to produce PIC... -fPIC -DPIC checking if gcc PIC flag -fPIC -DPIC works... yes checking if gcc static flag -static works... no checking if gcc supports -c -o file.o... yes checking if gcc supports -c -o file.o... (cached) yes checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes checking whether -lc should be explicitly linked in... no checking dynamic linker characteristics... GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... yes checking whether to build static libraries... yes checking how to run the C++ preprocessor... g++ -E checking for ld used by g++... /usr/bin/ld -m elf_x86_64 checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes checking for g++ option to produce PIC... -fPIC -DPIC checking if g++ PIC flag -fPIC -DPIC works... yes checking if g++ static flag -static works... no checking if g++ supports -c -o file.o... yes checking if g++ supports -c -o file.o... (cached) yes checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes checking dynamic linker characteristics... (cached) GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking for ar... /usr/bin/ar checking for ranlib... /usr/bin/ranlib checking for strip... /usr/bin/strip checking for gcov... /usr/bin/gcov checking for lcov... no checking for java... /usr/bin/java checking for python... /usr/bin/python checking for genhtml... no checking for git... /usr/bin/git checking for ccache... no checking for xgettext... no checking for hexdump... /usr/bin/hexdump checking for readelf... /usr/bin/readelf checking for c++filt... /usr/bin/c++filt checking for pkg-config... /usr/bin/pkg-config checking pkg-config is at least version 0.9.0... yes checking whether byte ordering is bigendian... no checking if compiler needs -Werror to reject unknown flags... no checking for the pthreads library -lpthreads... no checking whether pthreads work without any flags... no checking whether pthreads work with -Kthread... no checking whether pthreads work with -kthread... no checking for the pthreads library -llthread... no checking whether pthreads work with -pthread... yes checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE checking if more special flags are required for pthreads... no checking for PTHREAD_PRIO_INHERIT... yes checking for special C compiler options needed for large files... no checking for _FILE_OFFSET_BITS value needed for large files... no checking whether strerror_r is declared... yes checking for strerror_r... yes checking whether strerror_r returns char *... yes checking whether the linker accepts -Wl,--large-address-aware... no checking for __attribute__((visibility))... yes checking for __attribute__((dllexport))... no checking for __attribute__((dllimport))... no checking for library containing clock_gettime... none required checking whether C++ compiler accepts -fPIC... yes checking whether C++ compiler accepts -Wstack-protector... yes checking whether C++ compiler accepts -fstack-protector-all... yes checking whether C++ preprocessor accepts -D_FORTIFY_SOURCE=2... yes checking whether C++ preprocessor accepts -U_FORTIFY_SOURCE... yes checking whether the linker accepts -Wl,--dynamicbase... no checking whether the linker accepts -Wl,--nxcompat... no checking whether the linker accepts -Wl,-z,relro... yes checking whether the linker accepts -Wl,-z,now... yes checking whether C++ compiler accepts -fPIE... yes checking whether the linker accepts -pie... yes checking endian.h usability... yes checking endian.h presence... yes checking for endian.h... yes checking sys/endian.h usability... no checking sys/endian.h presence... no checking for sys/endian.h... no checking byteswap.h usability... yes checking byteswap.h presence... yes checking for byteswap.h... yes checking stdio.h usability... yes checking stdio.h presence... yes checking for stdio.h... yes checking for stdlib.h... (cached) yes checking for unistd.h... (cached) yes checking for strings.h... (cached) yes checking for sys/types.h... (cached) yes checking for sys/stat.h... (cached) yes checking sys/select.h usability... yes checking sys/select.h presence... yes checking for sys/select.h... yes checking sys/prctl.h usability... yes checking sys/prctl.h presence... yes checking for sys/prctl.h... yes checking for library containing getaddrinfo_a... -lanl checking for library containing inet_pton... none required checking whether strnlen is declared... yes checking whether le16toh is declared... yes checking whether le32toh is declared... yes checking whether le64toh is declared... yes checking whether htole16 is declared... yes checking whether htole32 is declared... yes checking whether htole64 is declared... yes checking whether be16toh is declared... yes checking whether be32toh is declared... yes checking whether be64toh is declared... yes checking whether htobe16 is declared... yes checking whether htobe32 is declared... yes checking whether htobe64 is declared... yes checking whether bswap_16 is declared... yes checking whether bswap_32 is declared... yes checking whether bswap_64 is declared... yes checking for MSG_NOSIGNAL... yes checking for visibility attribute... yes checking for Berkeley DB C++ headers... /usr/include/db4/ checking for main in -ldb_cxx-4.8... yes checking miniupnpc/miniwget.h usability... no checking miniupnpc/miniwget.h presence... no checking for miniupnpc/miniwget.h... no checking miniupnpc/miniupnpc.h usability... no checking miniupnpc/miniupnpc.h presence... no checking for miniupnpc/miniupnpc.h... no checking miniupnpc/upnpcommands.h usability... no checking miniupnpc/upnpcommands.h presence... no checking for miniupnpc/upnpcommands.h... no checking miniupnpc/upnperrors.h usability... no checking miniupnpc/upnperrors.h presence... no checking for miniupnpc/upnperrors.h... no checking for QT... no checking for QT... no configure: WARNING: Qt dependencies not found; bitcoin-qt frontend will not be built checking whether to build Bitcoin Classic GUI... no (Qt5) checking for boostlib >= 1.20.0... yes checking whether the Boost::System library is available... yes checking for exit in -lboost_system... yes checking whether the Boost::Filesystem library is available... yes checking for exit in -lboost_filesystem... yes checking whether the Boost::Program_Options library is available... yes checking for exit in -lboost_program_options-mt... yes checking whether the Boost::Thread library is available... yes checking for exit in -lboost_thread... yes checking whether the Boost::Chrono library is available... yes checking for exit in -lboost_chrono-mt... yes checking whether the Boost::Unit_Test_Framework library is available... yes checking for dynamic linked boost test... yes checking for SSL... yes checking for CRYPTO... yes checking for EVENT... yes checking for EVENT_PTHREADS... yes checking for ZMQ... no configure: WARNING: libzmq version 4.x or greater not found, disabling checking openssl/ec.h usability... yes checking openssl/ec.h presence... yes checking for openssl/ec.h... yes checking whether to build bitcoind... yes checking whether to build utils (bitcoin-cli bitcoin-tx)... yes checking whether to build libraries... yes checking if ccache should be used... no checking if wallet should be enabled... yes checking whether to build with support for UPnP... no checking whether to build test_bitcoin... yes checking whether to reduce exports... no checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating libbitcoinconsensus.pc config.status: creating Makefile config.status: creating src/Makefile config.status: creating share/setup.nsi config.status: creating share/qt/Info.plist config.status: creating src/test/buildenv.py config.status: creating qa/pull-tester/run-bitcoind-for-test.sh config.status: creating qa/pull-tester/tests_config.py config.status: creating qa/pull-tester/rpc-tests.py config.status: creating src/config/bitcoin-config.h config.status: executing depfiles commands config.status: executing libtool commands === configuring in src/secp256k1 (/usr/src/bitcoinclassic/src/secp256k1) configure: running /bin/sh ./configure --disable-option-checking '--prefix=/usr/local'  '--disable-shared' '--with-pic' '--with-bignum=no' '--enable-module-recovery' --cache-file=/dev/null --srcdir=. checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking how to print strings... printf checking for style of include used by make... GNU checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables...  checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking dependency style of gcc... gcc3 checking for a sed that does not truncate output... /bin/sed checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for fgrep... /bin/grep -F checking for ld used by gcc... /usr/bin/ld checking if the linker (/usr/bin/ld) is GNU ld... yes checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B checking the name lister (/usr/bin/nm -B) interface... BSD nm checking whether ln -s works... yes checking the maximum length of command line arguments... 1572864 checking whether the shell understands some XSI constructs... yes checking whether the shell understands "+="... yes checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop checking for /usr/bin/ld option to reload object files... -r checking for objdump... objdump checking how to recognize dependent libraries... pass_all checking for dlltool... no checking how to associate runtime and link libraries... printf %s\n checking for ar... ar checking for archiver @FILE support... @ checking for strip... strip checking for ranlib... ranlib checking command to parse /usr/bin/nm -B output from gcc object... ok checking for sysroot... no checking for mt... no checking if : is a manifest tool... no checking how to run the C preprocessor... gcc -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking for dlfcn.h... yes checking for objdir... .libs checking if gcc supports -fno-rtti -fno-exceptions... no checking for gcc option to produce PIC... -fPIC -DPIC checking if gcc PIC flag -fPIC -DPIC works... yes checking if gcc static flag -static works... no checking if gcc supports -c -o file.o... yes checking if gcc supports -c -o file.o... (cached) yes checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes checking dynamic linker characteristics... GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... no checking whether to build static libraries... yes checking whether make supports nested variables... (cached) yes checking for pkg-config... /usr/bin/pkg-config checking pkg-config is at least version 0.9.0... yes checking for ar... /usr/bin/ar checking for ranlib... /usr/bin/ranlib checking for strip... /usr/bin/strip checking for gcc... gcc checking whether we are using the GNU C compiler... (cached) yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... (cached) none needed checking dependency style of gcc... (cached) gcc3 checking how to run the C preprocessor... gcc -E checking whether gcc and cc understand -c and -o together... yes checking for gcc option to accept ISO C89... (cached) none needed checking if gcc supports -std=c89 -pedantic -Wall -Wextra -Wcast-align -Wnested-externs -Wshadow -Wstrict-prototypes -Wno-unused-function -Wno-long-long -Wno-overlength-strings... yes checking if gcc supports -fvisibility=hidden... yes checking for __int128... yes checking for __builtin_expect... yes checking for x86_64 assembly availability... yes checking for CRYPTO... yes checking for main in -lcrypto... yes checking for EC functions in libcrypto... yes checking whether byte ordering is bigendian... no configure: Using assembly optimizations: x86_64 configure: Using field implementation: 64bit configure: Using bignum implementation: no configure: Using scalar implementation: 64bit configure: Using endomorphism optimizations: no configure: Building ECDH module: no configure: Building Schnorr signatures module: no configure: Building ECDSA pubkey recovery module: yes checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating Makefile config.status: creating libsecp256k1.pc config.status: creating src/libsecp256k1-config.h config.status: executing depfiles commands config.status: executing libtool commands === configuring in src/univalue (/usr/src/bitcoinclassic/src/univalue) configure: running /bin/sh ./configure --disable-option-checking '--prefix=/usr/local'  '--disable-shared' '--with-pic' '--with-bignum=no' '--enable-module-recovery' --cache-file=/dev/null --srcdir=. checking whether make supports nested variables... yes checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking how to print strings... printf checking for style of include used by make... GNU checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables...  checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking dependency style of gcc... gcc3 checking for a sed that does not truncate output... /bin/sed checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for fgrep... /bin/grep -F checking for ld used by gcc... /usr/bin/ld checking if the linker (/usr/bin/ld) is GNU ld... yes checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B checking the name lister (/usr/bin/nm -B) interface... BSD nm checking whether ln -s works... yes checking the maximum length of command line arguments... 1572864 checking whether the shell understands some XSI constructs... yes checking whether the shell understands "+="... yes checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop checking for /usr/bin/ld option to reload object files... -r checking for objdump... objdump checking how to recognize dependent libraries... pass_all checking for dlltool... no checking how to associate runtime and link libraries... printf %s\n checking for ar... ar checking for archiver @FILE support... @ checking for strip... strip checking for ranlib... ranlib checking command to parse /usr/bin/nm -B output from gcc object... ok checking for sysroot... no checking for mt... no checking if : is a manifest tool... no checking how to run the C preprocessor... gcc -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking for dlfcn.h... yes checking for objdir... .libs checking if gcc supports -fno-rtti -fno-exceptions... no checking for gcc option to produce PIC... -fPIC -DPIC checking if gcc PIC flag -fPIC -DPIC works... yes checking if gcc static flag -static works... no checking if gcc supports -c -o file.o... yes checking if gcc supports -c -o file.o... (cached) yes checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes checking dynamic linker characteristics... GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... no checking whether to build static libraries... yes checking for g++... g++ checking whether we are using the GNU C++ compiler... yes checking whether g++ accepts -g... yes checking dependency style of g++... gcc3 checking how to run the C++ preprocessor... g++ -E checking for ld used by g++... /usr/bin/ld -m elf_x86_64 checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes checking for g++ option to produce PIC... -fPIC -DPIC checking if g++ PIC flag -fPIC -DPIC works... yes checking if g++ static flag -static works... no checking if g++ supports -c -o file.o... yes checking if g++ supports -c -o file.o... (cached) yes checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes checking dynamic linker characteristics... (cached) GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating Makefile config.status: creating pc/libunivalue.pc config.status: creating pc/libunivalue-uninstalled.pc config.status: creating univalue-config.h config.status: executing depfiles commands config.status: executing libtool commands Fixing libtool for -rpath problems. $

- Make:

sudo make

(At this point taking a coffee is advised)


- Install:

sudo make install

- Run:

This time in our folder and with our user (no sudo).

cd

nohup bitcoind -externalip=`wget --quiet -O- http://169.254.169.254/latest/meta-data/public-ipv4` &

head -50 .bitcoin/debug.log

Notes:
-externalip is used to tell bitcoind which Public IP to advertise on the Bitcoin network. This is extremely important when running a full-node in EC2.
- To obtain the Public IP of our node we use a standard EC2 Meta-Data API call. It can be used as shown on this example. No adjustments needed. You can learn more about EC2 Meta-Data in here.

2016-04-13 13:31:36 Bitcoin version v0.12.0.0-84d82a8 (2016-04-06 12:21:13 +0100) 2016-04-13 13:31:36 InitParameterInteraction: parameter interaction: -externalip set -> setting -discover=0 2016-04-13 13:31:36 InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1 2016-04-13 13:31:36 Using OpenSSL version OpenSSL 1.0.1k-fips 8 Jan 2015 2016-04-13 13:31:36 Using BerkeleyDB version Berkeley DB 4.8.30: (February 21, 2011) 2016-04-13 13:31:36 Default data directory /home/ec2-user/.bitcoin 2016-04-13 13:31:36 Using data directory /home/ec2-user/.bitcoin 2016-04-13 13:31:36 Using config file /home/ec2-user/.bitcoin/bitcoin.conf 2016-04-13 13:31:36 Using at most 125 connections (1024 file descriptors available) 2016-04-13 13:31:36 Using 4 threads for script verification 2016-04-13 13:31:36 scheduler thread start 2016-04-13 13:31:36 HTTP: creating work queue of depth 16 2016-04-13 13:31:36 No rpcpassword set - using random cookie authentication 2016-04-13 13:31:36 Generated RPC authentication cookie /home/ec2-user/.bitcoin/.cookie 2016-04-13 13:31:36 HTTP: starting 4 worker threads 2016-04-13 13:31:36 Using wallet wallet.dat 2016-04-13 13:31:36 init message: Verifying wallet... 2016-04-13 13:31:36 CDBEnv::Open: LogDir=/home/ec2-user/.bitcoin/database ErrorFile=/home/ec2-user/.bitcoin/db.log 2016-04-13 13:31:36 Bound to [::]:8333 2016-04-13 13:31:36 Bound to 0.0.0.0:8333 2016-04-13 13:31:36 Cache configuration: 2016-04-13 13:31:36 * Using 2.0MiB for block index database 2016-04-13 13:31:36 * Using 32.5MiB for chain state database 2016-04-13 13:31:36 * Using 65.5MiB for in-memory UTXO set 2016-04-13 13:31:36 init message: Loading block index... 2016-04-13 13:31:36 Opening LevelDB in /home/ec2-user/.bitcoin/blocks/index 2016-04-13 13:31:36 Opened LevelDB successfully 2016-04-13 13:31:36 Using obfuscation key for /home/ec2-user/.bitcoin/blocks/index: 0000000000000000 2016-04-13 13:31:36 Opening LevelDB in /home/ec2-user/.bitcoin/chainstate 2016-04-13 13:31:36 Opened LevelDB successfully 2016-04-13 13:31:36 Using obfuscation key for /home/ec2-user/.bitcoin/chainstate: 0000000000000000 ...

Reference: https://github.com/bitcoinclassic/bitcoinclassic/blob/develop/doc/build-unix.md

Deploy SSH Authorized Keys using S3 and AWS CLI with temporary credentials

Disclaimer: Modifying security credentials could render in loosing access to your server in case of problems. I strongly suggest you test the method described here in your Development environment before using it in Production.


Key-Pairs is the standard method to authenticate SSH access to our EC2 Instances based on AWS AMI Linux. We can easily create new Key-Paris for our team using the ssh-keygen command and manually adding them to the file /home/ec2-user/.ssh/authorized_keys for those with root access.
Format:

/home/ec2-user/.ssh/authorized_keys

ssh-rsa AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA== main-key ssh-rsa BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB== juan ssh-rsa CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC== pedro ssh-rsa DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD== luis

But when the number of instances and members grows we need a centralized method of distribution of this file.

Goal

- Store an authorized_keys file in S3 encrypted "at-rest".
- Transport this file from S3 to the instance securely.
- Give access to this file only to the right instances.
- Do not store any API Access Keys into the involved script.
- Store all the temporary files in RAM.


S3

- Create a bucket. In this example is "tarro".

- Create in your local an authorized_keys file and upload it to the new bucket.

- Select the file properties with S3 Console and select Server Side Encryption = AES256 and Save.

- Calculate the MD5 of the file with md5sum. 
Example:

$ md5sum authorized_keys 690f9d901801849f6f54eced2b2d1849  authorized_keys

- Create a file called authorized_keys.md5, copy the md5sum result in (only the hexadecimal string of numbers and letters) and upload it to the same S3 bucket.


IAM

We will use an EC2 IAM instance role. This way we don't need to store a copy of our API Access Key into the instances who will be accessing the secured files. AWS Command Line Interface (AWS CLI) will automatically access to the EC2 Instance Metadata and retrieve a temporary security credential needed to connect to S3. We will specify a role policy to grant read access to the bucket that contains those files.

- Create a role using the IAM Console. In my example is "demo-role".
- Select Role Type = Amazon EC2.
- Select Custom Policy.
- Create a role policy to grant read access only to "tarro" bucket. Example:

{   "Version": "2012-10-17",   "Statement": [     {       "Action": [         "s3:GetObject",         "s3:ListBucket"       ],       "Sid": "Stmt1380102067000",       "Resource": [         "arn:aws:s3:::tarro/*"       ],       "Effect": "Allow"     }   ] }

EC2

We will use Amazon Linux AMI 2013.09 which includes AWS CLI.

- Launch your instance as you usually do but now select the IAM Role and choose the appropriate one. In my example is "demo-role" but you could have different roles for every application tier like: web servers, data bases, test, etc.

- Under root, create /root/bin/

- In /root/bin/ create the file deploy-keys.sh with the following content:

#!/bin/bash # # /root/bin/deploy-keys.sh # Install centralized authorized_keys file from S3 securely using temporary security credentials # blog.domenech.org # ### User defined variables BUCKET="tarro" TMPFOLDER="/media/tmpfs/" # (Finish TMPFOLDER variable with slash) # Note: # The temporary folder in RAM size is 1 Megabyte. # If you are planning to deal with files bigger than that you have to change line #24 accordingly # Create temporary folder if [ ! -e $TMPFOLDER ] then mkdir $TMPFOLDER fi # Mount temporary folder in RAM mount -t tmpfs -o size=1M,mode=700 tmpfs $TMPFOLDER # Get-Object from S3 COMMAND=`aws s3api get-object --bucket $BUCKET --key "authorized_keys" $TMPFOLDER"authorized_keys"` if [ ! $? -eq 0 ] then umount $TMPFOLDER logger "deploy-keys.sh: aws s3api get-object authorized_keys failed! Exiting..." exit 1 fi # Get-Object from S3 (MD5) COMMAND=`aws s3api get-object --bucket $BUCKET --key "authorized_keys.md5" $TMPFOLDER"authorized_keys.md5"` if [ ! $? -eq 0 ] then umount $TMPFOLDER logger "deploy-keys.sh: aws s3api get-object authorized_keys.md5 failed! Exiting..." exit 1 fi # Check MD5, copy the new file if matches and clean up MD5=`cat $TMPFOLDER"authorized_keys.md5"` MD5NOW=`md5sum $TMPFOLDER"authorized_keys" | awk '{print $1}'` if [ $MD5 == $MD5NOW ] then mv --update /home/ec2-user/.ssh/authorized_keys /home/ec2-user/.ssh/authorized_keys.original cp --force $TMPFOLDER"authorized_keys" /home/ec2-user/.ssh/authorized_keys chown ec2-user:ec2-user /home/ec2-user/.ssh/authorized_keys chmod go-rwx /home/ec2-user/.ssh/authorized_keys # The unmount command will delete all the files in RAM but we are extra cautious here shredding and removing shred $TMPFOLDER"authorized_keys"; shred $TMPFOLDER"authorized_keys.md5" rm $TMPFOLDER"authorized_keys"; rm $TMPFOLDER"authorized_keys.md5"; umount $TMPFOLDER logger "deploy-keys.sh: Keys updated successfully." exit 0 else shred $TMPFOLDER"authorized_keys"; shred $TMPFOLDER"authorized_keys.md5" rm $TMPFOLDER"authorized_keys"; rm $TMPFOLDER"authorized_keys.md5"; umount $TMPFOLDER logger "deploy-keys.sh: MD5 check failed! Exiting..." exit 1 fi

- And give execution permissions to root and remove unnecessary Read/Write permissions.

OR you can do it all at once more easily executing this command:

mkdir /root/bin/; cd /root/bin/; wget -q http://www.domenech.org/files/deploy-keys.sh; chmod u+x deploy-keys.sh; chmod go-rwx deploy-keys.sh; chown root:root deploy-keys.sh

and test the script. You can check the script results at /var/log/messages

- Trigger after reboot the script by adding the line /root/bin/deploy-keys.sh at the system init script /etc/rc.local

#!/bin/sh # # This script will be executed *after* all the other init scripts. # You can put your own initialization stuff in here if you don't # want to do the full Sys V style init stuff. touch /var/lock/subsys/local /root/bin/deploy-keys.sh