Wednesday, October 20, 2021

Logstash: How to check if a field exists?

 


TL;DR: 


 if [yourField] or [yourField] =~ /.*/ {
    # Your code
 }


Why?


In Logstash [yourField] could come in one of these datatypes:
  • [yourField] = String
  • [yourField] = Null
  • [yourField] = Number
  • [yourField] = Boolean
i.e. [level] = "ERROR"

And also could have sub-fields and therefore these combinations:
  • [yourField][yourSub] = String
  • [yourField][yourSub] = Null
  • [yourField][yourSub] = Number
  • [yourField][yourSub] = Boolean
i.e. [request][time_milliseconds] = 73


If we use "if [yourField]" to check if the field exists, turns out that not always matches and if we use "if [yourField] =~ /.*/" not always matches.

Result:


if [yourField]

if [yourField] =~ /.*/




[yourField] = "foo"

Match

Match

[yourField] = ""

NO

Match

[yourField] = 0

Match

NO

[yourField] = True

Match

NO




[yourField][yourSub] = "foo"

Match

NO

[yourField][yourSub] = ""

Match

NO

[yourField][yourSub] = 0

Match

NO

[yourField][yourSub] = True

Match

NO


  • "if [yourField]" won't work when the field exists but is Null.
  • "if [yourField] =~ /.*/" won't work when is a number, boolean or has sub-fields.

The safer approach is:

if [yourField] or [yourField] =~ /.*/ {

    # Your code

}






Monday, February 27, 2017

Unix like " tail -f " command for Elasticsearch (using Python)

elasticsearch-logo-tail-command-linux-blog-domenech-org



Humble implementation of a Unix like "tail" command for Elasticsearch using Python.

Tested with Logstash indexed content.

Install


$ git clone https://github.com/juan-domenech/elasticsearch-python.git
$ cd elasticsearch-python/
$ python elasticsearch-tail.py



Basic usage

The only mandatory parameter is --endpoint

Example:

$ python elasticsearch-tail.py --endpoint http://elk.example.com



By default the last 10 lines of log are displayed. You can change this behaviour with --docs or -n switch.

Example: To display the last 50 lines.

$ python elasticsearch-tail.py --endpoint http://elk.example.com -n 50



To have continuous output use -f or --nonstop

Example:

$ python elasticsearch-tail.py --endpoint http://elk.example.com -f



By default ES type = apache is used. You can select other types with --type

Examples:

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type java

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type apache


Advanced


By default the more recent Logstash Index is used. Optionally you can specify the desired index name using --index

Example:

$ python elasticsearch-tail.py --endpoint http://elk.example.com --index logstash-2016.08.08



When using --type java there are two other selectors available: --javalevel and --javaclass

Examples:

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type java --javalevel ERROR

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type java --javaclas error.handler.java.class



When using --type apache there are two other selectors available: --httpresponse and --httpmethod

Examples:

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type apache --httpresponse 404

$ python elasticsearch-tail.py --endpoint http://elk.example.com --type apache --httpmethod POST



To display the native Elasticsearch timestamp of each event use --showheaders (Convenient when adding a |grep on the output to do additional filtering):

Example:

python elasticsearch-tail.py --endpoint http://elk.example.com --showheaders



To display events belonging to a particular host and ignore the rest use --hostname

Example:

$ python elasticsearch-tail.py --endpoint http://elk.example.com --hostname server1.example.com



Connection modes


The default protocol is HTTP (Port 80) but any other port can be specified under --endpoint

Example for HTTPS:

$ python elasticsearch-tail.py -f --endpoint https://elk.example.com 

or

$ python elasticsearch-tail.py -f --endpoint https://elk.example.com:443



Example for Elasticsearch native port:

$ python elasticsearch-tail.py -f --endpoint http://elk.example.com:9200




Known issues

You need elasticsearch Python module installed


If you don't have it, run 'sudo pip install elasticsearch' to install it.


TLS error: "elasticsearch.exceptions.ConnectionError: ConnectionError(('Connection failed.', CannotSendRequest())) caused by: ConnectionError(('Connection failed.', CannotSendRequest()))"


Update urllib3 running 'sudo pip install --upgrade urllib3' or use a non HTTPS Endpoint URL.


Error: "check_index: No index found! Exiting"


The current script only cares for indices with the "logstash-" prefix. If you use any other prefix to index your logs it will fail.
If you think this shouldn't be this way please let me know in the comments.



elasticsearch-logo-tail-command-linux-python-blog-domenech-org

Done for fun. Feel free to comment on bugs or additional desired features.

Thank you!

Wednesday, April 20, 2016

Outbound IPv6 on AWS EC2 Amazon Linux How-To (Not Production Grade)




It is well known that we can deploy an application in AWS and be fully IPv6 compliant thanks to the AAAA DNS records that every EC2 Elastic Load Balancer have at our disposal, but this does not apply to Outbound Internet connections (connections that are originated in our EC2 boxes).
The arrival of IPv6 to EC2 could be near but meanwhile there is a way to provide outbound IPv6 connectivity to our servers thanks to Hurricane Electric tunnel broker service.

I call this solution "Not Production Grade" because it is provided for free for experimentation purposes. Please read the Terms of Service (I have to say that is pretty fast and stable though).



Important Security Note:
With no additional measures in place, the configuration described here will open your TCP/IP services to Internet. Deploying a TCP tunnel will bypass the EC2 Security Group security layer.
IPv6 has no Network Address Translation (NAT) and your server will be directly connected to Internet to all effects.
Enabling and configuring ip6tables is advised.




Register:

- Get your free IPv6 tunnel at https://www.tunnelbroker.net




- Open your EC2 Security Group to receive ICMP traffic from Hurricane Electric (This is a requisite for this tunnel provider).




- Fill the field "IPv4 Endpoint (Your side)" with the Public IP of your instance.

- Select an IPv4 tunnel endpoint close to your AWS region.




- Once the tunnel is created we can access its details. No other changes are required, the tunnel is ready to use.




Configure:

- Click on "Example Configurations" to obtain the configuration guidelines for our Operative System (In our case: "Linux-net-tools" option).




Important Security Note:
With no additional measures in place, the configuration described here will open your TCP/IP services to Internet. Deploying a TCP tunnel will bypass the EC2 Security Group security layer.
IPv6 has no Network Address Translation (NAT) and your server will be directly connected to Internet to all effects.
Enabling and configuring ip6tables is advised.


(with sudo)


sudo ifconfig sit0 up
sudo ifconfig sit0 inet6 tunnel ::216.66.88.98
sudo ifconfig sit1 up
sudo ifconfig sit1 inet6 add 2001:470:1f1c:666::2/64
sudo route -A inet6 add ::/0 dev sit1


Note: In your case these IP addresses will vary.

- At this point the new interface and the tunnel are ready.



Test:

- Check our new interface sit1 and its IPv6 configuration. In this example the IP 2001:470:1f1c:666::2 is our Public IPv6 address for this server.


$ ifconfig sit1

sit1      Link encap:IPv6-in-IPv4  
          inet6 addr: 2001:470:1f1c:666::2/64 Scope:Global
          inet6 addr: fe80::a52:b404/64 Scope:Link
          UP POINTOPOINT RUNNING NOARP  MTU:1480  Metric:1
          RX packets:111 errors:0 dropped:0 overruns:0 frame:0
          TX packets:110 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:14575 (14.2 KiB)  TX bytes:11293 (11.0 KiB)




- ping6 (against Google IPv6 DNS server)


$ ping6 -c 5 2001:4860:4860::8888

PING 2001:4860:4860::8888(2001:4860:4860::8888) 56 data bytes
64 bytes from 2001:4860:4860::8888: icmp_seq=1 ttl=56 time=18.9 ms
64 bytes from 2001:4860:4860::8888: icmp_seq=2 ttl=56 time=19.0 ms
64 bytes from 2001:4860:4860::8888: icmp_seq=3 ttl=56 time=19.0 ms
64 bytes from 2001:4860:4860::8888: icmp_seq=4 ttl=56 time=19.1 ms
64 bytes from 2001:4860:4860::8888: icmp_seq=5 ttl=56 time=19.0 ms

--- 2001:4860:4860::8888 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4034ms
rtt min/avg/max/mdev = 18.993/19.062/19.125/0.131 ms



- route


$ route -n -A inet6

Kernel IPv6 routing table
Destination                     Next Hop                                Flags Metric Ref    Use Iface
::/96                              ::                                      U     256    0        0 sit0
2001:470:1f1c:666::/64             ::                                      U     256    0        0 sit1
fe80::/64                          ::                                      U     256    0        0 eth0
fe80::/64                          ::                                      U     256    0        0 sit1
::/0                               ::                                      U     1      3469     1 sit1
::1/128                            ::                                      U     0      22       2 lo
::10.82.180.4/128                  ::                                      U     0      0        1 lo
::127.0.0.1/128                    ::                                      U     0      0        1 lo
2001:470:1f1c:666::2/128           ::                                      U     0      3397     2 lo
fe80::a52:b404/128                 ::                                      U     0      0        1 lo
fe80::2000:aff:fe52:b404/128       ::                                      U     0      0        1 lo
ff00::/8                           ::                                      U     256    0        0 eth0

ff00::/8                           ::                                      U     256    0        0 sit1


::/0 is the Default route in IPv6 (equivalent to 0.0.0.0/0 in IPv4).
::1 host is our localhost interface (equivalent to 127.0.0.1). 

In IPv6 one or more leading zeroes from any groups of hexadecimal digits are removed and consecutive sections of zeroes are replaced with a double colon (::). 
This 0000:0000:0000:0000:0000:0000:0000:0001 is equal to ::1


- netstat


$ telnet www.google.com 80 &
$ netstat -nat -A inet6

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address              State      
tcp        0      0 :::22                       :::*                         LISTEN      
tcp        0      0 :::38030                    :::*                         LISTEN      
tcp        0      0 :::111                      :::*                         LISTEN      
tcp        0      0 2001:470:1f1c:666::2:35960  2a00:1450:400b:802::2004:80  ESTABLISHED 


Notice that our telnet command has created an IPv6 connection. We didn't specify any IPv6 parameter. How come? More about that later. Check DNS Considerations at the bottom of this article to know more.



- traceroute


$ traceroute -6 2001:4860:4860::8888

traceroute to 2001:4860:4860::8888 (2001:4860:4860::8888), 30 hops max, 80 byte packets
 1  juandomenech-1.tunnel.tserv1.lon2.ipv6.he.net (2001:470:1f1c:666::1)  13.331 ms  13.703 ms  14.143 ms
 2  ge3-20.core1.lon2.he.net (2001:470:0:320::1)  42.622 ms  75.028 ms  75.002 ms
 3  2001:7f8:4::3b41:1 (2001:7f8:4::3b41:1)  15.652 ms  15.614 ms  15.386 ms
 4  2001:4860::1:0:ab9e (2001:4860::1:0:ab9e)  14.648 ms 2001:4860::1:0:ab9d (2001:4860::1:0:ab9d)  14.697 ms 2001:4860::1:0:9914 (2001:4860::1:0:9914)  15.386 ms
 5  2001:4860::8:0:aba0 (2001:4860::8:0:aba0)  26.167 ms 2001:4860::8:0:ab9f (2001:4860::8:0:ab9f)  26.139 ms 2001:4860::8:0:aba0 (2001:4860::8:0:aba0)  26.113 ms
 6  2001:4860::8:0:83d2 (2001:4860::8:0:83d2)  26.079 ms 2001:4860::8:0:507c (2001:4860::8:0:507c)  18.276 ms 2001:4860::8:0:83d2 (2001:4860::8:0:83d2)  17.891 ms
 7  2001:4860::2:0:7a79 (2001:4860::2:0:7a79)  18.432 ms 2001:4860::2:0:79fb (2001:4860::2:0:79fb)  20.376 ms  20.369 ms

 8  google-public-dns-a.google.com (2001:4860:4860::8888)  19.208 ms  18.261 ms  17.545 ms



Notice Hop#1. It is the other site of the tunnel. The address :666::1 is the gateway of our network. 




- Type in your IPv6 Address, hit Submit and wait for 10 seconds. 

Do you see something interesting? Yes, as mentioned before, the ports 22 and 111 are open to the network over IPv6 bypassing the security provided by the EC2 Security Groups. 
Creating a TCP/IP tunnel has the same effect as adding another Internet connection to our instance. That traffic is encapsulated over TCP/IP and is out of control of the traditional EC2 Security Group firewall layer. 
Configuring ip6tables is advised.



DNS considerations:

We have added new interfaces to this box and we are routing IPv6 through a tunnel but we haven't changed its DNS configuration. It has the standard EC2 DNS configuration unchanged (EC2-Classic):



$ cat /etc/resolv.conf

; generated by /sbin/dhclient-script
search eu-west-1.compute.internal
options timeout:2 attempts:5
nameserver 172.16.0.23



Despite that, our previous telnet (telnet www.google.com 80) is connecting to Google's IPv6. Let's take a look to the dialog between our box and the EC2-Classic DNS server 172.16.0.23 to understand why:



$ sudo tcpdump -i eth0 -nn -s0 -A port 53

13:31:52.961142 IP 10.104.229.189.47624 > 172.16.0.23.53: 61300+ A? www.google.com. (32)
E..<.c@...1.
h.........5.(...t...........www.google.com.....
13:31:52.961158 IP 10.104.229.189.47624 > 172.16.0.23.53: 48977+ AAAA? www.google.com. (32)
E..<.d@...0.
h.........5.(...Q...........www.google.com.....
13:31:52.962279 IP 172.16.0.23.53 > 10.104.229.189.47624: 48977 1/0/0 AAAA 2a00:1450:400b:802::2004 (60)
E..X....@..N....
h...5...Ds3.Q...........www.google.com..............w..*..P@......... .
13:31:52.963683 IP 172.16.0.23.53 > 10.104.229.189.47624: 61300 6/0/0 A 209.85.203.104, A 209.85.203.105, A 209.85.203.106, A 209.85.203.147, A 209.85.203.99, A 209.85.203.103 (128)
E.......@.. ....
h...5....u..t...........www.google.com..............,...U.h.........,...U.i.........,...U.j.........,...U...........,...U.c.........,...U.g




- Packets #1 and #2 are our requests and packets #3 and #4 are the answers coming back from the DNS server.
- Our Linux box is resolving www.google.com twice. First with IPv4 (A) and second with IPv6 (AAAA).
- Each request receives a different answer. The A record receives a list of IPv4 addresses and the record AAAA receives a single IPv6 address (2a00:1450:400b:802::2004).  This is the address our box has decided to use.

In other words, during the DNS resolution our system determines whether this host is accessible using IPv6 or not. The way to do that is asking for the AAAA DNS record and use it when present.

We can do the same using dig.


- dig


$ dig AAAA www.linkedin.com

;; ANSWER SECTION:
www.linkedin.com. 88 IN CNAME glb-any-eu.www.linkedin.com.
glb-any-eu.www.linkedin.com. 88 IN CNAME any-eu.www.linkedin.com.
any-eu.www.linkedin.com. 1869 IN AAAA 2a04:f540:1::b93f:930a


Migrated = Yes



$ dig AAAA www.facebook.com


;; ANSWER SECTION:
www.facebook.com. 14 IN CNAME star-mini.c10r.facebook.com.
star-mini.c10r.facebook.com. 41 IN AAAA 2a03:2880:2130:7f20:face:b00c:0:25de


Migrated = Yes



$ dig AAAA github.com


;; AUTHORITY SECTION:
github.com. 60 IN SOA ns1.p16.dynect.net. hostmaster.github.com. 1461162636 3600 600 604800 60


Not migrated yet.


Wednesday, April 13, 2016

How to install Bitcoin Classic Full Node on Amazon Linux EC2 (also valid for RHEL and CentOS)


bitcoin-classic


These are installation instructions (with compilation) for Bitcoin Classic (Full Node) under Amazon Linux 2016.03.0
They should also work on Red Hat RHEL and CentOS.


- Make sure your box is up to date:

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/

sudo yum update



- Install all this:

(This command is a single line)

sudo yum install git boost boost-devel autoconf automake gcc-c++ libtool openssl openssl-devel libevent libevent-devel



- Add PUIAS repository (needed for Berkeley DB when compiling the Bitcoin Wallet):

* Create this file:

sudo vim /etc/yum.repos.d/puias-computational.repo


* And add this content to it and save:
[puias-computational]
name=PUIAS Computational
baseurl=http://puias.math.ias.edu/data/puias/computational/6/x86_64/
enabled=1
gpgcheck=0


- Install DB48


sudo yum install db48 db48-devel



- Download Bitcoin Classic latest version from GitHub:


cd /usr/src

sudo git clone -b v0.12.0cl1 https://github.com/bitcoinclassic/bitcoinclassic.git

cd bitcoinclassic/


Note: Using label v0.12.0cl1 to avoid issues with lib-boost-1.55 (a 0.12.1 requisite)


- Configure:


sudo ./autogen.sh

sudo ./configure



Expected output:
$ sudo ./configure
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether to enable maintainer-specific portions of Makefiles... yes
checking whether make supports nested variables... (cached) yes
checking for g++... g++
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking for style of include used by make... GNU
checking dependency style of g++... gcc3
checking for g++... g++
checking whether we are using the GNU Objective C++ compiler... no
checking whether g++ accepts -g... no
checking dependency style of g++... gcc3
checking how to print strings... printf
checking for gcc... gcc
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking dependency style of gcc... gcc3
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... @
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for mt... no
checking if : is a manifest tool... no
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking how to run the C++ preprocessor... g++ -E
checking for ld used by g++... /usr/bin/ld -m elf_x86_64
checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking for g++ option to produce PIC... -fPIC -DPIC
checking if g++ PIC flag -fPIC -DPIC works... yes
checking if g++ static flag -static works... no
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... (cached) GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking for ar... /usr/bin/ar
checking for ranlib... /usr/bin/ranlib
checking for strip... /usr/bin/strip
checking for gcov... /usr/bin/gcov
checking for lcov... no
checking for java... /usr/bin/java
checking for python... /usr/bin/python
checking for genhtml... no
checking for git... /usr/bin/git
checking for ccache... no
checking for xgettext... no
checking for hexdump... /usr/bin/hexdump
checking for readelf... /usr/bin/readelf
checking for c++filt... /usr/bin/c++filt
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking whether byte ordering is bigendian... no
checking if compiler needs -Werror to reject unknown flags... no
checking for the pthreads library -lpthreads... no
checking whether pthreads work without any flags... no
checking whether pthreads work with -Kthread... no
checking whether pthreads work with -kthread... no
checking for the pthreads library -llthread... no
checking whether pthreads work with -pthread... yes
checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
checking if more special flags are required for pthreads... no
checking for PTHREAD_PRIO_INHERIT... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking whether strerror_r is declared... yes
checking for strerror_r... yes
checking whether strerror_r returns char *... yes
checking whether the linker accepts -Wl,--large-address-aware... no
checking for __attribute__((visibility))... yes
checking for __attribute__((dllexport))... no
checking for __attribute__((dllimport))... no
checking for library containing clock_gettime... none required
checking whether C++ compiler accepts -fPIC... yes
checking whether C++ compiler accepts -Wstack-protector... yes
checking whether C++ compiler accepts -fstack-protector-all... yes
checking whether C++ preprocessor accepts -D_FORTIFY_SOURCE=2... yes
checking whether C++ preprocessor accepts -U_FORTIFY_SOURCE... yes
checking whether the linker accepts -Wl,--dynamicbase... no
checking whether the linker accepts -Wl,--nxcompat... no
checking whether the linker accepts -Wl,-z,relro... yes
checking whether the linker accepts -Wl,-z,now... yes
checking whether C++ compiler accepts -fPIE... yes
checking whether the linker accepts -pie... yes
checking endian.h usability... yes
checking endian.h presence... yes
checking for endian.h... yes
checking sys/endian.h usability... no
checking sys/endian.h presence... no
checking for sys/endian.h... no
checking byteswap.h usability... yes
checking byteswap.h presence... yes
checking for byteswap.h... yes
checking stdio.h usability... yes
checking stdio.h presence... yes
checking for stdio.h... yes
checking for stdlib.h... (cached) yes
checking for unistd.h... (cached) yes
checking for strings.h... (cached) yes
checking for sys/types.h... (cached) yes
checking for sys/stat.h... (cached) yes
checking sys/select.h usability... yes
checking sys/select.h presence... yes
checking for sys/select.h... yes
checking sys/prctl.h usability... yes
checking sys/prctl.h presence... yes
checking for sys/prctl.h... yes
checking for library containing getaddrinfo_a... -lanl
checking for library containing inet_pton... none required
checking whether strnlen is declared... yes
checking whether le16toh is declared... yes
checking whether le32toh is declared... yes
checking whether le64toh is declared... yes
checking whether htole16 is declared... yes
checking whether htole32 is declared... yes
checking whether htole64 is declared... yes
checking whether be16toh is declared... yes
checking whether be32toh is declared... yes
checking whether be64toh is declared... yes
checking whether htobe16 is declared... yes
checking whether htobe32 is declared... yes
checking whether htobe64 is declared... yes
checking whether bswap_16 is declared... yes
checking whether bswap_32 is declared... yes
checking whether bswap_64 is declared... yes
checking for MSG_NOSIGNAL... yes
checking for visibility attribute... yes
checking for Berkeley DB C++ headers... /usr/include/db4/
checking for main in -ldb_cxx-4.8... yes
checking miniupnpc/miniwget.h usability... no
checking miniupnpc/miniwget.h presence... no
checking for miniupnpc/miniwget.h... no
checking miniupnpc/miniupnpc.h usability... no
checking miniupnpc/miniupnpc.h presence... no
checking for miniupnpc/miniupnpc.h... no
checking miniupnpc/upnpcommands.h usability... no
checking miniupnpc/upnpcommands.h presence... no
checking for miniupnpc/upnpcommands.h... no
checking miniupnpc/upnperrors.h usability... no
checking miniupnpc/upnperrors.h presence... no
checking for miniupnpc/upnperrors.h... no
checking for QT... no
checking for QT... no
configure: WARNING: Qt dependencies not found; bitcoin-qt frontend will not be built
checking whether to build Bitcoin Classic GUI... no (Qt5)
checking for boostlib >= 1.20.0... yes
checking whether the Boost::System library is available... yes
checking for exit in -lboost_system... yes
checking whether the Boost::Filesystem library is available... yes
checking for exit in -lboost_filesystem... yes
checking whether the Boost::Program_Options library is available... yes
checking for exit in -lboost_program_options-mt... yes
checking whether the Boost::Thread library is available... yes
checking for exit in -lboost_thread... yes
checking whether the Boost::Chrono library is available... yes
checking for exit in -lboost_chrono-mt... yes
checking whether the Boost::Unit_Test_Framework library is available... yes
checking for dynamic linked boost test... yes
checking for SSL... yes
checking for CRYPTO... yes
checking for EVENT... yes
checking for EVENT_PTHREADS... yes
checking for ZMQ... no
configure: WARNING: libzmq version 4.x or greater not found, disabling
checking openssl/ec.h usability... yes
checking openssl/ec.h presence... yes
checking for openssl/ec.h... yes
checking whether to build bitcoind... yes
checking whether to build utils (bitcoin-cli bitcoin-tx)... yes
checking whether to build libraries... yes
checking if ccache should be used... no
checking if wallet should be enabled... yes
checking whether to build with support for UPnP... no
checking whether to build test_bitcoin... yes
checking whether to reduce exports... no
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating libbitcoinconsensus.pc
config.status: creating Makefile
config.status: creating src/Makefile
config.status: creating share/setup.nsi
config.status: creating share/qt/Info.plist
config.status: creating src/test/buildenv.py
config.status: creating qa/pull-tester/run-bitcoind-for-test.sh
config.status: creating qa/pull-tester/tests_config.py
config.status: creating qa/pull-tester/rpc-tests.py
config.status: creating src/config/bitcoin-config.h
config.status: executing depfiles commands
config.status: executing libtool commands
=== configuring in src/secp256k1 (/usr/src/bitcoinclassic/src/secp256k1)
configure: running /bin/sh ./configure --disable-option-checking '--prefix=/usr/local'  '--disable-shared' '--with-pic' '--with-bignum=no' '--enable-module-recovery' --cache-file=/dev/null --srcdir=.
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking how to print strings... printf
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking dependency style of gcc... gcc3
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... @
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for mt... no
checking if : is a manifest tool... no
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... no
checking whether to build static libraries... yes
checking whether make supports nested variables... (cached) yes
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for ar... /usr/bin/ar
checking for ranlib... /usr/bin/ranlib
checking for strip... /usr/bin/strip
checking for gcc... gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... (cached) none needed
checking dependency style of gcc... (cached) gcc3
checking how to run the C preprocessor... gcc -E
checking whether gcc and cc understand -c and -o together... yes
checking for gcc option to accept ISO C89... (cached) none needed
checking if gcc supports -std=c89 -pedantic -Wall -Wextra -Wcast-align -Wnested-externs -Wshadow -Wstrict-prototypes -Wno-unused-function -Wno-long-long -Wno-overlength-strings... yes
checking if gcc supports -fvisibility=hidden... yes
checking for __int128... yes
checking for __builtin_expect... yes
checking for x86_64 assembly availability... yes
checking for CRYPTO... yes
checking for main in -lcrypto... yes
checking for EC functions in libcrypto... yes
checking whether byte ordering is bigendian... no
configure: Using assembly optimizations: x86_64
configure: Using field implementation: 64bit
configure: Using bignum implementation: no
configure: Using scalar implementation: 64bit
configure: Using endomorphism optimizations: no
configure: Building ECDH module: no
configure: Building Schnorr signatures module: no
configure: Building ECDSA pubkey recovery module: yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating libsecp256k1.pc
config.status: creating src/libsecp256k1-config.h
config.status: executing depfiles commands
config.status: executing libtool commands
=== configuring in src/univalue (/usr/src/bitcoinclassic/src/univalue)
configure: running /bin/sh ./configure --disable-option-checking '--prefix=/usr/local'  '--disable-shared' '--with-pic' '--with-bignum=no' '--enable-module-recovery' --cache-file=/dev/null --srcdir=.
checking whether make supports nested variables... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking how to print strings... printf
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking dependency style of gcc... gcc3
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... @
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for mt... no
checking if : is a manifest tool... no
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... no
checking whether to build static libraries... yes
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... gcc3
checking how to run the C++ preprocessor... g++ -E
checking for ld used by g++... /usr/bin/ld -m elf_x86_64
checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking for g++ option to produce PIC... -fPIC -DPIC
checking if g++ PIC flag -fPIC -DPIC works... yes
checking if g++ static flag -static works... no
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... (cached) GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating pc/libunivalue.pc
config.status: creating pc/libunivalue-uninstalled.pc
config.status: creating univalue-config.h
config.status: executing depfiles commands
config.status: executing libtool commands
Fixing libtool for -rpath problems.
$


- Make:


sudo make


(At this point taking a coffee is advised)


- Install:


sudo make install



- Run:

This time in our folder and with our user (no sudo).


cd

nohup bitcoind -externalip=`wget --quiet -O- http://169.254.169.254/latest/meta-data/public-ipv4` &

head -50 .bitcoin/debug.log


Notes:
-externalip is used to tell bitcoind which Public IP to advertise on the Bitcoin network. This is extremely important when running a full-node in EC2.
- To obtain the Public IP of our node we use a standard EC2 Meta-Data API call. It can be used as shown on this example. No adjustments needed. You can learn more about EC2 Meta-Data in here.



2016-04-13 13:31:36 Bitcoin version v0.12.0.0-84d82a8 (2016-04-06 12:21:13 +0100)
2016-04-13 13:31:36 InitParameterInteraction: parameter interaction: -externalip set -> setting -discover=0
2016-04-13 13:31:36 InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1
2016-04-13 13:31:36 Using OpenSSL version OpenSSL 1.0.1k-fips 8 Jan 2015
2016-04-13 13:31:36 Using BerkeleyDB version Berkeley DB 4.8.30: (February 21, 2011)
2016-04-13 13:31:36 Default data directory /home/ec2-user/.bitcoin
2016-04-13 13:31:36 Using data directory /home/ec2-user/.bitcoin
2016-04-13 13:31:36 Using config file /home/ec2-user/.bitcoin/bitcoin.conf
2016-04-13 13:31:36 Using at most 125 connections (1024 file descriptors available)
2016-04-13 13:31:36 Using 4 threads for script verification
2016-04-13 13:31:36 scheduler thread start
2016-04-13 13:31:36 HTTP: creating work queue of depth 16
2016-04-13 13:31:36 No rpcpassword set - using random cookie authentication
2016-04-13 13:31:36 Generated RPC authentication cookie /home/ec2-user/.bitcoin/.cookie
2016-04-13 13:31:36 HTTP: starting 4 worker threads
2016-04-13 13:31:36 Using wallet wallet.dat
2016-04-13 13:31:36 init message: Verifying wallet...
2016-04-13 13:31:36 CDBEnv::Open: LogDir=/home/ec2-user/.bitcoin/database ErrorFile=/home/ec2-user/.bitcoin/db.log
2016-04-13 13:31:36 Bound to [::]:8333
2016-04-13 13:31:36 Bound to 0.0.0.0:8333
2016-04-13 13:31:36 Cache configuration:
2016-04-13 13:31:36 * Using 2.0MiB for block index database
2016-04-13 13:31:36 * Using 32.5MiB for chain state database
2016-04-13 13:31:36 * Using 65.5MiB for in-memory UTXO set
2016-04-13 13:31:36 init message: Loading block index...
2016-04-13 13:31:36 Opening LevelDB in /home/ec2-user/.bitcoin/blocks/index
2016-04-13 13:31:36 Opened LevelDB successfully
2016-04-13 13:31:36 Using obfuscation key for /home/ec2-user/.bitcoin/blocks/index: 0000000000000000
2016-04-13 13:31:36 Opening LevelDB in /home/ec2-user/.bitcoin/chainstate
2016-04-13 13:31:36 Opened LevelDB successfully
2016-04-13 13:31:36 Using obfuscation key for /home/ec2-user/.bitcoin/chainstate: 0000000000000000
...


Reference: https://github.com/bitcoinclassic/bitcoinclassic/blob/develop/doc/build-unix.md