Set up Apache Httpd 2.2 to cache static contents from Weblogic / Tomcat / Jboss backend

Configuration:

Add below to conf.d/backend.conf

<IfModule mod_cache.c>
<IfModule mod_disk_cache.c>
CacheRoot “/var/cache/mod_proxy/”
CacheEnable disk “/”
CacheDirLevels 2
CacheDirLength 1

CacheLock on
CacheLockPath “/tmp/mod_cache-lock”
CacheLockMaxAge 5
</IfModule>

# These only valid for Apache httpd 2.4
#CacheQuickHandler off
#CacheHeader on
#CustomLog “logs/cached-requests.log” common env=cache-hit
#CustomLog “logs/uncached-requests.log” common env=cache-miss
#CustomLog “logs/revalidated-requests.log” common env=cache-revalidate
#CustomLog “logs/invalidated-requests.log” common env=cache-invalidate

</IfModule>

 

How to check:

1) Check access logs for Apache as well as the backend

2) Confirm that there are files in CacheRoot

#ls -R /var/cache/mod_proxy

 

References:

https://httpd.apache.org/docs/2.2/caching.html

https://httpd.apache.org/docs/current/mod/mod_cache.html

 

Advertisements

Apache Httpd 2.2.x proxying request to Tomcat 8

Add below entries to /etc/httpd/conf.d/tomcat.conf:

SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1
SetEnv proxy-initial-not-pooled 1
RequestHeader unset Expect early

ProxyPass “/abc” “balancer://mycluster/”
<Proxy balancer://mycluster>
ProxySet failonstatus=502,503 maxattempts=100
BalancerMember ajp://host1:9999 timeout=10 retry=10 ping=10 disablereuse=on keepalive=on
BalancerMember ajp://host2:9999 timeout=10 retry=10 ping=10 disablereuse=on keepalive=on
BalancerMember ajp://host3:9999 timeout=10 retry=10 ping=10 disablereuse=on keepalive=on
</Proxy>

 

Some explanation:

SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1
– Force the request to use HTTP/1.0 with no keepalive.

SetEnv proxy-initial-not-pooled 1
– If this variable is set, no pooled connection will be reused if the client request is the initial request on the frontend connection. This avoids the “proxy: error reading status line from remote server” error message caused by the race condition that the backend server closed the pooled connection after the connection check by the proxy and before data sent by the proxy reached the backend. It has to be kept in mind that setting this variable downgrades performance, especially with HTTP/1.0 clients.

RequestHeader unset Expect early
– The issue is that some clients set the Expect header and only send the request headers before a PUT or POST of data. This allows the server to respond with errors/redirects/security violations prior to the client sending the request body (PUT or POST data). Apparently some clients does not wait until it gets a response and just pushes out the body of the request, which results in the 417 error.
http://stackoverflow.com/questions/3889574/apache-and-mod-proxy-not-handling-http-100-continue-from-client-http-417

 

ProxySet failonstatus=502,503 maxattempts=10
failonstatus – A single or comma-separated list of HTTP status codes. Will force the worker into error state when the backend returns any status code in the list
maxattempts – Maximum number of failover attempts before giving up.

BalancerMember ajp://host1:9999 timeout=10 retry=10 ping=10 disablereuse=on keepalive=on
timeout – maximum time to wait for a free worker. The default is to not wait.
retry – Connection pool worker retry timeout in seconds. If the connection pool worker to the backend server is in the error state, Apache httpd will not forward any requests to that server until the timeout expires. This enables to shut down the backend server for maintenance and bring it back online later. A value of 0 means always retry workers in an error state with no timeout.
ping – Delay in seconds to wait for the reply of “ping test” to the backend
disablereuse – force mod_proxy to immediately close a connection to the backend after being used
keepalive – This parameter should be used when you have a firewall between your Apache httpd and the backend server, which tends to drop inactive connections. This flag will tell the Operating System to send KEEP_ALIVE messages on inactive connections and thus prevent the firewall from dropping the connection

References:

https://bz.apache.org/bugzilla/show_bug.cgi?format=multiple&id=57520

https://httpd.apache.org/docs/current/mod/mod_proxy.html

Simple SSL Perfomance Comparison

Command used to test:

# openssl s_time -connect localhost:$PORT -new

Tomcat APR/Native:
2011 connections in 10.67s; 188.47 connections/user sec, bytes read 0
2011 connections in 31 real seconds, 0 bytes read per connection

JBoss Java connector:
460 connections in 11.01s; 41.78 connections/user sec, bytes read 0
460 connections in 31 real seconds, 0 bytes read per connection

Apache:
2753 connections in 14.43s; 190.78 connections/user sec, bytes read 0
2753 connections in 31 real seconds, 0 bytes read per connection

Weblogic:
797 connections in 3.81s; 209.19 connections/user sec, bytes read 0
797 connections in 31 real seconds, 0 bytes read per connection

Conclusion:

  1. Using APR/Native is better for performance.
  2. Weblogic is quite OK, performance better than JBoss, considering that it is using JDK 6 (will try with JDK 7 & 8 later).

Updates (26 Oct 2016)

Weblogic + JDK 7
1103 connections in 2.86s; 385.66 connections/user sec, bytes read 0
1103 connections in 31 real seconds, 0 bytes read per connection

Weblogic + JDK 8
889 connections in 4.27s; 208.20 connections/user sec, bytes read 0
889 connections in 31 real seconds, 0 bytes read per connection

Testing Apache Httpd SSL with testssl.sh

Default SSL configuration of apache out-of-the-box is not that secure.

From testssl.sh report, there were “NOT ok” items as below:

–> Testing protocols (via sockets except TLS 1.2 and SPDY/NPN)

SSLv2 not offered (OK)
SSLv3 offered (NOT ok)
TLS 1 offered
TLS 1.1 offered
TLS 1.2 offered (OK)
SPDY/NPN not offered

–> Testing ~standard cipher lists

Null Ciphers not offered (OK)
Anonymous NULL Ciphers not offered (OK)
Anonymous DH Ciphers not offered (OK)
40 Bit encryption not offered (OK)
56 Bit encryption Local problem: No 56 Bit encryption configured in /usr/bin/openssl
Export Ciphers (general) not offered (OK)
Low (<=64 Bit) not offered (OK)
DES Ciphers not offered (OK)
Medium grade encryption offered (NOT ok)
Triple DES Ciphers offered (NOT ok)
High grade encryption offered (OK)

–> Testing server preferences

Has server cipher order? nope (NOT ok)
Negotiated protocol TLSv1.2
Negotiated cipher ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (limited sense as client will pick)
Negotiated cipher per proto (limited sense as client will pick)
ECDHE-RSA-AES256-SHA: SSLv3, TLSv1, TLSv1.1
ECDHE-RSA-AES256-GCM-SHA384: TLSv1.2
No further cipher order check has been done as order is determined by the client

–> Testing vulnerabilities

Heartbleed (CVE-2014-0160) not vulnerable (OK) (timed out)
CCS (CVE-2014-0224) not vulnerable (OK)
Secure Renegotiation (CVE-2009-3555) not vulnerable (OK)
Secure Client-Initiated Renegotiation not vulnerable (OK)
CRIME, TLS (CVE-2012-4929) not vulnerable (OK)
BREACH (CVE-2013-3587) no HTTP compression (OK) (only supplied “/” tested)
POODLE, SSL (CVE-2014-3566) VULNERABLE (NOT ok), uses SSLv3+CBC (check TLS_FALLBACK_SCSV mitigation below)
TLS_FALLBACK_SCSV (RFC 7507), experim. Downgrade attack prevention supported (OK)
FREAK (CVE-2015-0204) not vulnerable (OK) (tested with 4/9 ciphers)
LOGJAM (CVE-2015-4000), experimental not vulnerable (OK) (tested w/ 2/4 ciphers only!), common primes not checked. See below for any DH ciphers + bit size
BEAST (CVE-2011-3389) SSL3: ECDHE-RSA-DES-CBC3-SHA EDH-RSA-DES-CBC3-SHA
DES-CBC3-SHA
TLS1: ECDHE-RSA-DES-CBC3-SHA EDH-RSA-DES-CBC3-SHA
DES-CBC3-SHA
— but also supports higher protocols (possible mitigation): TLSv1.1 TLSv1.2
RC4 (CVE-2013-2566, CVE-2015-2808) VULNERABLE (NOT ok): ECDHE-RSA-RC4-SHA RC4-SHA RC4-MD5 RC4-MD5
And there’s way too many insecure ciphers:

–> Testing all locally available 121 ciphers against the server, ordered by encryption strength

Hexcode Cipher Suite Name (OpenSSL) KeyExch. Encryption Bits Cipher Suite Name (RFC)
———————————————————————————————————————–
xc030 ECDHE-RSA-AES256-GCM-SHA384 ECDH 256 AESGCM 256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
xc028 ECDHE-RSA-AES256-SHA384 ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
xc014 ECDHE-RSA-AES256-SHA ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
x9f DHE-RSA-AES256-GCM-SHA384 DH 1024 AESGCM 256 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
x6b DHE-RSA-AES256-SHA256 DH 1024 AES 256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
x39 DHE-RSA-AES256-SHA DH 1024 AES 256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA
x88 DHE-RSA-CAMELLIA256-SHA DH 1024 Camellia 256 TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA
x9d AES256-GCM-SHA384 RSA AESGCM 256 TLS_RSA_WITH_AES_256_GCM_SHA384
x3d AES256-SHA256 RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA256
x35 AES256-SHA RSA AES 256 TLS_RSA_WITH_AES_256_CBC_SHA
x84 CAMELLIA256-SHA RSA Camellia 256 TLS_RSA_WITH_CAMELLIA_256_CBC_SHA
xc02f ECDHE-RSA-AES128-GCM-SHA256 ECDH 256 AESGCM 128 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
xc027 ECDHE-RSA-AES128-SHA256 ECDH 256 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
xc013 ECDHE-RSA-AES128-SHA ECDH 256 AES 128 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
x9e DHE-RSA-AES128-GCM-SHA256 DH 1024 AESGCM 128 TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
x67 DHE-RSA-AES128-SHA256 DH 1024 AES 128 TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
x33 DHE-RSA-AES128-SHA DH 1024 AES 128 TLS_DHE_RSA_WITH_AES_128_CBC_SHA
xc012 ECDHE-RSA-DES-CBC3-SHA ECDH 256 3DES 168 TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
x45 DHE-RSA-CAMELLIA128-SHA DH 1024 Camellia 128 TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA
x16 EDH-RSA-DES-CBC3-SHA DH 1024 3DES 168 TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
x9c AES128-GCM-SHA256 RSA AESGCM 128 TLS_RSA_WITH_AES_128_GCM_SHA256
x3c AES128-SHA256 RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA256
x2f AES128-SHA RSA AES 128 TLS_RSA_WITH_AES_128_CBC_SHA
x41 CAMELLIA128-SHA RSA Camellia 128 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA
x0a DES-CBC3-SHA RSA 3DES 168 TLS_RSA_WITH_3DES_EDE_CBC_SHA
xc011 ECDHE-RSA-RC4-SHA ECDH 256 RC4 128 TLS_ECDHE_RSA_WITH_RC4_128_SHA
x05 RC4-SHA RSA RC4 128 TLS_RSA_WITH_RC4_128_SHA
x04 RC4-MD5 RSA RC4 128 TLS_RSA_WITH_RC4_128_MD5
x010080 RC4-MD5 RSA RC4 128 SSL_CK_RC4_128_WITH_MD5

 

Remediation:

1) Disable SSLv3
– edit ssl.conf
SSLProtocol all -SSLv2 -SSLv3

You can also enable only TLS1.2 if desired
SSLProtocol +TLSv1.2

 

[root@localhost testssl]# ./testssl.sh –quiet -p localhost:443

–> Testing protocols (via sockets except TLS 1.2 and SPDY/NPN)

SSLv2 not offered (OK)
SSLv3 not offered (OK)
TLS 1 not offered
TLS 1.1 not offered
TLS 1.2 offered (OK)
SPDY/NPN not offered

 

2) Remove insecure ciphers
– edit ssl.conf
SSLCipherSuite EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4

 

# ./testssl.sh –quiet -e -f localhost:443

–> Testing ~standard cipher lists

Null Ciphers not offered (OK)
Anonymous NULL Ciphers not offered (OK)
Anonymous DH Ciphers not offered (OK)
40 Bit encryption not offered (OK)
56 Bit encryption Local problem: No 56 Bit encryption configured in /usr/bin/openssl
Export Ciphers (general) not offered (OK)
Low (<=64 Bit) not offered (OK)
DES Ciphers not offered (OK)
Medium grade encryption not offered (OK)
Triple DES Ciphers not offered (OK)
High grade encryption offered (OK)

–> Testing all locally available 121 ciphers against the server, ordered by encryption strength

Hexcode Cipher Suite Name (OpenSSL) KeyExch. Encryption Bits Cipher Suite Name (RFC)
———————————————————————————————————————–
xc030 ECDHE-RSA-AES256-GCM-SHA384 ECDH 256 AESGCM 256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
xc028 ECDHE-RSA-AES256-SHA384 ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
xc014 ECDHE-RSA-AES256-SHA ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
x9f DHE-RSA-AES256-GCM-SHA384 DH 1024 AESGCM 256 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
x6b DHE-RSA-AES256-SHA256 DH 1024 AES 256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
x39 DHE-RSA-AES256-SHA DH 1024 AES 256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA
xc02f ECDHE-RSA-AES128-GCM-SHA256 ECDH 256 AESGCM 128 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
x9e DHE-RSA-AES128-GCM-SHA256 DH 1024 AESGCM 128 TLS_DHE_RSA_WITH_AES_128_GCM_SHA256

 

3) Ciphers order
– edit ssl.conf
SSLHonorCipherOrder on

 

Final report:

[root@localhost testssl]# ./testssl.sh –quiet localhost:443

No engine or GOST support via engine with your /usr/bin/openssl

Testing now (2016-10-20 09:49) —> 127.0.0.1:443 (localhost) <—

A record via /etc/hosts
rDNS (127.0.0.1): localhost.
Service detected: HTTP
–> Testing protocols (via sockets except TLS 1.2 and SPDY/NPN)

SSLv2 not offered (OK)
SSLv3 not offered (OK)
TLS 1 not offered
TLS 1.1 not offered
TLS 1.2 offered (OK)
SPDY/NPN not offered

–> Testing ~standard cipher lists

Null Ciphers not offered (OK)
Anonymous NULL Ciphers not offered (OK)
Anonymous DH Ciphers not offered (OK)
40 Bit encryption not offered (OK)
56 Bit encryption Local problem: No 56 Bit encryption configured in /usr/bin/openssl
Export Ciphers (general) not offered (OK)
Low (<=64 Bit) not offered (OK)
DES Ciphers not offered (OK)
Medium grade encryption not offered (OK)
Triple DES Ciphers not offered (OK)
High grade encryption offered (OK)

–> Testing (perfect) forward secrecy, (P)FS — omitting 3DES, RC4 and Null Encryption here

PFS is offered (OK) ECDHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES256-SHA256 DHE-RSA-AES256-SHA ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES256-SHA ECDHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES128-GCM-SHA256

–> Testing server preferences

Has server cipher order? yes (OK)
Negotiated protocol TLSv1.2
Negotiated cipher ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH
Cipher order
TLSv1.2: ECDHE-RSA-AES256-GCM-SHA384 ECDHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES256-SHA DHE-RSA-AES256-SHA256 DHE-RSA-AES256-SHA

–> Testing server defaults (Server Hello)

TLS server extensions renegotiation info, EC point formats, session ticket, heartbeat
Session Tickets RFC 5077 300 seconds
Server key size 1024 bit
Signature Algorithm SHA1 with RSA
Fingerprint / Serial SHA1 3D61611F09363757125AF4120367098362264B88 / 5300
SHA256 64BA84D86682F6A8DB9326822C5165C246C00E2CFF4D6259B8A7ED479E979048
Common Name (CN) localhost (CN in response to request w/o SNI: localhost)
subjectAltName (SAN) —
Issuer localhost (SomeOrganization from –)
EV cert (experimental) no
Certificate Expiration expired! (2014-08-20 10:28 –> 2015-08-20 10:28 +0800)
# of certificates provided 1
Chain of trust (experim.) Your /usr/bin/openssl is too old, needed is version >=1.0.2
Certificate Revocation List —
OCSP URI —
OCSP stapling not offered
TLS clock skew -1 sec from localtime
–> Testing HTTP header response @ “/”

HTTP Status Code 200 OK
HTTP clock skew 0 sec from localtime
Strict Transport Security —
Public Key Pinning —
Server banner Apache/2.2.15 (Red Hat)
Application banner —
Cookie(s) (none issued at “/”)
Security headers —
Reverse Proxy banner —
–> Testing vulnerabilities

Heartbleed (CVE-2014-0160) not vulnerable (OK) (timed out)
CCS (CVE-2014-0224) not vulnerable (OK)
Secure Renegotiation (CVE-2009-3555) not vulnerable (OK)
Secure Client-Initiated Renegotiation not vulnerable (OK)
CRIME, TLS (CVE-2012-4929) not vulnerable (OK)
BREACH (CVE-2013-3587) no HTTP compression (OK) (only supplied “/” tested)
POODLE, SSL (CVE-2014-3566) not vulnerable (OK)
TLS_FALLBACK_SCSV (RFC 7507), experim. Check failed: seems like TLS 1.2 is the only protocol
FREAK (CVE-2015-0204) not vulnerable (OK) (tested with 4/9 ciphers)
LOGJAM (CVE-2015-4000), experimental not vulnerable (OK) (tested w/ 2/4 ciphers only!), common primes not checked. See below for any DH ciphers + bit size
BEAST (CVE-2011-3389) no SSL3 or TLS1
RC4 (CVE-2013-2566, CVE-2015-2808) no RC4 ciphers detected (OK)
–> Testing all locally available 121 ciphers against the server, ordered by encryption strength

Hexcode Cipher Suite Name (OpenSSL) KeyExch. Encryption Bits Cipher Suite Name (RFC)
———————————————————————————————————————–
xc030 ECDHE-RSA-AES256-GCM-SHA384 ECDH 256 AESGCM 256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
xc028 ECDHE-RSA-AES256-SHA384 ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
xc014 ECDHE-RSA-AES256-SHA ECDH 256 AES 256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
x9f DHE-RSA-AES256-GCM-SHA384 DH 1024 AESGCM 256 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
x6b DHE-RSA-AES256-SHA256 DH 1024 AES 256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
x39 DHE-RSA-AES256-SHA DH 1024 AES 256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA
xc02f ECDHE-RSA-AES128-GCM-SHA256 ECDH 256 AESGCM 128 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
x9e DHE-RSA-AES128-GCM-SHA256 DH 1024 AESGCM 128 TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
Done now (2016-10-20 09:50) —> 127.0.0.1:443 (localhost) <—

 

References:

https://raymii.org/s/tutorials/Strong_SSL_Security_On_Apache2.html
http://security.stackexchange.com/questions/51680/optimal-web-server-ssl-cipher-suite-configuration