D4Science Infrastructure

It does not work, really. I can add a calendar only when a single tomcat instance is running, and this explains why it never worked in production and why it works when you connect directly to one of the servers.
The behaviour can be reproduced easily: stop one of the backend, add a calendar (success). Remove the calendar and start the stopped backend. Add the calendar again (fail).

From the logs and the answer returned when I try to connect with curl, the error is

HTTP/1.1 401 Unauthorized

But the authentication header is there:

WWW-Authenticate: Digest realm="PortalRealm", nonce=<hash>"

and I did not find any report saying that haproxy do not let pass that header (it would be huge, breaking almost every site that uses authentication)

I just made another test, to add confusion. On both the pre and production load balancers I activated a frontend that listens on port 8443 and uses raw tcp to contact the backend servers. That mode works for me on the preproduction servers: I'm able to add a calendar even when both the instances are active. The same configuration does not work on the production servers. I even bypassed nginx there, so from the 8443 port the connection goes directly to tomcat

I also tried to start a http frontend on port 8443 and use it to backend a single tomcat instance. It does not work (we are talking about production)

The difference between liferay/tomcat preproduction cluster and the production one is that in case of preproduction the ha-proxy (AFAIK) resides in one node of the cluster (preprod1). In Production this is not true, the ha proxy is not in either of the cluster nodes. Perhaps, this could be the reason why you have 2 different behaviours?

I don't think so: from the haproxy point of view they are all remote targets, it never points to 'localhost' or something. And the called hostname is different from any VM hostname.

Andrea Dell'Amico wrote:

I've fixed a discrepancy in the web server and haproxy versions between the preprod and the production servers. Now they are aligned and I was able to successfully add the production calendar. It seems that the path must end with a /

A clarification: the haproxy versions were 1.6 on pre and 1.7 on the production server (and on the dev one). I only moved preproduction to 1.7, and this is the reason why I excluded that those modification caused the OCSP problem on the production load balancers. Now all the load balancers run haproxy 1.7 but infra dev, where 1.8 is running (pointlessly, because I discovered that the openssl library installed on Ubuntu 14.04 does not support the alpn API, so we cannot activate http/2 without upgrading to a newer Ubuntu release).

The big difference was instead the nginx version, where on preprod1 a more modern one - from a separate repository - was installed. Now the same repository is been used on the prod infra servers too.