D4Science Infrastructure

The dataminer[1:2]-d-d4s.d4science.org nodes are still directly reachable on port 80, as requested.

Calls to dataminer-d-d4s.d4science.org are always sent to dataminer1. @andrea.dellamico@isti.cnr.it could you please give it a look?

Can you try from a different IP? Even if it's a round robin configuration haproxy tries to deliver the requests to the same backend when the source does not change.

And a request: what URL should the load balance check to be sure that the service is working? I'm now testing /wps, but it's a 302.

Well, you need to chose one for the load balancer :). A light one possibly, they're executed once per second. We can use some of the others to produce a detailed nagios check, maybe.

According to the WPS specifications, timeout should be decided by the user's client. Processing can last also days and theoretically could be executed also in "synchronous" mode by the client. Is it possible to either set the timeout to infinite or to 10/20 days?

There is no 'infinite timeout' in tcp or http.
We only need to ensure that the proxy keep alive timeout is longer than the client (or the standard tcp) one.

<>
Not according to Apache: https://hc.apache.org/httpcomponents-client-4.2.x/tutorial/html/connmgmt.html

Anyway, if infinite is not contemplated by haproxy, is it possible to set it to 20 days at least?

Gianpaolo Coro wrote:

<>
Not according to Apache: https://hc.apache.org/httpcomponents-client-4.2.x/tutorial/html/connmgmt.html

Anyway, if infinite is not contemplated by haproxy, is it possible to set it to 20 days at least?

That only says that the client will wait indefinitely. Servers have timeouts too. At http level, and tcp level. What we are interested in for this case is called http persistent connections and the way for them to work reliably is to have keepalive doing the job correctly.
Yesterday I raised the haproxy keepalive timeouts so that they should be much longer of the keepalive interval, and your tests that weren't reliable now are always completing.
We now need tests that last longer, and only if they fail because of timeouts we can try and raise them to an unreasonble high level.

Since we are talking about a computational service, the longest run I have lasts 7 days in the development environment. I don't know the computational time of all the possible algorithms we will integrate in the future.
Is it possible to quantify the current timeout or shall we wait 7 days to understand if another tuning is needed?

Gianpaolo Coro wrote:

Since we are talking about a computational service, the longest run I have lasts 7 days in the development environment. I don't know the computational time of all the possible algorithms we will integrate in the future.
Is it possible to quantify the current timeout or shall we wait 7 days to understand if another tuning is needed?

I only have to check the client behaviour. A test lasting some then minutes is sufficient. If it fails I have a last haproxy configuration options to try, before falling back to increase the timeouts.