D4Science Infrastructure

If it is feasible the proxy should balance trying to redirect the same machine requesting machine to the same accounting-service instance. This will improve aggregation.
Production cluster was already created (see #8750), maybe you can check if such haproxy instance is configured as I'm suggesting.

Please note the client already support https.

please note that API to check the status has been changed.
From

GET/HEAD /accounting-service/gcube/service/status/getStatus?gcube-token=XXXXXX

to

GET /accounting-service/state?gcube-token=XXXXXX

Luca Frosini wrote:

If it is feasible the proxy should balance trying to redirect the same machine requesting machine to the same accounting-service instance

You are asking for 1 VM. Is it correct?

please note that API to check the status has been changed.

Also in production?

Production cluster was already created (see #8750), maybe you can check if such haproxy instance is configured as I'm suggesting.

Yes, we already discussed the configuration. Without sessions, what we can do is to use the leastconn balancer and make the sessions sticky - based on the IP source address - for an arbitrary amount of time (we chose 60 minutes)

Andrea Dell'Amico wrote:

Luca Frosini wrote:

If it is feasible the proxy should balance trying to redirect the same machine requesting machine to the same accounting-service instance

You are asking for 1 VM. Is it correct?

If you mean a VM to host the HAProxy the answer is yes.
Moreover, the HAProxy must balance between 2 VMs. One has to be created (accounting-service1-d), the other one (accounting-service-d.d4science.org) is already present and we also made some tests and benchmark on that see #10953.
When the proxy is available please notify me the host url that I have to change the Service Endpoint.

please note that API to check the status has been changed.

Also in production?

In production they will change when we will deploy the release 4.10. Actually the production HAProxy is not used.

Production cluster was already created (see #8750), maybe you can check if such haproxy instance is configured as I'm suggesting.

Yes, we already discussed the configuration. Without sessions, what we can do is to use the leastconn balancer and make the sessions sticky - based on the IP source address - for an arbitrary amount of time (we chose 60 minutes)

It sounds good.

No you need a preprod or dev token.
@lucio.lelii@isti.cnr.it can you provide it to @andrea.dellamico@isti.cnr.it

I have given the token to Andrea via skype

@andrea.dellamico@isti.cnr.it I'm confused because I'm able to successfully get the state by using
curl -k https://accounting-service1-d.dev.d4science.org:443/accounting-service/state?gcube-token=XXXXXXXXXXXXXXXx
I have to pass the -k argument to baypass the certificate verification.

$ curl -k https://accounting-service1-d.dev.d4science.org:443/accounting-service/state?gcube-token=7c66c94c-7f6e-49cd-9a34-909cd3832f3e-98187548
{"queryConnection":[false],"service":["running"],"insertConnection":[true],"context":["/gcube/devNext/NextNext"]}

Instead wget answer 503 Service Unavailable even I use --no-check-certificate obtaining the following error:

$wget --no-check-certificate https://accounting-service1-d.dev.d4science.org:443/accounting-service/state?gcube-token=XXXXXXXXXXXXXXX
--2018-01-22 09:16:21--  https://accounting-service1-d.dev.d4science.org/accounting-service/state?gcube-token=XXXXXXXXXXXXXXX
Resolving accounting-service1-d.dev.d4science.org (accounting-service1-d.dev.d4science.org)... 146.48.122.56
Connecting to accounting-service1-d.dev.d4science.org (accounting-service1-d.dev.d4science.org)|146.48.122.56|:443... connected.
WARNING: cannot verify accounting-service1-d.dev.d4science.org's certificate, issued by ‘CN=accounting-service1-d.dev.d4science.org self signed’:
  Self-signed certificate encountered.
    WARNING: certificate common name ‘accounting-service1-d.dev.d4science.org self signed’ doesn't match requested host name ‘accounting-service1-d.dev.d4science.org’.
HTTP request sent, awaiting response... 503 Service Unavailable
2018-01-22 09:16:21 ERROR 503: Service Unavailable.

In your cURL test you didn't check the http status code. Here it is:

curl -v -S -k 'https://accounting-service1-d.dev.d4science.org/accounting-service/state?gcube-token=<dev_token>'
*   Trying 146.48.122.56...
* TCP_NODELAY set
* Connected to accounting-service1-d.dev.d4science.org (146.48.122.56) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* NPN, negotiated HTTP1.1
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Unknown (67):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=accounting-service1-d.dev.d4science.org self signed
*  start date: Jan 17 17:38:28 2018 GMT
*  expire date: Jan 17 17:38:28 2019 GMT
*  issuer: CN=accounting-service1-d.dev.d4science.org self signed
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /accounting-service/state?gcube-token=a34b486c-874c-4539-97c9-be8d9e00fd7f-98187548 HTTP/1.1
> Host: accounting-service1-d.dev.d4science.org
> User-Agent: curl/7.57.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< Server: nginx
< Date: Mon, 22 Jan 2018 16:55:19 GMT
< Content-Type: application/json;charset=UTF-8
< Content-Length: 96
< Connection: keep-alive
<
* Connection #0 to host accounting-service1-d.dev.d4science.org left intact
{"queryConnection":[false],"service":["running"],"insertConnection":[true],"context":["/gcube"]}

As you can see, the 503 error is there with curl too. It does not depend by the self signed certificate, the test over plain http fails in the same way.

You are right. The response content is correct but the HTTP code is wrong. I'm going to check it.

I fixed the code and redeployed the artifact on snapshot repository. I'm going to compile it on etics to have the artifact on staging too. I'll advise you when the artifact will be available on staging

The artifact is available on staging too.

There is no reason to limit it the access to the service.
Moreover the client use the direct connection to the service as fallback if not able to discover the proxy address (please note if it is not able to discover the URL of the proxy but not if the proxy is not working). HA proxy is used to guarantee HA and load balancing during normal operation to be be able to scale horizontally.

So if I correctly understand, at the moment:

• accounting-service-d.dev.d4science.org (HAProxy) NOT YET AVAILABLE waiting for accounting-service2-d.dev.d4science.org
• accounting-service1-d.dev.d4science.org (Service) running and tested.
• accounting-service-d.d4science.org (Service) TO BE RENAMED to accounting-service2-d.dev.d4science.org #10995

Right?

Luca Frosini wrote:

There is no reason to limit it the access to the service.
Moreover the client use the direct connection to the service as fallback if not able to discover the proxy address (please note if it is not able to discover the URL of the proxy but not if the proxy is not working). HA proxy is used to guarantee HA and load balancing during normal operation to be be able to scale horizontally.

OK, I asked because I've found firewall rules to limit the accesses to our own networks.

Luca Frosini wrote:

So if I correctly understand, at the moment:

• accounting-service-d.dev.d4science.org (HAProxy) NOT YET AVAILABLE waiting for accounting-service2-d.dev.d4science.org

Not yet available, but it does not depend by the availability of accounting-service2-d.dev.d4science.org. I just waited for a working accounting-service1-d.dev.d4science.org, I'm now moving the hostname to the haproxy instance.

• accounting-service1-d.dev.d4science.org (Service) running and tested.
• accounting-service-d.d4science.org (Service) TO BE RENAMED to accounting-service2-d.dev.d4science.org #10995

Those are OK.

I'm in trouble with the HAProxy hostname. Is it:

• accounting-service-d.dev.d4science.org

or

• accounting-service-d.d4science.org (no dev subdomain)

The second one:

$ host accounting-service-d.d4science.org
accounting-service-d.d4science.org is an alias for dataminer-d-d4s.d4science.org.
dataminer-d-d4s.d4science.org has address 146.48.123.63

Ok. I confirm that also the proxy works:

$ host accounting-service-d.d4science.org
accounting-service-d.d4science.org is an alias for dataminer-d-d4s.d4science.org.
dataminer-d-d4s.d4science.org has address 146.48.123.63

$ ab -n 10000 -C 10 -T 'application/json' -p ~/workspace/Varie/accounting-data-simple.txt -H 'gcube-token: 7c66c94c-7f6e-49cd-9a34-909cd3832f3e-98187548'  https://accounting-service-d.d4science.org:443/accounting-service/record  
This is ApacheBench, Version 2.3 <$Revision: 1796539 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking accounting-service-d.d4science.org (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx
Server Hostname:        accounting-service-d.d4science.org
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-ECDSA-AES128-GCM-SHA256,256,128
TLS Server Name:        accounting-service-d.d4science.org

Document Path:          /accounting-service/record
Document Length:        0 bytes

Concurrency Level:      1
Time taken for tests:   61.482 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      5360000 bytes
Total body sent:        7940000
HTML transferred:       0 bytes
Requests per second:    162.65 [#/sec] (mean)
Time per request:       6.148 [ms] (mean)
Time per request:       6.148 [ms] (mean, across all concurrent requests)
Transfer rate:          85.14 [Kbytes/sec] received
                        126.12 kb/s sent
                        211.25 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        3    4  10.2      4    1026
Processing:     2    2   0.6      2      38
Waiting:        2    2   0.6      2      38
Total:          5    6  10.3      6    1029

Percentage of the requests served within a certain time (ms)
  50%      6
  66%      6
  75%      6
  80%      6
  90%      6
  95%      6
  98%      7
  99%      7
 100%   1029 (longest request)