Actions
VM Creation #10930
closed
Task #10929: Accounting Service Cluster dev/preprod
Create accounting-service1-d
Status:
Closed
Priority:
Urgent
Assignee:
_InfraScience Systems Engineer
Target version:
Start date:
Jan 15, 2018
Due date:
% Done:
100%
Infrastructure:
Development, Pre-Production
RAM:
4
CPU:
2
DiskSpace:
25
Smartgears:
Yes
Smartgears Version:
latest
Application Instructions:
Smartgears scopes:
/gcube
/gcube/preprod
/gcube/preprod/preVRE
/gcube/preprod/preECO
/gcube/preprod/Dorne
/gcube/devNext
/gcube/devNext/NextNext
/gcube/devsec
/gcube/devsec/devVRE
Applications list and version:
accounting-service (latest)
Applications context path:
External storage data disk:
Smartgears auth connector:
data transfer enabled:
No
PostgreSQL database:
No
Number of VMs:
1
Description
The machine will host accounting-service as 'accounting-service-d' (playbook is already available)
Related issues
Updated by Luca Frosini over 7 years ago
- Blocks Task #10931: Please create HAProxy instance in front of accounting-service-d and accounting-service1-d added
Updated by Roberto Cirillo over 7 years ago
- Priority changed from Normal to Urgent
Updated by Andrea Dell'Amico over 7 years ago
- Status changed from New to In Progress
Updated by Andrea Dell'Amico over 7 years ago
- RAM changed from 2 to 4
- DiskSpace changed from 10 to 25
- Infrastructure Development added
- Infrastructure deleted (
Production)
Updated by Andrea Dell'Amico over 7 years ago
On accounting-service-d.d4science.org the scopes listed into container.xml are:
/gcube /gcube/preprod /gcube/preprod/preVRE /gcube/preprod/preECO /gcube/preprod/Dorne /gcube/devNext /gcube/devNext/NextNext
What are the correct ones?
Updated by Luca Frosini over 7 years ago
Yes should also add
/gcube/devsec
/gcube/devsec/devVRE
Updated by Andrea Dell'Amico over 7 years ago
- Smartgears scopes updated (diff)
hostname and IP address are going to be: accounting-service1-d.dev.d4science.org 146.48.122.56
Updated by Luca Frosini over 7 years ago
If we use dev.d4science.org domain we can avoid to use -d but I don't any problem if you prefer with -d suffix in hostname
Updated by Andrea Dell'Amico over 7 years ago
Luca Frosini wrote:
If we use dev.d4science.org domain we can avoid to use -d but I don't any problem if you prefer with -d suffix in hostname
The -d suffix is needed because otherwise when connected via ssh it would be impossible to distinguish between a dev and a prod server. The BP for the server naming are seldom used, but they exist: https://support.d4science.org/projects/aginfraplut/wiki/Best_practices_for_the_servers_naming
Updated by Luca Frosini over 7 years ago
Andrea Dell'Amico wrote:
Luca Frosini wrote:
If we use dev.d4science.org domain we can avoid to use -d but I don't any problem if you prefer with -d suffix in hostname
The
-dsuffix is needed because otherwise when connected via ssh it would be impossible to distinguish between a dev and a prod server. The BP for the server naming are seldom used, but they exist: https://support.d4science.org/projects/aginfraplut/wiki/Best_practices_for_the_servers_naming
You are right, I forget it.
Updated by Andrea Dell'Amico over 7 years ago
The Oracle JDK is failing to download from the Oracle site. I'll try again later.
Updated by Andrea Dell'Amico over 7 years ago
The oracle java8 installer is still broken, I guess the reason is that Oracle just released jdk-8u161. I asked the package author for a fix.
Otherwise we should try OpenJDK.
Updated by Andrea Dell'Amico over 7 years ago
The service starts, but then I see this exception:
2018-01-18 16:39:47,088 [catalina-exec-6] ERROR org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/accounting-service].[org.gcube.accounting.service.AccountingResource]- Servlet.service() for servlet [org.gcube.accounting.service.AccountingResource] in context with path [/accounting-service] threw exception
java.lang.IllegalStateException: Cannot call sendError() after the response has been committed
at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:450)
at org.gcube.smartgears.managers.RequestManager.handleError(RequestManager.java:227)
at org.gcube.smartgears.managers.RequestManager.doFilter(RequestManager.java:98)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Updated by Andrea Dell'Amico over 7 years ago
- % Done changed from 0 to 70
And, in catalina.out:
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: org/gcube/smartgears/ApplicationManager
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:393)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.gcube.smartgears.managers.RequestManager.doFilter(RequestManager.java:95)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.glassfish.jersey.server.ContainerException: java.lang.NoClassDefFoundError: org/gcube/smartgears/ApplicationManager
at org.glassfish.jersey.servlet.internal.ResponseWriter.rethrow(ResponseWriter.java:256)
at org.glassfish.jersey.servlet.internal.ResponseWriter.failure(ResponseWriter.java:238)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:439)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373)
... 23 more
Updated by Luca Frosini over 7 years ago
- % Done changed from 70 to 0
I think is better to restore the use of Oracle JDK
Updated by Luca Frosini over 7 years ago
I found that the problem is in the service with has common-smartgears-app to provided.
The strange thing is that it worked with Oracle JDK.
I'm going to fix dependencies on accounting-service
Updated by Luca Frosini over 7 years ago
I lauched the compilation of the accounting-service on ETICS. I'll advise you as soon as it will succeed to deploy the service again.
Updated by Luca Frosini over 7 years ago
Launching the compilation from GUI fails due to use of java 1.7 instead of 1.8.
@mariaantonietta.digirolamo@eng.it is investigating such an issue.
Updated by Maria Di Girolamo over 7 years ago
- Status changed from In Progress to Closed
- % Done changed from 0 to 100
Dear @luca.frosini@isti.cnr.it
due the updating of the commpon configuration scanner , it was needed the re-build of accounting-analytics.
Infact accounting-service try the old configuration of accounting-analytics (and it not found).
The build of accounting-analytics and accounting-service have been executed with success.
The report of the last remote build of accounting-serivce is available here: http://etics.esl.eng.it/etics-repository-ws/reports/id/72f3958f-604b-4213-b8ee-3e15a064355e/centos6.9_x86_64_gcc447/-/reports/build-status.xml.
Updated by Luca Frosini over 7 years ago
- Status changed from Closed to In Progress
- % Done changed from 100 to 0
Thank you @mariaantonietta.digirolamo@eng.it
Updated by Luca Frosini over 7 years ago
@andrea.dellamico@isti.cnr.it can you please retry?
Updated by Andrea Dell'Amico over 7 years ago
It seems that the new build is not present on nexus yet.
Updated by Andrea Dell'Amico over 7 years ago
I tried to force a download, but I still got the same version as yesterday.
Updated by Andrea Dell'Amico over 7 years ago
- Status changed from In Progress to Feedback
- % Done changed from 0 to 100
Now it worked. The exceptions are gone.
Updated by Andrea Dell'Amico over 7 years ago
- Blocked by Task #10934: Please fix ansible playbook to copy document-store-lib-couchbase in tomcat/lib and remove document-store-lib-accounting-service in accountign-service installation added
Updated by Andrea Dell'Amico over 7 years ago
I just run the playbook again with the fixed tasks.
Updated by Luca Frosini over 7 years ago
- Status changed from Feedback to Closed
the service seems working.
Here the stress tests results I made:
$ ab -n 1000 -T 'application/json' -p ~/workspace/Varie/accounting-data-simple.txt -H 'gcube-token: XXXXXXXXXXXXXXXXXxx' https://accounting-service1-d.dev.d4science.org:443/accounting-service/record
This is ApacheBench, Version 2.3 <$Revision: 1796539 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking accounting-service1-d.dev.d4science.org (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software: nginx
Server Hostname: accounting-service1-d.dev.d4science.org
Server Port: 443
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128
TLS Server Name: accounting-service1-d.dev.d4science.org
Document Path: /accounting-service/record
Document Length: 0 bytes
Concurrency Level: 1
Time taken for tests: 5.260 seconds
Complete requests: 1000
Failed requests: 0
Total transferred: 536000 bytes
Total body sent: 787000
HTML transferred: 0 bytes
Requests per second: 190.11 [#/sec] (mean)
Time per request: 5.260 [ms] (mean)
Time per request: 5.260 [ms] (mean, across all concurrent requests)
Transfer rate: 99.51 [Kbytes/sec] received
146.11 kb/s sent
245.62 kb/s total
Connection Times (ms)
min mean[+/-sd] median max
Connect: 3 3 0.3 3 7
Processing: 1 2 2.5 2 68
Waiting: 1 2 2.5 2 68
Total: 4 5 2.5 5 72
Percentage of the requests served within a certain time (ms)
50% 5
66% 5
75% 5
80% 5
90% 6
95% 7
98% 8
99% 9
100% 72 (longest request)
and
$ ab -n 10000 -C 10 -T 'application/json' -p ~/workspace/Varie/accounting-data-simple.txt -H 'gcube-token: xxxxxxxxxxxxxxxxxxxxxxx' https://accounting-service-d.d4science.org:443/accounting-service/record
This is ApacheBench, Version 2.3 <$Revision: 1796539 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking accounting-service-d.d4science.org (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
Server Software: nginx
Server Hostname: accounting-service-d.d4science.org
Server Port: 443
SSL/TLS Protocol: TLSv1.2,ECDHE-ECDSA-AES128-GCM-SHA256,256,128
TLS Server Name: accounting-service-d.d4science.org
Document Path: /accounting-service/record
Document Length: 0 bytes
Concurrency Level: 1
Time taken for tests: 50.595 seconds
Complete requests: 10000
Failed requests: 0
Total transferred: 5360000 bytes
Total body sent: 7940000
HTML transferred: 0 bytes
Requests per second: 197.65 [#/sec] (mean)
Time per request: 5.059 [ms] (mean)
Time per request: 5.059 [ms] (mean, across all concurrent requests)
Transfer rate: 103.46 [Kbytes/sec] received
153.25 kb/s sent
256.71 kb/s total
Connection Times (ms)
min mean[+/-sd] median max
Connect: 2 4 14.7 3 1034
Processing: 1 1 4.4 1 419
Waiting: 1 1 4.4 1 419
Total: 3 5 15.5 4 1036
Percentage of the requests served within a certain time (ms)
50% 4
66% 5
75% 5
80% 5
90% 5
95% 6
98% 7
99% 10
100% 1036 (longest request)
Updated by Andrea Dell'Amico over 7 years ago
- Status changed from Closed to In Progress
- % Done changed from 100 to 80
I'm reopening this one because the discussion was ongoing on the wrong ticket. There's a new exception now:
2018-01-23 14:24:57,326 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO org.apache.catalina.startup.HostConfig- Undeploying context [/accounting-service]
2018-01-23 14:24:57,330 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] ERROR org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/accounting-service]- Servlet org.gcube.accounting.service.AccountingResource threw unload() exception
javax.servlet.ServletException: Servlet.destroy() for servlet org.gcube.accounting.service.AccountingResource threw exception
at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1502)
at org.apache.catalina.core.StandardWrapper.stopInternal(StandardWrapper.java:1842)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5684)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
at org.apache.catalina.core.ContainerBase.removeChild(ContainerBase.java:1028)
at org.apache.catalina.startup.HostConfig.undeploy(HostConfig.java:1478)
at org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:1406)
at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1626)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:328)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1374)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1530)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1540)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1519)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/jvnet/hk2/internal/ServiceLocatorImpl$7
at org.jvnet.hk2.internal.ServiceLocatorImpl.shutdown(ServiceLocatorImpl.java:811)
at org.glassfish.jersey.internal.inject.Injections.shutdownLocator(Injections.java:285)
at org.glassfish.jersey.server.internal.ConfigHelper$ServiceLocatorShutdownListener.onShutdown(ConfigHelper.java:149)
at org.glassfish.jersey.server.internal.ConfigHelper$2.onShutdown(ConfigHelper.java:115)
at org.glassfish.jersey.servlet.ServletContainer.destroy(ServletContainer.java:352)
at org.apache.catalina.core.StandardWrapper.unload(StandardWrapper.java:1481)
... 16 more
Caused by: java.lang.ClassNotFoundException: org.jvnet.hk2.internal.ServiceLocatorImpl$7
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1719)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
... 22 more
Updated by Andrea Dell'Amico over 7 years ago
- Status changed from In Progress to Closed
- % Done changed from 80 to 100
Sorry, I was wrong. It was from an older run. The service has been deployed correctly.
Updated by Luca Frosini over 7 years ago
I confirm that the service is running correctly:
$ ab -n 10000 -C 10 -T 'application/json' -p ~/workspace/Varie/accounting-data-simple.txt -H 'gcube-token: 7c66c94c-7f6e-49cd-9a34-909cd3832f3e-98187548' https://accounting-service1-d.dev.d4science.org:443/accounting-service/record
This is ApacheBench, Version 2.3 <$Revision: 1796539 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking accounting-service1-d.dev.d4science.org (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
Server Software: nginx
Server Hostname: accounting-service1-d.dev.d4science.org
Server Port: 443
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128
TLS Server Name: accounting-service1-d.dev.d4science.org
Document Path: /accounting-service/record
Document Length: 0 bytes
Concurrency Level: 1
Time taken for tests: 46.901 seconds
Complete requests: 10000
Failed requests: 0
Total transferred: 5360000 bytes
Total body sent: 7990000
HTML transferred: 0 bytes
Requests per second: 213.22 [#/sec] (mean)
Time per request: 4.690 [ms] (mean)
Time per request: 4.690 [ms] (mean, across all concurrent requests)
Transfer rate: 111.60 [Kbytes/sec] received
166.37 kb/s sent
277.97 kb/s total
Connection Times (ms)
min mean[+/-sd] median max
Connect: 3 4 0.4 4 17
Processing: 1 1 1.0 1 75
Waiting: 1 1 1.0 1 75
Total: 4 5 1.1 5 79
Percentage of the requests served within a certain time (ms)
50% 5
66% 5
75% 5
80% 5
90% 5
95% 5
98% 6
99% 7
100% 79 (longest request)
Actions
