Project

General

Profile

Actions
Copy link

Incident #5591

closed

The new authorization service stops responding

Added by Andrea Dell'Amico over 8 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
Immediate
Assignee:
Lucio Lelii
Category:
Application
Target version:
D4Science Infrastructure Upgrade to gCube 4.1.0
Start date:
Oct 28, 2016
Due date:
% Done:

100%

Estimated time:
Infrastructure:
Production

Description

It seems that the authorization service is not able to handle the amount of requests that come from the tagme service.
It stops responding to any request and the logs report these exceptions (into the standard error, they are not passed to the logging framework, btw):

[EL Severe]: cache: 2016-10-28 07:35:49.899--ClientSession(2093855182)--MAX TIME 600 seconds EXCEEDED FOR WRITELOCKMANAGER WAIT.  Waiting on Entity type: org.gcube.common.
authorizationservice.persistence.entities.UserAuthorizationEntitywith pk: [[/d4science.research-infrastructures.eu/SoBigData/TagMe, TOKEN, valerio.arnaboldi]: 1489348741]
currently locked by thread: catalina-exec-23 with the following trace:
                atorg.eclipse.persistence.sessions.server.ConnectionPool.releaseConnection(ConnectionPool.java:325)
                atorg.eclipse.persistence.sessions.server.ServerSession.releaseClientSession(ServerSession.java:827)
                atorg.eclipse.persistence.sessions.server.ClientSession.releaseWriteConnection(ClientSession.java:696)
                atorg.eclipse.persistence.sessions.server.ClientSession.basicCommitTransaction(ClientSession.java:182)
                atorg.eclipse.persistence.internal.sessions.AbstractSession.commitTransaction(AbstractSession.java:962)
                atorg.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransaction(UnitOfWorkImpl.java:1600)
                atorg.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitTransaction(RepeatableWriteUnitOfWork.java:647)
                atorg.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1615)
                atorg.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:284)
                atorg.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
                atorg.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:132)
                atorg.gcube.common.authorizationservice.persistence.RelationDBPersistence.getAuthorizationEntry(RelationDBPersistence.java:67)
                atorg.gcube.common.authorizationservice.TokenManager.retrieveToken(TokenManager.java:53)
                atsun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
                atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                atjava.lang.reflect.Method.invoke(Method.java:606)
                atorg.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
                atorg.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
                atorg.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
                atorg.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
                atorg.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
                atorg.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
                atorg.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
                atorg.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
                atorg.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
                atorg.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
                atorg.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
                atorg.glassfish.jersey.internal.Errors.process(Errors.java:315)
                atorg.glassfish.jersey.internal.Errors.process(Errors.java:297)
                atorg.glassfish.jersey.internal.Errors.process(Errors.java:267)
                atorg.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
                atorg.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
                atorg.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
                atorg.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373)
                atorg.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381)
                atorg.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344)
                atorg.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221)
                atorg.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
                atorg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
                atorg.gcube.common.authorizationservice.filters.AuthorizedCallFilter.doFilter(AuthorizedCallFilter.java:88)
                atorg.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
                atorg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
                atorg.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
                atorg.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
                atorg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
                atorg.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
                atorg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
                atorg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
                atorg.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683)
                atorg.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
                atorg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
                atorg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
                atorg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
                atorg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
                atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                atjava.lang.Thread.run(Thread.java:745)
 (There is no English translation for this message.)
[EL Severe]: cache: 2016-10-28 07:35:49.9--ServerSession(1609249471)--Current object locks:
Thread : catalina-exec-23
Locked Object : AuthorizationEntity [id=AuthorizationId [context=/d4science.research-infrastructures.eu/SoBigData/TagMe, clientId=valerio.arnaboldi qualifier =TOKEN], token=06d303a2-fb1c-4f62-857f-c1beb45a0d13, creationTime=java.util.GregorianCalendar[time=?,areFieldsSet=false,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="Europe/Rome",offset=3600000,dstSavings=3600000,useDaylight=true,transitions=169,lastRule=java.util.SimpleTimeZone[id=Europe/Rome,offset=3600000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=2,startMonth=2,startDay=-1,startDayOfWeek=1,startTime=3600000,startTimeMode=2,endMode=2,endMonth=9,endDay=-1,endDayOfWeek=1,endTime=3600000,endTimeMode=2]],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2016,MONTH=9,WEEK_OF_YEAR=44,WEEK_OF_MONTH=5,DAY_OF_MONTH=26,DAY_OF_YEAR=300,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=6,HOUR_OF_DAY=18,MINUTE=27,SECOND=20,MILLISECOND=27,ZONE_OFFSET=3600000,DST_OFFSET=3600000], entryType=UserInfo]
PK: [[/d4science.research-infrastructures.eu/SoBigData/TagMe, TOKEN, valerio.arnaboldi]: 1489348741]
Depth : 1

End of locked objects.

Related issues

Related to D4Science Infrastructure - Incident #5589: The tagme webapp doesn't work anymore after the smartgears ugpradeClosedMarco CornoltiOct 27, 2016

Actions
Actions
Copy link
 #1

Updated by Andrea Dell'Amico over 8 years ago

  • Related to Incident #5589: The tagme webapp doesn't work anymore after the smartgears ugprade added
Actions
Copy link
 #2

Updated by Andrea Dell'Amico over 8 years ago

There is no out of memory or other symptom of resource exhaustion on the servers.

Actions
Copy link
 #3

Updated by Marco Cornolti over 8 years ago

Hi, I have updated (solved) Incident #5589 but I do not believe the two issues are related, except from the fact that they are both caused by the update of SG. Let me know if I can be of any help.

Actions
Copy link
 #4

Updated by Lucio Lelii over 8 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

We have increased the possible connection from auhtorization to DB and modified the authorization service to avoid blocks on token retrieving.
The problem should be solved.

Actions
Copy link

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 8.91 MB)