D4Science Infrastructure

It seems that the authorization service is not able to handle the amount of requests that come from the tagme service.
It stops responding to any request and the logs report these exceptions (into the standard error, they are not passed to the logging framework, btw):

[EL Severe]: cache: 2016-10-28 07:35:49.899--ClientSession(2093855182)--MAX TIME 600 seconds EXCEEDED FOR WRITELOCKMANAGER WAIT.  Waiting on Entity type: org.gcube.common.
authorizationservice.persistence.entities.UserAuthorizationEntitywith pk: [[/d4science.research-infrastructures.eu/SoBigData/TagMe, TOKEN, valerio.arnaboldi]: 1489348741]
currently locked by thread: catalina-exec-23 with the following trace:
                atorg.eclipse.persistence.sessions.server.ConnectionPool.releaseConnection(ConnectionPool.java:325)
                atorg.eclipse.persistence.sessions.server.ServerSession.releaseClientSession(ServerSession.java:827)
                atorg.eclipse.persistence.sessions.server.ClientSession.releaseWriteConnection(ClientSession.java:696)
                atorg.eclipse.persistence.sessions.server.ClientSession.basicCommitTransaction(ClientSession.java:182)
                atorg.eclipse.persistence.internal.sessions.AbstractSession.commitTransaction(AbstractSession.java:962)
                atorg.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransaction(UnitOfWorkImpl.java:1600)
                atorg.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitTransaction(RepeatableWriteUnitOfWork.java:647)
                atorg.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1615)
                atorg.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:284)
                atorg.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
                atorg.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:132)
                atorg.gcube.common.authorizationservice.persistence.RelationDBPersistence.getAuthorizationEntry(RelationDBPersistence.java:67)
                atorg.gcube.common.authorizationservice.TokenManager.retrieveToken(TokenManager.java:53)
                atsun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
                atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                atjava.lang.reflect.Method.invoke(Method.java:606)
                atorg.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
                atorg.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
                atorg.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
                atorg.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
                atorg.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
                atorg.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
                atorg.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
                atorg.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
                atorg.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
                atorg.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
                atorg.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
                atorg.glassfish.jersey.internal.Errors.process(Errors.java:315)
                atorg.glassfish.jersey.internal.Errors.process(Errors.java:297)
                atorg.glassfish.jersey.internal.Errors.process(Errors.java:267)
                atorg.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
                atorg.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
                atorg.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
                atorg.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373)
                atorg.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381)
                atorg.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344)
                atorg.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221)
                atorg.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
                atorg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
                atorg.gcube.common.authorizationservice.filters.AuthorizedCallFilter.doFilter(AuthorizedCallFilter.java:88)
                atorg.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
                atorg.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
                atorg.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
                atorg.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
                atorg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
                atorg.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
                atorg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
                atorg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
                atorg.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683)
                atorg.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
                atorg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
                atorg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
                atorg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
                atorg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
                atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                atjava.lang.Thread.run(Thread.java:745)
 (There is no English translation for this message.)
[EL Severe]: cache: 2016-10-28 07:35:49.9--ServerSession(1609249471)--Current object locks:
Thread : catalina-exec-23
Locked Object : AuthorizationEntity [id=AuthorizationId [context=/d4science.research-infrastructures.eu/SoBigData/TagMe, clientId=valerio.arnaboldi qualifier =TOKEN], token=06d303a2-fb1c-4f62-857f-c1beb45a0d13, creationTime=java.util.GregorianCalendar[time=?,areFieldsSet=false,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="Europe/Rome",offset=3600000,dstSavings=3600000,useDaylight=true,transitions=169,lastRule=java.util.SimpleTimeZone[id=Europe/Rome,offset=3600000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=2,startMonth=2,startDay=-1,startDayOfWeek=1,startTime=3600000,startTimeMode=2,endMode=2,endMonth=9,endDay=-1,endDayOfWeek=1,endTime=3600000,endTimeMode=2]],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2016,MONTH=9,WEEK_OF_YEAR=44,WEEK_OF_MONTH=5,DAY_OF_MONTH=26,DAY_OF_YEAR=300,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=6,HOUR_OF_DAY=18,MINUTE=27,SECOND=20,MILLISECOND=27,ZONE_OFFSET=3600000,DST_OFFSET=3600000], entryType=UserInfo]
PK: [[/d4science.research-infrastructures.eu/SoBigData/TagMe, TOKEN, valerio.arnaboldi]: 1489348741]
Depth : 1

End of locked objects.