Actions
Incident #10203
closed
Something wrong in communication between EDISON Portal production instance and MongoDB Production
Status:
Closed
Priority:
High
Assignee:
Category:
-
Target version:
Start date:
Nov 03, 2017
Due date:
Nov 03, 2017
% Done:
100%
Estimated time:
Infrastructure:
Production
Description
See the exception below (java.net.ConnectException: Connection refused (Connection refused)}), unclear if related to the Authorization service:
14:01:16,168 INFO [GWTWorkspaceServiceImpl:388] Special folder name updated as: VRE Folders
com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches PrimaryServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo1-p-d4s.d4science.org:27017, type=REPLICA_SET_ARBITER, roundTripTime=48.9 ms, state=CONNECTED}, {address=mongo2-p-d4s.d4science.org:27017, type=REPLICA_SET_SECONDARY, roundTripTime=47.7 ms, state=CONNECTED}, {address=mongo3-p-d4s.d4science.org:27017, type=REPLICA_SET_SECONDARY, roundTripTime=47.4 ms, state=CONNECTED}, {address=mongo4-p-d4s.d4science.org:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
at com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:370)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:75)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:71)
at com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:175)
at com.mongodb.operation.CreateIndexesOperation.execute(CreateIndexesOperation.java:111)
at com.mongodb.operation.CreateIndexesOperation.execute(CreateIndexesOperation.java:66)
at com.mongodb.Mongo.execute(Mongo.java:747)
at com.mongodb.Mongo$2.execute(Mongo.java:730)
at com.mongodb.DBCollection.createIndex(DBCollection.java:1492)
at com.mongodb.DBCollection.createIndex(DBCollection.java:1478)
at org.gcube.contentmanagement.blobstorage.transport.backend.DefaultMongoClient.<init>(DefaultMongoClient.java:63)
at org.gcube.contentmanagement.blobstorage.transport.TransportManagerFactory.load(TransportManagerFactory.java:56)
at org.gcube.contentmanagement.blobstorage.transport.TransportManagerFactory.getTransport(TransportManagerFactory.java:43)
at org.gcube.contentmanagement.blobstorage.service.operation.GetUserTotalVolume.doIt(GetUserTotalVolume.java:24)
at org.gcube.contentmanagement.blobstorage.service.operation.GetUserTotalVolume.doIt(GetUserTotalVolume.java:13)
at org.gcube.contentmanagement.blobstorage.service.operation.OperationManager.startOperation(OperationManager.java:76)
at org.gcube.contentmanagement.blobstorage.service.impl.ServiceEngine.getTotalUserVolume(ServiceEngine.java:276)
at org.gcube.common.homelibrary.jcr.repository.external.GCUBEStorage.getDiskUsageByUser(GCUBEStorage.java:191)
at org.gcube.common.homelibrary.jcr.workspace.JCRWorkspace.getDiskUsage(JCRWorkspace.java:3614)
at org.gcube.portlets.user.workspace.server.GWTWorkspaceServiceImpl.getUserWorkspaceQuote(GWTWorkspaceServiceImpl.java:3694)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:561)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at sun.reflect.GeneratedMethodAccessor1323.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.liferay.portal.kernel.bean.ClassLoaderBeanHandler.invoke(ClassLoaderBeanHandler.java:67)
at com.sun.proxy.$Proxy749.doFilter(Unknown Source)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:165)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
at com.liferay.portal.kernel.servlet.PortalClassLoaderFilter.doFilter(PortalClassLoaderFilter.java:74)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:204)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:109)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:957)
at org.gcube.portal.threadlocalexec.SmartGearsPortalValve.invoke(SmartGearsPortalValve.java:69)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
14:01:41,612 INFO [GWTWorkspaceServiceImpl:3695] Root size is: 29664529407 formatting..
14:01:41,612 INFO [GWTWorkspaceServiceImpl:3750] Getting total items..
com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches PrimaryServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo1-p-d4s.d4science.org:27017, type=REPLICA_SET_ARBITER, roundTripTime=47.6 ms, state=CONNECTED}, {address=mongo2-p-d4s.d4science.org:27017, type=REPLICA_SET_SECONDARY, roundTripTime=47.5 ms, state=CONNECTED}, {address=mongo3-p-d4s.d4science.org:27017, type=REPLICA_SET_SECONDARY, roundTripTime=47.3 ms, state=CONNECTED}, {address=mongo4-p-d4s.d4science.org:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
at com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:370)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:75)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:71)
at com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:175)
at com.mongodb.operation.CreateIndexesOperation.execute(CreateIndexesOperation.java:111)
at com.mongodb.operation.CreateIndexesOperation.execute(CreateIndexesOperation.java:66)
at com.mongodb.Mongo.execute(Mongo.java:747)
at com.mongodb.Mongo$2.execute(Mongo.java:730)
at com.mongodb.DBCollection.createIndex(DBCollection.java:1492)
at com.mongodb.DBCollection.createIndex(DBCollection.java:1478)
at org.gcube.contentmanagement.blobstorage.transport.backend.DefaultMongoClient.<init>(DefaultMongoClient.java:63)
at org.gcube.contentmanagement.blobstorage.transport.TransportManagerFactory.load(TransportManagerFactory.java:56)
at org.gcube.contentmanagement.blobstorage.transport.TransportManagerFactory.getTransport(TransportManagerFactory.java:43)
at org.gcube.contentmanagement.blobstorage.service.operation.GetUserTotalVolume.doIt(GetUserTotalVolume.java:24)
at org.gcube.contentmanagement.blobstorage.service.operation.GetUserTotalVolume.doIt(GetUserTotalVolume.java:13)
at org.gcube.contentmanagement.blobstorage.service.operation.OperationManager.startOperation(OperationManager.java:76)
at org.gcube.contentmanagement.blobstorage.service.impl.ServiceEngine.getTotalUserVolume(ServiceEngine.java:276)
at org.gcube.common.homelibrary.jcr.repository.external.GCUBEStorage.getDiskUsageByUser(GCUBEStorage.java:191)
at org.gcube.common.homelibrary.jcr.workspace.JCRWorkspace.getDiskUsage(JCRWorkspace.java:3614)
at org.gcube.portlets.user.workspace.server.GWTWorkspaceServiceImpl.getUserWorkspaceQuote(GWTWorkspaceServiceImpl.java:3694)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:561)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at sun.reflect.GeneratedMethodAccessor1323.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.liferay.portal.kernel.bean.ClassLoaderBeanHandler.invoke(ClassLoaderBeanHandler.java:67)
at com.sun.proxy.$Proxy749.doFilter(Unknown Source)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:165)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
at com.liferay.portal.kernel.servlet.PortalClassLoaderFilter.doFilter(PortalClassLoaderFilter.java:74)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:204)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:109)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:957)
at org.gcube.portal.threadlocalexec.SmartGearsPortalValve.invoke(SmartGearsPortalValve.java:69)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches PrimaryServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=mongo1-p-d4s.d4science.org:27017, type=REPLICA_SET_ARBITER, roundTripTime=47.3 ms, state=CONNECTED}, {address=mongo2-p-d4s.d4science.org:27017, type=REPLICA_SET_SECONDARY, roundTripTime=47.5 ms, state=CONNECTED}, {address=mongo3-p-d4s.d4science.org:27017, type=REPLICA_SET_SECONDARY, roundTripTime=47.2 ms, state=CONNECTED}, {address=mongo4-p-d4s.d4science.org:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
at com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:370)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:75)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:71)
at com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:175)
at com.mongodb.operation.CreateIndexesOperation.execute(CreateIndexesOperation.java:111)
at com.mongodb.operation.CreateIndexesOperation.execute(CreateIndexesOperation.java:66)
at com.mongodb.Mongo.execute(Mongo.java:747)
at com.mongodb.Mongo$2.execute(Mongo.java:730)
at com.mongodb.DBCollection.createIndex(DBCollection.java:1492)
at com.mongodb.DBCollection.createIndex(DBCollection.java:1478)
at org.gcube.contentmanagement.blobstorage.transport.backend.DefaultMongoClient.<init>(DefaultMongoClient.java:63)
at org.gcube.contentmanagement.blobstorage.transport.TransportManagerFactory.load(TransportManagerFactory.java:56)
at org.gcube.contentmanagement.blobstorage.transport.TransportManagerFactory.getTransport(TransportManagerFactory.java:43)
at org.gcube.contentmanagement.blobstorage.service.operation.GetUserTotalVolume.doIt(GetUserTotalVolume.java:24)
at org.gcube.contentmanagement.blobstorage.service.operation.GetUserTotalVolume.doIt(GetUserTotalVolume.java:13)
at org.gcube.contentmanagement.blobstorage.service.operation.OperationManager.startOperation(OperationManager.java:76)
at org.gcube.contentmanagement.blobstorage.service.impl.ServiceEngine.getTotalUserVolume(ServiceEngine.java:276)
at org.gcube.common.homelibrary.jcr.repository.external.GCUBEStorage.getDiskUsageByUser(GCUBEStorage.java:191)
at org.gcube.common.homelibrary.jcr.workspace.JCRWorkspace.getDiskUsage(JCRWorkspace.java:3614)
at org.gcube.portlets.user.workspace.server.GWTWorkspaceServiceImpl.getUserWorkspaceQuote(GWTWorkspaceServiceImpl.java:3694)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:561)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at sun.reflect.GeneratedMethodAccessor1323.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.liferay.portal.kernel.bean.ClassLoaderBeanHandler.invoke(ClassLoaderBeanHandler.java:67)
at com.sun.proxy.$Proxy749.doFilter(Unknown Source)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:165)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
at com.liferay.portal.kernel.servlet.PortalClassLoaderFilter.doFilter(PortalClassLoaderFilter.java:74)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:204)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:109)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:957)
at org.gcube.portal.threadlocalexec.SmartGearsPortalValve.invoke(SmartGearsPortalValve.java:69)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
java.lang.Exception: error contacting authorization service
at org.gcube.common.authorization.client.proxy.DefaultAuthorizationProxy.generateUserToken(DefaultAuthorizationProxy.java:175)
at org.gcube.common.portal.PortalContext.generateAuthorizationToken(PortalContext.java:420)
at org.gcube.common.portal.PortalContext.getCurrentUserToken(PortalContext.java:388)
at org.gcube.portlets.user.workspace.server.util.WsUtil.getPortalContext(WsUtil.java:101)
at org.gcube.portlets.user.workspace.server.util.WsUtil.getWorkspace(WsUtil.java:147)
at org.gcube.portlets.user.workspace.server.GWTWorkspaceServiceImpl.getWorkspace(GWTWorkspaceServiceImpl.java:144)
at org.gcube.portlets.user.workspace.server.GWTWorkspaceServiceImpl.getUserWorkspaceTotalItems(GWTWorkspaceServiceImpl.java:3751)
at org.gcube.portlets.user.workspace.server.GWTWorkspaceServiceImpl.getUserWorkspaceQuote(GWTWorkspaceServiceImpl.java:3697)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:561)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at sun.reflect.GeneratedMethodAccessor1323.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.liferay.portal.kernel.bean.ClassLoaderBeanHandler.invoke(ClassLoaderBeanHandler.java:67)
at com.sun.proxy.$Proxy749.doFilter(Unknown Source)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:165)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
at com.liferay.portal.kernel.servlet.PortalClassLoaderFilter.doFilter(PortalClassLoaderFilter.java:74)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:204)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:109)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:957)
at org.gcube.portal.threadlocalexec.SmartGearsPortalValve.invoke(SmartGearsPortalValve.java:69)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
14:01:48,319 INFO [GWTWorkspaceServiceImpl:3695] Root size is: 29664529407 formatting..
14:01:48,319 INFO [GWTWorkspaceServiceImpl:3750] Getting total items..
14:01:48,412 INFO [GWTWorkspaceServiceImpl:3695] Root size is: 29664529407 formatting..
14:01:48,413 INFO [GWTWorkspaceServiceImpl:3750] Getting total items..
java.lang.Exception: error contacting authorization service
at org.gcube.common.authorization.client.proxy.DefaultAuthorizationProxy.generateUserToken(DefaultAuthorizationProxy.java:175)
at org.gcube.common.portal.PortalContext.generateAuthorizationToken(PortalContext.java:420)
at org.gcube.common.portal.PortalContext.getCurrentUserToken(PortalContext.java:388)
at org.gcube.portlets.user.workspace.server.util.WsUtil.getPortalContext(WsUtil.java:101)
at org.gcube.portlets.user.workspace.server.util.WsUtil.getWorkspace(WsUtil.java:147)
at org.gcube.portlets.user.workspace.server.GWTWorkspaceServiceImpl.getWorkspace(GWTWorkspaceServiceImpl.java:144)
at org.gcube.portlets.user.workspace.server.GWTWorkspaceServiceImpl.getUserWorkspaceTotalItems(GWTWorkspaceServiceImpl.java:3751)
at org.gcube.portlets.user.workspace.server.GWTWorkspaceServiceImpl.getUserWorkspaceQuote(GWTWorkspaceServiceImpl.java:3697)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:561)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at sun.reflect.GeneratedMethodAccessor1323.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.liferay.portal.kernel.bean.ClassLoaderBeanHandler.invoke(ClassLoaderBeanHandler.java:67)
at com.sun.proxy.$Proxy749.doFilter(Unknown Source)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:165)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
at com.liferay.portal.kernel.servlet.PortalClassLoaderFilter.doFilter(PortalClassLoaderFilter.java:74)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:204)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:109)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:957)
at org.gcube.portal.threadlocalexec.SmartGearsPortalValve.invoke(SmartGearsPortalValve.java:69)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
java.lang.Exception: error contacting authorization service
at org.gcube.common.authorization.client.proxy.DefaultAuthorizationProxy.generateUserToken(DefaultAuthorizationProxy.java:175)
at org.gcube.common.portal.PortalContext.generateAuthorizationToken(PortalContext.java:420)
at org.gcube.common.portal.PortalContext.getCurrentUserToken(PortalContext.java:388)
at org.gcube.portlets.user.workspace.server.util.WsUtil.getPortalContext(WsUtil.java:101)
at org.gcube.portlets.user.workspace.server.util.WsUtil.getWorkspace(WsUtil.java:147)
at org.gcube.portlets.user.workspace.server.GWTWorkspaceServiceImpl.getWorkspace(GWTWorkspaceServiceImpl.java:144)
at org.gcube.portlets.user.workspace.server.GWTWorkspaceServiceImpl.getUserWorkspaceTotalItems(GWTWorkspaceServiceImpl.java:3751)
at org.gcube.portlets.user.workspace.server.GWTWorkspaceServiceImpl.getUserWorkspaceQuote(GWTWorkspaceServiceImpl.java:3697)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:561)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at sun.reflect.GeneratedMethodAccessor1323.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.liferay.portal.kernel.bean.ClassLoaderBeanHandler.invoke(ClassLoaderBeanHandler.java:67)
at com.sun.proxy.$Proxy749.doFilter(Unknown Source)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:116)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDirectCallFilter(InvokerFilterChain.java:165)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:96)
at com.liferay.portal.kernel.servlet.PortalClassLoaderFilter.doFilter(PortalClassLoaderFilter.java:74)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.processDoFilter(InvokerFilterChain.java:204)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilterChain.doFilter(InvokerFilterChain.java:109)
at com.liferay.portal.kernel.servlet.filters.invoker.InvokerFilter.doFilter(InvokerFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:957)
at org.gcube.portal.threadlocalexec.SmartGearsPortalValve.invoke(SmartGearsPortalValve.java:69)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
14:01:48,834 INFO [GWTWorkspaceServiceImpl:3753] returning total items value: 1066
14:01:48,834 INFO [GWTWorkspaceServiceImpl:3703] returning user quote: WorkspaceUserQuote [diskSpace=29664529407, diskSpaceFormatted=27.63 GB, totalItems=1066]
14:01:50,353 INFO [GWTWorkspaceServiceImpl:3753] returning total items value: 1066
Updated by Massimiliano Assante over 7 years ago
the EDISON Portal production instance has an IP address 194.149.136.227
Updated by Massimiliano Assante over 7 years ago
Weird, the exception disappeared now, I cannot reproduce
Updated by Andrea Dell'Amico over 7 years ago
I can only confirm that all the mongo servers are accessible from the EDISON portal:
$ for n in $( seq 1 5 ) ; do ssh mongo${n}-p-d4s.d4science.org iptables -nvL | grep 194.149.136.227 ; done
327 19620 ACCEPT tcp -- * * 194.149.136.227 0.0.0.0/0 state NEW tcp dpt:27017
335 20100 ACCEPT tcp -- * * 194.149.136.227 0.0.0.0/0 state NEW tcp dpt:27017
258 15480 ACCEPT tcp -- * * 194.149.136.227 0.0.0.0/0 state NEW tcp dpt:27017
603 36180 ACCEPT tcp -- * * 194.149.136.227 0.0.0.0/0 state NEW tcp dpt:27017
0 0 ACCEPT tcp -- * * 194.149.136.227 0.0.0.0/0 state NEW tcp dpt:27017
Updated by Massimiliano Assante over 7 years ago
@lucio.lelii@isti.cnr.it can you look at the exception and tell us what could have happened? It seems it could be related to the Auth Service?
java.lang.Exception: error contacting authorization service
at org.gcube.common.authorization.client.proxy.DefaultAuthorizationProxy.generateUserToken(DefaultAuthorizationProxy.java:175)
at org.gcube.common.portal.PortalContext.generateAuthorizationToken(PortalContext.java:420)
at org.gcube.common.portal.PortalContext.getCurrentUserToken(PortalContext.java:388)
at org.gcube.portlets.user.workspace.server.util.WsUtil.getPortalContext(WsUtil.java:101)
Updated by Massimiliano Assante over 7 years ago
- Assignee changed from _InfraScience Systems Engineer to Lucio Lelii
Updated by Lucio Lelii over 7 years ago
I just checked the authorization service and everything is working fine.
Updated by Massimiliano Assante over 7 years ago
- Status changed from New to Closed
- Assignee changed from Lucio Lelii to Roberto Cirillo
I close the ticket, let's wait if it happens again.
Updated by Andrea Dell'Amico over 7 years ago
I can confirm that the authorization service is configured to allow requests from the edison portal.
The failures seem network related to me.
Actions
