Support #12034
closed
ckan-bb1.d4science.org: too many opened connections to node66.p.d4science
100%
Description
I see a lot of concurred opened connections from ckan-bb1.d4science.org to node66.p.d4science.research-infrastructures.eu
This host should be monitored.
Related issues
Updated by Roberto Cirillo almost 7 years ago
- Related to Task #12031: gCubeApps - IS-Collector: too many open connections added
Updated by Roberto Cirillo almost 7 years ago
- Status changed from New to In Progress
I see a lot of opened connection to node66.p, sometimes over 50. This is not normal.
From logs (INFO level) I don't see nothing.
I've also checked the connections from this host to the others IS-Collector (FARM- D4Research) but I don't see more than 1 opened connection.
I want to try to decrease the log level to DEBUG level, if possible, without restart the container. @andrea.dellamico@isti.cnr.it could I do that on ckan-bb1.d4science.org? The distro deployed here is 4.11
Updated by Andrea Dell'Amico almost 7 years ago
You should ask @francesco.mangiacrapa@isti.cnr.it as I only install the CKAN nodes. The smartgears part is separate from the main service, CKAN is not a tomcat application. So you can run the ckan.yml playbook with the smartgears tag without touching anything CKAN related.
Updated by Roberto Cirillo almost 7 years ago
In this host there is the smartgears distribution 4.11 but I don't remember if the logback version released in 4.11 allows to set the log level at runtime without restart the container.
Do you remember how to check it @andrea.dellamico@isti.cnr.it ? I need to check the logback library version or the logback configuration file?
Updated by Andrea Dell'Amico almost 7 years ago
The first line of the logback.xml file that automatically reload itself is:
<configuration scan="true" debug="false">
Updated by Roberto Cirillo almost 7 years ago
Andrea Dell'Amico wrote:
The first line of the
logback.xmlfile that automatically reload itself is:<configuration scan="true" debug="false">
Thanks @andrea.dellamico@isti.cnr.it the line above is present. I'm going to set the level to DEBUG
Updated by Roberto Cirillo almost 7 years ago
@andrea.dellamico@isti.cnr.it I guess the logback reloading doesn't work. I've set the log to DEBUG three hours ago but at the moment I don't see any DEBUG log. Any hint?
Updated by Andrea Dell'Amico almost 7 years ago
No. I tested it and it worked in the past, but I don't remember the specific smartgears version and the logback libraries (don't know if they have been changed again).
Ask @francesco.mangiacrapa@isti.cnr.it if you can restart the tomcat server in the meantime, but I'm afraid that you are going to miss the cause of the problem because the restart could fix it.
Updated by Roberto Cirillo almost 7 years ago
- Status changed from In Progress to Feedback
- Assignee changed from Roberto Cirillo to Luca Frosini
- % Done changed from 0 to 50
@luca.frosini@isti.cnr.it could I restart the container or do you want to check the behavior of this host?
I think that if the container will be restarted the behavior return to normal as happened on rstudio2.d4science.org: #12033
This line show the connection opened now from rstudio2.d4science.org (restarted)
root@node66:~# netstat -ano | grep 146.48.122.146:8080 | grep ESTA | grep -c 146.48.123.73 0
This one show the connections opened now from ckan-bb1.d4science.org:
root@node66:~# netstat -ano | grep 146.48.122.146:8080 | grep ESTA | grep -c 146.48.123.164 40
Updated by Luca Frosini almost 7 years ago
From my point of view, you can restart it
Updated by Roberto Cirillo almost 7 years ago
- Status changed from Feedback to Closed
- % Done changed from 50 to 100
The service has been restarted with log level set to DEBUG/TRACE. Now, the high number of opened connections to node66.p are disappeared. I'm going to close this ticket, if the problem returns, I'll reopen it
