D4Science Infrastructure

Could it be related to #10279?

Roberto Cirillo wrote:

It cannot be the cause of the issue #10279 since it is a Secondary member and the writing are done on the primary member. But, of course, the high workload in these days could be the cause of the high lag on that node. The high lag was happened only on this node and not on mongo4 and mongo5 (both secondaries nodes) as I would have expected. This maybe is due to the fact that this node has a disk storage more slow compared to the other two secondaries nodes. Is it could be possible?

The lags followed the crash and the restart, so I linked them to the fact that it restarted. I did not check if the mongo server also crashed on mongo4 and mongo5.

Anyway for preventing this kind of problem we should increase the HW resource of cluster nodes. Now we have 3 GB RAM and 4 CPU. I think we should increase the RAM to 4 GB on every node.

It's doable. Both Tommaso and me will be back in office on Thursday, but we can manage to do it anyway. Not today, but tomorrow morning maybe?

Andrea Dell'Amico wrote:

Roberto Cirillo wrote:

It cannot be the cause of the issue #10279 since it is a Secondary member and the writing are done on the primary member. But, of course, the high workload in these days could be the cause of the high lag on that node. The high lag was happened only on this node and not on mongo4 and mongo5 (both secondaries nodes) as I would have expected. This maybe is due to the fact that this node has a disk storage more slow compared to the other two secondaries nodes. Is it could be possible?

The lags followed the crash and the restart, so I linked them to the fact that it restarted. I did not check if the mongo server also crashed on mongo4 and mongo5.

Anyway for preventing this kind of problem we should increase the HW resource of cluster nodes. Now we have 3 GB RAM and 4 CPU. I think we should increase the RAM to 4 GB on every node.

It's doable. Both Tommaso and me will be back in office on Thursday, but we can manage to do it anyway. Not today, but tomorrow morning maybe?

it's ok for me. If it's needed only a VM restart you can do it at any time, if it's needed more time, it's better to do this operation after 6:00 pm when the worload decreases.

Roberto Cirillo wrote:

it's ok for me. If it's needed only a VM restart you can do it at any time, if it's needed more time, it's better to do this operation after 6:00 pm when the worload decreases.

Only a restart is needed.

I'm going to restart mongo3 with 4GB of ram just now

I think the problem is on mongo3 node. This node has the same configuration and the same workload of mongo2 but the load on mongo3 is very high while the load on mongo2 is very low.
I think this depends on the physical machine when mongo3 is running. If this is the case, I think we should transfer mongo3 on another machine.

last hour news: there are connections to mongod on mongo3-p-d4s.d4science.org from the social-isti portal.

portal-si.isti.cnr.it shows established connection on port 27017 to all social-isti mongod nodes but mongoR4-si.isti.cnr.it and there are instead connections to mongo3-p-d4s

Tommaso Piccioli wrote:

portal-si.isti.cnr.it shows established connection on port 27017 to all social-isti mongod nodes but mongoR4-si.isti.cnr.it and there are instead connections to mongo3-p-d4s

Could someone fix this and restart the service to stop these connections?
It is the jackrabbit tomcat app on portal-si.isti.cnr.it

Tommaso Piccioli wrote:

Tommaso Piccioli wrote:

portal-si.isti.cnr.it shows established connection on port 27017 to all social-isti mongod nodes but mongoR4-si.isti.cnr.it and there are instead connections to mongo3-p-d4s

Could someone fix this and restart the service to stop these connections?
It is the jackrabbit tomcat app on portal-si.isti.cnr.it

I've restarted the service. @tommaso.piccioli@isti.cnr.it, please, could you check if there are other connections from portal-si to mongo3?