Support #13083
closed
mongo2-d-d4s, mongo4-d-d4s: Certificate expired
100%
Description
The certificate seems to be expired on the hosts above and the lag is very high now. I see the following logs:
2018-12-27T10:44:17.766+0100 E NETWORK [conn3434] SSL peer certificate validation failed: certificate has expired 2018-12-27T10:44:17.766+0100 I - [conn3434] end connection 146.48.123.14:60718 (92 connections now open)
Could anyone check asap? Otherwise I should disable ssl from the cluster
Updated by Andrea Dell'Amico over 6 years ago
- Status changed from New to In Progress
- % Done changed from 0 to 50
The certificates were correctly renewed on all the dev mongo instances, Dec 1st. But I see that the restart action is not present in the letsencrypt hooks. I also see:
d4science-ghn-cluster//group_vars/mongo_vol_dev/mongo_vol_dev.yml:#mongodb_ssl_enabled: False d4science-ghn-cluster//group_vars/mongo_vol_dev/mongo_vol_dev.yml:mongodb_ssl_enabled: True
So I guess that the variable was switched but the playbook was not run again. @roberto.cirillo@isti.cnr.it You can run it applying the tag mongodb_letsencrypt to fix the script, but you also must manually restart the mongodb service.
I'm going to check the status of preprod and prod.
Updated by Roberto Cirillo over 6 years ago
OK thanks. I'm going to run the playbook as requested
Updated by Roberto Cirillo over 6 years ago
After the restart I see the same error on mongo2-d-d4s and mongo4-d-d4s
Updated by Andrea Dell'Amico over 6 years ago
- Infrastructure Pre-Production, Production added
On the other mongo instances is installed the wrong hook script too.
A recap: mongodb does not support a service reload, so after a certificate change a restart is required. The cron job that renews the certificates runs at different times on the various servers to avoid downtimes.
Updated by Andrea Dell'Amico over 6 years ago
Roberto Cirillo wrote:
After the restart I see the same error on mongo2-d-d4s and mongo4-d-d4s
But mongodb on mongo1-d-d4s and mongo3-d-d4s was not restarted.
Updated by Roberto Cirillo over 6 years ago
Andrea Dell'Amico wrote:
Roberto Cirillo wrote:
After the restart I see the same error on mongo2-d-d4s and mongo4-d-d4s
But mongodb on mongo1-d-d4s and mongo3-d-d4s was not restarted.
You are right. After the restart it works properly.
Updated by Roberto Cirillo over 6 years ago
Andrea Dell'Amico wrote:
On the other mongo instances is installed the wrong hook script too.
A recap: mongodb does not support a service reload, so after a certificate change a restart is required. The cron job that renews the certificates runs at different times on the various servers to avoid downtimes.
So we should manually restart the production instances after the certificate renew?
Updated by Andrea Dell'Amico over 6 years ago
Roberto Cirillo wrote:
Andrea Dell'Amico wrote:
On the other mongo instances is installed the wrong hook script too.
A recap: mongodb does not support a service reload, so after a certificate change a restart is required. The cron job that renews the certificates runs at different times on the various servers to avoid downtimes.So we should manually restart the production instances after the certificate renew?
After you run the playbook as indicated above, the mongo servers will be restarted by the letsencrypt hook itself.
Updated by Roberto Cirillo over 6 years ago
- Status changed from In Progress to Closed
- % Done changed from 50 to 100
Done also on production environment. I'm going to close this ticket
