Project

General

Profile

Actions

Incident #10063

closed

Mongo5-p-d4s: disk space is critical

Added by Roberto Cirillo over 7 years ago. Updated over 7 years ago.

Status:
Closed
Priority:
Urgent
Category:
System Application
Start date:
Oct 25, 2017
Due date:
% Done:

100%

Estimated time:
Infrastructure:
Production

Description

We have just added 50 GB to mongo5 but the workload of the cluster is very increased in the last days so we need a strategy for recovering this node.Now we have only 50 GB and the disk space on the server is very critical.


Related issues

Related to D4Science Infrastructure - Incident #10101: Mongo5-p-d4s: full backup not completed after 24hClosedTommaso PiccioliOct 30, 2017

Actions
Actions #1

Updated by Tommaso Piccioli over 7 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 50

After some disk reallocation:

                Size  Used Avail Use% Mounted on
mongo1-p-d4s     10G  5.4G  4.7G  54% /         (Arbiter)

mongo2-p-d4s    1.4T  433G  966G  31% /data    (Secondary)
mongo3-p-d4s    1.4T  437G  961G  32% /data    (Secondary)
mongo4-p-d4s    1.4T  434G  964G  32% /data    (Primary at the moment)

mongo5-p-d4s    1.9T  875G  989G  47% /data    (Backup node)

mongo5-p-d4s will be fully resync with the rest of the cluster, this will free up more disk space.

Actions #2

Updated by Roberto Cirillo over 7 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 50 to 100

Unfortunately the first full resync attempt was failed because this operation needs more RAM. The second attempt, with RAM increased from 4 to 8 GB has been successfully performed.
Now mongo5 is sinchronized with the others nodes.

Actions #3

Updated by Roberto Cirillo over 7 years ago

  • Related to Incident #10101: Mongo5-p-d4s: full backup not completed after 24h added
Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 8.91 MB)