D4Science Infrastructure

In a meeting today we have agreed to have the following configurations:

1 - each VRE will have two proxies: DataMiner (Front-End) and DataMinerGenericWorker (GW) (should be already so in most VREs)
2 - there will be two proxies for the prototype and other two proxies for the production environment (already so)
3 - the old GenericWorker machines will be deleted along with their provisioning scripts
4 - the old GenericWorker will be replaced by multi-core DMs having internal jobs queue = 1
5 - The distribution (and the architectures) of the machines over the clusters will be the following:

Prototype:

6 DMs on the Front End-Proxy (16 core, 16GB RAM) + 6 DMs on the GW-Proxy (16 core, 16GB RAM). All the machines will be CNR machines.

Production:

15 DMs on the Front End-Proxy (16 core, 32GB RAM) + 15 DMs on the GW-Proxy (16 core, 32GB RAM). The machines will be CNR and Garr machines.

We will need to find resources for the production configuration.

Gianpaolo Coro wrote:

4 - the old GenericWorker will be replaced by multi-core DMs having internal jobs queue = 1

Just a reminder of the fact that this is not going to happen until that parameter will be transformed in a configuration property.

Production:

15 DMs on the Front End-Proxy (16 core, 32GB RAM) + 15 DMs on the GW-Proxy (16 core, 32GB RAM). The machines will be CNR and Garr machines.

Aren't they way too many?

Andrea Dell'Amico wrote:

Gianpaolo Coro wrote:

4 - the old GenericWorker will be replaced by multi-core DMs having internal jobs queue = 1

Just a reminder of the fact that this is not going to happen until that parameter will be transformed in a configuration property.

We need to be flexible since we cannot continue supporting AGInfra in this way. We need a solution for the next week since otherwise, they will block all other VREs belonging to other communities.
We have already a JAR with that property set to 1. I am wondering if we cannot provision next week two different DM distributions. The standard ones with the current provisioning and the new set of 6 DM proto-GW with that DM jar.

Production:

15 DMs on the Front End-Proxy (16 core, 32GB RAM) + 15 DMs on the GW-Proxy (16 core, 32GB RAM). The machines will be CNR and Garr machines.

Aren't they way too many?

Accordingly to what we discussed they are not too many since AgInfra is starting exploiting the infrastructure and this is very important for our future sustainability since we need to demonstrate that we can support the community that will create the so-called FoodCloud.

Talking about the numbers, currently, we have 21 DM, each with 16 cores and 32 GB ram + 9 GW, each with 2 cores and 8 GB ram. In total means 744 GB Ram and 354 cores.

With the new configuration, we will have 15 + 15 = 30 DMs, each with 16 cores and 32 GB ram. In total means 480 cores and 960 GB ram.

We need therefore additional 216 GB ram. It would be also better if possible to provision the clusters in two distinct GARR sites.

what do you mean?
The proxies will provide access to a static set of servers.

Pasquale Pagano wrote:

what do you mean?
The proxies will provide access to a static set of servers.

I did not understand the above phrase, @g.panichi@isti.cnr.it just explained to me that nothing changes. The new gw prototypes just replace the old instance.

people more informed than me can explain better.

What I understood is that we need 4 proxy backend, 2 for the production and 2 for the prototypes.
Each proxy backend provides access to different set of servers.

proxy 1 for accessing 6 proto DM with queue = 4, 16 CPU, 16 GB Ram
proxy 2 for accessing 6 proto DM with queue = 1, 16 CPU, 16 GB Ram

proxy 3 for accessing 15 DM with queue = 4, 16 CPU, 32 GB Ram
proxy 4 for accessing 15 DM with queue = 1, 16 CPU, 32 GB Ram

Pasquale Pagano wrote:

people more informed than me can explain better.

What I understood is that we need 4 proxy backend, 2 for the production and 2 for the prototypes.
Each proxy backend provides access to different set of servers.

proxy 1 for accessing 6 proto DM with queue = 4, 16 CPU, 16 GB Ram
proxy 2 for accessing 6 proto DM with queue = 1, 16 CPU, 16 GB Ram

proxy 3 for accessing 15 DM with queue = 4, 16 CPU, 32 GB Ram
proxy 4 for accessing 15 DM with queue = 1, 16 CPU, 32 GB Ram

Yes, it's the same configuration that we have currently, what changes is the number and resources of the DM instances.