D4Science Infrastructure

Hey Andrea,

thank you for your answer. I checked your links and I can see some nice graphs, but I am not sure if that really helps.

With the workflow I provided in the dataminer you could easily check if your changed settings work as expected, the possibility to use 16-1 cores at once.

I tried to access dataminer0-gw-proto which was used in one of the processes I run, but I could not find it.

I saw one graph in the linked servers which showed 3 peaks a 75%. That would fit to the old settings with 4 cores use max. According to that it might not have changed.

Best,
Lars

Lars Valentin wrote:

Hey Andrea,

thank you for your answer. I checked your links and I can see some nice graphs, but I am not sure if that really helps.

With the workflow I provided in the dataminer you could easily check if your changed settings work as expected, the possibility to use 16-1 cores at once.

We did not change any settings on the nodes. We gave you access to a cluster that was already configured that way. The nodes were already there, but they were not meant to be directly accessible.

I tried to access dataminer0-gw-proto which was used in one of the processes I run, but I could not find it.

The dashboard for the generic worker prototypes nodes is https://grafana.d4science.org/d/B63Xb3Gmk/dataminer-ct1-gw-proto?orgId=1

I saw one graph in the linked servers which showed 3 peaks a 75%. That would fit to the old settings with 4 cores use max. According to that it might not have changed.

Even if the configuration was wrong (and it's not), there are no hard limits on the cpu usage. So, if there's only one job running on a node, it's free to use all the available CPUs.
A usage of 75% means that all the jobs running on the node are not able to use more CPU than that, not that some policy is limiting them.

Hi Andrea,

I still don't see a climb in the Grafana CPU fields with the links provided above after executing my testworkflow.

Apart from that there seems to be a difference in the version of our plugin installed and the developing version I use locally, which is causing that only 10 cores are used in the test. We are working on a new release version and as soon as we have it published I will ask you to install it in your infrastructure and I test again.

Best,
Lars

Lars Valentin wrote:

Hi Andrea,

I still don't see a climb in the Grafana CPU fields with the links provided above after executing my testworkflow.

Apart from that there seems to be a difference in the version of our plugin installed and the developing version I use locally, which is causing that only 10 cores are used in the test. We are working on a new release version and as soon as we have it published I will ask you to install it in your infrastructure and I test again.

Your plugin is automatically upgraded every night, as you asked.

Thats great! We will then have it right after the new version is published from our side.

But still, I can't even monitor the use of the current 10 cores. Therefore I need support on how to see that in the tool provided. I suggest that we do that together in the bilateral webmeeting which will be set up by CNR in the near future to get support in diffrent issues.

Best,
Lars

Lars Valentin wrote:

Unfortunately, I still have no feedback at all by the links you provided me.

I need to proof that each of the 16 cores is used while my 'challenge workflow' is executed, like in htop (eg. core 1 87%; core 2 77%; core 3 98% ...)
Could you please check that for me if it works as expected?

Furthermore, I am wondering why non of the CPU graphs reacted in any way while I was running the workflow, using your links and setting last 15 minutes. I assume they do not work properly or target other servers then those we are using.

They work properly, and they are related to the servers you are using.

I cannot spend time staring at a bunch of htop sessions on request by every users at any time, if you can run your workflow at a certain time I'll have a session ready to wach the execution. Later today, or in the late morning tomorrow. Otherwise we must go to the next week.

They work properly, and they are related to the servers you are using.

I cannot spend time staring at a bunch of htop sessions on request by every users at any time, if you can run your workflow at a certain time I'll have a session ready to wach the execution. Later today, or in the late morning tomorrow. Otherwise we must go to the next week.

Good morning Andrea,
I expected that you are able to run the workflow yourself at any time it suits you, in order to check in htop if really all cores can be used by the model/workflow in your infrastructure. But if you need my help for that, for sure, I am very willing to provide that.

Today, I would like to invest time to finally solve that issue. If you don't mind we can have a webmeeting were we share screens and you show me that.

In addition, I would like to learn how I would have extracted this information from the grafana tool.

Please let me know what time. I think Leonardo can setup a GOTO meeting for us.

Best,
Lars

Hi @g.panichi@isti.cnr.it

looks like my previous message got lost while uploading.

Thank you for taking over! The workflow was already mentioned in the ticket opening. Here is the get request from SHOW:

http://dataminer-genericworkers-proto.d4science.org/wps/WebProcessingService?request=Execute&service=WPS&Version=1.0.0&gcube-token=USERTOKEN&lang=en-US&Identifier=org.gcube.dataanalysis.wps.statisticalmanager.synchserver.mappedclasses.transducerers.DM15CORESUSE&DataInputs=dummy=dummy;

Since many tickets are public I removed the user token.

As already explained above, all I need to know is how many cores are really used by the workflow on the server. I usually easily check that with HTOP.

HTH,

Lars

@lars.valentin@bfr.bund.de thanks for the explanation. What was puzzling me was the comment "multicore use does now behave as expected" ... this sound like CNR changed something to make the cluster working as expected ... this is not the case since the beginning of this thread.

BTW, I'm happy to close this ticket and to consider the activity complete.

I add examples of what CNR said in the past:

https://support.d4science.org/issues/12614#note-5
https://support.d4science.org/issues/12614#note-8

Lars Valentin wrote:

Why do you say I am fighting? I was kindly asking why we have this gap in what CNR said before and does say now.

As you know, this thread was opened because you suggested "open a new ticket declaring the testing / assessment plan (e.g. algorithms to be integrated and tested, load, concurrency, number of users served in parallel) and use the new ticket to report on the effectiveness of the proposed solution." here https://support.d4science.org/issues/12614#note-8 .

All I wanted to do was following your suggestion and trying to test if the NEW setting is working as expected. And I can't say that CNR was not really supportive in that topic for a long time.

So please start to consider the ticket in the context it was and is, I am just following your advise.

@leonardo.candela@isti.cnr.it and @lars.valentin@bfr.bund.de we have a meeting in 20 minutes can we discuss there the issue and resolve any misunderstanding? I don't think this is a fruitful discussion

I will add a bit of technical information since I believe it is not yet clear.

D4Science operated by CNR supports two configurations for the computing clusters assigned to the VREs. We shortly called: (joblength =4) and (joblength =1).

With the first (joblength =4), each server allocated to the cluster accepts up to 4 jobs at a time. This does mean that
-- each server may run 1, 2, 3, 4 jobs at a timet. 4 jobs are the maximum number of jobs accepted by the server before queueing.
-- each job may use up to 15 cores. Clearly, if 4 jobs are running at the same time, they will compete to access the same set of cores and the OS will change contexts several times. This will not be efficient but still supported.

With the first (joblength =1), each server allocated to the cluster accepts up to 1 job at a time. Each job may use up to 15 cores without competing for the computing resources.

Both configurations have pros and cons and this is why we support both. It really depends on the use cases. The first case makes a better use of the computational resources, while with the second configuration we risk to unexploit computational resources that are kept unallocated.

@pasquale.pagano@isti.cnr.it

Thanks a lot for the detailed explanation!

In the past I always understood that the joblength=4 is somehow setup with the limitation that 1 job can use only 4 cores max. Now I know it just behaves like a normal system. I see both advantages and disadvantages and will take them into account for the further development plans.

Best,
Lars