D4Science Infrastructure

@massimiliano.assante@isti.cnr.it, do we need another server? Can we allocate it on an existing one?

@massimiliano.assante@isti.cnr.it: according to the requirements I have, Apache is a good solution. In general it opens a proxy from our internal network to the external (from a security point of view I would be worried by the opposite): anyway, if the HTTP dynamic content is limited we could limit the endpoints.
However I am very far from saying that we have the solution: for this reason I would like to test this idea in an environment similar to the destination one with the portlets that currently are problematic (or a subset of them). If there are more requirements that I could have missed, and could make this solution unapplyable please let me know.

Ciro Formisano wrote:

@massimiliano.assante@isti.cnr.it: according to the requirements I have, Apache is a good solution. In general it opens a proxy from our internal network to the external (from a security point of view I would be worried by the opposite): anyway, if the HTTP dynamic content is limited we could limit the endpoints.

The proxy will be accessed by the portlets, correct? So we can limit the access using iptables rules and maybe apache ACLs

However I am very far from saying that we have the solution: for this reason I would like to test this idea in an environment similar to the destination one with the portlets that currently are problematic (or a subset of them). If there are more requirements that I could have missed, and could make this solution unapplyable please let me know.

Is the 2.4.8 version mandatory? Ubuntu 14.04 provides apache 2.4.7 and that's our platform right now.

Hi Andrea,
I am not sure that the proxy will be accessed only by the Portlets. As far as I know it is not a "backend" of the Portal, but it should hide some services accessed directly by the client (browser but not only) that are in HTTP and this leads to errors since the browser expects to use HTTPs. Anyway @francesco.mangiacrapa@isti.cnr.it probably can be more detailed than me.

Concerning the version of apache, unluckily I need some features available from 2.4.8: I have a Debian and the version available in the repositories is 2.4.10 (on which I made the tests)

Ciro Formisano wrote:

Hi Andrea,
I am not sure that the proxy will be accessed only by the Portlets. As far as I know it is not a "backend" of the Portal, but it should hide some services accessed directly by the client (browser but not only) that are in HTTP and this leads to errors since the browser expects to use HTTPs. Anyway @francesco.mangiacrapa@isti.cnr.it probably can be more detailed than me.

It that's the case @massimiliano.assante@isti.cnr.it is right: this is going to be a huge security hole because we are activating an open proxy and letting it available to everyone. So it's not a viable solution.

Concerning the version of apache, unluckily I need some features available from 2.4.8: I have a Debian and the version available in the repositories is 2.4.10 (on which I made the tests)

nginx could be an option, possibly?

It that's the case @massimiliano.assante@isti.cnr.it is right: this is going to be a huge security hole because we are activating an open proxy and letting it available to everyone. So it's not a viable solution.

The problem you are pointing could be related to the solution, not to its implementation. For the moment we have some services accessed in HTTP on behalf of HTTPS: actually I don't know how these services are protected at portal level and if the new solution adds or remove current security issues (if present). I would evaluate also these problems by the test session.

nginx could be an option, possibly?

Version 2.4.x is OK for everything except for backreferences of locationmatch (https://httpd.apache.org/docs/2.4/mod/core.html#locationmatch), available from 2.4.8.

Ciro Formisano wrote:

The problem you are pointing could be related to the solution, not to its implementation. For the moment we have some services accessed in HTTP on behalf of HTTPS: actually I don't know how these services are protected at portal level and if the new solution adds or remove current security issues (if present). I would evaluate also these problems by the test session.

So what you're saying is that only a (fixed) list of target URLs will be accepted?

Version 2.4.x is OK for everything except for backreferences of locationmatch (https://httpd.apache.org/docs/2.4/mod/core.html#locationmatch), available from 2.4.8.

I've found a PPA for apache2, btw: https://launchpad.net/~ondrej/+archive/ubuntu/apache2?field.series_filter=trusty

So what you're saying is that only a (fixed) list of target URLs will be accepted?

No, I am saying that at this stage we have to test the candidate solution and evaluate functional and not functional problems (if present). Then we will fix them (if possible) otherwise we will find another solution. Your proposal could be OK but I have to test it.

I've found a PPA for apache2, btw: https://launchpad.net/~ondrej/+archive/ubuntu/apache2?field.series_filter=trusty

It seems that it provides apache 2.4.25. Should be OK

@andrea.dellamico@isti.cnr.it and @ciro.formisano@eng.it thanks for your replies

A question:
Can we add one or more configuration (security policies) through which the proxy will be accessible: either from LAN or by a list of ip addresses otherwise by gcube token and/or etc..??
I think yes.
So the security is not a problem (at least I hope).
At the moment, the issue is that It is not possible to access to dynamic HTTP content from our portals.. and a solution could be an https to http proxy adding obviously our policies to the security.

Francesco Mangiacrapa wrote:

@andrea.dellamico@isti.cnr.it and @ciro.formisano@eng.it thanks for your replies

A question:
Can we add one or more configuration (security policies) through which the proxy will be accessible: either from LAN or by a list of ip addresses otherwise by gcube token and/or etc..??
I think yes.
So the security is not a problem (at least I hope).

It's what I was asking in #note-10. network or hosts restrictions are not a problem. I don't know how it could be possible using a gcube token.

@francesco.mangiacrapa@isti.cnr.it thank you for the reply.

I would only avoid, for the moment, to add any specific security policy while we are performing tests. I think that we will need and I also think that it will not be difficult to define and apply them, but if we apply security at this time I fear that we will have problems to clearly understand if the functionalities of the solution are OK.
For this reason I asked @andrea.dellamico@isti.cnr.it to not think about security for the moment (even if we consider the problem).

Andrea Dell'Amico wrote:

It's what I was asking in #note-10. network or hosts restrictions are not a problem. I don't know how it could be possible using a gcube token.

You're right @andrea.dellamico@isti.cnr.it. Security based on gcube-token could be very hard (impossible?) to be implemented/integrated via Apache.
I think that network or hosts restrictions are sufficient even because is not need to perform accounting for the request passed via proxy.. :-) Right? :-)

two short comments:
a) we have a concrete issue and we need a solution. Please don't try to solve all possible problems.
the issue is that It is not possible to access to dynamic HTTP content from our portals running on HTTPS.
The solution we though as to define an https to http proxy.
I am not an expert but the proxy is only for outgoing calls from our network to the web and from our portal only. So rules can be formulated under those considerations.

b) security is an issue even during the test phase. We cannot open to the world risking to compromise the infrastructure we run. So, the testing phase cannot be started without the right security configuration

Pasquale Pagano wrote:

two short comments:
a) we have a concrete issue and we need a solution. Please don't try to solve all possible problems.
the issue is that It is not possible to access to dynamic HTTP content from our portals running on HTTPS.
The solution we though as to define an https to http proxy.
I am not an expert but(alls from our network to the web and from our portal only. So rules can be formulated under those considerations.

Nope, any browser contacts the proxy from anywhere.

I was clearly wrong but even accepting requests from anywhere we know the target of the calls to accept. So the filters can be set on the target and not on the source.

Thank you Andrea,
I will work on that machine as soon as @massimiliano.assante@isti.cnr.it , @francesco.mangiacrapa@isti.cnr.it and me will have agreed on the best solution.
Meanwhile I should get access to this machine: may you help me?

Ciro Formisano wrote:

Thank you Andrea,
I will work on that machine as soon as @massimiliano.assante@isti.cnr.it , @francesco.mangiacrapa@isti.cnr.it and me will have agreed on the best solution.
Meanwhile I should get access to this machine: may you help me?

I forgot to say that you should have ssh access as root already. I've found a ssh key of yours on manage.research-infrastructures.eu. If it's not valid anymore you can add a ssh key into your profile data on the portal.

@andrea.dellamico@isti.cnr.it ASAP we need a smartgears machine running in the scope "/gcube/devNext/NextNext"
which is the scope where is up a Proxy Portlet for testing it https://dev4.d4science.org/group/nextnext/test-proxy, thx

The VM is going to be converted to smartgears, apache removed

Note that as it's a development server, it will run Java 8.

looking at ghn.log I see several exceptions, continuously produced such as:

 ERROR ProfileManager: cannot publish container (see details)
com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: java.net.ConnectException: Connection refused (Connection refused)
        at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:117) ~[na:1.8.0_121]
        at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:208) ~[na:1.8.0_121]
        at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:130) ~[na:1.8.0_121]
        at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:124) ~[na:1.8.0_121]
        at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:1121) ~[na:1.8.0_121]
        at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:1035) ~[na:1.8.0_121]
        at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:1004) ~[na:1.8.0_121]
        at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:862) ~[na:1.8.0_121]
        at com.sun.xml.internal.ws.client.Stub.process(Stub.java:448) ~[na:1.8.0_121]
        at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(SEIStub.java:178) ~[na:1.8.0_121]
        at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:93) ~[na:1.8.0_121]
        at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:77) ~[na:1.8.0_121]
        at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:147) ~[na:1.8.0_121]
        at com.sun.proxy.$Proxy48.update(Unknown Source) ~[na:na]
        at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source) ~[na:na]

Tomcat autodeploy was missed, could you please check again?