Project

General

Profile

Actions

Incident #232

closed

node51.p.d4science.research-infrastructures.eu webapp unresponsive. The ajp port is open with connections from China

Added by Andrea Dell'Amico almost 10 years ago. Updated almost 10 years ago.

Status:
Closed
Priority:
Urgent
Category:
System Application
Target version:
Start date:
Jun 06, 2015
Due date:
% Done:

100%

Estimated time:
Infrastructure:
Production

Description

The 8080 (and 8005) tomcat port has been closed since last night.
The actual 'netstat -na' output is:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:4949            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:5666            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:8627            0.0.0.0:*               LISTEN
tcp        0      0 146.48.122.127:22       146.48.123.11:54985     ESTABLISHED
tcp6       0      0 :::22                   :::*                    LISTEN
tcp6       0      0 :::45339                :::*                    LISTEN
tcp6       0      0 :::4000                 :::*                    LISTEN
tcp6       0      0 :::4001                 :::*                    LISTEN
tcp6       0      0 146.48.122.127:8009     61.240.144.65:60000     ESTABLISHED
tcp6       0      0 146.48.122.127:8009     61.240.144.67:60000     ESTABLISHED
tcp6       0      0 146.48.122.127:42774    146.48.122.255:6166     ESTABLISHED
udp        0      0 146.48.122.127:55702    239.2.10.67:8627        ESTABLISHED
udp        0      0 146.48.122.127:8627     0.0.0.0:*
udp        0      0 0.0.0.0:8627            0.0.0.0:*
udp6  124860      0 :::4446                 :::*
udp6       0      0 :::4446                 :::*

The two IPs connected to the 8009 (ajp) port are from a chinese network.
The 4446 udp port has been opened by tomcat or one of the running apps, but I don't know the reason.
The load average is costantly around 1.00, while there's no memory pressure.

I completely stopped the tomcat container to get rid of the rogue connections.

Actions #1

Updated by Roberto Cirillo almost 10 years ago

On node51.p (pre production host) there are the following services:

  • executionengineservice-search
  • searchsystemservice

I'm going to analyze the logs

Actions #2

Updated by Roberto Cirillo almost 10 years ago

If there isn't an apache frontend on this host, I think is better to disable the 8009 port in the server.xml file. I think someone could abuse of 8009 port as happened in the latest days.
What do you think about it?

Actions #3

Updated by Andrea Dell'Amico almost 10 years ago

It surely needs to be closed.

The ajp port is always of no use on our systems, because we always use the http port even for the apache frontends.
Even if we wanted to use it, it should be configured to bind on localhost only.

I'm more worried about the udp port 4446, is it open by one of the d4science services?

Actions #4

Updated by Roberto Cirillo almost 10 years ago

I don't know this. The d4science services hosted on this VM are related to an elastic search cluster and, maybe, this port is used for this. I've added John to this ticket, maybe John can answer to this question.

Actions #5

Updated by John Gerbesiotis almost 10 years ago

Searchsystem service communicates with elasticsearch, which is on another vm and with pottal.

To my knowledge, elasticsearch does not use the port 4446. Moreover, searchsystem exploits elasticsearch through http rest api, so there is no need for other communication to elasticsearch node. Portal exploits searchsearvice throught resultset which communicates at another port (e.g. 4000-4010).

I searched logs of elasticsearch, searchsystem and portal of the development environment and didn't find 4446 port exploitation.

At least, we could limit access to the production subnet? Otherwise, if there is a change in firewall, we have to check the status of the services.

Actions #6

Updated by Roberto Cirillo almost 10 years ago

as John said, for this VM I think is better to limit access to our subnet. The services hosted on node51.p have to communicate with: DTS, index-service, portal, Information-System and Registry of preproduction environment. All these services are deployed to CNR, so, I think we could try to restrict the access to the CNR net.

Actions #7

Updated by John Gerbesiotis almost 10 years ago

Well, search services should be publicly available for REST APIs also, as such, 8080 should be public at least. We could only protect irrelevant ports with the services, such as 8009. Correct me if I am wrong.

Actions #8

Updated by Andrea Dell'Amico almost 10 years ago

Port 8009 is now closed. What's worrying me is udp 4446, it's opened by one of the tomcat webapps but I don't know if it's used by some services.

Actions #9

Updated by John Gerbesiotis almost 10 years ago

I suggest limiting 4446 to subnet only.

Actions #10

Updated by Roberto Cirillo almost 10 years ago

I've notice on node51.p there are the ehcache jars. I don't know how the ehcache is configured but I've seen that a kind of setting is:

<cacheManagerPeerProviderFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
properties="peerDiscovery=automatic, multicastGroupAddress=230.0.0.1,
multicastGroupPort=4446, timeToLive=32"/>

with multicastGroupPort=4446....
Maybe the 4446 port is used by ehcache ?

Actions #11

Updated by John Gerbesiotis almost 10 years ago

Yes searchsystem uses ehcache and the default port of ehcache is 4446.

Actions #12

Updated by Roberto Cirillo almost 10 years ago

  • % Done changed from 0 to 100

I think we can close the ticket now

Actions #13

Updated by Roberto Cirillo almost 10 years ago

  • Status changed from New to In Progress
  • % Done changed from 100 to 90

Andrea, do you want to add an iptables rule for this port?

Actions #14

Updated by Andrea Dell'Amico almost 10 years ago

John Gerbesiotis wrote:

Yes searchsystem uses ehcache and the default port of ehcache is 4446.

And that port is effectively used to talk with other instances?
In a tcpdump session I've seen only broadcast traffic from other - completely unrelated - servers that happen to live on the same network.

Andrea, do you want to add an iptables rule for this port?

Yes. I'd limit it to localhost only if it does not need to talk to anyone else.

Actions #15

Updated by John Gerbesiotis almost 10 years ago

No it is a feature of ehcache that is not being exploited. Searchsystem's ehcache does not talk with other service instances. It is enabled by default.

Actions #16

Updated by Andrea Dell'Amico almost 10 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 90 to 100

The iptables rules are active.

Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 8.91 MB)