Incident #232
closednode51.p.d4science.research-infrastructures.eu webapp unresponsive. The ajp port is open with connections from China
100%
Description
The 8080 (and 8005) tomcat port has been closed since last night.
The actual 'netstat -na' output is:
Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:4949 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:8627 0.0.0.0:* LISTEN tcp 0 0 146.48.122.127:22 146.48.123.11:54985 ESTABLISHED tcp6 0 0 :::22 :::* LISTEN tcp6 0 0 :::45339 :::* LISTEN tcp6 0 0 :::4000 :::* LISTEN tcp6 0 0 :::4001 :::* LISTEN tcp6 0 0 146.48.122.127:8009 61.240.144.65:60000 ESTABLISHED tcp6 0 0 146.48.122.127:8009 61.240.144.67:60000 ESTABLISHED tcp6 0 0 146.48.122.127:42774 146.48.122.255:6166 ESTABLISHED udp 0 0 146.48.122.127:55702 239.2.10.67:8627 ESTABLISHED udp 0 0 146.48.122.127:8627 0.0.0.0:* udp 0 0 0.0.0.0:8627 0.0.0.0:* udp6 124860 0 :::4446 :::* udp6 0 0 :::4446 :::*
The two IPs connected to the 8009 (ajp) port are from a chinese network.
The 4446 udp port has been opened by tomcat or one of the running apps, but I don't know the reason.
The load average is costantly around 1.00, while there's no memory pressure.
I completely stopped the tomcat container to get rid of the rogue connections.
Updated by Roberto Cirillo almost 10 years ago
On node51.p (pre production host) there are the following services:
- executionengineservice-search
- searchsystemservice
I'm going to analyze the logs
Updated by Roberto Cirillo almost 10 years ago
If there isn't an apache frontend on this host, I think is better to disable the 8009 port in the server.xml file. I think someone could abuse of 8009 port as happened in the latest days.
What do you think about it?
Updated by Andrea Dell'Amico almost 10 years ago
It surely needs to be closed.
The ajp port is always of no use on our systems, because we always use the http port even for the apache frontends.
Even if we wanted to use it, it should be configured to bind on localhost only.
I'm more worried about the udp port 4446, is it open by one of the d4science services?
Updated by Roberto Cirillo almost 10 years ago
I don't know this. The d4science services hosted on this VM are related to an elastic search cluster and, maybe, this port is used for this. I've added John to this ticket, maybe John can answer to this question.
Updated by John Gerbesiotis almost 10 years ago
Searchsystem service communicates with elasticsearch, which is on another vm and with pottal.
To my knowledge, elasticsearch does not use the port 4446. Moreover, searchsystem exploits elasticsearch through http rest api, so there is no need for other communication to elasticsearch node. Portal exploits searchsearvice throught resultset which communicates at another port (e.g. 4000-4010).
I searched logs of elasticsearch, searchsystem and portal of the development environment and didn't find 4446 port exploitation.
At least, we could limit access to the production subnet? Otherwise, if there is a change in firewall, we have to check the status of the services.
Updated by Roberto Cirillo almost 10 years ago
as John said, for this VM I think is better to limit access to our subnet. The services hosted on node51.p have to communicate with: DTS, index-service, portal, Information-System and Registry of preproduction environment. All these services are deployed to CNR, so, I think we could try to restrict the access to the CNR net.
Updated by John Gerbesiotis almost 10 years ago
Well, search services should be publicly available for REST APIs also, as such, 8080 should be public at least. We could only protect irrelevant ports with the services, such as 8009. Correct me if I am wrong.
Updated by Andrea Dell'Amico almost 10 years ago
Port 8009 is now closed. What's worrying me is udp 4446, it's opened by one of the tomcat webapps but I don't know if it's used by some services.
Updated by John Gerbesiotis almost 10 years ago
I suggest limiting 4446 to subnet only.
Updated by Roberto Cirillo almost 10 years ago
I've notice on node51.p there are the ehcache jars. I don't know how the ehcache is configured but I've seen that a kind of setting is:
<cacheManagerPeerProviderFactory class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory" properties="peerDiscovery=automatic, multicastGroupAddress=230.0.0.1, multicastGroupPort=4446, timeToLive=32"/>
with multicastGroupPort=4446....
Maybe the 4446 port is used by ehcache ?
Updated by John Gerbesiotis almost 10 years ago
Yes searchsystem uses ehcache and the default port of ehcache is 4446.
Updated by Roberto Cirillo almost 10 years ago
- % Done changed from 0 to 100
I think we can close the ticket now
Updated by Roberto Cirillo almost 10 years ago
- Status changed from New to In Progress
- % Done changed from 100 to 90
Andrea, do you want to add an iptables rule for this port?
Updated by Andrea Dell'Amico almost 10 years ago
John Gerbesiotis wrote:
Yes searchsystem uses ehcache and the default port of ehcache is 4446.
And that port is effectively used to talk with other instances?
In a tcpdump session I've seen only broadcast traffic from other - completely unrelated - servers that happen to live on the same network.
Andrea, do you want to add an iptables rule for this port?
Yes. I'd limit it to localhost only if it does not need to talk to anyone else.
Updated by John Gerbesiotis almost 10 years ago
No it is a feature of ehcache that is not being exploited. Searchsystem's ehcache does not talk with other service instances. It is enabled by default.
Updated by Andrea Dell'Amico almost 10 years ago
- Status changed from In Progress to Closed
- % Done changed from 90 to 100
The iptables rules are active.