Hello,
I am trying to limit users from hijacking other ip’s other than assigned by Opennebula.
Testcase:
Automatically assign a ip to a VM. Let’s say ip 10.0.0.1
User want to abuse the system and manually sets the ip in the VM to ip 10.0.0.2 once booted and online
This new ip 10.0.0.2 is working and reachable(!)
I am using Centos 7 on my Hypervisors and Firewalld for port control/security.
For Virtual Network I use the following config (Bridge + security), see image. Settings network
I am NOT using ‘FILTER = “clean-traffic”’ in my template in this scenario. The problem with this setting is that this prevents hijacking BUT I can’t make a secondary ip reachable…?
Maybe there is a setting that I am not aware of that needs to be set to make FILTER_IP_SPOOFING working?
We really want to use Opennebula, but the ability to hijack ip’s prevents us from using it in production.
Thanks in advance!!
ipspoof1|690x448
ahuertas
April 15, 2019, 11:16am
2
Hello @boomstammetje
Could you please send me the output of:
onevnet show 0 -x.
onevm show <VM_ID> -x.
iptables-save Note: execute the command in the hypervisor where the VM is running.
Thanks for getting back to me @ahuertas
onevnet show 0 -x
<VNET>
<ID>0</ID>
<UID>0</UID>
<GID>0</GID>
<UNAME>oneadmin</UNAME>
<GNAME>oneadmin</GNAME>
<NAME>public1</NAME>
<PERMISSIONS>
<OWNER_U>1</OWNER_U>
<OWNER_M>1</OWNER_M>
<OWNER_A>0</OWNER_A>
<GROUP_U>0</GROUP_U>
<GROUP_M>0</GROUP_M>
<GROUP_A>0</GROUP_A>
<OTHER_U>0</OTHER_U>
<OTHER_M>0</OTHER_M>
<OTHER_A>0</OTHER_A>
</PERMISSIONS>
<CLUSTERS>
<ID>0</ID>
</CLUSTERS>
<BRIDGE><![CDATA[br0]]></BRIDGE>
<BRIDGE_TYPE><![CDATA[linux]]></BRIDGE_TYPE>
<PARENT_NETWORK_ID/>
<VN_MAD><![CDATA[fw]]></VN_MAD>
<PHYDEV/>
<VLAN_ID/>
<OUTER_VLAN_ID/>
<VLAN_ID_AUTOMATIC>0</VLAN_ID_AUTOMATIC>
<OUTER_VLAN_ID_AUTOMATIC>0</OUTER_VLAN_ID_AUTOMATIC>
<USED_LEASES>1</USED_LEASES>
<VROUTERS/>
<TEMPLATE>
<BRIDGE><![CDATA[br0]]></BRIDGE>
<BRIDGE_TYPE><![CDATA[linux]]></BRIDGE_TYPE>
<DESCRIPTION><![CDATA[serverius]]></DESCRIPTION>
<DNS><![CDATA[8.8.8.8 8.8.4.4]]></DNS>
<FILTER_IP_SPOOFING><![CDATA[YES]]></FILTER_IP_SPOOFING>
<FILTER_MAC_SPOOFING><![CDATA[YES]]></FILTER_MAC_SPOOFING>
<GATEWAY><![CDATA[5.18.165.161]]></GATEWAY>
<NETWORK_ADDRESS><![CDATA[5.18.165.160]]></NETWORK_ADDRESS>
<NETWORK_MASK><![CDATA[255.255.255.240]]></NETWORK_MASK>
<PHYDEV><![CDATA[]]></PHYDEV>
<SECURITY_GROUPS><![CDATA[0]]></SECURITY_GROUPS>
<VN_MAD><![CDATA[fw]]></VN_MAD>
</TEMPLATE>
<AR_POOL>
<AR>
<AR_ID><![CDATA[0]]></AR_ID>
<IP><![CDATA[5.18.165.169]]></IP>
<MAC><![CDATA[02:00:05:b2:41:a9]]></MAC>
<SIZE><![CDATA[3]]></SIZE>
<TYPE><![CDATA[IP4]]></TYPE>
<MAC_END><![CDATA[02:00:05:b2:41:ab]]></MAC_END>
<IP_END><![CDATA[5.18.165.171]]></IP_END>
<USED_LEASES>1</USED_LEASES>
<LEASES>
<LEASE>
<IP><![CDATA[5.18.165.169]]></IP>
<MAC><![CDATA[02:00:05:b2:41:a9]]></MAC>
<VM><![CDATA[1]]></VM>
</LEASE>
</LEASES>
</AR>
</AR_POOL>
</VNET>
onevm show 1 -x
<VM>
<ID>1</ID>
<UID>2</UID>
<GID>1</GID>
<UNAME>martin</UNAME>
<GNAME>users</GNAME>
<NAME>CentOS 7 - KVM-1</NAME>
<PERMISSIONS>
<OWNER_U>1</OWNER_U>
<OWNER_M>1</OWNER_M>
<OWNER_A>0</OWNER_A>
<GROUP_U>0</GROUP_U>
<GROUP_M>0</GROUP_M>
<GROUP_A>0</GROUP_A>
<OTHER_U>0</OTHER_U>
<OTHER_M>0</OTHER_M>
<OTHER_A>0</OTHER_A>
</PERMISSIONS>
<LAST_POLL>1555326307</LAST_POLL>
<STATE>8</STATE>
<LCM_STATE>0</LCM_STATE>
<PREV_STATE>8</PREV_STATE>
<PREV_LCM_STATE>0</PREV_LCM_STATE>
<RESCHED>0</RESCHED>
<STIME>1555321516</STIME>
<ETIME>0</ETIME>
<DEPLOY_ID>one-1</DEPLOY_ID>
<MONITORING>
<CPU><![CDATA[0.0]]></CPU>
<DISKRDBYTES><![CDATA[155862270]]></DISKRDBYTES>
<DISKRDIOPS><![CDATA[8857]]></DISKRDIOPS>
<DISKWRBYTES><![CDATA[28488192]]></DISKWRBYTES>
<DISKWRIOPS><![CDATA[1402]]></DISKWRIOPS>
<DISK_SIZE>
<ID><![CDATA[0]]></ID>
<SIZE><![CDATA[535]]></SIZE>
</DISK_SIZE>
<DISK_SIZE>
<ID><![CDATA[1]]></ID>
<SIZE><![CDATA[1]]></SIZE>
</DISK_SIZE>
<MEMORY><![CDATA[0]]></MEMORY>
<NETRX><![CDATA[845169]]></NETRX>
<NETTX><![CDATA[863575]]></NETTX>
<STATE><![CDATA[a]]></STATE>
</MONITORING>
<TEMPLATE>
<AUTOMATIC_DS_REQUIREMENTS><![CDATA[("CLUSTERS/ID" @> 0)]]></AUTOMATIC_DS_REQUIREMENTS>
<AUTOMATIC_NIC_REQUIREMENTS><![CDATA[("CLUSTERS/ID" @> 0)]]></AUTOMATIC_NIC_REQUIREMENTS>
<AUTOMATIC_REQUIREMENTS><![CDATA[(CLUSTER_ID = 0) & !(PUBLIC_CLOUD = YES)]]></AUTOMATIC_REQUIREMENTS>
<CONTEXT>
<DISK_ID><![CDATA[1]]></DISK_ID>
<ETH0_CONTEXT_FORCE_IPV4><![CDATA[]]></ETH0_CONTEXT_FORCE_IPV4>
<ETH0_DNS><![CDATA[8.8.8.8 8.8.4.4]]></ETH0_DNS>
<ETH0_EXTERNAL><![CDATA[]]></ETH0_EXTERNAL>
<ETH0_GATEWAY><![CDATA[5.18.165.161]]></ETH0_GATEWAY>
<ETH0_GATEWAY6><![CDATA[]]></ETH0_GATEWAY6>
<ETH0_IP><![CDATA[5.18.165.169]]></ETH0_IP>
<ETH0_IP6><![CDATA[]]></ETH0_IP6>
<ETH0_IP6_PREFIX_LENGTH><![CDATA[]]></ETH0_IP6_PREFIX_LENGTH>
<ETH0_IP6_ULA><![CDATA[]]></ETH0_IP6_ULA>
<ETH0_MAC><![CDATA[02:00:05:b2:41:a9]]></ETH0_MAC>
<ETH0_MASK><![CDATA[255.255.255.240]]></ETH0_MASK>
<ETH0_MTU><![CDATA[]]></ETH0_MTU>
<ETH0_NETWORK><![CDATA[5.18.165.160]]></ETH0_NETWORK>
<ETH0_SEARCH_DOMAIN><![CDATA[]]></ETH0_SEARCH_DOMAIN>
<ETH0_VLAN_ID><![CDATA[]]></ETH0_VLAN_ID>
<ETH0_VROUTER_IP><![CDATA[]]></ETH0_VROUTER_IP>
<ETH0_VROUTER_IP6><![CDATA[]]></ETH0_VROUTER_IP6>
<ETH0_VROUTER_MANAGEMENT><![CDATA[]]></ETH0_VROUTER_MANAGEMENT>
<NETWORK><![CDATA[YES]]></NETWORK>
<SSH_PUBLIC_KEY></SSH_PUBLIC_KEY>
<TARGET><![CDATA[hda]]></TARGET>
</CONTEXT>
<CPU><![CDATA[1]]></CPU>
<DISK>
<ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS>
<CLONE><![CDATA[YES]]></CLONE>
<CLONE_TARGET><![CDATA[SYSTEM]]></CLONE_TARGET>
<CLUSTER_ID><![CDATA[0]]></CLUSTER_ID>
<DATASTORE><![CDATA[default]]></DATASTORE>
<DATASTORE_ID><![CDATA[1]]></DATASTORE_ID>
<DEV_PREFIX><![CDATA[vd]]></DEV_PREFIX>
<DISK_ID><![CDATA[0]]></DISK_ID>
<DISK_SNAPSHOT_TOTAL_SIZE><![CDATA[0]]></DISK_SNAPSHOT_TOTAL_SIZE>
<DISK_TYPE><![CDATA[FILE]]></DISK_TYPE>
<DRIVER><![CDATA[qcow2]]></DRIVER>
<IMAGE><![CDATA[CentOS 7 - KVM]]></IMAGE>
<IMAGE_ID><![CDATA[0]]></IMAGE_ID>
<IMAGE_STATE><![CDATA[2]]></IMAGE_STATE>
<LN_TARGET><![CDATA[SYSTEM]]></LN_TARGET>
<ORIGINAL_SIZE><![CDATA[8192]]></ORIGINAL_SIZE>
<READONLY><![CDATA[NO]]></READONLY>
<SAVE><![CDATA[NO]]></SAVE>
<SIZE><![CDATA[8192]]></SIZE>
<SOURCE><![CDATA[/var/lib/one//datastores/1/ec272b699d89ee4cfd7e519e54a1100a]]></SOURCE>
<TARGET><![CDATA[vda]]></TARGET>
<TM_MAD><![CDATA[ssh]]></TM_MAD>
<TYPE><![CDATA[FILE]]></TYPE>
</DISK>
<GRAPHICS>
<LISTEN><![CDATA[0.0.0.0]]></LISTEN>
<PORT><![CDATA[5901]]></PORT>
<TYPE><![CDATA[VNC]]></TYPE>
</GRAPHICS>
<MEMORY><![CDATA[1024]]></MEMORY>
<NIC>
<AR_ID><![CDATA[0]]></AR_ID>
<BRIDGE><![CDATA[br0]]></BRIDGE>
<BRIDGE_TYPE><![CDATA[linux]]></BRIDGE_TYPE>
<CLUSTER_ID><![CDATA[0]]></CLUSTER_ID>
<FILTER_IP_SPOOFING><![CDATA[YES]]></FILTER_IP_SPOOFING>
<FILTER_MAC_SPOOFING><![CDATA[YES]]></FILTER_MAC_SPOOFING>
<IP><![CDATA[5.18.165.169]]></IP>
<MAC><![CDATA[02:00:05:b2:41:a9]]></MAC>
<NAME><![CDATA[NIC0]]></NAME>
<NETWORK><![CDATA[public1]]></NETWORK>
<NETWORK_ID><![CDATA[0]]></NETWORK_ID>
<NIC_ID><![CDATA[0]]></NIC_ID>
<SECURITY_GROUPS><![CDATA[0]]></SECURITY_GROUPS>
<TARGET><![CDATA[one-1-0]]></TARGET>
<VN_MAD><![CDATA[fw]]></VN_MAD>
</NIC>
<OS>
<ARCH><![CDATA[x86_64]]></ARCH>
<BOOT><![CDATA[]]></BOOT>
</OS>
<SECURITY_GROUP_RULE>
<PROTOCOL><![CDATA[ALL]]></PROTOCOL>
<RULE_TYPE><![CDATA[OUTBOUND]]></RULE_TYPE>
<SECURITY_GROUP_ID><![CDATA[0]]></SECURITY_GROUP_ID>
<SECURITY_GROUP_NAME><![CDATA[default]]></SECURITY_GROUP_NAME>
</SECURITY_GROUP_RULE>
<SECURITY_GROUP_RULE>
<PROTOCOL><![CDATA[ALL]]></PROTOCOL>
<RULE_TYPE><![CDATA[INBOUND]]></RULE_TYPE>
<SECURITY_GROUP_ID><![CDATA[0]]></SECURITY_GROUP_ID>
<SECURITY_GROUP_NAME><![CDATA[default]]></SECURITY_GROUP_NAME>
</SECURITY_GROUP_RULE>
<TEMPLATE_ID><![CDATA[0]]></TEMPLATE_ID>
<TM_MAD_SYSTEM><![CDATA[ssh]]></TM_MAD_SYSTEM>
<VMID><![CDATA[1]]></VMID>
</TEMPLATE>
<USER_TEMPLATE>
<INPUTS_ORDER><![CDATA[]]></INPUTS_ORDER>
<LOGO><![CDATA[images/logos/centos.png]]></LOGO>
<MEMORY_UNIT_COST><![CDATA[MB]]></MEMORY_UNIT_COST>
</USER_TEMPLATE>
<HISTORY_RECORDS>
<HISTORY>
<OID>1</OID>
<SEQ>0</SEQ>
<HOSTNAME>5.18.165.166</HOSTNAME>
<HID>1</HID>
<CID>0</CID>
<STIME>1555321540</STIME>
<ETIME>1555326307</ETIME>
<VM_MAD><![CDATA[kvm]]></VM_MAD>
<TM_MAD><![CDATA[ssh]]></TM_MAD>
<DS_ID>0</DS_ID>
<PSTIME>1555321540</PSTIME>
<PETIME>1555321547</PETIME>
<RSTIME>1555321547</RSTIME>
<RETIME>1555326307</RETIME>
<ESTIME>0</ESTIME>
<EETIME>0</EETIME>
<ACTION>19</ACTION>
<UID>2</UID>
<GID>1</GID>
<REQUEST_ID>5664</REQUEST_ID>
</HISTORY>
</HISTORY_RECORDS>
</VM>
iptables-save
ahuertas
April 15, 2019, 11:40am
4
It seems that everything is ok.
Could you please stop the firewalld (systemctl stop firewalld) and disable it (systemctl disable firewalld)
@ahuertas
Ok the spoofing protection works now. Only when I assign a second ip, it isn’t reachable. The main/first ip remains reachable.
Do you also know the answer to that?
Thanks for helping out!
ahuertas
April 15, 2019, 2:27pm
6
That is working correctly, the first IP will be always reachable.
Is it possible to assign more than one IP that is protected by IP spoofing?
It doesn’t make sense why the first IP is reachable and the second (alias or not) IP isn’t reachable.
ahuertas
April 16, 2019, 7:42am
8
If you want to have no IP reachable, you need to launch the VM without IP.
OpenNebula will allow traffic on the IPs you assign when creating the VM, that’s the reason why your first IP is reachable.
Hi @ahuertas ,
I want to have all my assigned IP’s to be reachable and protected with IP spoofing. That is the problem, the second IP is not reachable and I want to have it reachable… Only the first assigned IP is reachable, no matter if I select Alias or just a second NIC/adapter.
ahuertas
April 16, 2019, 11:03am
10
Could you please send me again the iptables-save, I need to check it without firewalld now.
iptables-save
# Generated by iptables-save v1.4.21 on Tue Apr 16 13:08:05 2019
*filter
:INPUT ACCEPT [8234:99496761]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [12141:20595253]
:one-15-0-i - [0:0]
:one-15-0-o - [0:0]
:one-15-1-i - [0:0]
:one-15-1-o - [0:0]
:opennebula - [0:0]
-A FORWARD -m physdev --physdev-is-bridged -j opennebula
-A one-15-0-i -m state --state RELATED,ESTABLISHED -j ACCEPT
-A one-15-0-i -j RETURN
-A one-15-0-i -j DROP
-A one-15-0-o -m mac ! --mac-source 02:00:05:B2:41:A9 -j DROP
-A one-15-0-o -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -j ACCEPT
-A one-15-0-o -m set ! --match-set one-15-0-ip-spoofing src -j DROP
-A one-15-0-o -m state --state RELATED,ESTABLISHED -j ACCEPT
-A one-15-0-o -j RETURN
-A one-15-0-o -j DROP
-A one-15-1-i -m state --state RELATED,ESTABLISHED -j ACCEPT
-A one-15-1-i -j RETURN
-A one-15-1-i -j DROP
-A one-15-1-o -m mac ! --mac-source 02:00:05:B2:41:AA -j DROP
-A one-15-1-o -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -j ACCEPT
-A one-15-1-o -m set ! --match-set one-15-1-ip-spoofing src -j DROP
-A one-15-1-o -m state --state RELATED,ESTABLISHED -j ACCEPT
-A one-15-1-o -j RETURN
-A one-15-1-o -j DROP
-A opennebula -m physdev --physdev-in one-15-1 --physdev-is-bridged -j one-15-1-o
-A opennebula -m physdev --physdev-out one-15-1 --physdev-is-bridged -j one-15-1-i
-A opennebula -m physdev --physdev-in one-15-0 --physdev-is-bridged -j one-15-0-o
-A opennebula -m physdev --physdev-out one-15-0 --physdev-is-bridged -j one-15-0-i
-A opennebula -j ACCEPT
COMMIT
# Completed on Tue Apr 16 13:08:05 2019
ahuertas
April 17, 2019, 10:39am
12
Hello @boomstammetje
I can see this:
-A one-15-0-o -m set ! --match-set one-15-0-ip-spoofing src -j DROP
-A one-15-1-o -m set ! --match-set one-15-1-ip-spoofing src -j DROP
It should work, so please:
Check the guest configuration.
You can use the command ipset list to check it.
Check that you didn’t change any MAC, because you have MAC spoofing protection enabled.
Check the routes in the guest.
I assume firewalld and iptables should not be running?
ipset list
Name: one-25-1-ip-spoofing
Name: one-25-2-ip-spoofing
Only the first ip is reachable. I am really clueless where this goes wrong.
Even with IP-spoofing and MAC-spoofing set to OFF, a secondary or third IP isn’t reachable
ahuertas
April 22, 2019, 9:02am
15
Hello @boomstammetje
It seems that there is a configuration problem in the guest or in the switch. OpenNebula created the rules correctly.
You can use tcpdump to debug and see where the traffic is being filtered out.
darkfader
April 24, 2019, 11:03pm
16
In my case last year, with OVS on 5.6, I had to hack the flow template, in my experience the IP-spoofing “OFF” setting didn’t work (the ovs flows didn’t change)
IP spoofing in this case is working it is only allowing the one IP address as intended .
I do mine through dhcp and Mac reservations so set as ethernet enable Mac spoofing only flag and then reserve Mac to IP , same for alias just set as dhcp otherwise you will need to edit the source for the iptables rules it sets up.
I suppose it has to work for each IP that I assign to it? That is the problem. I can’t get multiple IP’s to work with protection, only the first IP that I assign to it is reachable. It doesn’t matter if I add the second as an alias or separate NIC. I use the Centos 7 image from the marketplace by the way.
What I don’t understand is that this isn’t resolved yet. It isn’t like I am using a very exclusive or strange network setup. It’s just a Bridged hypervisor without VLAN’s or other network related techniques that might get in the way.
I’ll have a look and see if I can reproduce it , I haven’t had any issues with dual nics before , e.g. both are reachable.
When you add a second nic do you add another gateway ? You should only have one default route.
Is the whole idea of this two public IP addresses on 1 nic but you don’t want users to use any IP other than the one allocated by opennebula ? So you want to use anti ipspoofing. E.g. it’s on a shared subnet and you want to make sure users don’t steal other users or spare IP addresses ?
That’s how I am reading it , if so I’ll lab it up and have a look see if I can replicate it , I am sure it is possible someway or another
Is the whole idea of this two public IP addresses on 1 nic but you don’t want users to use any IP other than the one allocated by opennebula ? So you want to use anti ipspoofing. E.g. it’s on a shared subnet and you want to make sure users don’t steal other users or spare IP addresses ?
This is exactly what I want.
Please see if you can replicate it.
