Cloud on a cloud!
Yes! You read it correctly. This post will guide you to install an OpenStack on top of AWS EC2.
Installing OpenStack on nested hypervisor environment is not a big deal when we use the QEMU emulator for launching virtual machines inside the Virtual machine. However, unlike usual nested hypervisor setup, Installing OpenStack on AWS EC2 instances has few restrictions on networking part for the OpenStack setup to work properly. This blog outlines the limitations and its solutions to run OpenStack on top of AWS EC2 Virtual Machine.
Limitations:
AWS environment will allow the packets to flow in their network only when the MAC address is known/registered in the AWS network environment.
Also, the MAC address and the IP address are tightly mapped. So, the AWS environment will not allow the packet flow if the MAC address registered for the given IP address is different.
You may wonder why the above restrictions will impact the OpenStack setup on AWS EC2.
Yes…! It will!
While configuring Neutron networking, we would be creating a virtual bridge (Say, br-ex ) for the Provider Network where all the VMs traffic will reach the internet via external bridge followed by the actual physical NIC (say,eth1).
In that case, we usually configure external interface (NIC) with a special type of configuration as follows.
|
The provider interface uses a special configuration without an IP address assigned to it. Configure the second interface as the provider interface: Replace INTERFACE_NAME with the actual interface name. For example, eth1 or ens224.
# The provider network interface auto INTERFACE_NAME iface INTERFACE_NAME inet manual up ip link set dev $IFACE up down ip link set dev $IFACE down |
Ref: – http://docs.openstack.org/mitaka/install-guide-ubuntu/environment-networking-controller.html
Due to this special type interface configuration, the restriction in AWS will hit the OpenStack networking. In mainstream OpenStack setup, the above-mentioned provider interface would be configured with special NIC configuration that will have no IP for that interface and will allow all packets via that specially configured NIC.
Moreover, the VM packets reaching internet via this specially configured NIC will have the IP of OpenStack tenant router’s gateway IP address as the Source IP address in each packet.
Like I mentioned earlier in the limitations above, AWS will only allow the packet flow when the MAC address is known/registered in their environment. Also, the IP address must match the MAC address.
In our case, the packet from the above-mentioned OpenStack tenant router will have the IP address of router’s Gateway in every single packet and the packet source MAC address will be the mac address of router’s interface.
Note: – You could see these details using the “ip netns show” followed by “ip netns exec qr-<router_ID> ifconfig” command in the OpenStack controller’s terminal.
Since the MAC address in unknown/not registered in the AWS environment, the packets will be dropped when it reaches the AWS Switch. To allow the VM packets to reach the internet via AWS switch, we need to do some tricks/hacks in our OpenStack setup.
Make Use of What we have:
The possible ways are,
To register the router’s MAC address and its IP address with AWS environment. However, certainly, this is not feasible. AWS do not have these features available till date to register any random MAC address and IP address inside the VPC. Moreover, allowing this type of functionality will be a severe security threat to their environment.
So, the other method is to make use of what we have. Since we have used a special type of interface configuration for the provider NIC, you could note that the IP address assigned to the provider NIC (Say eth1) is left unused. We could use this available/unused IP address for the OpenStack router’s gateway.
The below command will do that trick,
|
neutron router-gateway-set router provider —fixed-ip ip_address=<Registered_IP_address*>
|
IP address and MAC address Mismatch:
After configuring router gateway with the AWS registered IP address, each packet from router’s gateway will have the AWS registered IP address as the source IP address but with OVS generated unregistered MAC address.
Like I mentioned in the AWS restriction session above, the IP address must match the MAC address registered else all the packets with mismatched mac and IP address will be dropped by the AWS switch.
To make the registered MAC address match with the IP address, we need to change the MAC address of router’s interface. The below–mentioned steps will do that magic.
Step 1) Install macchanger tool.
Step 2) Note down the actual / Original MAC address of provider NIC (eth1)
Step 3) Change the MAC address of Provider NIC (eth1)
Step 4) Change the MAC address of Router’s Gateway Interface to the original MAC address of eth1.
Step 5) now, try to ping 8.8.8.8 from router namespace
If you get a successful ping response, then we are done with Cloud on Cloud setup.
Key points to remember:
1) Change Mac address:
In my case, I had worked in Ubuntu 14.04 LTS server which has no issue in changing the MAC address using macchanger tool. However, when I tried in Ubuntu 16.04 LTS, I got an error saying “No permission to modify the mac address .“ I suspect the error was due to the cloud-init tool not allowing the mac address configuration.
So before installing OpenStack setup, try changing mac address of the NIC.
2) Floating IP disabled:
Associating floating IP to any OpenStack’s VM will send the packet via router’s Gateway with the source IP address as floating IP’s IP address. This will make the packets to hit the AWS switch with non-registered IP and MAC address which results in dropping the packets.
So, we could not use the floating IP functionality in this setup. However, still, we could access the VM publicly using the below mentioned NAT process.
3) NAT to access OpenStack VM:
Like I mentioned above, we could access the OpenStack VM publicly using the registered IP address that we have assigned for Router’s gateway.
Use the below NAT command to access the Openstack VM using the AWS EC2 instance’s elastic IP:
|
$ ip netns exec qrouter-f85bxxxx-61b2-xxxx-xxxx-xxxxba0xxxx iptables -t nat -A PREROUTING -p tcp -d 172.16.20.101 –dport 522 -j DNAT –to-destination 192.168.20.5:22
|
Note: In the above command, I had NAT for forwarding all packets for 172.16.20.101 with port 522
Using the above NAT command, All the packets reaching 172.16.20.101 with port number 522 will be forwarded to 192.168.20.5:22
Here,
172.16.20.101 – Registered IP address of AWS EC2 instance which was assigned for Router’s gateway.
192.168.20.5 – Local IP of the OpenStack VM
Notably, 172.16.20.101 has already NAT with AWS Elastic IP which means all the traffic that comes to the elastic IP (Public IP) will be forwarded to this VPC local IP (172.16.20.101).
In short:
[Elastic IP]:522 172.16.20.101:522 192.168.20.5:22
It means you could SSH the OpenStack VM globally by using the elastic IP address and the respective port number.
4) Elastic IP address:
For this type of customised OpenStack installation, we required at least 2 NIC for AWS EC2 instance.
One for accessing the VM terminal for the installation and for accessing the dashboaIn short, it acts as a Management network/ VM Tunnel network / API network.
Later one is for an external network with unique type interface configuration and mapped with provider network bridge. (Say br-ex with eth1).
AWS will not allow any packets to travel out of the VPC unless the elastic IP is attached with that IP address.
To overcome this problem, we must attach the elastic IP for this NIC. So, that the packets of OpenStack’s VM will reach OpenStack router’s gateway and from the gateway, the packets get embedded with registered mac address and matching IP address will reach the AWS switch (VPC environment) via br-ex and eth1 (Special type interface configuration) and then hit the AWS actual VPC gateway. From there the packets will reach the internet.
Other Cloud Platforms:
In my analysis, I could see most of the cloud providers like Dreamhost, Auro-cloud has the same limitations for OpenStack Networking. So we could use the tricks/hacks mentioned above in any of those cloud providers to run an OpenStack cloud on top of it.
Note: – Since we are using QEMU emulator without KVM for the nested hypervisor environment, the VM performance will be slow.
If you want to try OpenStack on AWS. Please register in our Cloud Lab as a Service offering available at https://claas.corestack.io/clab#/clab/0 which provides a consistent and on demand lab environment for OpenStack in AWS.
Cheers,
Vinoth Kumar Selvaraj
Hi Vinoth,
Useful information.
When I tried, I am stuck at this “Step 4) Change the MAC address of Router’s Gateway Interface to the original MAC address of eth1”
I get below error:
root@ip-10-0-0-**:~# neutron port-update 8efc1a39-931f-4f2e-8296-ecb7bdc2be3d –mac-address 0a:**:df:75:**:43
Unable to complete operation on port 8efc1a39-931f-4f2e-8296-ecb7bdc2be3d, port is already bound, port type: bridge, old_mac fa:16:3e:6b:66:d9, new_mac 0a:**:df:75:**:43
Setting Up the Environment:
1. Create new VPC in AWS environment.
2. Launch new VM with at least 6GB RAM and 20GB HDD in the newly created VPC.
3. Configure the VM with 2 Primary NICs. (Say eth0 & eth1)
4. Attach Elastic IP to both the NIC.
(Say, eth0:172.16.10.101 < --> 189.32.xx.21 & eth1:172.16.20.101 < --> 189.32.x.34)
5. Configure Security group to allow the required ports.
6. Install Macchanger tool.
7. Note the original MAC address of the NIC you have planned for provider’s network. (Say eth1)
8. Check whether you can change the MAC address using Macchanger tool.
9. Start installing the OpenStack setup. In my case, I choose OpenStack – Mitaka.
10. While creating the OpenStack provider network, do consider creating the provider’s subnet with the allocation pools covering the registered IP address of your EC2 instance earlier that was assigned for eth1 (Say, –allocation-pool start=172.16.20.100, end=172.16.20.250).
11. After creating a router, execute the above-mentioned “neutron router-gateway-set ….” command to add the Gateway with the fixed IP address (Say, $ neutron router-gateway-set router provider –fixed-ip ip_address=172.16.20.101).
neutron router-gateway-set router provider –fixed-ip ip_address=
Hi Vinoth, I’m also stuck at step 4. Can you confirm exactly how you change the router gateway interface MAC? I thought it would be via neutron port-update, however, like Sabarish above it doesn’t work. Is this MAC change achieved via macchanger somehow??