Edit: Ivan in the comments mentioned: “Looks like in ESXi 7.0 U3j you cannot use the same Port Group for VMs and for Management interface (VMKernel NIC)”, which is super unfortunate. I’ve been playing around with ESXi, XCP-ng, and Proxmox lately. If I wind up going with ESXi again, I’ll see if I can update this tutorial to the new version.
OVH Eco, formerly Kimsufi, offers a line of super cheap dedicated servers, that are a great entry point into running your own server in the cloud. They’re typically older, lower power hardware, that is near the end of it’s useful life… but that just means it can be had for cheap!
Installing Ubuntu, or Debian on these servers is super useful, however being as many of these servers have 64gigs of ram, and multi-TB hard drive space, running a single service on them would be a waste of resources. Enter: ESXi. If you don’t know what VMWare ESXi is, google it.
The OVH Eco servers typically only allocate you a single IP address, without the capability to add extras. This is an issue when using ESXi, since the management interface binds to the primary IP on installation, so any VMs that you install wouldn’t have any way to access to the internet (“ESXi is not a proxy”, as the sshd_config points out). With other dedicated servers, you would often grab a second IP address, bind it to a router instance inside ESXi, and use that for traffic management on the server, but with only one IP address available, that is not an option.
Fortunately there IS a way to do this! The tl; dr: of the process I’m going to describe here:
- Install ESXi using OVH’s dashboard
- Configure ESXi networking
- Create vSwitch called “LANSWITCH” to provide connectivity between VMs
- Create LANPG port group using LANSWITCH vSwitch for VMs to use.
- Create OPNSense VM:
- Single 4 core CPU
- 2GB of ram
- 8GB hard drive
- WAN ethernet with host MAC, in the default “VM Network” portgroup, default disabled
- LAN ethernet with random MAC tied to LANPG portgroup, default enabled
- Install OPNSense from ISO file uploaded to datastore
- Set secure password for OPNSense, since it will be open to the world for a short time
- Set WAN IP to ESXi host IP
- Edit config xml
- Adjust WAN rules to open everything to the world
- Adjust LAN to 10.0.0.1/24 with DHCP
- Set WAN IP/MAC/Gateway identical to host
- Enable ESXi Auto-Start service, and enable Auto-Start on OPNSense VM
- Create second management interface on LANPG with IP 10.0.0.254
- SSH into ESXi and run command:
esxcli network ip interface remove --interface-name=vmk0 && esxcfg-route -a default gw_ip_here && reboot
- Log into the publicly available OPNSense Router, set up VPN, test it, and disable WAN public access to the web interface.
Install ESXI using OVH’s dashboard
The first step is to log into your dedicated server dashboard, and install ESXi. Before doing anything, you should disable monitoring! Circled on the right hand side of the below image. This prevents OVH from intervening in the middle of your operations and rebooting the server into maintenance mode,,, or worse: Sending you a snarky email about forgetting to disable monitoring while you’re in KVM mode. (j/k, I loved the snark)
Once you have monitoring disabled, click on the three dots next to “last Operating System”, and select “Install”
You want to install from OVHcloud template, so click “Next”
Select “Virtualization” as the type of OS, and “VMWare ESXi 7.0” from the “UNIX” dropdown. You can choose other versions, I’ve tested this process with both 6.7 and 7.0, and both seem to work nearly identically.
Keep all defaults, and you should be prompted with an installation progress screen
After 5-30 minutes, it’ll register as complete on the main screen
At this point you should receive an email from OVH with instructions on how to log into your server, including a link to retrieve the password. Once you access your server by typing the IP in the browser, and enter username root with the password provided, you’ll be prompted with the standard ESXi web UI:
Configure ESXi networking
The next step is to configure the internal server networking to accommodate what we are trying to do. Click on Networking on the left side, then “Virtual Switches” to see a list of the available vSwitches
Click “Add Standard virtual switch” to add a new switch. This switch will be for our internal LAN network, between our OPNSense router, and the various VMs that we want to run onboard. Call it LANSWITCH, or whatever you want, but enable the bottom three security options to “Accept” (This MAY not be necessary?)
Once you click “Add”, navigate to the Port Group page, and click “Add Port Group” This will create the portgroup that the VMs will actually be able to connect to, and call it LANPG, and connect it to the Virtual Switch LANSWITCH
Click on the “VMKernel NICs, and select “Add VMKernel NIC”. Connect it o Port Group LANPG, select Static IPV4, and configure it with a “10.0.0.254/24” IP address and mask. This is the internal network address that you will use to configure ESXi, once it is locked down to only being accessible from the internet network. Be sure to check the “Management” checkbox before clicking Create.
Create OPNSense VM
On the left hand menu, click on “Virtual Machines”, then click “Create/Register VM”.
Give it a name like OPNSense, select “Other” for Guest OS Family, and “FreeBSD 13 or later (64bit) for Guest OS Version.
I selected 4 CPU, 4 cores per CPU, and enabled CPU Hot Add so I could change this value later without shutting off the VM. You will NOT be able to turn this VM off, without losing complete access to your system, so make sure you get the setting right he first time, you’ll be doing this several times (At least 3-4!)
I also gave mine 2GB of ram, and 8GB hard disk. The other very important configuration is the network adapters. You’ll need to click “Add Network Adapter” at he top of the screen to create a second one, and then set them up so the first one is connected to the default “VM Network”, and the MAC address is manually set to the same as your host adapter. BE SURE TO UNCHECK “Connect at power on”! This adapter should be DISCONNECTED for the first part of this process to avoid IP conflicts!
The second network adapter should be connected to the LANPG network, and set to connect at power-on.
Once you are finished configuring, click next then Finish to complete the VM creation.
At this point you need to install OPNSense, which I will cover in another post another time, but generally you want the following configuration:
- WAN port STATIC, with same IP,gateway, and mask as host adapter
- DNS set identical to host
- LAN DHCP, 10.0.0.1/24
If you run into an issue where the OPNSense installer complains it cannot mount the boot efi partition
Shut down OPNSense, go into the VM settings, change SCSI controller0 to be LSI Logic Parallel, and in VM Options change the Boot Options Firmware to EFI.
Enable ESXi Auto-Start service, and enable Auto-Start on OPNSense VM
Once OPNSense is installed and configured, go ahead and shut it off, configure the disabled ethernet adapter to be connected, and save.
Under Manage on the left, System, Auto-Start, you can see the overall system auto-start configuration, as well as individual VM Auto-Start config.
Click on “Edit Settings”, and change “Enabled” to Yes, and click Save.
You should then click on the OPNSense VM in the list at the bottom, and click “Enable” to enable Auto-Start for that VM specifically.
SSH into ESXi and run command
At this point you’re ready to reboot! SSH into the ESXi system, and run the following command:
esxcli network ip interface remove --interface-name=vmk0 && esxcfg-route -a 10.0.0.1 && reboot
This will remove the primary management interface from the system, leaving only the one bound to “10.0.0.254”. It will also set the ESXi default gateway to the OPNSense LAN IP of 10.0.0.1, ensuring that you can read the ESXi host from within the LAN segment. Finally, it will reboot the whole system, which takes several minutes to come back up.
Once it comes back up, you can go to the primary IP Address in your browser. Don’t use the hostname, as that will throw a DNS poisoning error. You should see the standard OPNSense login screen, which you can use root/whateverpasswordyouused to login. If you do NOT see this screen… then something went wrong with the config, and you should go back and validate all your configuration is correct. You can get in via KVM and reset the management interface if need be.
Once you’re logged in, you’ll want to set up a VPN server. I didn’t take screenshots of this process, but the basic procedure is:
- Set up CA Root in System->Trust->Authorities
- Create User in System->Access->Users
- Create User Cert for user in System->Trust->Certificates (Or during the user creation process)
- Create OVPN Server Cert in System->Trust->Certificates
- Create OVPN Server in VPN->OpenVPN->Servers
- I highly recommend making 10.0.1.0/24 your tunnel network, so you can make 10.0.0.0/24 your local network. This allows you to access the OPNSense and ESXi hosts through the VPN with split-home.
- Export OVPN Client file from VPN->OpenVPN->Client Export
Next you’ll need to allow OpenVPN traffic through WAN, as well as allow traffic to flow from OVPN to the LAN network.
There may be better ways to do this, but generally these are the OpenVPN rules I set up:
And these are the WAN rules
Once these two rule sets are complete, try to connect to OpenVPN, and ensure web access to 10.0.0.1 and 10.0.0.254 is available. If so, then you can go into your WAN and remove the “* *” rule that allows ALL traffic through to OPNSense. Disable VPN and make sure you can no longer access the web interface via the primary external IP address, then Re-enable it and make sure you can access it via 10.0.0.1
At this point, you’re set up! You can spin up some more VMs from 10.0.0.254, you can forward ports using 10.0.0.1 to those VMs, and go! I currently run a HAProxy VM with all web traffic forwarded to it, and then use that to load balance off various subnet of this site. This blog is actually running on a second VM, so your connection is going OPNSense->HAProxy->BlogVM. I’ll go into more details on the HAProxy setup in another post, another time.
I’d love to hear if this process worked for you, if you have another way that achieves the same effect, or any constructive criticism of my methods. Leave a comment below, or email me (Info on my github on my contact page)