DevOps Concepts #6: Jump Hosts
What is Bastion? What is Jump Host? Find out how Bastion Works, and how you can avoid it using AWS System Manager or GCP IAP.
Dragan Jovanovic
September 16, 2023
Jump hosts are something that is pretty common at the moment, but engineers sometimes don’t know what they are, or get confused. Today, we will cover how we deploy jump hosts to AWS, and what’s their purpose.
Securing resources
When you want to give engineers access to the resources through SSH, you have to put some restrictions on it. If you do it through multiple servers, it can be really annoying to manage those keys, and you forget to remove users, trust me, people do it. And if you have to manage each server separately, it’s annoying and leaves space for mistakes.
The solution to it is usually to introduce something we call Jump Server or Jump Host which allows us to control who has access to our infrastructure, and to protect only one part, instead of every single server. This way, your internal services are exposed only through bastion, instead of exposing each one.
Jump Host / Bastion Illustration
But if we are speaking in terms of AWS and Cloud, the ideal solution is that the application is inside a private subnet and that our Bastion Server is inside a Public Subnet. That way, we don’t expose our instances, there are better solutions from AWS like Systems Manager or GCP IAP where you have a way to open a session toward the instance without exposing instances and managing SSH keys, but will speak about that later, still, sometimes you want to use Bastion.
With this infrastructure, you can see that we have a couple of things now.
Our Bastion VPC is exposed to the internet and you should usually restrict traffic to be coming from your company office for example, you can restrict on the security group level and allow TCP Traffic on port 22 for a particular IP. That way, engineers can SSH into Jump Host, which is just a server and allows users to connect if their key is allowed. It’s going to be an EC2 instance.
App VPC subnet is allowing traffic on port 22 only from our Bastion VPC, which ensures that you can’t SSH into it from other places.
Well, this is better than exposing and managing each server, and you have less possibility of attacks.
Cloud Ways
Bastion works, but when you are in the Cloud, you might not need it. Because there are some managed ways to connect, like using AWS System Manager.
It works in that way, that you shouldn’t open the SSH 22 Port, and with two subnets. Basically, you have one Public Subnet and one Private Subnet in which you deploy your instances.
Session Manager to Login to EC2 Instances
Okay, a lot of things are happening in the above image, but a few important things are that. In our Public Subnet, we deployed NAT Gateway which allows outbound traffic from the private subnets, and our public subnet routes all internet traffic back to Internet Gateway.
Private Subnet doesn’t go to Internet Gateway, it instead routes all internet traffic to the NAT Gateway, so it remains private. And we configure these things of course using Route Tables.
There are some things that are not visible in the image. Our EC2 Instances have a role which is named EC2SystemManagerRole which just has permissions AmazonSSMManagedInstanceCore and we need that because our EC2 instance has to connect to System Manager on the bootstrap. I’ve used Amazon Linux AMI, but if you have your own, you have to add SSM Agent to it and configure it. SSM that way can track your instances.
So everything is nice, but in order to connect, the engineer should have proper permission to connect to it. Ideally, you should create an IAM Group and attach a policy to it that allows user to start sessions with SSM. After you have IAM Group, you can just add users to it and enable them access. That policy looks like this, you can also specify resources instead of all resources:
So our final architecture looks something like this:
Before you connect to your instance, you have to configure AWS CLI and SSM Plugin for CLI which you can find here. Once you have this configured, you can use CLI:
aws ssm start-session --target INSTANCE_ID --region YOUR_REGION
And you are in.
GCP
Apparently, Google has GCP IAP which should allow us something similar to this. So I wondered how much different it is from AWS System Manager.
We are going to try to create a Bastion server and connect to it, without it having the Public IP. And then, we are going to have a separate server to which we can connect from Bastion.
As you can see in the image above. We have two instances in two different subnets, and in order to enable IAP, we have first to enable that API in GCP and configure the OAuth Consent Screen.
Once that’s in place, and instances are deployed. We have to add firewall rules that enable traffic on port 22 to the bastion instance, from the source IP 35.235.240.0/20 which is a fixed IP for IAP.
We can allow SSH traffic for app instances from our bastion app. And still, our Bastion server doesn’t have a public IP, we connect to it using the command like this:
gcloud compute ssh --tunnel-through-iap --zone=europe-central2-a bastion
You have to configure GCP CLI using gcp init and after that, you should be able to do SSH into them. You can also control the access using IAM, but I won’t be covering it.
End
I kind of liked to manage this using IAM, but I think most people are still on the classic Bastion host. I remember managing SSH keys with Ansible script and updating the server through CI runners. It worked, but I think with IAM we can manage them more granularly. Even from the compliance perspective, I think it helps because as in everything, you most likely have to rotate those SSH keys.
In the example above, we enabled all resources, like that we can also limit access per team on the IAM Group and allow them access only to the resources they need instead of letting them to jump from Bastion where they want.
How did you like today's issue? |