Combining FleetDM with reverse ssh shell
Why would you want a reverse SSH shell
So in a previous post I wrote about using FleetDM to help you manage a device running Linux of a friend or family member. Fleet handles a lot for you, but if you really need to troubleshoot, you want to login and get a shell. Now what you don’t want is to expose their device listening to ssh connections on the internet and open ports in their firewall or ISP provided router. You can, but aside from security reasons, you may not want to also change anything in their router. They may not even have a static IP address. These days ISPs even use tech to share IP addresses amongst multiple clients, and if the laptop is not in the home Lan, it will by definition have a new IP Address.
As a reminder, I will use the term host for the device managed by Fleet.
So what you want is:
- to be able to start a connection from the host (triggered by Fleet)
- than login over that connection into a shell on host
- enable that connection only when you need it. So toggle this on and off using Fleet
To do this you can use a reverse ssh tunnel. The link I provided explains the concept well enough for me not to repeat it here.
How to set this up
Preparations on the host
Although in theory if you have the time and are creative you could set this up using the approach I explained yesterday using Fleet. But that would be quite a challenge. Best is to prepare this when you have actual access to the device. A small investment of your time which can save you many trips later on. So it makes sense to this when you install the Fleet client software anyway.
Steps to prepare the host
- make sure you have a ssh server in your control that can be reached from the host.
- enable passwordless ssh connection to this ssh server
- install a script on the host that sets up the reverse shell when needed from Fleet
Steps on the Fleet server
- install a script that calls the script on the host
Make sure you have an ssh server under your control
The host needs to connect to an ssh server and you need to be able connect to this as well. As it sits in the middle you need to trust it. There are many ways to achieve this. If you have something running in your own network make sure it can be reached from the internet. Obviously make sure this machine stays secure. You can also setup a server somewhere on the internet, i.e. a virtual at a hoster (VPS) or even the cheapest free tier at AWS will do. As long as it can run openssh and you control the config and can add users.
Furthermore, best is to set up a (virtual) machine that only functions for this. For security reasons you want to isolate this machine as much as possible, rigorously keep it up to date and harden it. It helps if the machine is not used for anything else which may hinder you with updates. update windows, hardening rules etc.
Enable passwordless ssh connection to the ssh server
- on the SSH server create a user for this purpose. In this example I will use fleet as an example. From a security point of view you should create a separate user. On the host you will have to place a private key. You want that tied only to that device. As you want to start the tunnel from a script, it should be used without a passphrase. That makes this key vulnerable. That’s why you want a separate user for this device that can only login to the ssh server and there do nothing else.
If you want to secure this a bit more, you could use Fleet to inject the private key just before you start to use it or even automate that a temporary user is created on the ssh server, keys are generated and placed which can all be deleted when done, but that is outside the scope of this write-up.
- create an ssh keypair
- copy it to the ssh server
Code:
1
2
3
mkdir -p $HOME/fleet
ssh-keygen -t ed25519 -C "fleetdm_hostname@example.com" -f $HOME/fleet/id_ed25519_fleet
scp $HOME/fleet/id_ed25519_fleet.pub fleet@yoursshserver:/home/fleet/.ssh/
Note that the ‘email address’ is just a comment so it does not need to be valid, it is added to the keys to help you identify them. Furthermore you obviously need to replace yoursshserver with actual address of your ssh server.
Test via:
1
2
sudo su -
ssh -F /dev/null -o IdentitiesOnly=yes -i /home/$USER/fleet/id_ed25519_fleet fleet@yoursshserver
The first time you should have to accept the key of the server. Try this a second time to ensure that the next time you won’t have to do it again.
Create the script for Fleet
- first ensure that the private key is in the place to be used by the script
1 2 3 4
sudo mkdir -p /etc/fleet/keys sudo mv $HOME/fleet/id_ed25519_fleet /etc/fleet/keys/ sudo chown root:root /etc/fleet/keys/id_ed25519_fleet sudo chmod 600 /etc/fleet/keys/id_ed25519_fleet
- create a script:
/usr/local/sbin/fleet-ssh-tunnel.sh
Past the following content and adapt to your setup:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
#!/usr/bin/env bash
# fleet-ssh-tunnel.sh
# Usage: fleet-ssh-tunnel.sh start|stop|status
REMOTE_USER="fleet" # SSH user on your SSH server
REMOTE_HOST="yoursshserver" # your SSH server
REMOTE_PORT="7022" # remote port on SSH server
LOCAL_PORT="22" # port on the Fleet host
SSH_KEY="/etc/fleet/keys/id_ed25519_fleet"
PID_FILE="/var/run/fleet-ssh-tunnel.pid"
case "$1" in
start)
if [ -f "$PID_FILE" ] && kill -0 $(cat "$PID_FILE") 2>/dev/null; then
echo "Tunnel already running (PID $(cat $PID_FILE))"
exit 0
fi
echo "Starting tunnel..."
nohup ssh -N -i "$SSH_KEY" -o ExitOnForwardFailure=yes \
-o ServerAliveInterval=30 -o ServerAliveCountMax=3 \
-R "0.0.0.0:${REMOTE_PORT}:localhost:${LOCAL_PORT}" \
"$REMOTE_USER@$REMOTE_HOST" >/var/log/fleet-ssh-tunnel.log 2>&1 &
echo $! > "$PID_FILE"
echo "Tunnel started (PID $!)"
;;
stop)
if [ -f "$PID_FILE" ] && kill -0 $(cat "$PID_FILE") 2>/dev/null; then
echo "Stopping tunnel..."
kill $(cat "$PID_FILE")
rm -f "$PID_FILE"
echo "Tunnel stopped."
else
echo "Tunnel is not running."
fi
;;
status)
if [ -f "$PID_FILE" ] && kill -0 $(cat "$PID_FILE") 2>/dev/null; then
echo "Tunnel running (PID $(cat $PID_FILE))"
else
echo "Tunnel not running."
fi
;;
*)
echo "Usage: $0 start|stop|status"
exit 1
;;
esac
- make executable:
sudo chmod 700 /usr/local/sbin/fleet-ssh-tunnel.sh
- Test via:
1 2 3
sudo su - /usr/local/sbin/fleet-ssh-tunnel.sh start /usr/local/sbin/fleet-ssh-tunnel.sh status
The last command should show the tunnel running. The real test:
ssh <useraccountathost>@yoursshserver -p 7702
This part is important and can be confusing. The <useraccountathost> means a user account on the host. If you setup a user for yourself with sudo rights, use that and the credentials belong to the account on the host.
Setup scripts on Fleet
The paid for version of Fleet allows you to easily add an argument to a script. As I work from the free/open version I will have to do without. So to run the script you need three different scripts: one to start, one to stop and one to tell you the status. Here are the scripts:
reverse_shell_start.sh
1
2
#!/bin/bash
/usr/local/sbin/fleet-ssh-tunnel.sh start
reverse_shell_stop.sh
1
2
#!/bin/bash
/usr/local/sbin/fleet-ssh-tunnel.sh stop
reverse_shell_status.sh
1
2
#!/bin/bash
/usr/local/sbin/fleet-ssh-tunnel.sh status
Upload to Fleet as usual and run on the host as needed.
The actual troubleshooting
- Obviously start the reverse shell
- login to youraccount_at_host@yoursshserver -p 7702
Now your logged in and can do what you can from the shell as if you are sitting behind the device :-)