Mar 05 2025 18 mins 8
This show has been flagged as Clean by the host.
Maintaining The Remote System
I have renamed the project Libre Indie Archive because the name theindiearchive is already someone else's domain. I never would have renamed The Indie Archive but I do think that Libre Indie Archive is more descriptive, hence, better.
I am getting close to a pre beta push up to codeberg. Anyone following along who wants to help test, you can do this with two or three old systems. Let me know. Email [email protected] or on Mastodon I am @[email protected].
I have decided to develop and document for Xubuntu first and here's the reasons why.
I bought an older HP small form factor office system with 4 Gigabytes of ram.
HP Compaq 4000 Pro Pentium Dual-Core E6600 3.06GHz 4GB RAM
Thirty dollars on ebay with shipping and taxes.
I was testing Libre Indie Archive on it. Because of the age of the system Ubuntu wouldn't install. I tested it with some BSD systems and installed Indie Archive without a GUI. Ghost BSD didn't install but Midnight BSD did install so I used the Midnight BSD GUI and installed Indie Archive. None of this was easy for me because I'm a BSD newb and unless you already use BSD I can't recommend it for Libre Indie Archive. Remember, not all indie producers are computer programmers, and I want Indie Archive to work for those producers as well as for the computer savvy.
Then on a whim I thought I would try the Xubuntu 24.04 distro and it installed no problems. Thanks XFCE for keeping it light.
The other reason I am developing and documenting for Xubuntu is that I can use the Xubuntu install document and install on Ubuntu or Debian with only minor differences. I know because I tried it. This is probably also true for other Debian and Ubuntu derived distributions.
So, if you want to help, you could take the Xubuntu install document and see if it works on other distributions. Write down what you had to change and let me know. I plan on making an install checklist out of the install document and it would be great to have a checklist with the actual commands for several distributions.
So, that was the intro. Now on to the topic.
I am planning on installing remotenear and remotefar systems, remotenear being a short drive away (or maybe in your home if your studio is not in your home, like mine) and the remotefar further away to avoid losing data in the case of a regional catastrophe like flood, fire, tornado, or hurricane.
Still even a short drive is not what I want to do any time there might be something I need to check on a remote system so I have devised a way to manage it from the secondary system.
When a remote system is delivered to a new location it will be headless. No monitor, no keyboard, and no mouse. At the remote location it is plugged into a UPS and attached to the network with an ethernet cable and attached to the UPS with a usb cable. Then it is turned on.
Even without a keyboard or a mouse there is still some local control of the system available. As part of the remote system install we go into the power management settings and next to "when power button is pressed" we select shutdown.
So, a short press on the power button initiates a Xubuntu shutdown just like the shutdown that you get from the menu or Alt F4.
If that doesn't work a long press of the power button will turn the system off. This is like unplugging the system or losing power and is not recommended but Xubuntu will rebuild the file structure when the system is restarted.
And if you do lose power the UPS will send a signal to the computer shutting it down with a controlled shutdown, just like a short press of the power button or a shutdown from the menu.
I would like to carry this one step further and enable automatic power up for the computer. A quick search shows cyberpower PowerPanel software for linux. Also you can set a power restore function in the BIOS to restart the system when the power is restored. I just checked and this worked on my little HP.
So ... with just the power button and an attached UPS you can get both manual and automatic control of shutting the remote system down and restarting it.
Pretty cool for a rather sparse interface.
If you know more about how to set this up please let me know. There's a big jump between doing a search to see if something is possible and actually implementing it.
Okay, that was the easy part. Now for the fun part.
First off, the remote system is probably not going to be at your place but at the home or business of friends or family. And they probably don't have a static IP, and they may not be able to implement port forwarding in their router, and they may not be able to control their firewall.
So we can't go, "I'll just ssh in when I need to fix a problem". And you don't really want to change their setup anyway because all of the above add to their security risk.
Also their router undoubtedly gives dynamic IP addresses so we want the remote system to use that because when we are setting it up we might not even know what subnet their LAN uses.
But, at the same time it doesn't make any sense at all to try to maintain a remote system that you can't log into.
So, the tool for setting up a terminal session on the remote system is called a remote tunnel reverse shell. The remote system is already connecting to the secondary system with rsync ssh when the cron job fires off every day to update the files. So, the secondary system is running an ssh server and the remote system has the public key that allows access without entering a password.
There are two parts to setting up a remote tunnel reverse shell. The secondary system has to be listening for the remote system on a port, I use port 7070. And then the remote system runs a bash command with the -i parameter that means reverse shell, and with the port, 7070.
I'm using nc to set up the listener.
nc -lvnp 7070
-l is --listen
-v is --verbose
-n means the port is restricted to numeric values.
-p is --port
7070 is the port
I chose the port number, 7070. You can use any available port but the listener has to use the same port as the remote system uses in the bash call.
Which is this.
bash -i >& /dev/tcp/your-static-ip-from-your-isp/7070 0>&1
This is the order of events.
On the secondary system I start listening.
nc -lvnp 7070
Then a script runs on the remote system.
bash -i >& /dev/tcp/your-static-ip-from-your-isp/7070 0>&1
And then a command prompt opens up in the terminal on the secondary system that's listening. And you are logged into the remote system and you can look around and check things out and even move or delete files until you exit.
Except it didn't work.
Of course not, nothing ever works the first time.
Two other things have to be changed that we're going to talk about now, the firewall and port forwarding.
These things are already discussed in install.txt because we had to fix the firewall and port forwarding for the remote system to log into the secondary system to pick up the new files.
To set up port forwarding, log into your router from a browser attached to the router. Like, for instance, a browser on your secondary system.
You open the browser and type into the address bar, 192.168.1.1
Which is right most of the time. On my setup I type 192.168.2.1 because the isp's router uses the 192.168.1 subnet.
How do I know which to use???
This also is covered in install.txt because to connect from the primary system to the secondary system I have to connect to the static ip that I assigned to the secondary system.
So my primary system has the static ip 192.168.2.11 and my secondary system has the static ip 192.168.2.12 which allows me to ssh into the secondary system from the primary system.
And this means my router is at 192.168.2.1
Your router is likely at 192.168.1.1 because that's the most common LAN subnet.
Anyway, in the browser I open the router's control console and then I have to enter the password. If you don't know what it is you have to find out and write it down.
Check what the defaults are for your router by searching on the internet. The defaults might work.
If they do change your login and password and write them down! Do not leave your router defaults in place. That's a big security risk.
After you're logged into the control console check around in the menus for Port Forwarding.
I already had to do this to make ssh work from the remote system to the secondary system. In that case I had to forward port 22 (the ssh port) from the internet to the secondary system. Here's how that works.
On the remote system I type.
ssh indiearchive@your-static-ip-from-your-isp
Since it's coming in as ssh that means the router sees port 22.
The router checks the port forwarding table and sees that incoming traffic using port 22 should go to the secondary system, in my case 192.168.2.12
So the incoming ssh goes to the secondary system which is my ssh server. What a coincidence.
So in order to use port 7070 to open a tunnel from the remote system to the secondary system I have to add a row to the port forwarding table with 7070 as the port and 192.168.2.12 as the ip. Except on your LAN the ip address may be different.
Except it doesn't work.
I bet you guessed why. It's the firewall.
On the secondary system type.
sudo ufw status
It should show you that port 22 is allowed because otherwise you wouldn't be getting ssh traffic.
It probably won't show you that port 7070 is allowed. So type.
sudo ufw allow 7070
Then check the status again and see if it shows 7070.
Here's a nice firewall link with instructions.
https://www.digitalocean.com/community/tutorials/ufw-essentials-common-firewall-rules-and-commands
It still might not work even though it should. Why? Operator error. You may have typed 7000 instead of 7070. (I did that.) Or any other little typo in any of the commands.
When this works you are ready to test the reverse shell. The remote system can ssh into the secondary system and we have added port 7070 to the port forwarding table on the router and to the firewall on the secondary system.
This is great!
But how do I know when to listen and how do I get the remote system to issue the bash command that sets up the reverse shell? Remember, in the future the remote system is going to be sitting somewhere with no monitor, keyboard, or mouse.
Only computer programmers are required to remember the future.
After all that setup, here's the clever bit.
I have a text file on the secondary system named letmein.txt and it's a flag with two values. The text file either reads yes or no.
If it reads yes it means I'm here at the secondary system and I want to log into the remote system. If it reads no. Not so much. I'm not really trying to log in to the remote system at all.
The remote system has ssh access to the secondary system since that's the way it picks up the new files, with rsync ssh. So the remote system can use rsync to copy the letmein.txt file over to it's hard drive. And it does this every five minutes, with a cron job.
On the remote system type sudo -s to become root.
crontab -e to edit the root crontab.
Add this line
*/5 * * * * /home/indiearchive/check.sh
Every 5 minutes the remote system runs check.sh which grabs the letmein.txt file and checks to see if it says yes or no. If it says yes it starts the reverse shell, assuming I remembered to start listening to port 7070 on the secondary system.
After I'm done working on the remote system while sitting at the secondary system I type exit to close the remote terminal and come back to the terminal on the secondary system. If I forgot to do something I can start listening again but if I'm done I edit letmein.txt to say no and the remote system will quit trying to set up a reverse shell every 5 minutes.
But wait! There's more.
Email notifications.
I set up email notifications with mailersend for file integrity reports using curl.
To do that I wrote a script called send.sh that takes a file name as an argument and then sends me an email with the contents of the file in the body of the email. So when I run my file integrity program if the log files are larger than they should be, it means there is a discrepancy and that log file gets emailed to me so I can check things out. (Maybe with my remote tunnel reverse shell.)
I also check diskspace with df and send a disk space report.
Using send.sh when I run check.sh and detect a yes in letmein.txt I call send.sh with letmein.txt as the parameter and I get an email that says yes, meaning the remote system is trying to set up a reverse shell.
So if I change letmein.txt to yes on the secondary system and I wait five or ten minutes without getting notified I may just have to make a call.
Maybe the nice people who are hosting my remote system have lost power. Or internet. Or maybe they will have to push a button. If that doesn't work I may have to make a trip. I hope it's remotenear and not remotefar.
So when I was testing the email notifications part of check.sh and fiddling around with the code all of a sudden I quit getting notifications at all. I learned a lot about bash scripting trying to figure out what I did wrong and it turned out it wasn't me.
After I sent myself numerous emails saying yes from a weird email address gmail decided they were spam. So I went into my spam folder and marked the notification email as not spam. That fixed it for me but if you are setting up email notifications for Libre Indie Archive or for anything be sure you white list the email address so that the email powers that be don't suddenly decide that your notifications are spam and you quit getting important notifications.
In gmail you set up a filter entry with the notifier's email address and set the action to be "Never send it to Spam".
Because getting these emails is important. First they remind me to have the secondary system listen. Then they remind me to change letmein.txt from yes to no after I'm done with the remote terminal. And while you're changing letmein.txt to no make sure the listener is off. Leaving it listening for an extended period of time is a security risk.
So there's a lot of little moving parts involved in this. Kind of complicated but still fascinating.
Almost done.
I didn't think this would be so long and now I'm exhausted. I am including slightly redacted and well commented copies of check.sh and send.sh in the show notes which will be on Hacker Public Radio and on my Delta Boogie Network-Gamer+ blog at home.gamerplus.org.
As always, I appreciate your comments.
Thanks
