Imagine this: you’re running a web hosting service and you need to connect in to add another client who has specific needs your setup script doesn’t account for. You try to connect via SSH, but it hangs. After hanging, you escape out of the session and reconnect with the verbose flag set. You notice the wrong IP address, and you realize that you set up Cloudflare proxying. So you SSH in to the server with the public facing IP address, to find out it’s still not connecting. You can ping the server just fine and you can even access the website just fine. Maybe you accidentally banned yourself from accessing the server via SSH. So you log in to your server provider’s website and open up the server. You see that your IP isn’t banned and the SSH service is running just fine. The variable you didn’t account for was where you’re logging in from: the network you’re connecting from has outbound SSH blocked.
As that opening paragraph suggests, this is a situation I had to deal with when I first transferred to SEMO. The image that I used to deploy multiple WordPress websites with used PHP 7.4, and I had to upgrade many of those websites to PHP 8.1. However, I was unable to SSH in to those servers due to a firewall rule on SEMO’s network. While I was able to get those servers upgraded to use PHP 8.1 that day by logging in through the console, ensuring I had proper backups prior to those upgrades was made drastically more difficult.
It’s not just SSH
Around the same time as running those upgrades, I decided to embark on another project: writing my own Twitch chat bot. Twitch’s chat utilizes the IRC protocol, which is usually a plain text service. Among other things, IRC is still used today for real time support channels akin to a Discord server. However, when trying to test things to see if I was able to successfully connect to the server, I kept getting connection timed out warnings. After generating a GitHub personal access token to push my code to a repository since outbound SSH was blocked and running it in my labbing cloud server, I discover that it works fine there. Seeing as how widespread IRC is, I try a generic IRC client on my main computer to see if I can connect in to a Twitch chat. That answer was no.
Similarly, around that time I got the idea that I wanted to try figuring out how to send and receive emails using my own mail server, with the intention on learning how email works on a lower level and what security practices can be put in place for a secure but effective email server. Naturally, this was harder to diagnose as I was not only reliant on the firewall allowing outbound SMTP, but also that the cloud provider’s firewall wasn’t blocking SMTP and my configuration was working correctly. I could rule out my cloud provider’s firewall wasn’t blocking SMTP as I submitted a ticket to have that restriction removed from my account. Additionally, I was able to confirm I could send emails via Postfix to a Gmail account. That suggested that port 25 was open. However, trying to use netcat or telnet to connect to the instance, I couldn’t access it.
Determining why the blocks existed
After having experienced those issues and being able to rule out it was neither my computer nor the servers at fault, I did what any sane person did: submit a ticket about the rules. After all, I’m trying to do my job and manage these web servers that clients are paying me for. Two months later, a possible lead turns in to pushing me to use the student VPN. A VPN that I am not able to use because of my personal machines running Linux and using a Windows exclusive installation method, not to mention still not solving the root issue. Attempting to circle back on the root issue results in radio silence. Following up after a month and discovering yet another blocked service: 9001 for the portainer agent, results in more radio silence. Following up two months later again (we’re now in December, four months after I moved in), still silence. At this point, I escalate it to a Missouri Sunshine request. You can read some more information on my post on the Open Information Committee for more details on that, as that is outside the scope of this post. However, in spite of raising the issue initially a year ago as of the writing of this, it still hasn’t been addressed.
Circumvention, Discrepancies, and the Consequences
Out of curiosity, I decide to try changing the SSH port one my labbing cloud server to a different port. That curiously works, suggesting the firewall merely blocks ports, not services. The easiest solution would be to change each of the ports that are blocked to ones that aren’t blocked. Further, this process was facilitated by a script I wrote, Port Knocker, to see what ports were available. Ports 3306, 1433, 1434 are blocked, which usually are associated with database servers like MySQL and Microsoft SQL. Normally it would be trivial to change the ports that they’re listening on, however services that depend on a SQL server usually assume the respective ports they use, with a layer of difficulty trying to change that in the event you use a different port. Furthermore, you’re heavily restricted to using third party servers like GitHub, as they made the decision to remove password based authentication in August 2021, favoring git over SSH instead or using a personal access token if you decide to stick with using git over HTTPS.
Using the aforementioned script, I discovered several discrepancies. For instance, port 22, the port commonly used by SSH, is blocked but port 23, the port commonly used by telnet, isn’t blocked. Further, port 21, the port commonly used for FTP, isn’t blocked either. While a cynical person might point out that blocking SSH and not blocking Telnet or FTP would allow snooping on data, port 1723 appears to be blocked as well, the port used for PPTP with the L2TP, IPsec, OpenVPN, and Wireguard ports all being left open. Additionally, while SSH would be blocked in its default settings, you can still use the remote desktop protocol’s default port of 3389 or VNC’s default port of 5900.
Hypocrisy
One of the additional issues in this particular context is the hypocrisy. A couple of SEMO’s majors are Computer Networking Systems Administrator and Cybersecurity, with both of those having a fundamental need to remotely manage networking appliances and servers. Furthermore, a required course for Computer Science, Computer Information Systems, and Cybersecurity is a database class, which the course description describes it as the following:
Basic concepts of database and database architecture. Discussion of entity-relationship and relational database models. Study of the SQL query language. Study of database design methodology.
https://semo.edu/student-support/academic-support/registrar/bulletin/courses/index.php
While it doesn’t inherently call out a database software used, it could be assumed that something akin to MySQL or Microsoft SQL is being used to learn on. As I haven’t taken the course, that is purely speculation, but is a worth while detail to mention.
Furthermore, your physical location appears to make a difference, as the networks available to you on campus if you’re a student vary on if you’re in a dorm building or not, with the latter being more permissive, allowing services like outbound SSH.
My Plea: Configure Your Firewalls Correctly
If your intents are to block access to external devices, please configure your firewalls consistently and block each of the remote ports, including VNC, SSH, Telnet, and RDP. If you wish to block file transfers, don’t just block SSH and NFS, but also block FTP and FTPS. Even then, those dedicated to getting access to their external services will find ways around, such as changing ports or doing things over a VPN.
I’m not asking for access to computers inside of the network using computers from outside, but access to computers outside of the network from the inside. To add to it, the inconsistent rules of the network based solely on where you’re located on campus makes it more difficult to predict what we will be able to access. I’m not asking for no firewall rules, as a firewall is fundamental to our network security. We’re not asking to host various services in our dorms, since we agreed in the terms of service we signed that we would not be hosting any types of servers on campus. What I’m asking for is for us to use the tools available to us and should be actively practiced, especially if it’s part of our major. What I’m asking for are consistent rules across campus. What I’m asking for is to allow us to work from home in this increasingly work-from-home environment and to keep our hobbies such as homelabbing or programming.