Me: so, ifconfig, what is my gateway?

ifconfig: [ip address]

Me: nmap, what is this IP address?

nmap: it's a network switch with an open telnet port.

Me: what happens if I connect to it?

switch: WHAT IS THE PASSWORD?!?!

Me: is it blank?

switch: correct. what do you want to do?

Me: can I look at all the IP and MAC addresses on the network?

switch: WHAT IS THE ADMIN PASSWORD?!?!

Me: is it... admin?

switch: correct. Here's everyone that's connected to the network: [400+ IPs and MACs]

Me: ok python, would you filter through these and tell me what manufacturer each one belongs to?

python: sure.
[~50 manufacturer lookups later]
python: there's a bunch of apple product, a bunch of miscellaneous laptop and printer manufacturers, and some raspberry pis.

Me: raspberry pis?

python: yep. about 20 of them.

Me: What happens if I connect to one?

rpi: WHAT IS THE PASSWORD?!?!

Me: raspberry?

rpi: correct. what do you want to do?

Me: can I make you do my bidding in the background when you aren't being used?

rpi: sure, sounds fine.

I love ignorant sysadmins.

Happy SysAdmin day ... even if I’m wondering if sysadmins can be happy.

Source : xkcd (of course...)

sysadmins ;)

Sysadmins life

If all you have is a hammer, everything looks like a nail!

This was something which my tech lead used to tell me when I was so obsessed with nosql databases a few years back. I would try to find problems to solve that has a use case for nosql databases or even try to convince me(I didn’t realise it back then) that I need to use nosql db for this new idea that I have, without really thinking deep enough whether the data in question is better represented using an sql schema or not.

Now, leading a team of young developers, I come across similar suggestions from few of my team members who just discovered this new and shiny tech and want to use it in production projects.

While I am not against new and shiny, it’s not a good practice to jump right in to it without exploring it deep enough or considering all the shortcomings. The most important question to ask is, whether some of the problems you are trying to solve can be solved with the current stack.

Modifying your stack requires more than just a week’s experience of playing around with the getting started guide and stack overflow replies. This is something which need to be carefully considered after taking inputs from the people who would be supporting it, that include operations, sysadmins and teams that are gonna interface with your stack indirectly.

I am not talking about delaying adoption by waiting for long list of approvals to get some thing that would bring immediate value, but a carefully orchestrated plan for why and how to migrate to a new stack.

Just because one of the tech giants made a move to a new stack and wrote about it in their engineering blog doesn’t mean that you need to make a switch in the same direction. Take a moment to analyse the possible reasons that motivated them to do it, ask yourself if your organisation is struggling with the exact same problems, observe how others facing the same issue are addressing it, and then make an informed decision.
Collect enough data to support your proposal.
Ask yourself again if you are the one holding the hammer.
If the answer is no, forge ahead!

Every time I hear a developer say "works for me", I'm gonna hide a service of mine that they use behind the VPN.

Dev: "oy mate, this server is down"
Me (with VPN connection): "sorry mate, works for me"
Dev: "but here, check this out, it's down!!"
Me: "mate, check your network connection. You must have a shitty network connection."
Dev: -_-
Me: "Maybe shitty hardware? Driver issues on your network card? 🤭"

Because you know, we sysadmins can do that too 😉

A dating site for devs/sysadmins (etc) with privacy in mind.

I'm getting ridiculously pissed off at Intel's Management Engine (etc.), yet again. I'm learning new terrifying things it does, and about more exploits. Anything this nefarious and overreaching and untouchable is evil by its very nature.

(tl;dr at the bottom.)

I also learned that -- as I suspected -- AMD has their own version of the bloody thing. Apparently theirs is a bit less scary than Intel's since you can ostensibly disable it, but i don't believe that because spy agencies exist and people are power-hungry and corrupt as hell when they get it.

For those who don't know what the IME is, it's hardware godmode. It's a black box running obfuscated code on a coprocessor that's built into Intel cpus (all Intell cpus from 2008 on). It runs code continuously, even when the system is in S3 mode or powered off. As long as the psu is supplying current, it's running. It has its own mac and IP address, transmits out-of-band (so the OS can't see its traffic), some chips can even communicate via 3g, and it can accept remote commands, too. It has complete and unfettered access to everything, completely invisible to the OS. It can turn your computer on or off, use all hardware, access and change all data in ram and storage, etc. And all of this is completely transparent: when the IME interrupts, the cpu stores its state, pauses, runs the SMM (system management mode) code, restores the state, and resumes normal operation. Its memory always returns 0xff when read by the os, and all writes fail. So everything about it is completely hidden from the OS, though the OS can trigger the IME/SMM to run various functions through interrupts, too. But this system is also required for the CPU to even function, so killing it bricks your CPU. Which, ofc, you can do via exploits. Or install ring-2 keyloggers. or do fucking anything else you want to.

tl;dr IME is a hardware godmode, and if someone compromises this (and there have been many exploits), their code runs at ring-2 permissions (above kernel (0), above hypervisor (-1)). They can do anything and everything on/to your system, completely invisibly, and can even install persistent malware that lives inside your bloody cpu. And guess who has keys for this? Go on, guess. you're probably right. Are they completely trustworthy? No? You're probably right again.

There is absolutely no reason for this sort of thing to exist, and its existence can only makes things worse. It enables spying of literally all kinds, it enables cpu-resident malware, bricking your physical cpu, reading/modifying anything anywhere, taking control of your hardware, etc. Literal godmode. and some of it cannot be patched, meaning more than a few exploits require replacing your cpu to protect against.

And why does this exist?
Ostensibly to allow sysadmins to remote-manage fleets of computers, which it does. But it allows fucking everything else, too. and keys to it exist. and people are absolutely not trustworthy. especially those in power -- who are most likely to have access to said keys.

The only reason this exists is because fucking power-hungry doucherockets exist.

Now, I work at a hosting company in the UK, as a linux support engineer. I've seen many cases where a number of clients ran one of the following:

rm -rf / something
rm -rf /var/cache (attempt to magento)
chmod 777 /var
chown -R user:user /*

Half the time, they're like "Hey guys, I dun did fuck up, please help!". The other half of the time, they piss me off. Here's a number of responses that really grinds my gears:

"Such a harmful command should really prompt for input before running" -- From the buy you "forced" a recursive rm command, which mutes such a feature.

Client: "I did no such thing"
Me: "I've seen the command history, and at the time the command was run, you were the only person logged in"
Client: "You're mistaken, You're reading the information wrong"
Me: "I assure you, I'm not, I know what I'm looking at"
Client: "Well you're a shit engineer"
Me (thought): "Says the fucker who doesn't know how to linux"

I like people who own up to fuck ups. But the ones that don't, are just making their lives harder, since we have all the evidence in front of us.

Most of these people are the developers, and in some cases, the sysadmins...

vegans and sysadmins likely the same

Rip WPA2
Rip SysAdmins

I have no idea why but I think all ranters with long hair are Linux sysadmins

Long story short, I'm unofficially the hacker at our office... Story time!

So I was hired three months ago to work for my current company, and after the three weeks of training I got assigned a project with an architect (who only works on the project very occasionally). I was tasked with revamping and implementing new features for an existing API, some of the code dated back to 2013. (important, keep this in mind)

So at one point I was testing the existing endpoints, because part of the project was automating tests using postman, and I saw something sketchy. So very sketchy. The method I was looking at took a POJO as an argument, extracted the ID of the user from it, looked the user up, and then updated the info of the looked up user with the POJO. So I tried sending a JSON with the info of my user, but the ID of another user. And voila, I overwrote his data.

Once I reported this (which took a while to be taken seriously because I was so new) I found out that this might be useful for sysadmins to have, so it wasn't completely horrible. However, the endpoint required no Auth to use. An anonymous curl request could overwrite any users data.

As this mess unfolded and we notified the higher ups, another architect jumped in to fix the mess and we found that you could also fetch the data of any user by knowing his ID, and overwrite his credit/debit cards. And well, the ID of the users were alphanumerical strings, which I thought would make it harder to abuse, but then realized all the IDs were sequentially generated... Again, these endpoints required no authentication.

So anyways. Panic ensued, systems people at HQ had to work that weekend, two hot fixes had to be delivered, and now they think I'm a hacker... I did go on to discover some other vulnerabilities, but nothing major.

It still amsues me they think I'm a hacker 😂😂 when I know about as much about hacking as the next guy at the office, but anyways, makes for a good story and I laugh every time I hear them call me a hacker. The whole thing was pretty amusing, they supposedly have security audits and QA, but for five years, these massive security holes went undetected... And our client is a massive company in my country... So, let's hope no one found it before I did.

Sysadmins should not be doctors

3 rants for the price of 1, isn't that a great deal!

1. HP, you braindead fucking morons!!!

So recently I disassembled this HP laptop of mine to unfuck it at the hardware level. Some issues with the hinge that I had to solve. So I had to disassemble not only the bottom of the laptop but also the display panel itself. Turns out that HP - being the certified enganeers they are - made the following fuckups, with probably many more that I didn't even notice yet.

- They used fucking glue to ensure that the bottom of the display frame stays connected to the panel. Cheap solution to what should've been "MAKE A FUCKING DECENT FRAME?!" but a royal pain in the ass to disassemble. Luckily I was careful and didn't damage the panel, but the chance of that happening was most certainly nonzero.
- They connected the ribbon cables for the keyboard in such a way that you have to reach all the way into the spacing between the keyboard and the motherboard to connect the bloody things. And some extra spacing on the ribbon cables to enable servicing with some room for actually connecting the bloody things easily.. as Carlos Mantos would say it - M-m-M, nonoNO!!!
- Oh and let's not forget an old flaw that I noticed ages ago in this turd. The CPU goes straight to 70°C during boot-up but turning on the fan.. again, M-m-M, nonoNO!!! Let's just get the bloody thing to overheat, freeze completely and force the user to power cycle the machine, right? That's gonna be a great way to make them satisfied, RIGHT?! NO MOTHERFUCKERS, AND I WILL DISCONNECT THE DATA LINES OF THIS FUCKING THING TO MAKE IT SPIN ALL THE TIME, AS IT SHOULD!!! Certified fucking braindead abominations of engineers!!!

Oh and not only that, this laptop is outperformed by a Raspberry Pi 3B in performance, thermals, price and product quality.. A FUCKING SINGLE BOARD COMPUTER!!! Isn't that a great joke. Someone here mentioned earlier that HP and Acer seem to have been competing for a long time to make the shittiest products possible, and boy they fucking do. If there's anything that makes both of those shitcompanies remarkable, that'd be it.

2. If I want to conduct a pentest, I don't want to have to relearn the bloody tool!

Recently I did a Burp Suite test to see how the devRant web app logs in, but due to my Burp Suite being the community edition, I couldn't save it. Fucking amazing, thanks PortSwigger! And I couldn't recreate the results anymore due to what I think is a change in the web app. But I'll get back to that later.

So I fired up bettercap (which works at lower network layers and can conduct ARP poisoning and DNS cache poisoning) with the intent to ARP poison my phone and get the results straight from the devRant Android app. I haven't used this tool since around 2017 due to the fact that I kinda lost interest in offensive security. When I fired it up again a few days ago in my PTbox (which is a VM somewhere else on the network) and today again in my newly recovered HP laptop, I noticed that both hosts now have an updated version of bettercap, in which the options completely changed. It's now got different command-line switches and some interactive mode. Needless to say, I have no idea how to use this bloody thing anymore and don't feel like learning it all over again for a single test. Maybe this is why users often dislike changes to the UI, and why some sysadmins refrain from updating their servers? When you have users of any kind, you should at all times honor their installations, give them time to change their individual configurations - tell them that they should! - in other words give them a grace time, and allow for backwards compatibility for as long as feasible.

3. devRant web app!!

As mentioned earlier I tried to scrape the web app's login flow with Burp Suite but every time that I try to log in with its proxy enabled, it doesn't open the login form but instead just makes a GET request to /feed/top/month?login=1 without ever allowing me to actually log in. This happens in both Chromium and Firefox, in Windows and Arch Linux. Clearly this is a change to the web app, and a very undesirable one. Especially considering that the login flow for the API isn't documented anywhere as far as I know.

So, can this update to the web app be rolled back, merged back to an older version of that login flow or can I at least know how I'm supposed to log in to this API in order to be able to start developing my own client?

> Customer logs Jira ticket claiming app is not working
< I restart the app, investigate and explain tht their server has issues
ø Client closes the ticket as Resolved
-- a couple of days pass by ---

> Customer logs Jira ticket claiming app is not working
< I restart the app, investigate and explain tht their server has issues
ø Client closes the ticket as Resolved
-- a couple of days pass by ---

> Customer logs Jira ticket claiming app is not working
< I restart the app, investigate and explain tht their server has issues
ø Client closes the ticket as Resolved
-- a couple of days pass by ---
<...>

< I log a JIRA ticket explaining what and how is wrong with the server with suggestions how to fix the problem so the app will not crash any longer (client own the server, has his own sysadmins -- I don't even had permissions to open syslog.. had to hack dmesg on their PROD server to pin-point the issue)
> no reaction from customer for weeks. I ping the ticket

× app crashes again

> no reaction from customer for weeks. I ping the ticket
> customer leaves a comment that their sysadmins are looking at it trying to figure out what might be wrong (ignoring what I wrote in ticket's description??? srsly?)

× app crashes again
< I post detail investigation details: snips from logs, screenshots, everything with crystal clear explanations.

> no reaction for weeks
......

well that's fun..

This happend to me around 2 weeks ago. For some reason, I decied to post this now.

I won the lottery, yey! I mean, bot really, but I am <19yo student, "less than junior dev" in my office, but sonce I am the only one who is capable of working with hardware, I was working month back as a sysadmin for a few days. Our last sysadmin was really good working but really, really toxic guy, so he got fired on a spot after argument with some manager or whatever, no big deal, we could have another guy hired in a week. But, our backup server literally was on fire, all data probably dead because bad capacitor or whatever. This was our only backup of everything at the time. Everyone in full fucking panic mode, we had literally no other working HW we could use for backup, but then comes me, intern employed on his first dev job for 3 months. That day I bought some HW for my own personal server at home (Intel NUC with some Celeron, 4GB DDR4 RAM and two 240GB SSDs for RAID 1. My manager asked everyone in the office for sollution how to survive next 4 days before new server arrives. People there had no idea what tk do and no knowedgle about HW, I just came from a break and offered my components for a week, since there was noone else who can work with HW, servers and stuff like this, manager offered me $500+HW cost if I, random intern, can make it work. I installed Debian on that little PC, created RAID1 from both SSDs, installed MySQL server and mirrored GIT server from our last standing server (we had two before one of them went lit 🔥), made simple Python script to copy all data on that RAID, with some help of our database guy copied whole DB from production to this little computer and edited some PHP so every SQL request made on our server will run on that NUC too. Everything after ±2 hours worked perfectly. Untill a fucking PSU burned in our server and took RAID controller with him in sillicon heaven next night, so we could not access any data unltill we got a new one. Thanks to every god out there, I was able to create software RAID from survived HDDs on our production server and copy all data from that NUC on the servers software RAID and make it working at 3 AM in the night before an exam 😂. Without this, we would be next ±40 hours without aerver running and we might loose soke of our data and customers. So my little skill with Linux, Python, MySQL and most importantly my NUC hardware I got that day running as a backup server saved maybe whole company 😂.
Btw, guess who is now employee of the year with $2500 bonus? 😀
Sorry for bragging and log post, but I was so lucky an so happy when everything worked out, good luck to all sysadmins out there! 👍

TL:DR: Random intern saved company and made some money 😂

When the company is treating u like shit... Blaming everything on you and you just get fed up with em.. You be like...

I'm seeing a pattern here... We devs/testers/sysadmins/etc. don't get to spend too much time outside... We talk about different stuff than most people... We are more intelligent than most people so we don't get their dumb jokes... Most of us like to work at night because that's the time when nobody bothers us...

We don't get a chance to find a girlfriend, we don't understand how it works...

We are doomed

Every year my team runs an award ceremony during which people win “awards” for mistakes throughout the year. This years was quite good.

The integration partner award- one of our sysAdmins was talking with a partner from another company over Skype and was having some issues with azure. He intended to send me a small rant but instead sent “fucking azure can go fuck itself, won’t let me update to managed disks from a vhd built on unmanaged” to our jv partner.

Sysadmin wannabe award (mine)- ran “Sudo chmod -R 700 /“ on one of our dev systems then had to spend the next day trying to fix it 😓

The ain’t no sanity clause award - someone ran a massive update query on a prod database without a where clause

The dba wannabe award - one of our support guys was clearing out a prod dB server to make some disk space and accidentally deleted one of the databases devices bringing it down.

The open source community award - one of the devs had been messing about with an apache proxy on a prod web server and it ended up as part of a botnet

There were others but I can’t remember them all

I didn't scream.. just told him to jump off of terrace..
What ticked me?! He was a support guy..slowest mofo ever..
I was in the middle of fixing major fuckup on prod, when our VPN to client disconnected. I rushed over to support to ask if it is 'just' an expired session (which he was in charge of renewing but constantly fucked up) or if there is some other problem, so I know how to proceed..do I need to contact our sysadmins, client's support guys etc..
He
started
to
slooooooooowly
explain
I
am
not
the
only
one
with
VPN
problems
...
Was that what I asked you?! // he had an annoying habit of slooooowly talking and explaining unrelated things & personal stuff that bothered him & most of the times he chose the most time sensitive period to drone off..
So I cut him of saying, that others were probably not 'tinkering' with production and that I need this back ASAP, so if he could tell me when the session will be renewed or if there is something else problematic..
He said he will check..I didn't move.. he looked at me insurprise, you want me to check *NOW*?! Yeah, it's urgent.. He proceeded very very veeeery slooooowly, taking the support phone../* he was even eating sandwich during that, so only one hand free, typing one letter at a min */
I was finaly notified that the session expired and that he will fix it soon (meaning in 15-20mins o.O which should not take him more than 5).. and was like 'can I do sth else for you'?! Yeah, do the backflip.. you know the rest..

I've been lurking for a while but I had it up to here with these goddamned "js sucks" posts.

I'm not gonna deny js has severe design problems,
or that chromium is a motherfucking vampire
or that it's a goddamn pain in the ass to understand how to babel webpack + plugins correctly

that is all true.

the problem is that it's just a lazy damn circlejerk at this point where no learning is gained, with no outlook on any possible solution of these problems, let alone ANY type of actual collaboration to help the situation.

sometimes people don't even care to specify what is specifically wrong with js. It's just "js sucks" and that's it, farm ++.

slack is a ram hog, yes, yes, we know... WE KNOW.
every 5 days someone has to remind that!

is there any solution? why is it a ram hog? is electron the problem, or is the slack source code doing weird shit?
are there any lightweight alternatives to electron?

That's actual good conversation, but no, apparently it's impossible to drop the snarky tone for 2 seconds.

I think it's fine to point out defficiencies in applications, but it's not ok to shitpost on and on.

I would very ok with someone shitcomplaining about js is if they were doing something about it.

I'm still ok with people letting of some steam, I'm fine with people expressing frustration from direct work experience with js. I'm not ok with people and their ignorance and snarky comments and non helpfulness while comfortably laughing from their own camp of totally unrelated technologies.

Hearing sysadmins or people that code exclusively in c shit on js makes me feel my insides twirl.

Imagine I didn't do shit for linux, but I went around forums pointing out the defficiencies, like the lack of standards, and saying that mac is way better.

Or I if yapped on and on about openvpn and having an obscure as fuck api, meanwhile not doing a single fucking thing about it, or not even using it in a day to day basis.

do you hate slack's ram usage? me too and js isn't going anywhere in the next 5 years, so either do something or provide smart conversation, diagnosis of the problem or possible alternstives/solutions, otherwise stfu

Baby cream. It will make your babies become sysadmins by the age of 3 😂

My school just tried to hinder my revision for finals now. They've denied me access just today of SSHing into my home computer. Vim & a filesystem is soo much better than pen and paper.

So I went up to the sysadmin about this. His response: "We're not allowing it any more". That's it - no reason. Now let's just hope that the sysadmin was dumb enough to only block port 22, not my IP address, so I can just pick another port to expose at home. To be honest, I was surprised that he even knew what SSH was. I mean, sure, they're hired as sysadmins, so they should probably know that stuff, but the sysadmins in my school are fucking brain dead.

For one, they used to block Google, and every other HTTPS site on their WiFi network because of an invalid certificate. Now it's even more difficult to access google as you need to know the proxy settings.

They switched over to forcing me to remote desktop to access my files at home, instead of the old, faster, better shared web folder (Windows server 2012 please help).

But the worst of it includes apparently having no password on their SQL server, STORING FUCKING PASSWORDS IN PLAIN TEXT allowing someone to hijack my session, and just leaving a file unprotected with a shit load of people's names, parents, and home addresses. That's some super sketchy illegal shit.

So if you sysadmins happen to be reading this on devRant, INSTEAD OF WASTING YOUR FUCKING TIME BLOCKING MORE WEBSITES THAN THEIR ARE LIVING HUMANS, HOW ABOUT TRY UPPING YOUR SECURITY, PASSWORDS LIKE "", "", and "gryph0n" ARE SHIT - MAKE IT BETTER SO US STUDENTS CAN ACTUALLY BROWSE MORE FREELY - I THINK I WANT TO PASS, NOT HAVE EVERY OTHER THING BLOCKED.

Thankfully I'm leaving this school in 3 weeks after my last exam. Sure, I could stay on with this "highly reputable" school, but I don't want to be fucking lied to about computer studies, I don't want to have to workaround your shitty methods of blocking. As far as I can tell, half of the reputation is from cheating. The students and sysadmins shouldn't have to have an arms race between circumventing restrictions and blocking those circumventions. Just make your shit work for once.

**On second thought, actually keep it like that. Most of the people I see in the school are c***s anyway - they deserve to have half of everything they try to do censored. I won't be around to care soon.**

This one's for all the SysAdmins out there.

About 4 years ago I was asked to take over a dental offices systems administration (~20 machines) after their previous guy had allowed their servers RAID 1 to fail and hadn't done any updates or general maintenance. (please take note this office is my parents dental office).

I since have been recovering from his poor configuration and setup by instating an active directory environment and installing up to date software as well as updating machines on the domain to Windows 10 since windows 7 is no longer supported. I have also been properly licensing everything.

My bosses (my parents) are annoyed with this because "it's more expensive" and "it's too complicated we don't know how to manage it" and I don't know how to explain to them that they aren't fucking systems admins. They asked why they could do it before and I tried to explain that now it's secure and things need to be rolled out on the network level. They had every user running full local admin on every workstation plus the server.

Some people don't fucking understand that just because it's simple doesn't make it a good fucking idea. And because it's cheap doesn't mean it will always be (just wait till Microsoft audits you).

Oh and they also don't understand fucking CAL licensing and refuse to pay for gsuite for all their staff who use it. Instead they just have two gsuite accounts and give everyone the fucking password.

I'm going to have an aneurysm

A cream for sysadmins :)

In ancient times, a friend and I made a new website for a golf course, in exchange for free golf whenever we wanted it. We were traveling to Texas for work(we were Linux sysadmins for a defense contractor at the time) and found out that mechanical/logistic issues at the airport in Houston would delay our departure by two hours, but this wasn't until after the plane was fully boarded and had begun to taxi. So we sat on the tarmac at Kansas City Airport for two hours with nothing to do but release that website. We finished some perl hooks to site resources, and pushed the site live. This was on a laptop tethered to a phone with a CDMA data connection, before even EVDO was released.

Even so, it went great! I sshed into the server(running netBSD), swung over the necessary tags, and the site was up.

My workflow today is largely the same, just with git and a more elaborate .vimrc.

When your sysadmins can't script a file compare and so you do the code for them.
"Sorry but we can't run unknown code on the server"
Read the code then you vile troglodytes!

Did a bunch more cowboy coding today as I call it (coding in vi on production). Gather 'round kiddies, uncle Logan's got a story fer ya…

First things first, disclaimer: I'm no sysadmin. I respect sysadmins and the work they do, but I'm the first to admit my strengths definitely lie more in writing programs rather than running servers.

Anyhow, I recently inherited someone else's codebase (the story of my profession career, but I digress) and let me tell you this thing has amateur hour written all over it. It's written in PHP and JavaScript by a self-taught programmer who apparently discovered procedural programming and decided there was nothing left to learn and stopped there (no disrespect to self-taught programmers).

I could rant for days about the various problems this codebase has, but today I have a very specific story to tell. A story about errors and logs.

And it all started when I noticed the disk space on our server was gradually decreasing.

So today I logged onto our API server (Ubuntu running Apache/PHP) and did a df -h to check the disk space, and was surprised to see that it had noticeably decreased since the last time I'd checked when everything was running smoothly. But seeing as this server does not store any persistent customer data (we have a separate db server) and purely hosts the stateless API, it should NOT be consuming disk space over time at all.

The only thing I could think of was the logs, but the logs were very quiet, just the odd benign message that was fully expected. Just to be sure I did an ls -Sh to check the size of the logs, and while some of them were a little big, nothing over a few megs. Nothing to account for gigabytes of disk space gradually disappearing.

What could it be? I wondered.

cd ../..
du . | sort --sort=numeric

What's this? 2671132 K in some log folder buried in the api source code? I cd into it and it turns out there are separate PHP log files in there, split up by customer, so that each customer of ours (we have 120) has their own respective error log! (Why??)

Armed with this newfound piece of (still rather unbelievable) evidence I perform a mad scramble to search the codebase for where this extra logging is happening and sure enough I find a custom PHP error handler that is capturing (most) errors and redirecting them to these individualized log files.

Conveniently enough, not ALL errors were being absorbed though, so I still knew the main error_log was working (and any time I explicitly error_logged it would go there, so I was none the wiser that this other error-catching was even happening).

Needless to say I removed the code as quickly as I found it, tail -f'd the error_log and to my dismay it was being absolutely flooded with syntax errors, runtime PHP exceptions, warnings galore, and all sorts of other things.

My jaw almost hit the floor. I've been with this company for 6 months and had no idea these errors were even happening!

The sad thing was how easy to fix all the errors ended up being. Most of them were "undefined index" errors that could have been completely avoided with a simple isset() check, but instead ended up throwing an exception, nullifying any code that came after it.

Anyway kids, the moral of the story is don't split up your log files. It makes absolutely no sense and can end up obscuring easily fixable bugs for half a year or more!

Happy coding.

I previously worked as a Linux/unix sysadmin. There was one app team owning like 4 servers accessible in a very speciffic way.
* logon to main jumpbox
* ssh to elevated-privileges jumpbox
* logon to regional jumpbox using custom-made ssh alternative [call it fkup]
* try to fkup to the app server to confirm that fkup daemon is dead
* logon to server's mgmt node [aix frame]
* ssh to server directly to find confirm sshd is dead too
* access server's console
* place root pswd request in passwords vault, chase 2 mangers via phone for approvals [to login to the vault, find my request and aprove it]
* use root pw to login to server's console, bounce sshd and fkupd
* logout from the console
* fkup into the server to get shell.

That's not the worst part... Aix'es are stable enough to run for years w/o needing any maintenance, do all this complexity could be bearable.

However, the app team used to log a change request asking to copy a new pdf file into that server every week and drop it to app directory, chown it to app user. Why can't they do that themselves you ask? Bcuz they 'only need this pdf to get there, that's all, and we're not wasting our time to raise access requests and chase for approvals just for a pdf...'

oh, and all these steps must be repeated each time a sysadmin tties to implement the change request as all the movements and decisions must be logged and justified.

Each server access takes roughly half an hour. 4 servers -> 2hrs.

So yeah.. Surely getting your accesses sorted out once is so much more time consuming and less efficient than logging a change request for sysadmins every week and wasting 2 frickin hours of my time to just copy a simple pdf for you.. Not to mention that threr's only a small team of sysadmins maintaining tens of thousands of servers and every minute we have we spend working. Lunch time takes 10-15 minutes or so.. Almost no time for coffee or restroom. And these guys are saying sparing a few hours to get their own accesses is 'a waste of their time'...

That was the time I discovered skrillex.

I miss old times rants...So i guess, here it goes mine:

Tomorrow is the day of the first demo to our client of a "forward-looking project" which is totally fucked up, because our "Technical Quality Assurance" - basically a developer from the '90-s, who gained the position by "he is a good guy from my last company where we worked together on sum old legacy project...".
He fucked up our marvellous, loose coupling, publish/subscribe microservice architecture, which was meant to replace an old, un-maintainable enormous monolitch app. Basically we have to replace some old-ass db stored functions.
Everyone was on our side, even the sysadmins were on our side, and he just walked in the conversation, and said: No, i don't like it, 'cause it's not clear how it would even work... Make it an RPC without loose coupling with the good-old common lib pattern, which made it now (it's the 4th 2 week/sprint, and it is a dependency hell). I could go on day and night about his "awesome ideas", and all the lovely e-mails and pull request comments... But back to business

So tomorrow is the demo. The client side project manager accidentally invited EVERYONE to this, even fucking CIO, legal department, all the designers... so yeah... pretty nice couple of swallowed company...

Today was a day, when my lead colleague just simply stayed home, to be more productive, our companys project manager had to work on other prjects, and can't help, and all the 3 other prject members were thinking it is important to interrupt me frequently...

I have to install our projects which is not even had a heart beat... not even on developer machines. Ok it is not a reeeeaaally big thing, but it is 6 MS from which 2 not even building because of tight coupling fucktard bitch..., But ok, i mean, i do my best, and make it work for the first time ever... I worked like 10 ours, just on the first fucking app to build, and deploy, run on the server, connect to db and rabbit mq... 10 FUCKING HOURS!!! (sorry, i mean) and it all was about 1, i mean ONE FUCKING LINE!

Let me explain: spring boot amqp with SSL was never tested before this time. I searched everything i could tought about, what could cause "Connection reset"... Yeah... not so helpful error message... I even have to "hack" into the demo server to test the keystore-truststore at localhost... and all the fucking configs, user names, urls, everything was correct... But one fucking line was missing...
EXCEPT ONE FUCKING LINE:
spring.rabbitmq.ssl.enabled=false # Whether to enable SSL support.

This little bitch took me 6 hours to figure out...so please guys, learn from my fault and check the spring boot appendix for default application properties, if everything is correct, but it is not working...

And of course, if you want SSL then ENABLE it...
spring.rabbitmq.ssl.enabled=true

BTW i really miss those old rants from angry devs, and i hope someone will smile on my fucking torture

Niantic need to hire a bunch of us a d some sysadmins, oh and some anti DDOS experts 😂

How come, that devs likes crapple products while sysadmins hate them?

I took like 3 years to my company to get this huge-ass client to ask us to remake their website (the client is already our client for other purposes).

The old website was hosted on their local machine, behind a proxy that was there for other 30 website servers.

The old website took like 30-40 seconds to load on a browser and had a google score of 3-6/100.

We made the new website in wordpress, since it was basically a blog and managed all of the older links to redirect to the new pages so that SEO wouldn't get affected.

We then asked the previous developers to let their domain redirect to the new one (it was like example.com => ex.example.com and now it's just example.com, so we needed them to make ex.example.com redirect to example.com).

What they did was making a redirection to the 404 page of the new website, making everything go to fuck itself.

Damn this might be the first time I despise other developers, but this move was fucking awful.

I mean, I get it, we stole your big client, but it's not our fault if we made the google score go up to 90/100 in a week just by changing server and CMS.

I'm not a programmer by trade, so the only language I know well is bash. But as sysadmins we do use bash often.

Looking at other sysadmins' scripts though, there are interesting things in it every so often. Like for example `touch file` which creates a file. I've seen some sysadmins just do this instead `> file`. Genius! Or perhaps a `cat file >> elsewhere`. You can do that with `< file >> elsewhere`. It's something that if I hadn't seen it elsewhere, I wouldn't have thought about. But yeah, it saves a program call and it works!

All sysadmins, PLEASE! For the love of God just block port 21 in any direction from anywhere, going anywhere.. FTP needs to die.. The f**king protocol predates tcp/ip for God's sake! We need to stop project managers using it, it's a nightmare!!

What is it with devs (not all, by any means!) who don't understand networks or basic computer operation? I'm not talking about anything complex, but things like the dev who asked if his IP address could be whitelisted so he could remote in from home. We asked what his public IP address is and he said 10.0.0.27.

Or the new dev who started and said her laptop camera didn't work and logged a ticket, only to be asked if she had the camera cover open or closed and said, "oh, that's what that lever is for."

Don't get me wrong - many devs and sysadmins and IT people of all fields are excellent. And there are some who are crap in every field. This is no rant about devs in general, just *these* crap devs that I can only throw my hands in the air and think, well, they scored ok in the SQL test.

Crappy Kryptonian SysAdmins

Die linux fanboy, die!

No seriously. I'm not trying to be salty, but i'm fed up with sysadmins and developers that proclaim that linux will one day save the world from the clutches of companies like microsoft.

STOP BOTHERING ME!

All my respect goes to sysadmins, ive been trying to hack together a cronjob for tha past 2 hours and its still not working.

Proxmox team, go fuck yourselves.

Now I'm sure that I'll receive a lot of flack for this, but hear me out.

I've tried Proxmox and was quite pleased with its web UI. But I hate how much it locks me into their own little ecosystem.

I want to use btrfs on my drives. Why is this impossible, yet the hack that is ZoL is your obvious alternative? An alternative wherein I can't even compile and run my own kernel, because then ZoL suddenly fails? And don't you tell me to compile your stock config, when it's well over 15GB large in your source tree.

Proxmox is literally the MacOS of Linux distributions. Which was even more so made clear by me being called an idiot by possibly wanting to run Same on the PVE host. Because why on Earth would sysadmins want to?! Why on Earth would sysadmins be competent for wanting to?!!

You know what? I'll just convert those Proxmox servers to Arch and say fuck you to all the bells and whistles that's Proxmox' web UI. Because at least Arch allows me to make my own fucking choices, limited only by what's supported by the Linux operating system.

Perhaps Proxmox will consider btrfs stable in 2021. Because you know, despite it being stable today in 2018, Debian and Proxmox alike live 3 years in the past, i.e. 2015. I hate the Debian ecosystem because of that, but boy do I hate Proxmox even more so. Bloody fucking piece of shit it is!!! 😡

Had to change password on computer for administrative reasons (sysadmins and infosec make us change our pass every quarter). Changes didn't sync to everything so now I can't even log into my computer.
Need to go to the office tomorrow so some guy can type in an admin password on my pc and do stuff to it. If that doesn't work I will just be given a new laptop.

Seriously fuck this week

fuck oracle. fuck my company.

Using Oracle VM Manager/Servers to host Oracle Phone transfer solution without support coverage from Oracle.

Requiring Unix sysadmins to update to latest release and not telling that we do not have coverage from Oracle if anything goes wrong.

Gues what.. We've updated to Oracle VM Manager/Server 3.4.5 which was released this year and it uses fucking XEN hypervisor version 4.4.4 which has been deprecated and dead since who knows when. Latest release of XEN is 4.11. But that is not an issue, whatever, enterprise, legacy software, etc.

This fucking update introduced memory leak on the hypervisor which has been reported as per xen 4.4.4 history. Furthermore, we have no support from Oracle which means that I have to dig through mailing lists and limited information on the net since oracle has freakin support wall on nearly each of the major bugs found on that shitty software.

I have no idea whether any newer version of xen will work with that old Oracle Linux kernel or not.

Furthermore, Oracle provided great documentation on how to rollback the fcking update. Reinstall the hypervisor. Riiiight. XEN does not have export/import feature.

eh

Canoncal.. buddy.. pal..

We need to talk about the content on the server image's login screen.

Now, I get that lots of developers will use the server image out of a desire to keep their environments minimal.. but at the same time, is the same server image that will be deployed on thousands of VMs all over the world really the place to be talking about "great IDEs available on Ubuntu" complete with smiley faces?

I'm dead serious I log in and there are fifty seven lines of crap on the screen. I don't need links to your docs or support pages, I definitely don't need cutesy links to "hey look at this cool stuff you can do on Ubuntu!", and I absolutely don't need advertisements for your paid services.

This is some of the tackiest stuff I've seen outside of Gitlab shilling for GKE in the paid enterprise version.

Stuff like this turns actual users off. Sysadmins, the ones who are going to be seeing this stuff since it's visible on SSH shells only do not care about your cutesy IDE advertising.

Grow up.

But fuck, why are so many so called devs so fucking stupid?
Or have life decided that I should get all the stupid ones?

If the last one is true, Am I the Jesus for sysadmins?

A peeve of mine is when someone in the software industry denigrates a technology/tool/framework outside of his role eg webdevs on sysadmin stuff or viceversa.

I'm not trying to shame anyone for having subjective experiences, I just think that if you're gonna talk about tools that are not on your domain, then you need to be twice as humble as usual.

I'm a webdev and I don't post around how I KNOW how to make ssh secure, while other people devote their entire careers to that and all related matters.

What prompted me is seeing some not webdevs do this here that seem to be sysadmins/devops (can't tell for sure since I don't know them), but in real life, I've seen people from any role do this, webdevs too, even testers!

Imagine you had cancer, and you had a tumor extraction, and the oncologist said to the surgeon "step aside son, let me show you how to deal with cancer".

Scary sysadmin is scary

alias fmljustdoit='sudo'

Any sysadmins on devrant?

I thought the “works on my machine” from sysadmins is mostly a joke...

But no. I’m attending the Polish high school and I have a lessons via the Internet. I wasn’t able to hear the voice of the teacher on Linux 4.18 and Chromium browser, so I sent the mail to support to report it. Of course it “works on their machines”...

is it a bad thing you hear your uni's sysadmins sing Aqours' Aozora Jumping Heart in the cringiest way possible?

Any Windows Sysadmins here? I have a question for you - How do you do it?

I only very rarely have to do something that would fall under "Windows System Administration", but when I do... I usually find something either completely baffling, or something that makes me want to tear our my hair.

This time, I had a simple issue - Sis brought me her tablet laptop (You know, the kind of tablets that come with a bluetooth keyboard and so can "technically" be called a laptop) and an SD card stating that it doesn't work.

Plugging it in, it did work, only issue was that the card contained file from a different machine, and so all the ACLs were wrong.

I... Dealt with Windows ACLs before, so I went right to the usual combination of takeown and icacls to give the new system's user rights to work with the files already present. Takeown worked fine... But icacls? It got stuck on the first error it encountered and didn't go any further - very annoying.

The issue was a found.000 folder (Something like lost+found folder from linux?) that was hidden by default, so I didn't spot it in the explorer.

Trying to take ownership of that folder... Worked for for files in there, safe for one - found.000\dir0000.chk$Txf; no idea what it is, and frankly neither do I care really.

Now... Me, coming from the Linux ecosystem, bang my head hard against the table whenever I get "Permission denied" as an administrator on the machine.

Most of the times... While doing something not very typical like... Rooting around (Hah... rooting... Get it?! I... Carry on) the Windows folder or system folders elsewhere. I can so-so understand why even administrators don't have access to those files.

But here, it was what I would consider a "common" situation, yet I was still told that my permissions were not high enough.

Seeing that it was my sister's PC, I didn't want to install anything that would let me gain system level permissions... So I got to writing a little forloop to skip the one hidden folder alltogether... That solved the problem.

My question is - Wtf? Why? How do you guys do this sort of stuff daily? I am so used to working as root and seeing no permission denied that situations like these make me loose my cool too fast too often...

Also - What would be the "optimal" way to go about this issue, aside for the forloop method?

The exact two commands I used and expected to work were:
takeown /F * /U user /S machine-name /R
icacls * /grant machine-name\user:F /T

When I was younger I had a decision to go into hardware or software. I chose software and have loved it.

Recentily I just spent 5 hours trying install a Linux distro on an old server. I made no progress.

I made the right decision. Hardware freaking sucks! You spend hours working on outdated pieces of crap and find that to fix your problem you need to sell you kidney to finance your project. Not to mention you have to wait for literally everything! It's like gradel builds everywhere! Want to install a new distro on your USB? Bam, 5min gone. Want to boot into bios and change one setting? BAM! more time wasted...

A note to the sysadmins out there: thank you. I love you. I am so happy you do this kind of work so I don't have to.

!Rant

Sysadmins don't have an easy job. Dev side is brighter.

Hey, sysadmins / legacy devs.

What is the cheapest way to get my hands on unices' shells? AIX 5.3/6.1/7.1, SunOS 5.x, HP-UX 11.*,..

Needed for playing around.

Security patches installed and servers blessed by the Pope... Ready for any threat!

Sadly, I’m not a good enough developer to have clever and hacky solutions to anything. In college I did once use Visual Basic to spoof a Novell login screen and steal other students’ passwords and write them to a diskette, which I’d recover after they walked away from the machine. The worst I did to them was log into their messaging and send them messages from themselves. Oh, and I also set up an “underground” web site that the campus sysadmins didn’t discover for a while. I used it to set up a forum where students could sell their used textbooks for better prices than the buy back program at the campus bookstore.

My work's website being unsupported with Django security patches. I bring this up with management and say we should upgrade ASAP. Apparently that wasn't possible because the sysadmins refused to upgrade their old version of RedHat so we can't use anything that doesn't support Python 2.6.... To this day it still runs on Django 1.6.

To the sysadmins:
We are currently installing and configuring dns, dhcp, smb, ftp and webserver on Ubuntu vms in class.
The teacher changed his mind and want us to install a system that has everything installed and configured.
He recommended Zentyal to us.
What would you use and why would you use it?

To the SysAdmins and Linux-Engineers here,
How you got into your job?
I'm a Linux-Fanboy by heart and love tinkering and fiddling with systems, so I want to work in the field?
Please tell me, how you got into your job, and how it is to work in the field.

Hello sysadmins, silly question but can I consider Python as a serious alternative to powershell/bash? I have always hidden myself from learning bash considering myself not that kind of guy hacking around in Linux. Thanks in advance

Ugh, I feel like fucking crying every time I have to explain to the other developers and network sysadmins that we can't have the same Redis server for both session and caching data in the current project we're working on. I wrote all this down in the specifications which NO ONE reads!!

There are times like these I wish I just had access to the AWS account so I can do all of this myself.

SYSADMINS!

Any good, modern alternatives to PuTTy?

- Every specialist is looking after his area of expertise

- Everyone is a specialist of everything and shall work on everything

- Every specialist is looking after his area of expertise, making improvements and automations in his area

- Everyone is a specialist of everything and is looking after everything, automating everything (devops)

- Everyone is a specialist of everything and is looking after everything, automating everything, in all the environments (SRE)

- ... I wonder what's next...

I miss the good old days when developers could be developers and rely on DBAs, sysadmins and networkists to do their job well. I miss the days when developers were developing applications, sytems, modules,.. Not troubleshooting ELBs, RDS latencies or building monitoring for servers.

!rant
Happy mother's day to all the mommy sysadmins and developers!

My best mentoring experience was the teacher that taught me puppet. The way he explained it, the way he walked you through a very intense course of three days consecutive puppet knowlegde rammed into your head...and still manage to actually learn almost all of it. Great attitude, and meanwhile, he is a esteemed collegue and friend...

Without you Johan, i would not be an independent entrepreneur and making bank at my current client

Johan the white, to me one of the greatest sysadmins ever

Hi Hi fellow devs and sysadmins hiding in the dark,

In august of next year I need to do my final internship for my IT Management Study in The Netherlands. My study level is between the US Community College and Associate Degree.

As you can see from the tags I'm looking for a International internship as I'm looking for experience and a challenge but its annoying and pretty hard to find something on the internet due to all of the places that "help you find a place" that ask 4.5 EUR.

So I wanted to ask you guys for help, I'm looking as you can guess for a IT Management internship and I have a slight preference to Asia(With very big preference to Japan) due to interest in the culture.

If you guys know of any places in your company or other places I would love to hear and if you have tips please share them.

Best regard,
Marcel aka inpothet

remember: the conficker table is still really useful in 2020 due to the prevalence of laziness in users and sysadmins. Use with permutations of some sort!

Is it just in the company I work for, that all sysadmins, after they login they type sudo -i. Kind of shows the powerhunger :)

Any former sysadmins here?
I am one for the past 4 years but I had enough. I really want to switch to the developer world. Currently self-learning in my free time. I already know java, python and javascript.
Any advices?

Bribing sysadmin with an ice cold Coke hoping he will deploy my latest tag sort of in a hurry since I fudged up last one :/

I hate coming to work and not seeing my team. I’m contractor & expected to be there amongst the other SysAdmins. This **** make ya quit. I’m discouraged rn. I have interview for automated warehouse on Thurs. My contract almost up at current & have heard no talk of in-house hire. Not sure if I’m supposed to negotiate again or what but I have had no training and came out of nowhere as they struggle to keep and find talent and I kill it. I have completed projects, organized assets, closed mid-high ticket count weekly, etc. I like showing up. Imma go. Anyone got wisdoms or words for meh? Maybe it’s a surprise!!!🎉 Like a birthday or sumn.

!rant
Yoo fellow devs... And sysAdmins??
We're tasked with migrating our apps from current HP UX server to RHEL...
Most of our apps run on Tomcat 7 and weblogic...
We also have Oracle DB Enterprise Edition...
Just wondering if any of you out there from the stone age have done this recently... Any tips.. Warnings... Advice on the matter??

DevRant should have another category for bashing/ranting about Sysadmins since we usually deal with many incompetent ones.

I see more and more posts about them as I see growing number of devranters.

@VirtualizationPeasants
@SysAdmins
XenServer or Proxmox? Or other solutions and why?

Any sysadmins/devops browsing here?

To all my sysadmins/devops, what's your opinion on GoAccess (if you've used it or are currently using it)?

This Post is German and a question relating to the German School System.

Also: Ich bin momentan ein Abiturient und merke das ich das Abitur vermutlich nicht schaffen werde. (Aufgrund einiger Fächer), stehe aber Informatik eins und möchte gerne ein Systemintegrator (sysadmin) werden. Die Frage ist ob ich das Abi durchziehen sollte auch wenn es grade zu einem 4,0 Durschnitt (oder keinem) kommen könnte oder ob ich lieber jetzt zu einer Ausbildung und somit ein Fachabitur machen sollte...

Ich weiß wirklich nicht was ich nehmen soll deswegen dachte ich da hier einige Sysadmins sind ob ich hier vieleicht Rat bekomme.