Hey, Root? How do you test your slow query ticket, again? I didn't bother reading the giant green "Testing notes:" box on the ticket. Yeah, could you explain it while I don't bother to listen and talk over you? Thanks.

And later:
Hey Root. I'm the DBA. Could you explain exactly what you're doing in this ticket, because i can't understand it. What are these new columns? Where is the new query? What are you doing? And why? Oh, the ticket? Yeah, I didn't bother to read it. There was too much text filled with things like implementation details, query optimization findings, overall benchmarking results, the purpose of the new columns, and i just couldn't care enough to read any of that. Yeah, I also don't know how to find the query it's running now. Yep, have complete access to the console and DB and query log. Still can't figure it out.

And later:

Hey Root. We pulled your urgent fix ticket from the release. You know, the one that SysOps and Data and even execs have been demanding? The one you finished three months ago? Yep, the problem is still taking down production every week or so, but we just can't verify that your fix is good enough. Even though the changes are pretty minimal, you've said it's 8x faster, and provided benchmark findings, we just ... don't know how to get the query it's running out of the code. or how check the query logs to find it. So. we just don't know if it's good enough.

Also, we goofed up when deploying and the testing database is gone, so now we can't test it since there are no records. Nevermind that you provided snippets to remedy exactly scenario in the ticket description you wrote three months ago.

And later:

Hey Root: Why did you take so long on this ticket? It has sat for so long now that someone else filed a ticket for it, with investigation findings. You know it's bringing down production, and it's kind of urgent. Maybe you should have prioritized it more, or written up better notes. You really need to communicate better. This is why we can't trust you to get things out.

*twitchy smile*

An important message:
PrOpErLy managing servers is HARD.

I get pissed off at customers with ZERO server knowledge who think they can manage their VPS. “Just get a control panel and a VPS” from some flashy provider that makes server management look way too easy.. Clicking around in their fancy control panel, until:

- they need help with their *self-managed* VPS;
- their email ends up in spam;
- they suffer from performance issues;
- they need to restore a backup;
- something breaks, because YES, things break

Way too little people are able to answer:

- when and how do you make backups?
- how do you monitor your servers and which services?
- how do you keep track of trend analysis?

Then I come by with necessary software. SNMP for trend analysis, Graphite for infrastructure health, Sensu for monitoring, Kibana, Ansible for configuration management..

Things that servers need but that customers have never even heard of.. because they can do everything in their control panel..

Until they come crying to me because it broke and they don’t even know how to get into SSH.

I think the ones to blame are VPS providers that tell the tale of how easy it is to install a control panel and never look at your server again.

Customers become responsible for something *business-critical*! Yet they don’t know how it works.

Highlights from my week:

Prod access: Needed it for my last four tickets; just got it approved this week. No longer need it (urgently, anyway). During setup, sysops didn’t sync accounts, and didn’t know how. Left me to figure out the urls on my own. MFA not working.

Work phone: Discovered its MFA is tied to another coworker’s prod credentials. Security just made it work for both instead of fixing it.

My merchant communication ticket: I discovered sysops typo’d my cronjob so my feature hasn’t run since its release, and therefore never alerted merchants. They didn’t want to fix it outside of a standard release. Some yelling convinced them to do it anyway.

AWS ticket: wow I seriously don’t give a crap. Most boring ticket I have ever worked on. Also, the AWS guy said the project might not even be possible, so. Weee, great use of my time.

“Tiny, easy-peasy ticket”: Sounds easy (change a link based on record type). Impossible to test locally, or even view; requires environments I can’t access or deploy to. Specs don’t cover the record type, nor support creating them. Found and patched it anyway.

Completed work: Four of my tickets (two high-priority) have been sitting in code review for over a month now.

Prod release: Release team #2 didn’t release and didn’t bother telling anyone; Release team #1 tried releasing tickets that relied upon it. Good times were had.

QA: Begs for service status page; VP of engineering scoffs at it and says its practically impossible to build. I volunteered. QA cheered; VP ignored me.

Retro: Oops! Scrum master didn’t show up.

Coworker demo: dogshit code that works 1 out of 15 times; didn’t consider UX or user preferences. Today is code-freeze too, so it’s getting released like this. (Feature is using an AI service to rearrange menu options by usage and time of day…)

Micromanager response: “The UX doesn’t matter; our consumers want AI-driven models, and we can say we have delivered on that. It works, and that’s what matters. Good job on delivering!”

Yep.
So, how’s your week going?

So, the company I’m working for is finally merging with its sister company. That means budget for upgrading our infrastructure. And guess who volunteered to be in charge of it, and future sysops. That’s me.

Previously we haven’t had anything close to a sysops, and our servers has been neglected and never updated nor upgraded since 2012. We even had a Windows Server 2012 running with rdp wide open...

I did a stream last night with 4 other devs/ sysops. The theme were ranting and alcohol, lasted for almost 8 hours.
When I woke up this afternoon, I regret it all, but totally doing it next month too

👧🏻 : Why CLI is faster than GUI? 🤔
👦🏻 : Terminal Velocity 😜

Joined a new team at work 6 months ago. Immediately set upon by a useless PO who was somehow set in her ways while still being around 30 years old. Absolutely refused to change the broken team dynamic or processes in any way whatsoever. Made terrible tickets, never did refinement on tickets so they were always missing stuff and constantly blocked. Generally unlikeable and difficult to work with, incompetent at her job and resolutely refused to change literally anything to make the team function better.

She finally leaves after 6 months and the team dynamic changes immediately. Suddenly we are improving our processes, getting stakeholder input, refining tickets, taking reasonable amounts of work in a sprint. We have discussions without her butting in and getting frustrated when you bring up legitimate concerns. No longer do you have to tiptoe around and appease her ego if you want to point out the obvious flaws in the work she drew up or even just examine it from a technical perspective.

It's insane how much things can improve once you shed the dead weight of people that are just determined to be difficult and won't budge an inch to change their ways. Good riddance.

Looks like it's time to update the old CV... Christ have I really been here for 8 years.

It's been fun, the most fun time of my life but with new owners breathing on everything stuffs starting to fall to shit.

To use a SysOps analogy there are category 1 - critical warnings ringing in my ears.

I can accept a lot, but I'm genuinely concerned for the future of this place, and after trying to fix things for long enough to realise the new owners are the ones drilling the holes in the ship it's time to sink or swim, and I don't feel like sinking.

To quote billy Joel,
It seems such a waste of time
If that's what it's all about
Mama if that's movin' up
Then I'm movin' out

Took the AWS SysOps Admin exam today and failed .
Preparing to retake it, with a different strategy.

Normally you would have:
- Management
- SysOps
- DevOps
- Devs

In the company (30-50 workers) we only have:
- Management
- SysOps (they don't know how to deploy apps beyond FTP to a webhost either)
- Devs

Jepp, management does not want a specific DevOps department, because he thinks every single "I just finished the Javascript course on codecademy" person knows how to deploy an app beyond dragging&dropping it to a webhost with FTP...

I tried to propose to them that I handle DevOps and teach it to others, so we can deploy code that we deem "production ready" in a more proper manner...
They refused...
They rather stick to "just use FTP to push any changes we made directly to the production server and test changes there"

It is great feeling, to leave company and leave all your crap code to others :D

500 lines bash generic wrapper to curl (just to catch and print errors, not just silently fail as most devs tell curl to do).

It was monster that used "function overload" and "subclasses" (based on dynamic source files). Also dynamically created inline AWK script to parse curl output. It kinda worked, but amount of high-level hacks I had to use was enormous.

Never use Bash when you do not have to. Even if you have experience with it. Others don't have it and will fail miserably trying to patch your code. Just leave bash for fast bridging between programs, leave python/java/c#/go or any other proper OOP language for a job. Please ?

Nice way to start devops work today:

(from ticket comments, seems someone did not minitor disk space usage :D)

Guys, have you experience in taking AWS Certificate as a SysOps Admin associate?
I am curious to know about the preparation s and all the matters during the exam?
Thanks

PM: Can we setup this new server today ?
Me: Sure, where is the ESXi setup CD ?
PM: ehh, wasn't in the box ?
Me: nah

So Finance team can't ctrl+c, ctrl+v from what we send them and actually buy stuff we need :D

Can anyone recommend books/courses about sysOps? Mainly security for web servers. Thanks!

i am sysops/devops whatever, how do you organize your scripts using scm, is it a good practice to put it in a monorepository and all the scripts and sources are just a submodule or every script should be as a separate project? any suggestions, hints and recommendations are welcome.