toxic workplace; leaving

I haven't wanted to write this rant. I haven't even wanted to talk to anyone (save my gf, ofc). I've just been silently fuming.

I wrote a much longer rant going into far too much detail, but none of that is relevant, so I deleted it and wrote this shorter (believe it or not) version instead. And then added in more details because details.

------

On Tuesday, as every Tuesday, I had a conference call with the rest of the company. For various, mostly stupid reasons, the boss yelled at and insulted me for twenty minutes straight in front of everyone, telling me how i'm disorganized, forgetful, how can't manage my time, can't manage myself let alone others, how I don't have my priorities straight, etc. He told the sales team to get off the call, and then proceeded to yell and chew at me for another twenty minutes in front of the frontend contractor about basically the same things. The call was 53 minutes, and he spent 40 minutes of it telling me how terrible I've been. No exaggeration, no spin. The issues? I didn't respond to an email (it got lost in my ever-filling inbox), and I didn't push a very minor update last week (untested and straight to prod, ofc). (Side note: he's yelled at me for ~15 minutes before for being horribly disorganized and unable to keep up on Trello -- because I had a single card in the wrong column. One card, out of 60+ over two boards. Never mind that most have time estimates, project tags, details, linked to cards on his boards, columns for project/qa/released, labels for deferred, released to / rejected from qa, finished, in production, are ordered by priority, .... Yep. I'm totes disorganized.)

Anyway, I spent most of conference call writing "Go fuck yourself," "Choke on a cat and die asshole," "Shit code, low pay, and broken promises. what a prize position," etc. or flipping him off under the camera on our conference-turn-video-call (switched due to connection issues, because ofc video is more stable than audio-only in his mind).

I'm just.
so, so done.

I did nothing the rest of the day on Tuesday, and basically just played games on Wednesday. I did one small ticket -- a cert replacement since that was to expire the next day -- but the rest was just playing CrossCode. (fun game, fyi; totally recommend.)

Today? It's 3:30pm and I can't be bothered to do anything. I have an "urgent" project to finish by Monday, literally "to give [random third party sales guy] a small win". Total actual wording. I was to drop all other tasks (even the expiring cert lol) and give this guy his small win. fucking whatever. But the project deals with decent code -- it's a minor extension to the first project I did for the company (see my much earlier rants), back when I was actually applying myself and learning something (everything) new, enjoying myself, and architecting+writing my own code. So I might actually do the project, but It's been two days and I haven't even opened single file yet.

But yeah. This place is total and complete shit. Dealing with the asshole reminds me of dealing with my parents while growing up, and that's a subject I don't want to broach -- far too many toxic memories.

So, I'm quitting as soon as I find something new.
and with luck, this will be before assface hires my replacement-to-be, and who will hopefully quit as soon as s/he sees the abysmal codebase. With even more luck, the asshole king himself will get to watch his company die due to horrible mismanagement. (though ofc he'll never attribute it to himself. whatever.)

I just never want to see or think about him again.
(nor this fetid landfill of a codebase. bleh.)

With luck, this will be one of my last rants about this toxic waste dump and its king of the pile.
Fourty fucking minutes, what the fuck.

Welcome back to practiseSafeHex's new life as a manager.

Episode 2: Why automate when you can spend all day doing it by hand

This is a particularly special episode for me, as these problems are taking up so much of my time with non-sensical bullshit, that i'm delayed with everything else. Some badly require tooling or new products. Some are just unnecessary processes or annoyances that should not need to be handled by another human. So lets jump right in, in no particular order:

- Jira ... nuff said? not quite because somehow some blue moon, planets aligning, act of god style set of circumstances lined up to allow this team to somehow make Jira worse. On one hand we have a gigantic Jira project containing 7 separate sub teams, a million different labels / epics and 4.2 million possible assignees, all making sure the loading page takes as long as possible to open. But the new country we've added support for in the app gets a separate project. So we have product, backend, mobile, design, management etc on one, and mobile-country2 on another. This delightfully means a lot of duplication and copy pasting from one to the other, for literally no reason what so ever.

- Everything on Jira is found through a label. Every time something happens, a new one is created. So I need to check for "iOS", "Android", "iOS-country2", "Android-country2", "mobile-<feature>", "mobile-<feature>-issues", "mobile-<feature>-prod-issues", "mobile-<feature>-existing-issues" and "<project>-July31" ... why July31? Because some fucking moron decided to do a round of testing, and tag all the issues with the current date (despite the fact Jira does that anyway), which somehow still gets used from time to time because nobody pays attention to what they are doing. This means creating and modifying filters on a daily basis ... after spending time trying to figure out what its not in the first one.

- One of my favourite morning rituals I like to call "Jira dumpster diving". This involves me removing all the filters and reading all the tickets. Why would I do such a thing? oh remember the 9000 labels I mentioned earlier? right well its very likely that they actually won't use any of them ... or the wrong ones ... or assign to the wrong person, so I have to go find them and fix them. If I don't, i'll get yelled at, because clearly it's my fault.

- Moving on from Jira. As some of you might have seen in your companies, if you use things like TestFlight, HockeyApp, AppCenter, BuddyBuild etc. that when you release a new app version for testing, each version comes with an automated change-log, listing ticket numbers addressed ...... yeah we don't do that. No we use this shitty service, which is effectively an FTP server and a webpage, that only allows you to host the new versions. Sending out those emails is all manual ... distribution groups?? ... whats that?

- Moving back to Jira. Can't even automate the changelog with a script, because I can't even make sense of the tickets, in order to translate that to a script.

- Moving on from Jira. Me and one of the remote testers play this great game I like to call "tag team ticketing". It's so much fun. Right heres how to play, you'll need a QA and a PM.

*QA creates a ticket, and puts nothing of any use inside it, and assigns to the PM.
*PM fires it back asking for clarification.
*QA adds in what he feels is clarification (hes wrong) and assigns it back to the PM.
*PM sends detailed instructions, with examples as to what is needed and assigns it back.
*QA adds 1 of the 3 things required and assigns it back.
*PM assigns it back saying the one thing added is from the wrong day, and reminds him about the other 2 items.
*QA adds some random piece of unrelated info to the ticket instead, forgetting about the 3 things and assigns it back.

and you just continue doing this for the whole dev / release cycle hahaha. Oh you guys have no idea how much fun it is, seriously give it a go, you'll thank me later ... or kill yourselves, each to their own.

- Moving back to Jira. I decided to take an action of creating a new project for my team (the mobile team) and set it up the way we want and just ignore everything going on around us. Use proper automation, and a kanban board. Maybe only give product a slack bot interface that won't allow them to create a ticket without what we need etc. Spent 25 minutes looking for the "create new project" button before finding the link which says I need to open a ticket with support and wait ... 5 ... fucking ... long ... painful ... unnecessary ... business days.

... Heres hoping my head continues to not have a bullet hole in it by then.

Id love to talk more, but those filters ain't gonna fix themselves. So we'll have to leave it here for today. Tune in again for another episode soon.

And remember to always practiseSafeHex

I login this morning

Everyone going crazy as prod is broken in a million ways. 2.5 million exceptions In 24 hours.

We talk for 1 hour and solve nothing.

I diagnose the 2 biggest sources of exceptions and explain every step I took with screen snippets to reach my conclusions

The one other competent person on my team agrees. Otherwise complete silence.

I'm told not to fix the issue because I am leaving in 6 weeks and other people have to start taking responsibility.

No one fixes the issues. Instead they leave early. Now it's the weekend and the product is fucked.

Fucking useless people. Can't wait to leave.

> Customer logs Jira ticket claiming app is not working
< I restart the app, investigate and explain tht their server has issues
ø Client closes the ticket as Resolved
-- a couple of days pass by ---

> Customer logs Jira ticket claiming app is not working
< I restart the app, investigate and explain tht their server has issues
ø Client closes the ticket as Resolved
-- a couple of days pass by ---

> Customer logs Jira ticket claiming app is not working
< I restart the app, investigate and explain tht their server has issues
ø Client closes the ticket as Resolved
-- a couple of days pass by ---
<...>

< I log a JIRA ticket explaining what and how is wrong with the server with suggestions how to fix the problem so the app will not crash any longer (client own the server, has his own sysadmins -- I don't even had permissions to open syslog.. had to hack dmesg on their PROD server to pin-point the issue)
> no reaction from customer for weeks. I ping the ticket

× app crashes again

> no reaction from customer for weeks. I ping the ticket
> customer leaves a comment that their sysadmins are looking at it trying to figure out what might be wrong (ignoring what I wrote in ticket's description??? srsly?)

× app crashes again
< I post detail investigation details: snips from logs, screenshots, everything with crystal clear explanations.

> no reaction for weeks
......

well that's fun..

PM (on slack): "we’re about to deploy to production".
Me: "ok"
… I keep on working on a task / remain available for any post deployment issues …
PM (5 minutes later on slack): "deployment broke production! We need to handle this NOW!"

My dev colleague has already called it a day, but I’m still online

Me: "ok I don’t have access to prod, can you describe what’s going on? I can’t reproduce on any other environment"

PM: …

10 minutes go by

Me: "anybody there?"

PM: …

45 minutes later, I realize PM is offline

The following day:
PM: "ok we got prod running again" (turns out it was client’s fault for not updating a config we as devs can’t access)
PM: "but we’re REALLY UPSET! You guys need to be available to intervene for any issues following deployment to production! At least one of you should be available!"

Me: "but, but…" 🫠

The riskiest dev choice...

How about "The riskiest thing you've done as a dev"? I have a great entry for that. and I suppose it was my choice to build the feature afterall.

I was working on an instance of a small MMO at a game company I worked for. The MMO boasted multiple servers, each of them a vastly different take on the base game. We could use, extend, or outright replace anything we wanted to, leading to everything from Zelda to pokemon to an RP haven to a top-down futuristic counterstrike. The server in this particular instance was a fantasy RPG, and I was building it a new leveling and experience system with most of the trimmings. (Talents, feats/perks, etc. were in a future update.)

A bit of background, first: the game's dev setup did not have the now-standard dev/staging/prod servers; everything ran on prod, devs worked on prod, players connected and played on prod, etc. Worse yet, there was no backup system implemented -- or not really. The CTO was really the only person with sufficient access. The techy CEO did as well, but he rarely dealt with anything technical except server hardware, occasionally. And usually just to troll/punish us devs (as in "Oops ! I pulled the cat5 ! ;)"). Neither of them were the most reliable of people, either. The CTO would occasionally remote in and make backups of each server -- we assumed whenever he happened to think of it -- and would also occasionally do it when asked, but it could take him a week, sometimes even up to a month to get around to it. So the backups were only really useful for retreiving lost code and assets, not so much for player data.

The lack of reliable backups and the lack of proper testing grounds (among the plethora of other issues at the company) made for an absolutely terrible dev setup, but that's just how it was, and that's what we dealt with. We were game devs, afterall. Terrible or not, we got to make games! What more could you ask for!? It was amazing and terrible and wonderful and the worst thing ever, all at the same time. (and no, I'm not sharing the company name, but it isn't EA or Nexon, surprisingly 😅)

Anyway, back to the story! My new leveling system also needed to migrate players' existing data, so... you can see where this is going.

I did as much testing and inspection of my code as I could, copied it from a personal dev script to the server's xp system, ... and debated if I really wanted to click [Apply]. Every time I considered it, I went back to check another part or do yet more testing. I ended up taking like 40 minutes to finally click it.

And when I did... that was the scariest button press of my life. And the scariest three seconds' wait afterwards. That one click could have ruined every single player's account, permanently lost us players ...

After applying it, I immediately checked my character to see if she was broken, checked the account data for corruption or botched flags, checked for broken interactions with the other systems....

Everything ended up working out perfectly, and the players loved all of the new features. They had no idea what went into building them, and certainly had no idea of what went into applying them, or what could have gone wrong -- which is probably a good thing.

Looking back, that entire environment was so fragile, it's a wonder things didn't go horribly wrong all the time. Really, they almost never did. Apocalypses did happen, but were exceedingly rare, and were ususally fixed quickly. I guess we were all super careful simply because everything was so fragile? or the decent devs were, at least. We never trusted the lessers with access 😅 at least on the main servers where it mattered. Some of the smaller servers... well, we never really cared about those.

But I'm honestly more surprised to realize I've never had nightmares of that button click. It was certainly terrifying enough.

But yay! Complete system overhaul and migration of stored and realtime player data! on prod! With no issues! And lots of happy players! Woooooo!

Thinking back on it makes me happy 😊

Be me, new dev on a team. Taking a look through source code to get up to speed.

Dev: **thinking to self** why is there no package lock.. let me bring this up to boss man

Dev: hey boss man, you’ve got no package lock, did we forget to commit it?

Manager: no I don’t like package locks.

Dev: ...why?

Manager: they fuck up computer. The project never ran with a package lock.

Dev: ..how will you make sure that every dev has the same packages while developing?

Manager: don’t worry, I’ve done this before, we haven’t had any issues.

**couple weeks goes by**

Dev: pushes code

Manager: hey your feature is not working on my machine

Dev: it’s working on mine, and the dev servers. Let’s take a look and see

**finds out he deletes his package lock every time he does npm install, so therefore he literally has the latest of like a 50 packages with no testing**

Dev: well you see you have some packages here that updates, and have broken some of the features.

Manager: >=|, fix it.

Dev: commit a working package lock so we’re all on the same.

Manager: just set the package version to whatever works.

Dev: okay

**more weeks go by**

Manager: why are we having so many issues between devs, why are things working on some computers and not others??? We can’t be having this it’s wasting time.

Dev: **takes a look at everyone’s packages** we all have different packages.

Manager: that’s it, no one can use Mac computers. You must use these windows computers, and you must install npm v6.0 and node v15.11. Everyone must have the same system and software install to guarantee we’re all on the same page

Dev: so can we also commit package lock so we’re all having the same packages as well?

Manager: No, package locks don’t work.

**few days go by**

Manager: GUYS WHY IS THE CODE DEPLOYING TO PRODUCTION NOT WORKING. IT WAS WORKING IN DEV

DEV: **looks at packages**, when the project was built on dev on 9/1 package x was on version 1.1, when it was approved and moved to prod on 9/3 package x was now on version 1.2 which was a change that broke our code.

Manager: CHANGE THE DEPLOYMENT SCRIPTS THEN. MAKE PROD RSYNC NODE_MODULES WITH DEV

Dev: okay

Manager: just trust me, I’ve been doing this for years

Who the fuck put this man in charge.

Every year my team runs an award ceremony during which people win “awards” for mistakes throughout the year. This years was quite good.

The integration partner award- one of our sysAdmins was talking with a partner from another company over Skype and was having some issues with azure. He intended to send me a small rant but instead sent “fucking azure can go fuck itself, won’t let me update to managed disks from a vhd built on unmanaged” to our jv partner.

Sysadmin wannabe award (mine)- ran “Sudo chmod -R 700 /“ on one of our dev systems then had to spend the next day trying to fix it 😓

The ain’t no sanity clause award - someone ran a massive update query on a prod database without a where clause

The dba wannabe award - one of our support guys was clearing out a prod dB server to make some disk space and accidentally deleted one of the databases devices bringing it down.

The open source community award - one of the devs had been messing about with an apache proxy on a prod web server and it ended up as part of a botnet

There were others but I can’t remember them all

That feeling when your client connection is more stable than the connection of a fucking game server... Incompetent pieces of shit!!! BEING ABLE TO PUT A COUPLE OF SPRITES DOESN'T MAKE YOU A FUCKING SYSADMIN!!!

Oh and I sent those very incompetent fucks a mail earlier, because my mailers are blocking their servers as per my mailers' security policy. A rant from the old box - their mail servers self-identify a fucking .local!!! Those incompetent shitheads didn't even properly change the values from test into those from prod!! So I sent them an email telling them exactly how they should fix it, as I am running the same MTA on my mailers (Postfix), at some point had to fix my mailers against the exact same issue as well, and clearly noticed in-game that they have deliverability problems (they explicitly mention to unblock their domain). Guess why?! Because their server's shitty configuration triggers fucking security mechanisms that are built against rogue mailers that attempt to spoof themselves as an internal mailer, with that fucking .local! And they STILL DIDN'T CHANGE IT!!!! Your fucking domain has no issues whatsoever, it's your goddamn fucking mail servers that YOU ASOBIMO FUCKERS SHOULD JUST FIX ALREADY!!! MOTHERFUCKERS!!!!!

On the first day of Christmas, the bossman gave to me: The fact that my new computer purchase order needs to be OKed by the CEO and I need to continue working on a 2014 Mac Mini (i5-4260U, 8 Gig RAM, GPU shot by an ESD on the case long ago) for the next year.
On the second day of Christmas, my family gave to me... a good reason to get shitfaced
On the third day of Christmas, getting shitfaced gave to me: A hangover and some urgent plastic welding job that had to be done with a soldering iron. FML, I've had a headache before breathing in pure hydro-cyano-whatthefuckyougetwhenyoumeltplastics
On the fourth day of Christmas, my team gave to me: A legacy, age-old Rails 2 project that was written by an intern and never reviewed, went to prod in 2014 and can't be changed anymore, but needs to be changed after the fact that it has zero test coverage and needs 100 % now to prevent issues and costly manual testing.
On the fifth day of Christmas, devrant gave to me: The Idea that making fun of Christmas songs to get over the sheer amount of dicks that working over the twelve days of Christmas sucks.

To be continued...

One of the biggest reality checks you will run into when starting your first dev related job - and which they don't teach you about in school - is that a lot of the time will be spent working with other people's code, and rewriting it into "your own" is rarely an option.

You might be super into making things, but not everyone manages to maintain that same spark while taking over a 15 year old project with fundamental issues that have to be triaged "for now" because you need a hotfix on this other specific thing out in prod before lunch.

There are no gods now. They left the company years ago and nobody knows why they used the windows registry as a user repo.

I JUST HAD ONE OF THOSE DAYS THAT MAKES ONE WANT TO BANG TWO BRICKS ON HEAD SND END THE PAIN THE STORY STARTS YESTETDAY WITH ISSUES AFTER A MIGRSTION AND THEY ASK ME TO HELP TROUBLESHOOT EVEN THOUGH I'M A DEV DBA AND THE ISSUE IS IN QA/SAT AND I HELP ANYWAY AND THEY CAN'T FIND A VIEW AND SO I LOOK EVERYWHERE AND CAN'T DOING IT EITHER AND IT DIDN'T EXIST IN PROD OR DEV SO I TELL THEM IT'S NOT THERE, AND THEY ARE LIKE, CAN YOU RETRIEVE IT FOR US AND I'M LIKE FROM WHERE? I DON'T KEEP VIEWS IN MY BUTT AND YOU GUYS ARE SMOKING CRACK AND THE GIVE ME THEIR QUERY WHICH CONTAIN THE VIEE ANYWAY AND THEY SAY CAN YOU RUN IT AND IT RUNS AND WORKS AND THEY CAN'T MAKE IT WORK AND IT WORKS BECAUSE IT DOESN'T CALL THE VIEW THEY HAVE ME SO NO PROBLEM THERE SO I FINALLY ASK THEM ARE YOU POINTING TO THE CORRECT DATABASE AND THEY'RE LIKE OH MAN WE TOLD YOU THE WRONG DATABASE AND SO I LOOK AT THE RIGHT DATABASE AND FIND THAT THE GRANTS ARE MISSING AND YEAH THANK YOU FOR TAKING EIGHT HOURS OF MY LIFE BECAUSE WE WERE IN THE WRONG DB YOU GAVE ME AND I HOPE THE FLAG OF A THOUSAND CAMELS INVEST YOUR ARMPITS AND THE CHIGGERS OF A THOUSAND SOUTHERN LAWNS INGEST YOUR SOCKS AND UNDERWEAR. YAAAAAA!!!!

Just learned that yesterday someone suggested putting the dev team on "workspace", when I was on leave.

My first question, "what the hell is workspace?"
"It's a remote environment..."
Okay I get it. Are you kidding me? Doing development on remote desktop?

My second question, "Why the hell did someone suggest that?"
"We have had issues with devs using MySQL but the target prod will be using PostgreSQL. That caused issues, inconsistencies... And we found some issues after deployment."

Okay so much for DB agnostic. I called it out that everyone now install PostgreSQL on local. Problem solved, hopefully.

Why we had MySQL in the first place? Yes DB agnostic is one of the reason. The other being I'm more familiar with MySQL so it's quicker to perform tasks (like "can you clone that environment for me" and "can you fix the data on XYZ"). But that's trivial.

Just some ridiculous suggestion that set me off.

FUCKING FUCK ANGULAR!!!!
LIKE FUCK IT IN THE ARSE AND BURN THE MOTHERFUCKER WHILE LAUNCHING A MISSILE ON IT TO BE SURE!

(ﾉ≧∇≦)ﾉ ﾐ ┸━┸

So I am making something on angular and I got everything running in ng serve(development environment) , after handling all issues and showing it to my boss man he approves and asked to put it up on prod for a demo , doesn’t sound like an issue , I make the prod build on cli and BAM! 16 errors ? No issues right?, I’ll just google the issue. Googles.... there aren’t no clear solutions to it as the angular version keeps changing and nobody knows what broke it, I mean people have the issue,but like 100 reasons that can cause it,
HOLY LORD RELEASE A NEWER VERSION AFTER MENDING THE OLD ONE
But nooooooo!

Angular Dev:We fucked this one, lol what should we do boss man?
Angular boss man: lol just leave it, we need to build the new version with newer bugs,

P.S. I like angular, but it’s like a underdeveloped framework, too many issues and too many changes

Me: Why are we spending time building reports for Support? AFAIK they never read or use them.

Boss: Seems they expect you to do it.

Me: Then what exactly are they supposed to be doing? All the issues seem to just escalate back to us.... We should just make sure issues never get into PROD

Boss: I agree, they're always firefighting... we gave them more funding so eventually they should catch up **Me: I highly doubt it, you should just stop hiring monkeys** esp. if we prevent PROD issues.

Me: Yea... we should prevent production issues because someone always has to pay off the (technical) debt and interest rates in PROD are very high

I hate dev politics...

PM: Hey there is a weird error happening when I upload this file on production, but it works on our test environments.

Me: After looking at this error, I don't find any issues with the code, but this variable is set when the application is first loaded, I bet it wasn't loaded correctly our last deployment and we just need to reload the application.

Senior Dev: We need to output all of the errors and figure out where this error is coming from. Dump out all the errors on everything in production!!

Me: That's dumb... the code works on test... it's not the code.. it's the application.

Senior dev: %$*^$>&÷^> $

Me: Hey I have an idea! If test works... I can go ahead and deploy last week's changes to prod and dump those errors you were talking about!!

Senior Dev: OK

Me: *runs Jenkins job the deploys the new code and restarts the application*

PM: YAY you fixed it!!

Senior Dev: Did you sump put those errors like I said.

Me: Nope didn't touch a thing... I just deployed my irrelevant changes to that error and reloaded the application.

Got a PROD issues caused by missing data from upstream. Told them about it last week. 3 days of no replies.

Finally got a reply that they will look into it. It's been almost a week since I first told them but no responses and no solution yet.

I couldn't fall asleep because I guess I drank too much coffee but also cuz the thought of this pissed me off.... So got up and sent a work email basically threatening to escalate this all the way up, put their asses in hot water.

Though not sure if I can but.... Maybe I can take a disability leave for needing to mentally recover from going mad at other people that can't do their jobs but aren't fired...

!rant

I've always been wondering why do tech companies need everyone to have a strong grasp of algos and data structures?

I've been coding most of my life but didn't get a CS degree so ended up in IT but I kind of want to get into a tech company as my thinking is the quality of code much higher (I spend a lot of time cleaning up other people's code and prod issues over the years...), I've been learning Algo/DS but when I see those technical questions on CareerCup, I go WTF.... it's this the kind of problems you guys do every day?

Working with LEGAL is the worst!

They say yes to all reviews and once it's in prod they want to completely change the whole thing cause they think every user reads the Terms & Condition!

To be honest I deployed 5 times. I have not read that shit once!

Microsoft :

"YOLO All Devops pipeline are now .net 8"

Me, lunching PROD release : "Whatever, I don't use any advanced shit"

Pipeline : "Can't connect to your nuget soure (private) : 401"

Me : What ? You just connected to it in prevous step in pipeline !

Azure : Yah, but prev st ep was using different syntax so it worked

Me : Ok, here you go, same syntax

Azure : Still nop.
Me : WHY ?!!!
Azure : I won't tell you. 401 !!

Me : Fuck you, I 'll rewrite this step from scrath !
30 mins later

Azure : Naaaan, stil 401.
Me : HOW ?!!! You literally use this nuget feed in 2 prev steps with no issues.
Azure : Fuck you
Me : fuck you too.
15 mins later
me to me... Hmm, I've already restored all packages. They "should" stay where they are..
Me : Take that azure : Ignore restoring for this step"
Azure : Oh yeah, that works ! All compiled !

WTF ?!!!! And why they release cha ges without even sending a mail. because fuck us apparently.

God I hate devops....

Mentoring someone in iOS from scratch, then teaching him how to maintain 3 different apps. He is able to maintain them without me now which is testament to how well I taught him, but it was challenging. Especially since I had to simultaneously work on other tickets and live prod issues.

Today I think it became apparent why we have so many data and prod issues...

No one understands SOLID design principals... Particularly Separation of Concern. No wonder all the Spring (DI) modules look like a mess of God functions that are impossible to reuse and difficult to extend...

Due to covid, mgrs decided to fire 10% but could not negotiate schedule increase with internal IT. With no promotions or hikes, few full stacks we have leave.
Now am working with 2 data engg doing cloud java microsvs work while learning. Their first delivery was applauded by their mgr who is under pressure to retain them.
I as arch review their code. No unit tests, print statements all around, shoddy exception handling, variable naming issues. We have Sonar by default in our build. They ignore the report. I ask them about it. Seems mgr told them he is getting a contract person from another team on part time basis to do/fix. I share my confusion.

Mgr calls me up and checks if we can put it as tech debt backlog and deploy to prod !!!

Happy Friday, the 13th, and merry new post-deployment prod issues!!

Long post, TLDR: Given a large team building large enterprise apps with many parts (mini-projects/processes), how do you reduce the bus-factor and the # of Brent's (Phoenix Project)?

# The detailed version #

We have a lot of people making changes, building in new processes to support new flows or changes in the requirements and data.

But we also have to support these except when it gets into Production there is little information to quickly understand:
- how it works
- what it does/supposed to do
- what the inputs and dependencies are

So often times, if there's an issue, I have to reverse engineer whatever logic I can find out of a huge mess.

I guess the saying goes: the only people that know how it works is whoever wrote it and God.

I'm a senior dev but i spend a lot of time digging thru source code and PROD issues to figure out why ... is broken and how to maybe fix it.

I think in Agile there's supposed to be artifacts during development but never seen em.

Personally whenever i work on a new project, I write down notes and create design diagrams so i can confirm things and have easy to use references while working.

I don't think anyone else does that. And afterwards, I don't have anywhere to put it/share it. There is no central repo for this stuff other than our Wiki but for the most part, is like a dumping ground. You have to dig for information and hoping there's something useful.

And when people leave, information is lost forever and well... we hire a lot of monkeys... so again I feel a lot of times i m trying to recover information from a corrupted hard drive...

The only way real information is transferred is thru word of mouth, special knowledge transfer sessions.

Ideally I would like anything that goes into PROD to have design docs as well as usage instructions in order for anyone to be able to quickly pick it up as needed but I'm not sure if that's realistic.

Even unit tests don't seem to help much as they just test specific functions but don't give much detail about how a whole process is supposed to work.

Yesterday I had a user reachout via email because his account became inaccessible when he updated it.

This was something I didn't realize could happen... so it was fun recreating this scenario locally.

Basically he somehow bypassed the lowercase requirement for the updateUsername endpoint. So in the DB, his name was uppercase. And the SQL query was searching for his exact username without doing LIKE (by design).

It was failing on that getProfile endpoint and showing a Whoops screen. 😬 Should be a 404 screen, I know. I'm working on it.

Since then, I added Sentry logging to the backend and frontend.

And this showed me there were endpoints still being triggered nightly to remove stale data. (I forgot to remove). And these endpoints were getting 401's. I'm thinking who TF is triggering these endpoints. I soon realized those specific endpoints needed keys that expired monthly. 🤦‍♂️

Had a definite week from hell... a bunch of prod issues that only I could fix (that's a whole other rant for another day!)... a piece of code totally kicking my ass for days... a hosting environment that was unstable seemingly every time I needed to do something in it (and that killer piece of code could ONLY be properly tested there, naturally!)... a service that my app depends on flaking out with no indication what the problem was and another team responsible for it that is based off-shore so aren't responsive when I need them to be... a metric shit-ton of procedural bullshit dropped on my head... an immense amount of stress due to the lead-up to a prod rollout next month that absolutely CANNOT fail without huge ramifications for the business but not enough help to ensure it gets done.

But, with all that said, I DID manage to get that killer piece of code working late on Friday after slamming my head against the wall for over a week on it (and ultimately re-writing it from the ground-up on Thursday and Friday)... so, the week of hell ended on a high note at least, which is always a Very Good Thing(tm)!

This is an actual transcript...
Since it's way too long for the normal 5000 characters, hence splitting it up...

Infra Guy: mr Dev, could you please give some rational for update of jjb?
Dev: sparse checkout support is missing
Infra Guy: is this support mandatory to achive whatever you trying to do?
Dev: yes
Infra Guy: u trying to get set of specific folder for set of specific components?
Dev: yes
Infra Guy: bash script with cp or mv will not work for you?
Dev: no
Infra Guy: ?
Dev: when you have already present functionality why reinvent the wheel
Dev: jenkins has support for it
Dev: the jjb is the bottle neck
Infra Guy: getting this functionality onto our infra would have some implications
Dev: why should I write bash script if jenkins allows me to do that
Dev: what implications ??
Infra Guy: will you commit to solve all the issues caused by new jjb?
Dev: you show me the implications first
Infra Guy: like a year ago i have tried to get new jjb <commit_url>
Infra Guy: no, the implications is a grey area
Infra Guy: i cant show all of them and they may hit like in week or eve month
Dev: then why was it not tackled
Dev: and why was it kept like that
Infra Guy: few jobs got broken on something
Dev: it will crop up some time later
Dev: if jobs get broken because of syntax
Dev: then jobs can be fixed
Dev: is it not ???
Infra Guy: ofc
Infra Guy: its just a question who will fix them
Dev: follow the syntax and follow the guidelines
Dev: put up a test server and try and lets see
Dev: you have a dev server
Dev: why not try on that one and see what all jobs fails
Dev: and why they fail
Dev: rather than saying it will fail and who will fix
Dev: let them fail and then lets find why
Dev: I manually define a job
Dev: I get it done
Infra Guy: i dont think we have test server which have the same workload and same attention as our prod
Dev: unless you test how would you know ??
Dev: and just saying that it broke one with a version hence I wont do it
Infra Guy: and im not sure if thats fair for us to deal with implication of upgrading of the major components just cause bash script is not good enough for u
Dev: its pretty bad
Infra Guy: i do agree
Infra TL Guy: Dev, what Infra Guy is saying is that its not possible to upgrade without downtime
Infra Guy: no
Dev: how long a downtime are we looking at ??
Infra Guy: im saying that after this upgrade we will have deal with consequences for long time
Infra Guy-2: No this is not testing the upgrade is the huge effort as we dont have dev resources to handle each job to run
Dev: if your jjb compiles all the yaml without error
Dev: I am not sure what consequences are we talking of
Infra Guy: so you think there will be no consequences, right?
Dev: unless you take the plunge will you know ??
Dev: you have a dev server running at port 9000
Infra Guy: this servers runs nothing
Dev: that is good
Dev: there you can take the risk
Infra Guy: and the fack we have managed to put something onto api doesnt mean it works
Dev: what API ?
Infra Guy: jenkins api
Infra Guy: hmmm
Dev: what have you put on Jenkins API ??
Infra Guy: (
Dev: jjb is a CLI
Infra Guy: ((
Dev: is what I understand
Dev: not a Jenkins API
Infra Guy: (((
Dev: (((((
Infra Guy: jjb build xmls and push them onto api
Infra Guy: and its doent matter
Dev: so you mean to say upgrading a CLI is goig to upgrade your core jenkisn API
Dev: give me a break
Infra Guy: the matter is that even if have managed to build something and put it onto api
Infra Guy: doesnt mean it will work
Dev: the API consumes the xml file and creates a job
Infra Guy: right
Dev: if it confirms to the options which it understands
Dev: then everything will work
Dev: I am actually not getting your point Infra Guy
Infra Guy: i do agree mr Dev
Dev: we are beating around the bush
Infra Guy: just want to be sure that if this upgrade will break something
Infra Guy: we will have a person who will fix it
Dev: that is what CICD is supposed to let me know with valid reasons
Dev: why can't that upgrade be done
Infra Guy: it can be done
Infra Guy: i even have commit in place

A while back I was looking for a new job and was given an interview by one company who shall remain nameless. Before the interview, they asked me look through their current site, nothing unusual there, so I started browsing. Then I received an email with all the details I needed to access their production server. Apparently they wanted me to look through the code, unusual but I did so.

First thing all the passwords, including those belonging to members of the public were stored in plain text and many were still the default passwords which were based on the Id so were sequential.

I highlighted these issues at the interview and they then asked me to do a test, not the usual test though, they asked me to add some charts to their prod site. Needless to say that didn’t happen and I got another job elsewhere.

So I'm assigned once again to fix a new someone else created and that seems to be the case whenever there's an issue...

Boss just assigns it to whoever is most likely to be able to investigate it... which is basically me. Other than the little time I can use to develop stuff, I'm usually cleaning up other people's messes.

And these other people are to busy working on new crap to properly explain how their existing code/processes/changes works.

And well the fact that anything breaks in production (that's not due to upstream one off issues) whoever does not think he needs to take responsibility for it.

So everyone else and especially me has to spend time understanding the shit they wrote and fixing it for them.

How do I tell my boss this nicely that we need clearly definitely ownership and whenever a component blows up in prod, the guy that wrote the code fixes it no matter what? Thereby incentivizing him to not write shit code in the first place and be more proactive in making sure it doesn't in the first place since he knows otherwise he's doing overtime to fix it?

Is it just me or is there really no such thing as a dev job where something doesn't blow up due to poorly tested and designed code every other day?

During the last two weeks I have come to realize that I dislike holidays. Or maybe I don't, maybe it's just now that almost everyone is forced to use their holidays at the same time... anyway, I just know some junior has made a hot spaghetti mess that needs a'fixin and I don't look fwd to attending to those issues when I get back to the office... worst case the buggy spaghetti poo is already in prod

Couldn't find it on Google but does Node 6.9.1 support ES6 syntax and const, let, etc?

The code runs on my machine which uses 6.9.1 I think but the Code Style report raises issues saying it needs a esversion: 6. I can add it but dunno if it's gonna break in PROD

So there is this owner team who reviewed my code recently. I don't have much context about the their system and architecture. We try to build our changes with less context and rely in owner team's knowledge for any review gap.
The guy from the owner team missed something in my review and changes went to prod, review already took more that it was expected to take. He took 1 week for small change reviews. Now, not him but with someone else's advice they had to revert.
I wrote a mail shooting to manager, the guy who reverted and the guy who reviewed, asking the reviewer guy to explain why didn't he mentioned about any issues at the time of the review.
I have tried best from my side. But all this, god!!!
Why everything I do has some kind of weird issue. I feel so bad blaming the guy, I just think that, the way I used to feel anxious he must be feeling the same, but what can I do? I don't want to take the blame I don't even see if I can and I shouldn't be. If it was a major issue it should have been raised but he didn't. I feel so bad that I am almost crying, I am feeling that like always I am going to be judged by my team that work is slow and on top of that I can't do anything for the guy I blamed it on.

I don't know, is it my mistake? but I cannot think of anyway I would have known this.

I've having issues trying to form a proper branching strategy for my mean stack app deployment.

Heroku creates staging and prod branches for my web app so I'm a bit confused if I need my own staging branch?

Currently I have this: feature -> dev -> staging -> heroku staging (the staging branch seems useless)

Also, Heroku allows you to promote heroku staging to heroku prod, so there's no point in making master push to heroku prod.

I'm thinking of making my strategy to the following, but wasn't sure of any pitfalls or anything I'm overlooking long term.

feature -> dev -> master -> heroku staging -> manual promote to heroku prod.

Any suggestions?

development progress is like chinese whispers; "we have encountered issues and will be delayed" becomes "we'll be ready to push to prod by COB" when it reaches the CTO. /facepalm

Recently joined new Android app (product) based project & got source code of existing prod app version.

Product source code must be easy to understand so that it could be supported for long term. In contrast to that, existing source structure is much difficult to understand.

Package structure is flat only 3 packages ui, service, utils. No module based grouped classes.

No memory release is done. So on each screen launch new memory leaks keep going on & on.

Too much duplication of code. Some lazy developer in the past had not even made wrappers to avoid direct usage of core classes like Shared Preference etc. So at each place same 4-5 lines were written.

Too much if-else ladders (4-5 blocks) & unnecessary repetitions of outer if condition in inner if condition. It looks like the owner of this nested if block implementation has trust issues, like that person thought computer 'forgets' about outer if when inside inner if.

Too much misuse of broadcast receiver to track activities' state in the era of activity, apપ life cycle related Android library.

Sometimes I think why people waste soooo... much efforts in the wrong direction & why can't just use library?!!

These things are found without even deep diving into the code, I don't know how much horrific things may come out of the closet.

This same app is being used by many companies in many different fields like banking, finance, insurance, govt. agencies etc.

Sometimes I surprise how this source passed review & reached the production.