One week, and it turned out to be worse than that.

I was put on a project for a COVID-19 program in America (The CARES Act). The financial team came to us on Monday morning and said they need to give away a couple thousand dollars.

No big deal. All they wanted was a single form that people could submit with some critical info. Didn't need a login/ registration flow or anything. You could have basically used Google Forms for this project.

The project landed in my lap just before lunch on Monday morning. I was a junior in a team with a senior and another junior on standby. It was going to go live the next Monday.

The scope of the project made it seem like the one week deadline wasn't too awful. We just had to send some high priority emails to get some prod servers and app keys and we were fine.

Now is the time where I pause the rant to express to you just how fine we were decidedly **not**: we were not fine.

Tuesday rolls around and what a bad Tuesday it was. It was the first of many requirement changes. There was going to need to be a review process. Instead of the team just reading submissions from the site, they needed accept and reject buttons. They needed a way to deny people for specific reasons. Meaning the employee dashboard just got a little more complicated.

Wednesday came around and yeah, we need a registration and login flow. Yikes.

Thursday came and the couple-thousand dollars turned into a tens of millions. The amount of users we expected just blew up.

Friday, and they needed a way for users to edit their submissions and re-submit if they were rejected. And we needed to send out emails for the status of their applications.

Every day, a new meeting. Every meeting, new requirements that were devastating given our timeframe.

We put in overtime. Came in on the weekend. And by Monday, we had a form that users could submit and a registration/ login flow. No reviewer dashboard. We figured we could take in user input on time and then finish the dashboard later.

Well, financial team has some qualms. They wanted a more complicated review process. They wanted roles; managers assign to assistants. Assistants review assigned items.

The deadline that we worked so hard on whizzed by without so much as a thought, much less the funeral it deserved.

Then, they wanted multiple people to review an application before it was final. Then, they needed different landing pages for a few more departments to be able to review different steps of the applications.

Ended up going live on Friday, close to a month after that faithful Monday which disrupted everything else I was working on, effective immediately.

I don't know why, but we always go live on a Friday for some reason. It must be some sort of conspiracy to force overtime out of our managers. I'm baffled.

But I worked support after the launch.

And there's a funny story about support too: we were asked to create a "submit an issue" form. Me and the other junior worked on it on a wednesday three weeks into the project. Finished it. And the next day it was scrapped and moved to another service we already had running. Poor management like that plagued the project and worked in tandem with the dynamic and ridiculous requirements to make this project hell.

Back to support.

Phone calls give me bad anxiety. But Friday, just before lunch, I was put on the support team. Sure, we have a department that makes calls and deal with users. But they can't be trained on this program: it didn't exist just a month ago, and three days ago it worked differently (the slippery requirements never stopped).

So all of Friday and then all of Saturday and all of Monday (...) I had extended panic attacks calling hundreds of people. And the team that was calling people was only two people. We had over 400 tickets in the first two days.

And fuck me, stupid me, for doing a good job. Because I was put on the call team for **another** COVID project afterwards. I knew nothing about this project. I have hated my job recently. But I'm a junior. What am I gonna say, no?

"Hey, Root, someone screwed up and now all of our prod servers are running this useless query constantly. I know I already changed your priorities six times in the past three weeks, but: Go fix it! This is higher priority! We already took some guesses at how and supplied the necessary code changes in the ticket, so this shouldn't take you long. Remember, HIGH PRIORITY!"

1. I have no idea how to reproduce it.
2. They have no idea how to reproduce it.
3. The server log doesn't include queries.
4. The application log doesn't include queries.
5. The tooling intercepts and strips out some log entries the legendary devs considered useless. (Tangent: It also now requires a tool to read the logs because log entries are now long json blobs instead of plain text.)
6. The codebase uses different loggers like everywhere, uses a custom logger by default, and often overwrites that custom logger with the default logger some levels in. gg
7. The fixes shown in the ticket are pretty lame. (I've fixed these already, and added one they missed.)
8. I'm sick and tired and burned out and just can't bring myself to care. I'm only doing this so i don't get fired.
9. Why not have the person who screwed this up fix it? Did they quit? I mean, I wouldn't blame them.

Why must everything this company does be so infuriatingly complicated?

Root: Fleshes out missing data in some factories. Tests affected code and finds the change breaks some specs (but shouldn’t).

Root: Reaches out to spec author.

Root: Messages thundercunt (the ticket’s code reviewer) on slack about the specs and the reaching out. No response.

Root: Works on another ticket while blocked.

Root: Logs off.

Root: Talks with spec author chick in the morning. Decide to pair on specs later.

TC: Still no slack response.

Root: Gives update in standup. Mentions factories and broken specs. Mentions pairing with spec chick.

TC: Still no slack response.

Root: Pulled off tickets in favor of prod issue. Gets ignored by everyone else diagnosing prod issue. Investigates prod issue by herself. Discovers prod issue isn’t from bad code, but bad requirements — code works as requested. Communicates this with details. Gets ignored by people still diagnosing prod issue. Tries again. Gets ignored. Gives up. Works on non-blocked tickets instead.

TC: Still no slack response.

Hours later:

TC: Comments on PR telling me I broke specs (how did I not notice?), that I need to reach out to spec chick and work with her, and that I can’t resolve the ticket until it’s fixed and passes code review.

TC: Still no slack response. (21 hours later at this point)

TC: Logs off. Still no response (25 hours at this point)

———

Ignoring the prod issue for the moment…

I broke specs. No shit.
I need to talk with spec chick. No shit.
I can’t resolve the ticket. No shit!

Bitch, I told you all of this 21 fucking hours prior, and again 3 hours prior during standup. But no, I clearly “don’t communicate” and obviously have no bloody clue what I’m doing, either, so I need everything spelled out for me.

And no, I didn’t resolve the fucking ticket. Why the fuck would I if it still has pending changes? Do you even check? Ugh!

And what the fuck with that prod issue? I’m literally giving you the answer. fucking listen! Stupid cunts.

Why is it all of the women I work with are useless or freaking awful people? Don’t get me wrong, many of the men are, too, but I swear it’s every single one of the women. (Am I awful, too?)

Just. Ugh.
I can’t wait to leave this sewer of a company.

Oddly still a good day, though. Probably because I talked to recruiters and sent out my resume again.

23:04 Client: We need these changes pushed to production before morning.
06:22 Staff: Changes pushed to prod.

<Tells story to manager>
Manager: Sum up the time you've worked on it and double it.

This isn't a special thing, we however have a policy stated in our contracts that all changes must be scheduled with a minimum of 24h before the time they're needed.

Me passing time on the weekend

Random call from unknown number

Turns out it's the manager

M: hey , how is your weekend going ...

Me: nothing much ... Whatsup ?

M : yeah well , we wanted to push some minor adhoc fixes as some clients wanted it urgently

The Devops folks need developer support . Can you pitch in and monitor

Me : I'm not aware of what changes are going , i don't think i can provide support

M : don't worry it's minor changes , it's already tested in pre prod , you just need to be on call for 30 mins

Me : ugh okay .. guess 1 hr won't hurt

M: thanks 👍🏽

Me: *logs in

*Notices the last merged PR
+ 400 lines , implemented by junior dev and merged by manager

*Wait , how is this a *minor* release...

*Release got triggered already and the CI CD pipeline is in progress

*5 mins later

*Pipeline fails , devops sends email - test coverage below 50%

Manager immediately pitches in ...

M: hey , i see test coverage is down , can you increase it ?

Me: and how do u suppose I do that ?

M : well it's simple just write UTC for the missing lines ... Will it take time ?

Me : * ah shit here we go again

Yeah it will take time , there are around 400 lines , I am not aware of this component all together

Can you ask junior dev to pitch in and write the UTC for this

*Actually junior dev is out on a vacation with his girlfriend

M : well he's out for the weekend , but
as a senior dev , i expect you to have holistic understanding of the codebase and not give excuses ,
this is a priority fix which client are demanding we need this released ASAP

Me : * wait wat ?

---

I ended up being online for next 3 hours figuring out the code change and bumping up the UTC 🤦🏾

Rant, but also want to know, how the fuck do you accidentally drop a database link. In production. On the most important database. Housing our biggest applications.

I really, really, fucking god damn it REALLY need to move a legacy project from the grave yard server and get it in git, and then build a dev environment for it, so I can stop making incredibly volatile changes direct to PROD (backend, frontend and DB all at once and then test it while it’s live and being used, but fuck me if I can be bothered digging through a 10GB code base and attempting to make it work in a multi-environment setup when it’s going to be a long trip down the error logs until it works again 😱🔫

I was asked to look into a site I haven't actively developed since about 3-4 years. It should be a simple side-gig.
I was told this site has been actively developed by the person who came after me, and this person had a few other people help out as well.
The most daunting task in my head was to go through their changes and see why stuff is broken (I was told functionality had been removed, things were changed for the worse, etc etc).
I ssh into the machine and it works. For SOME reason I still have access, which is a good thing since there's literally nobody to ask for access at the moment.
I cd into the project, do a git remote get-url origin to see if they've changed the repo location. Doesn't work. There is no origin. It's "upstream" now. Ok, no biggie. git remote get-url upstream. Repo is still there. Good.
Just to check, see if there's anything untracked with git status. Nothing. Good.
What was the last thing that was worked on? git log --all --decorate --oneline --graph. Wait... Something about the commit message seems familiar. git log. .... This is *my* last commit message. The hell?
I open the repo in the browser, login with some credentials my browser had saved (again, good because I have no clue about the password). Repo hasn't gotten a commit since mine. That can't be right.
Check branches. Oh....Like a dozen new branches. Lots of commits with text that is really not helpful at all. Looks like they were trying to set up a pipeline and testing it out over and over again.
A lot of other changes including the deletion of a database config and schema changes. 0 tests. Doesn't seem like these changes were ever in production.
...
At least I don't have to rack my head trying to understand someone else's code but.... I might just have to throw everything that was done into the garbage. I'm not gonna be the one to push all these changes I don't know about to prod and see what breaks and what doesn't break
.
I feel bad for whoever worked on the codebase after me, because all their changes are now just a waste of time and space that will never be used.

* Yesterday:*
PM: Yes, so, could you please do those changes on this page tomorrow by 3 pm and push to prod?
Me: Yeah sure! Noted :) (task is to "untick" a checkbox in a page's settings on our CMS)

* the next day -- 11 am *
PM: erm yes so please can you do the changes I told you about, it's getting urgent and you didn't start it already and it stresses the hell out of me because today is friday and it needs to be up and running fine for Monday 12pm and you don't work on weekend so I'd like yo-
Me:

There is a company providing a very speciffic service. And it has a core application for that svc, supported by a core app team.
That company also has other services, which are derivations of the core one. So every svc depends on core.

Now that we're clear on that... I was working in a team of one of the subservices. We very strongly depended on core. In fact, our svc was useless if integration w/ core broke down.

The core team had an annoying habbit. They refused to version their webservices and they LOVED to push api updates w/o any warnings. Our prod, test, other envs used to fail bcz of core api changes quite often. Mgmt, IT head was aware of the problem and customers' complaints as well.

So as a result, once core api changes we're all in a panic mode: all prior priorities are lowered and revival of prod is to be our main focus. Core api is not docummented, the changes are not clear, so we have to reverse engineer the shit out of it. We manage to patch our prod up w/ hotfixes, but now we have tech debt. While working on the debt, core api changed again, in test env. Mgmt pushes debt back and reallocates us to hotfix test. Hotfix is 80% done when another core api breaks. Now mgmt asks us to drop wtv we're working on and fix that new break. By the time we're to deploy the hotfix, another api breaks in another env. The mgmt..... You get the picture :)

2 years go by, nothing has changed so far.

Today in horror stories: setting up the local dev environment for a project I'm taking over creates changes and notifications in prod.

Oh and my local dev env is somehow receiving data from actual patients. I don't know how.

This was some time ago. A Legendary bug appeared. It worked in the dev environment, but not in the test and production environment.

It had been a week since I was working on the issue. I couldn't pinpoint the problem. We CANNOT change the code that was already there, so we needed to override the code that was written. As I was going at it, something happened.

---
Manager: "Hey, it's working now. What did you do?"
Me: *Very confused because I know I was nowhere close to finding the real source of the problem* Oh, it is? Let me check.
Also me: *Goes and check on the test and prod environment and indeed, it's already working*
Also me to the power of three: *Contemplates on life, the meaning of it, of why I am here, who's going to throw out the trash later, asking myself whether my buddies and I will be drinking tonight, only to realize that I am still on the phone with my manager*
Me again: "Oh wow, it's working."
Manager: "Great job. What were the changes in the code?"
Me: "All I did was put console logs and pushed the changes to test and prod if they were producing the same log results."
Manager: "So there were no changes whatsoever, is that what you mean?"
Me: "Yep. I've no idea why it just suddenly worked."
Manager: "Well, as long as it's working! Just remove those logs and deploy them again to the test and prod environment and add 'Test and prod fix' to the commit comment."
Me: "But what if the problem comes up again? I mean technically we haven't resolved the issue. The only change I made were like 20 lines of console logs! "
Manager: "It's working, isn't it? If it becomes a problem, we'll work it out later."
---

I did as I was told, and Lo and Behold, the problem never occurred again.

Was the system playing a joke on me? The system probably felt sorry for me and thought, "Look at this poor fucker, having such a hard time on a problem he can't even comprehend. That idiotic programmer had so many sleepless nights and yet still couldn't find the solution. Guess I gotta do my job and fix it for him. I'm the only one doing the work around here. Pathetic Homo sapiens!"

Don't get me wrong, I'm glad that it's over but..

What the fuck happened?

WHY THE FUCK ARE WE ONLY KEEPING THE LAST 10 BUILD ARTIFACTS IN ARTIFACTORY?!

THAT IS LITERALLY THE OPPOSITE OF HOW YOU'RE SUPPOSED TO USE IT

NOW I CAN'T ROLL BACK PROD CHANGES TO A WORKING VERSION BECAUSE IT LITERALLY DOESN'T EXIST ANYMORE

FUCK YOU, IT'S YOUR FIRE NOW

"Architect"(A) - Hey, StrucN, we have a bit of a problem on the module you are working on (which the previous "developers" seem to have given it roofies)

Me: Okay, what seems to be the problem?

A: There is a need to add some functionality to it, we need you to ...

Me: I see, well it can be done but it wouldn't be so simple - the module is a mess and the change would need to be well tested

A: I fear the clients deadline is for tomorrow

Me: Well he'll have to wait, rushing it is the worst possible option

A: I'll talk to him about it, thanks

After around half an hour A rushes back

A: Hey I passed a ticket to you about the additions we spoke about, it should be ready for tomorrow

Me: It won't be ready, it's too complex to complete is in such a shirt notice (considering it's already the end of the day and all the changes need to be pushed tommorow to prod)

A: I know *programmer from useless team B* did something similar so as it is close to what we need you should copy it.

My inner voice: FUCK YOU YOU USELESS FUCKING CUNT! THERE SHOULDN'T BE ANY COPY PASTE SHIT FROM SOME UNRELATED MODULE! YOU SHIT STAINED MEAT BAG ALREADY DID SUCH A SIN IN THE PAST AND I HAD TO FIX ALL OF IT. THE MODULE SHOULDN'T SUFFER ANY MORE AS IT IS ALREADY A GODDAMN RAPE VICTIM!

WHERE DID PROPER PROFESSIONALISM WENT? WHY IS IT THE INDUSTRY FILLED WITH STUPID WANNA BE "ARCHITECTS" WHILE OTHER MORE COMPETENT FOLK SHOULD ALWAYS BE IGNORED BECAUSE IT'S ALWAYS SHOULD BE READY FOR TOMMOROW?!

For fucks sake I miss my old Architect, he could really understand the essence of program development

Boss wanted me to make changes in company's website which was based on wordpres s.

I knew it could be done by tweaking some JS code, but I have very less experience with wordpress

But wordpress is easy man(Internet told me).

Give me 5 minutes, you will see the changes in production.
Being lazy af I directly logged in to ftp, checked out some files, updated some code, I was good to go.

Before pushing it, I opened the website and it was GONE ٩(๑´0`๑)۶

Now there was no public_html in the root.
I was fucked. I have accidentally deleted the website that had no backup.

And the best part I was on leave from
next day.

I was looking everywhere for backups, looked into google cache to get the contents. I have to recreate the complete site now.

Just when I was asking questions on choice of my profession and simultaneously looking here and there in FTP for backups,
I found the jewel "public_html".

It happens out that I have accidentally moved the folder to some other directory.

Phewww.

Moved it back to root. Site was up and running.

Reassured myself that I deserve to be a dev.

Backed up complete site, made the changes.

Uploaded it.

And the best part, amount of wordpress I learned in those three hours was way more than I could have learnt in many weeks.

Lessons Learnt :

A) ALWAYS keep backups.
B) You SHOULD NOT make changes on prod directly
C) You become superhuman when your brain know you are going to be fucked 😂

FUCKING FUCK ANGULAR!!!!
LIKE FUCK IT IN THE ARSE AND BURN THE MOTHERFUCKER WHILE LAUNCHING A MISSILE ON IT TO BE SURE!

(ﾉ≧∇≦)ﾉ ﾐ ┸━┸

So I am making something on angular and I got everything running in ng serve(development environment) , after handling all issues and showing it to my boss man he approves and asked to put it up on prod for a demo , doesn’t sound like an issue , I make the prod build on cli and BAM! 16 errors ? No issues right?, I’ll just google the issue. Googles.... there aren’t no clear solutions to it as the angular version keeps changing and nobody knows what broke it, I mean people have the issue,but like 100 reasons that can cause it,
HOLY LORD RELEASE A NEWER VERSION AFTER MENDING THE OLD ONE
But nooooooo!

Angular Dev:We fucked this one, lol what should we do boss man?
Angular boss man: lol just leave it, we need to build the new version with newer bugs,

P.S. I like angular, but it’s like a underdeveloped framework, too many issues and too many changes

I didn’t turn down a dev freelance project when the client decided against going with best practices because the solution I offered was a well-established design pattern but created a need for a financial management change she didn’t like. I stupidly built what she asked for. It worked fine in the 3rd party vendor test environment but failed on production. After hours of analysis of code to ensure no changes happened to my source during test->prod deployment, and the vendor denying they had config differences between them, and the client refusing to pay, all I could do was abandon the project.

Designers use plugin in WP to modify UI. CSS and JS is saved (somewhere?) in database. No version control. Changes are made across different envs at the same time and all need to be migrates to prod later.

Pls

IMHO technical dept is kind of like smoking cigarettes for some decades.

You were told that shit will hit the fan but you do not take proper action. And one day you'll realize that you fucked up (or not, also seen that).

Worked for a company in IT, where we maintained an ERP which was "in progress" for over a decade. The basic implementation was done by people with zero technical understanding. To clarify: not self coded. Software was bought. We are talking about integrating the system.

Therefore, the foundation was like a wet noodle. When I joined that company, I told them that they need to address that. I told them that things will get slower and slower and that shit will hit the fan if no proper actions taken.
Even made a list with flaws I found. With potential risk and actions to take, that could then be measured.

At that time, five people worked in said department (including me).

People did not want to listen. "Would be too expensive to rewrite stuff".

Nothing has changed about the wet noodle, but I tried to fix as many things in a working system as I could. Felt like heart surgery, because changes got implemented and "tested" in prod. No version control, no documentation, everyone implemented things like they felt (no guidelines for consistency).
A lot of small fuckups that summed up over the years.

I left the company after two years because I had the chance to land a job as a dev.

Been around two years now since I left. Now 9 people work in that department with around the same efficiency as us 5 people back then.

The new employees struggle to be productive, because things are just implemented poorly and not maintainable anymore.

Had some dialogs with them some time ago. Everything I told them would happen, actually happened. What a suprise :-|
I will not go into too much detail about all the shit that's going on there, as it would be just too much (and my morning coffe is almost finished).

I think that we all know the difference between "not beautiful, but does the job" and "oh, that will backfire - badly". And I wish that my communication skills increase so that people start listening in future.

My project manager just asked me if we need to do a release in order for the changes we have done to be seen in prod.

Yes… what do you think a release means?

Love my new job but fuck they are way behind in any kind of modernization.

Just saw a demo over zoom where someone was showing the team how to change the margin on an error page.

They literally changed the HTML directly in prod using the VIM. So first of all no web modernation because there was no build, no deployment, not git, no pipeline - NOTHING!

This project went from 40 people to around 200 in 6 mos. You can't have all these people in prod just making fucking changes.

If this job did not pay 110k a year I would bail.

Just needing somewhere to let some steam off

Tl;dr: perfectly fine commandline system is replaced by bad ui system because it has a ui.

For a while now we have had a development k8s cluster for the dev team. Using helm as composing framework everything worked perfectly via the console. Being able to quickly test new code to existing apps, and even deploy new (and even third party apps) on a simar-to-production system was a breeze.

Introducing Rancher

We are now required to commit every helm configuration change to a git repository and merge to master (master is used on dev and prod) before even being able to test the the configuration change, as the package is not created until after the merge is completed.

Rolling out new tags now also requires a VCS change as you have to point to the docker image version within a file.

As we now have this awesome new system, the ops didn't see a reason to give us access to kubectl. So the dev team is stuck with a ui, but this should give the dev team more flexibility and independence, and more people from the team can roll releases.

Back to reality: since the new system we have hogged more time from ops than we have done in a while, everyone needs to learn a new unintuitive tool, and the funny thing, only a few people can actually accept VCS changes as it impacts dev and prod. So the entire reason this was done, so it is reachable to more people, is out the window.

I used to be in an infrastructure maintenance team, and I worked with an old guy. We had a jump box we all used. This guy would work weekend maintenance windows and still be trying to get changes done at 7am, three hours after the end of the window. He was glacially slow. I remember watching him login to a prod weblogic server. He would open the Windows start menu, move his fucking mouse through two or three submenus, and finally click putty. Then, he would type out the FQDN of the jump server, and move his mouse to the connect/ok button. Then it would prompt him for his username and password, both of which took him about 90 seconds to single-finger type. Then, once loved into the jump box, he would then type ssh user@server.fqdn, rather than copying and pasting the server name.

It took him fully five minutes to get logged into the weblogic server. I could not take it. It would have taken me about ten seconds.

TL;DR: When picking vendors to outsource work to, vet them really well.

Backstory:
Got a large redesign project that involves rebuilding a website's main navigation (accessibility reasons).
Project is too big just for our dev team to handle with our workload so we got to bring a 3rd party vendor to help us. We do this often so no big deal.
But, this time the twist was Senior Management already had retained hours with a dev shop so they want us to use them for project. Okay...

It begins:
Have our scope / discovery meeting about the changes and our expected DevOps workflow.

Devs work Local and push changes to our Github, that kicks off the build and we test on Dev, then it goes to Staging for more testing & PM review. Once ready we can push to prod, or whenever needed. All is agreed, everyone was happy.

Emailed the vendors' project manager to ask for their devs Github accounts so we can add them to the project. Got no reply for 3 days.

4th day, I get back "Who sets up the Github accounts?"

fuck me. they've never used Github before but in our scope meeting 4 days ago you said Github was fine...??

Whatever, fuck it. I'll make the accounts and add them.

Added 4 devs to the repo and setup new branch. 40min later get an email that they can't setup dev environment now, the dev doesn't know how to setup our CMS locally, "not working for some reason."

So, they ask for permission to develop on our STAGING server.. "because it's already setup"... they want to actively dev on our staging where we get PM/Senior Management approvals?

We have dev, staging, production instances and you want to dev in staging, not dev?... nay nay good sir.

This is whom senior management wants us to use, already paid for via retainer no less. They are a major dev shop and they're useless...

😢😭

Cant wait for today's progress checkup meeting. 😐😐

/rant

ideal sprint fallacy.
total days 10 , total hours(excluding breaks ) 8 hrs per day= 80 hrs per dev

code freeze day = day 8, testing+ fixing days : 8,9,10. release day : day 10
so ideal dev time = 7days/56 hr
meetings= - 1hr per day => 49 hrs per dev

- 1 day for planning i.e d1 . so dev time left . 6 days 42 hrs.
-----------
all good planning. now here comes the messups

1. last release took some time. so planning could not happen on d1. all devs are waiting. . devtime = 5 days 35 hrs.

2. during planning:
mgr: hey devx what's the status on task 1?
d: i integrated mock apis. if server has made the apis, i will test them .
mgr : server says the apis are done. whats your guestimate for the task completion?
d : max 1-2 hrs?
m : cool. i assign you 4 hrs for this. now what about task 2?
d : task told to me is done and working . however sub mgr mentioned that a new screen will be added. so that will take time
m : no we probably won't be taking the screen. what's your giestimate?
d : a few more testing on existing features. maybe 1-2 hrs ?
m: cool
another 4 hrs for u. what about task 3?
d : <same story>
m : cool. another 4 hrs for u. so a total of 12 hrs out of 35 hrs? you must be relaxed this sprint.
d : yeah i guess.
m cool.
-------
timelines.
d1: wasted i previous sprint
d2 : sprint planning
d3 : 3+ hrs of meetings, apis for task 1 weren't available sub manager randomly decided that yes we can add another screen but didn't discussed. updates on all 3 tasks : no change in status
d4 : same story. dev apis starts failing so testing comes to halt.
d5 : apis for task1 available . task 3 got additional improvement points from mgr out of random. some prod issue happens which takes 4+ hrs. update on tasks : some more work done on task 3, task 1 and 2 remains same.
d6 : task1 apis are different from mocks. additionally 2 apis start breaking and its come to know thatgrs did not explain the task properly. finally after another 3+ hrs of discussion , we come to some conclusions and resolutions
d7 : prod issue again comes. 4+ hrs goes into it . task 2 and 3 are discussed for new screen additiona that can easily take 2+ days to be created . we agree tot ake 1 and drop 2nd task's changes i finish task 2 new screens in 6 hrs , hoping that finally everything will be fine.

d8 : prod issue again comes, and changes are requested in task 2 and 3

day 9 build finally goes to tester
day 10 first few bugs come with approval for some tasks
day 11(day 1 of new sprint) final build with fixes is shared. new bugs (unrelated to tasks. basically new features disguised as bugs) are raised . we reject and release the build.

day 2 sprint planning
mgr : hey dev x, u had only 12 hrs of work in your plate. why did the build got delayed?

🥲🫡

I hate having to deal with our IT service desk. Every time it takes enormous energy to get to the right people and make them understand that no, you are not an idiot, but you actually have a technical issue.

Sure thing they do have a few competent nice folks there too I've gotten to know over time and they indeed have to deal with a ton of dumb non-tech savvy idiots on a daily basis. However, if my job title mentions "software" and "engineer" they should at least assume I'm an idiot in tech. Or something. Every single time I need to open a ticket, even for the simplest "add x to env y", I need to quadruple check that the subject line is moron-friendly because otherwise they would take every chance to respond "nah we can't do that", "that's not us", or "sry that's not allowed". And then I would need to respond, "yes you do:) your slightly more competent colleague just did this for us 2 weeks ago".

Now you might imagine this is on even another level when the problem is complex.

One of our internal apps has been failing because one of the internal APIs managed by a service desk team responds a 500 status code randomly but only when called with a specific internal account managed by another service desk team.
(when I say "managed by", that doesn't mean they maintain it, it just mean they are the only ones who would have access to change something)

Yesterday I spent over a fucking hour writing a super precise essay detailing the issue, proving a million times it's not on our end and that they need to fix it. Now here is an insight to what beautiful "IT service" our service desk provides:

1) ticket gets assigned to a "Connectivity Engineer" lady

2) few hours later she responds and asks me to give her the app and environment IDs and grant her access to those
(naturally everything in my email was ignored including these two IDs)

3) since the app needs to be in prod for the issue, I make a copy isolating the failing part and grant her access to the original "for reference" and the copy to play with

4) few hours later I get an email from the env that some guy called P made changes to the actual app, no changes to the copy
(maybe they immediately fixed the app even though I asked them to only touch the copy)
I also check the env and the live app had been shared with another 2 people giving them editing rights:)

5) another few hours pass and the lady responds that she had been chatting with P (no mention of who tf that guy is) and that P has a suggestion that might work and I should test it, "please see screen shot" for details:

These motherfuckers sent me a fucking screenshot of the env config file where "P has edited a few parameters" that might help. The screenshot had a 16 line part of the config json with a bunch of IDs and Base64 params which HE EDITED LOCALLY.

Again, because I needed a few iterations to realise what I've just witnessed:
These idiots modified some things in the main app (not the copy) for hours. Then came to the conclusion that the config needs some IDs and params updated. They downloaded the config json. Edited it locally. Did not fucking upload it back to the main or test app. Did not test it live. Did not CC in or direct the guy with changes to me. Did not send me the modified config file. Did not even paste the new IDs into the email. But TOOK A FUCKING SCREENSHOT OF THE MODIFIED FILE AND SENT THAT SHIT TO ME. And then had the audacity to ask me to test it when they had access to it and that's literally their fucking job.

I had to compare the fucking screenshot to the live config file and manually type in the changes.

And no, it still doesn't work. And Now I have to get back to them showing it still fails the same way but I just can't deal with these people. Fuck. Was hoping by the time I write it all down it'd be better, and it does feel a bit better, but I still need to get this app fixed. And I can only do it through these... monkeys. I just can't. Talking to these people drains my life energy... I'm just sad.

I hate dev politics...

PM: Hey there is a weird error happening when I upload this file on production, but it works on our test environments.

Me: After looking at this error, I don't find any issues with the code, but this variable is set when the application is first loaded, I bet it wasn't loaded correctly our last deployment and we just need to reload the application.

Senior Dev: We need to output all of the errors and figure out where this error is coming from. Dump out all the errors on everything in production!!

Me: That's dumb... the code works on test... it's not the code.. it's the application.

Senior dev: %$*^$>&÷^> $

Me: Hey I have an idea! If test works... I can go ahead and deploy last week's changes to prod and dump those errors you were talking about!!

Senior Dev: OK

Me: *runs Jenkins job the deploys the new code and restarts the application*

PM: YAY you fixed it!!

Senior Dev: Did you sump put those errors like I said.

Me: Nope didn't touch a thing... I just deployed my irrelevant changes to that error and reloaded the application.

A loooong time ago...

I've started my first serious job as a developer. I was young yet enthusiastic as well as a kind of a greenhorn. First time working in a business, working with a team full of experienced full-lowered ultra-seniors which were waiting to teach me the everything about software engineering.

Kind of.

Beside one senior which was the team lead as well there were two other devs. One of them was very experienced and a pretty nice guy, I could ask him anytime and he would sit down with me a give me advice. I've learned a lot of him.

Fast forward three months (yes, three months).

I was not that full kind of greenhorn anymore and people started to give me serious tasks. I had some experience in doing deployments and stuff from my other job as a sysadmin before so I was soon known as the "deployment guy", setting up deployments for our projects the right way and monitoring as well as executing them. But as it should be in every good team we had to share our knowledge so one can be on vacation or something and another colleague was able to do the task as well.

So now we come to the other teammate. The one I was not talking about till now. And that for a reason.

He was very nice too and had a couple of years as a dev on his CV, but...yeah...like...

When I switched some production systems to Linux he had to learn something about Linux. Everytime he encountered an error message he turned around and asked me how to fix it. Even. For. The. Simplest. Error. He. Could. Google. Up.

I mean okay, when one's new to a system it's not that easy, but when you have an error message which prints out THE SOLUTION FOR THE ERROR and he asks me how to fix it...excuse me?

This happened over 30 times.
A. Week.

Later on I had to introduce him to the deployment workflow for a project, so he could eventually deploy the staging environment and the production environment by hisself.

I introduced him. Not for 10 minutes. I explained him the whole workflow and the very main techniques and tools used for like two hours. Every then and when I stopped and asked him if he had any questions. He had'nt! Wonderful!

Haha. Oh no.

So he had to do his first production deployment. I sat by his side to monitor everything. He did well. One or two questions but he did well.

The same when he did his second prod deploy. Everythings fine.

And then. It. Frikkin. Begins.

I was working on the project, did some changes to the code. Okay, deploy it to dev, time for testing.

Hm.

Error checking out git. Okay, awkward. Got to investigate...

On the dev server were some files changed. Strange. The repo was all up to date. But these changes seemed newer because they were fixing at least one bug I was working on.

This doubles the strangeness.

I want over to my colleague's desk.

I asked him about any recent changes to the codebase.

"Yeah, there was a bug you were working on right? But the ticket was open like two days so I thought I'll fix it"

What the Heck dude, this bug was not critical at all and I had other tasks which were more important. Okay, but what about the changed files?

"Oh yeah, I could not remember the exact deployment steps (hint from the author: I wrote them down into our internal Wiki, he wrote them done by hisself when introducing him and after all it's two frikkin commands), so I uploaded them via FTP"

"Uhm... that's not how we do it buddy. We have to follow the procedure to avoid..."

"The boss said it was fine so I uploaded the changes directly to the production servers. It's so much easier via FTP and not this deployment crap, sorry to say that"

You. Did. What?

I could not resist and asked the boss about this. But this had not Effect at all, was the long-time best-buddy-schmuddy-friend of the boss colleague's father.

So in the end I sat there reverting, committing and deploying.

Yep

It's soooo much harder this deployment crap.

Years later, a long time after I quit the job and moved to another company, I get to know that the colleague now is responsible for technical project management.

Hm.

Project Management.

Karma's a bitch, right?

I've now worked on both monolithic solutions and microapps/microservices. I gotta say I'm not sold on the new approach. There's so much overhead! You don't have to know your way around one solution -- no, now you need to know your way around 100 solutions. Debugging? Yeah, good luck with that. You don't have to provision one environment for dev, test, staging, and prod. No, now you need 100 environments per... environment. Now, you need a dedicated fulltime devops person. Now devs can check in breaking changes because their code compiles fine in that one tiny microapp. The extra costs go on and on and on. I get the theoretical benefits but holy crap you pay for it dearly. Going back to monolithic is so satisfying. You just address the bug or new feature head on without the ceremony and complexity. You know you're not crapping on other people's day (compilation-wise) because the entire solution compiles.

...and yeah, I'm getting old. So get off the lawn! ;)

In a previous job, I was trying to organize a common repository with our shitty business partner so we could both be able to contribute our part so our work would not overlap. Not like they cared anyways.

One thing I quickly noticed is those fuckers would just straight up commit untested changes on master and cripples our whole testing and prod deployment at times because we were depending on a shitty IoT service they provided us onto which we had no control whatsoever.

I told my boss, who was often complaining about them being unreliable in the first place, I would simply restrict them from merging and commiting to develop or to master without my approval. We cannot keep working like this.

He told me that we could not impose on them our work practices and that I should not try to piss them off. To be diplomatic.

I politely and professionally refused to do it, but he did change his mind in the end. He and I left not too long after. I guess he felt obliged to respond that having his job at stake but you cannot condone voluntarily shitty work.

So I made an update to my React Native app. I changed UI of a couple of screen, added a few animations here and there, refactored how my graphQL resolvers work in the backend(no breaking changes), changed how data gets loaded into the database etc.

It worked in dev so I figured hey let's deploy it. Today is(was because it's now 3am but more on that later) a national holiday so no one goes to work so no one will use my app so I have an entire day to deploy.

I started at 15:00(because i woke up at 13:00 lol). I tested the update once again in dev and proceeded to deploy it to prod. I merged backend to master, built docker images, did migrations on the db, restarted docker-compose with new images. And now for the app. I run ./gradlew assembleRelease and it starts complaining that react-native-gesture-handler is not installed. Ugh, rm -rf node_modules && yarn install. It worked. But now gradlew crashes and logs don't tell me anything. Google tells me to change a bunch of gradle settings but none of them work. Fast forward 5h, it's around 20:00 and I isolated the issue to, again, react-native-gesture-handler. They updated from 2.2.4 to 2.3.0 which didn't fucking compile. 2 more hours passed (now 22:00) and I got v2.3.1 working which fixed the problem in 2.3.0 but made my app crash on startup. YOUR FUCKING LIBRARY GETS 250K WEEKLY DOWNLOADS AND YOU DONT EVEN BOTHER CHECKING IF IT COMPILES IN PROD ON ANDROID?! WHAT THE FUCK software-mansion?

After I solved that, my app didn't crash. Now it threw an error "Type errors: Network Request Failed" every time I fetch my legacy REST API(older parts use rest and newer use graphql. I'll refactor that in the next update). I'll spare you the debugging hell i went through but another 5h passed. Its 3am. My config had misspelled url to prod but good for dev... I hate myself and even more so react-native-gesture-handler.

So we have this project that we are hosting on our testing server for presentation purposes ( already provisioning prod server ).

Our boss was presenting it to investors and my superior committed a bug there and was asking me help to figure out how to fix it (yeah.. he doesn't know how to checkout last commits in git... fml), and I realised the presentation might still be going on... so I asked: isn't boss showing it to investors?

superior: lol, idk maybe.

me: right... ( I proceed to roll back changes ) bye, have a good lunch.

And here I am having lunch considering my life choices.

I just joined a new company.
Their CI pipeline is to give root access to staging and prod servers to every developer in the company and the manually git pull each repository (8-10 repos per server) and manually set nginx and port configurations. And if this wasn't enough, all of the 30 sites they have are basically the same site and they make the changes manually for each tenant (no env file). I'm amazed at how hard some people are willing to work.

Small chaotic startup that never grew up (15 years atm).
Hosts/maintains a number of apps/sites for various customers.

At some point, someone decides that a CMS would be usefull to maintain the content across all products. Forgoing all sense, reason and the very notion of "additional maintenance and dev" it is decided that one should be built in-house.

Fast forward a number of years.

Ops performs routine maintenance on prod-servers. A java-patch accidently knocks out one of the pillars a 3rd party lib the CMS uses for storing images. CMS basically burst in to flames causing a.... significant incident.

Enter yours truly to fix the mess.
Spend a few days replacing the affected 3rd party lib. Run tests on CMS in test and staging environments. Apply java-patch. All seems fine.

When speaking to frontenders and app-devs, a significant hurdle present itself:
All test/staging instances of all websites/apps/etc ALL USE PRODUCTION CMS. Hardcoded. No way around.

There is -no- way to properly test and verify the functionality of any changes made to the home-brewed CMS.
My patch did indeed work in the end.

But did the company learn anything? Did they listen to my reasoning, pleading or even anguished screams for sanity?

No.

While addressing a Senior Dev's (SD) query from another team.

SD: why is this field mandatory? Can't it be just optional? Any other work around?
Me: Is your code changes already pushed in Devo? In that case, we provide a value which will work since you are not concerned about it.
SD: Yes. It's pushed till production. And, I want to test changes in Prod.
Me: (shared some codes) and explained that this feature for testing is only available in Devo.
SD: I know that. (Shared me a ticket) I want this field to be optional. That's it.
Me: (read the entire ticket. Didn't find anything related their) Told him, I will discuss with team. And meanwhile, for Devo, you can use this value.

Next morning, I accidentally came over some other ticket raised by him only which had the correct doubts regarding request to support this field in production
Now, I don't know why did he share a wrong ticket with me.
And, how will it even help him if that field was even optional.
THAT JUST WONT WORK IN PRODUCTION.

I will discuss with my team and see what can happen.

Long post, TLDR: Given a large team building large enterprise apps with many parts (mini-projects/processes), how do you reduce the bus-factor and the # of Brent's (Phoenix Project)?

# The detailed version #

We have a lot of people making changes, building in new processes to support new flows or changes in the requirements and data.

But we also have to support these except when it gets into Production there is little information to quickly understand:
- how it works
- what it does/supposed to do
- what the inputs and dependencies are

So often times, if there's an issue, I have to reverse engineer whatever logic I can find out of a huge mess.

I guess the saying goes: the only people that know how it works is whoever wrote it and God.

I'm a senior dev but i spend a lot of time digging thru source code and PROD issues to figure out why ... is broken and how to maybe fix it.

I think in Agile there's supposed to be artifacts during development but never seen em.

Personally whenever i work on a new project, I write down notes and create design diagrams so i can confirm things and have easy to use references while working.

I don't think anyone else does that. And afterwards, I don't have anywhere to put it/share it. There is no central repo for this stuff other than our Wiki but for the most part, is like a dumping ground. You have to dig for information and hoping there's something useful.

And when people leave, information is lost forever and well... we hire a lot of monkeys... so again I feel a lot of times i m trying to recover information from a corrupted hard drive...

The only way real information is transferred is thru word of mouth, special knowledge transfer sessions.

Ideally I would like anything that goes into PROD to have design docs as well as usage instructions in order for anyone to be able to quickly pick it up as needed but I'm not sure if that's realistic.

Even unit tests don't seem to help much as they just test specific functions but don't give much detail about how a whole process is supposed to work.

Week ago, leader of the artifacts/packages storage and mirroring with animal in Logo, fucked up our testing enabling new feature on theirs SaaS.

We created a ticket, they managed to fix that, although it took them long time to do so, probably due to timezones as fix was simple click on their Admin area.

Today they forwarded us email that there will be some changes that can impact prod ...

Great timing, great ..

What's better in general:
#1 master merge -> production
#2 master merge -> button -> production

And why?

At work we currently use #2 because sometimes we need to make manual changes or update images on prod before we can fully update the code

Edit: This should have been a 'question' not a joke/meme, sorry about that

So I'm assigned once again to fix a new someone else created and that seems to be the case whenever there's an issue...

Boss just assigns it to whoever is most likely to be able to investigate it... which is basically me. Other than the little time I can use to develop stuff, I'm usually cleaning up other people's messes.

And these other people are to busy working on new crap to properly explain how their existing code/processes/changes works.

And well the fact that anything breaks in production (that's not due to upstream one off issues) whoever does not think he needs to take responsibility for it.

So everyone else and especially me has to spend time understanding the shit they wrote and fixing it for them.

How do I tell my boss this nicely that we need clearly definitely ownership and whenever a component blows up in prod, the guy that wrote the code fixes it no matter what? Thereby incentivizing him to not write shit code in the first place and be more proactive in making sure it doesn't in the first place since he knows otherwise he's doing overtime to fix it?

Is it just me or is there really no such thing as a dev job where something doesn't blow up due to poorly tested and designed code every other day?

So there is this owner team who reviewed my code recently. I don't have much context about the their system and architecture. We try to build our changes with less context and rely in owner team's knowledge for any review gap.
The guy from the owner team missed something in my review and changes went to prod, review already took more that it was expected to take. He took 1 week for small change reviews. Now, not him but with someone else's advice they had to revert.
I wrote a mail shooting to manager, the guy who reverted and the guy who reviewed, asking the reviewer guy to explain why didn't he mentioned about any issues at the time of the review.
I have tried best from my side. But all this, god!!!
Why everything I do has some kind of weird issue. I feel so bad blaming the guy, I just think that, the way I used to feel anxious he must be feeling the same, but what can I do? I don't want to take the blame I don't even see if I can and I shouldn't be. If it was a major issue it should have been raised but he didn't. I feel so bad that I am almost crying, I am feeling that like always I am going to be judged by my team that work is slow and on top of that I can't do anything for the guy I blamed it on.

I don't know, is it my mistake? but I cannot think of anyway I would have known this.

- Change this, change that too, oh and that too.
* Ok, will do ( however unlikely it's going to be finished correctly, as you didn't consult me before or listen to me about the impact this might have )
- ( just stfu and do it )

Sometime at or after important cutoff:
- Hey this doesn't show up, is that right
* Yeah, you wanted too many things changed at once + fix it and notice a stupid error like an && switched with a ||, or other models that don't know about change x yet

/repeat

Fuck my company, sincerly.

So Im crunching my ass off, to make product, there is +- fuckton of changes that for example require refactoring flow of certain things, restructure of how shit work, Im +- 2nd weekend now, and most heavy features are cleared.

I work till late. constantly I have someone with stupid shit like calls, indeed Im needed for that stuff but also, that slows down progress of this project. Just sake of example friday 18:00 I had call (I work till 16:00) about new minor and frankly easy feature. Today, morning 8:30 one call, than 13:00 long call, Ive done the feature, didn't push it to alpha. yet though.

Now during that call that started 13:00 I get yelled on that all ordered features aren't on prod yet (I throw them to alpha becouse manual tests must be done as standard here).

Dude what the motherfuck. Im literally wearing my ass off to deliver your stupid product becouse I know its critical for company but it does not mean I can do it all in one fucking night.

F**k off and shut your mouth up and let me work for f**k sakes.

Ah also, stop f**king remotely micromanage me you little piece of sh*t.

Thanx for allowing me to vent out,
Peace.

Pushed some changes to PROD today. Go to login and check changes .. noooooope!

Still a bit new to Symfony 5... but I'm just not a fan right now. The login screen just jumps back to itself. No login failed message and prod log had a size of 0 so that was no help.

Traced this thing way down into the CSRF Authentication functions. \is_callable(...namespace) just returning null so no go on getting a token for isTokenValid() =/
ugh! This is truly the most torturous junk I've ever seen. Nothing in the logs so I decided to just use the good old ECHO'HERE' debugger.

What was the issue you might ask?... effin' yaml file
Fix for now is to set the session handler_id back to null

When you get an assignment by mail,
- you do those changes exactly as being asked, albeit one insignificant extra visual thing,

- you send back exactly what's changed (which is exactly what was being asked, but naming things as they are, clearer than I can ever explain changes),

- you go to the person that has assigned the very thing, explain it, he seems to get it,

...
Then.. a few hours later.. you get a reply via mail:
"Come explain what you're saying here, because I don't get it."

🤨

My team is pretty small right now. It's myself and two other guys. One lead, who's been here for five years. A senior who we brought on 2 weeks ago. And me, a regular app dev. The lead put his two weeks in last week and has been trying to brain dump as much as he can onto us.

I've been building a list of prioritization to compensate for when he leaves based on what he was saying was the most important. This list has gotten pretty massive after reviewing most of the processes in place.

I was hired mainly to quell new requests coming in and not to maintain our systems, so that's what I did. I didn't examine our prod code base too closely. I wish I had. It's in a sorry state. I'm pretty sure I have about 2 years of tech debt for a crew of two guys constantly working on it.

I've been trying to prioritize based on what gets the most bug fixes and change requests. These apps will see the biggest changes and will undergo the most maintenance.

Since I'm just a regular app dev it feels weird trying to come up with this and try to prioritize this and come up with a plan. It feels like someone else should have. If it needs done then I guess it needs done. I need to be able to collaborate and work with my co worker and be able to plan for what projects are coming next.

If anyone has any suggestions to tackle tech debt please make them. Or if there's any help for managing priorities in a different manner that may prove helpful I'm open. Honestly, I don't want to tackle this completely blind, it feels like a lot.