I've found and fixed any kind of "bad bug" I can think of over my career from allowing negative financial transfers to weird platform specific behaviour, here are a few of the more interesting ones that come to mind...

#1 - Most expensive lesson learned

Almost 10 years ago (while learning to code) I wrote a loyalty card system that ended up going national. Fast forward 2 years and by some miracle the system still worked and had services running on 500+ POS servers in large retail stores uploading thousands of transactions each second - due to this increased traffic to stay ahead of any trouble we decided to add a loadbalancer to our backend.

This was simply a matter of re-assigning the IP and would cause 10-15 minutes of downtime (for the first time ever), we made the switch and everything seemed perfect. Too perfect...

After 10 minutes every phone in the office started going beserk - calls where coming in about store servers irreparably crashing all over the country taking all the tills offline and forcing them to close doors midday. It was bad and we couldn't conceive how it could possibly be us or our software to blame.

Turns out we made the local service write any web service errors to a log file upon failure for debugging purposes before retrying - a perfectly sensible thing to do if I hadn't forgotten to check the size of or clear the log file. In about 15 minutes of downtime each stores error log proceeded to grow and consume every available byte of HD space before crashing windows.

#2 - Hardest to find

This was a true "Nessie" bug.. We had a single codebase powering a few hundred sites. Every now and then at some point the web server would spontaneously die and vommit a bunch of sql statements and sensitive data back to the user causing huge concern but I could never remotely replicate the behaviour - until 4 years later it happened to one of our support staff and I could pull out their network & session info.

Turns out years back when the server was first setup each domain was added as an individual "Site" on IIS but shared the same root directory and hence the same session path. It would have remained unnoticed if we had not grown but as our traffic increased ever so often 2 users of different sites would end up sharing a session id causing the server to promptly implode on itself.

#3 - Most elegant fix

Same bastard IIS server as #2. Codebase was the most unsecure unstable travesty I've ever worked with - sql injection vuns in EVERY URL, sql statements stored in COOKIES... this thing was irreparably fucked up but had to stay online until it could be replaced. Basically every other day it got hit by bots ended up sending bluepill spam or mining shitcoin and I would simply delete the instance and recreate it in a semi un-compromised state which was an acceptable solution for the business for uptime... until we we're DDOS'ed for 5 days straight.

My hands were tied and there was no way to mitigate it except for stopping individual sites as they came under attack and starting them after it subsided... (for some reason they seemed to be targeting by domain instead of ip). After 3 days of doing this manually I was given the go ahead to use any resources necessary to make it stop and especially since it was IIS6 I had no fucking clue where to start.

So I stuck to what I knew and deployed a $5 vm running an Nginx reverse proxy with heavy caching and rate limiting linked to a custom fail2ban plugin in in front of the insecure server. The attacks died instantly, the server sped up 10x and was never compromised by bots again (presumably since they got back a linux user agent). To this day I marvel at this miracle $5 fix.

Inherited a simple marketplace website that matches job seekers and hospitals in healthcare. Typically, all you need for this sort of thing is a web server, a database with search

But the precious devs decided to go micro-services in a container and db per service fashion. They ended up with over 50 docker containers with 50ish databases. It was a nightmare to scale or maintain!

With 50 database for for a simple web application that clearly needs to share data, integration testing was impossible, data loss became common, very hard to pin down, debugging was a nightmare, and also dangerous to change a service’s schema as dependencies were all tangled up.

The obvious thing was to scale down the infrastructure, so we could scale up properly, in a resource driven manner, rather than following the trend.

We made plans, but the CTO seemed worried about yet another architectural changes, so he invested in more infrastructure services, kubernetes, zipkin, prometheus etc without any idea what problems those infra services would solve.

In fact I'm a sinful dev, so that I can't easily decide which one is worst. From indenting with tabs, or using nano instead of vim/emacs, to hardcoding database credentials on server, to many hacks and workarounds I use as actual "fixes" when the deadline is upon me and I've tried all I could. But it always led only to my own regret. For instance, my latest sin was that I prefered Debian over Arch and used proprietary graphic drivers to speed up my new setup. But ended up with a curse from St. Ignucius. (check my last rant)

But my worst sin probably goes to when I was "printf-debugging" some issue for a GSM controller on a raspberry pi. I forgot to remove one little print line and deployed the new "fixed" version. I didn't follow that project after that for like a month or so, when the client posted back the device and said that "it just doesn't work anymore". It seemed that raspbian didn't boot beacause the sd card was curroptted. I dd'ed through the card and I noticed that there are billions of lines of "DEBUG:: reading stream from 192.some.shitty.ip", took almost all over the 32G sdcard. Just as I suddenly remembered the cursed line I just added a month ago, I declared the sd card dead with no hesitation, dunce-commented the line (so the history would remember), implemented a time out for the thread containing it, setup a journald unit for my service and removed the redirection of process output to a log file, found a new sd card and installed everything again, and finally posted back the new "fix" to the client.

Moral: Never comfort yourself for the sins you have commited in the past kids, they certainly will come back to you. And also not to do any io especially write to a file on an SD card with ext fs, in a potentially infinite loop with no timeout.

P.S: I'd posted my last rant just before the new week rant last nigh. I really liked the St. Ignucius meme so decided to create a new one. He's very adorable :)

So ok here it is, as asked in the comments.

Setting: customer (huge electronics chain) wants a huge migration from custom software to SAP erp, hybris commere for b2b and ... azure cloud
Timeframe: ~10 months….

My colleague and me had the glorious task to make the evaluation result of the B2B approval process (like you can only buy up till € 1000, then someone has to approve) available in the cart view, not just the end of the checkout. Well I though, easy, we have the results, just put them in the cart … hmm :-\
The whole thing is that the the storefront - called accelerator (although it should rather be called decelerator) is a 10-year old (looking) buggy interface, that promises to the customers, that it solves all their problems and just needs some minor customization. Fact is, it’s an abomination, which makes us spend 2 months in every project to „ripp it apart“ and fix/repair/rebuild major functionality (which changes every 6 months because of „updates“.

After a week of reading the scarce (aka non-existing) docs and decompiling and debugging hybris code, we found out (besides dozends of bugs) that this is not going to be easy. The domain model is fucked up - both CartModel and OrderModel extend AbstractOrderModel. Though we only need functionality that is in the AbstractOrderModel, the hybris guys decided (for an unknown reason) to use OrderModel in every single fucking method (about 30 nested calls ….). So what shall we do, we don’t have an order yet, only a cart. Fuck lets fake an order, push it through use the results and dismiss the order … good idea!? BAD IDEA (don’t ask …). So after a week or two we changed our strategy: create duplicate interface for nearly all (spring) services with changed method signatures that override the hybris beans and allow to use CartModels (which is possible, because within the super methods, they actually „cast" it to AbstractOrderModel *facepalm*).
After about 2 months (2 people full time) we have a working „prototype“. It works with the default-sample-accelerator data. Unfortunately the customer wanted to have it’s own dateset in the system (what a shock). Well you guess it … everything collapsed. The way the customer wanted to "have it working“ was just incompatible with the way hybris wants it (yeah yeah SAP, hybris is sooo customizable …). Well we basically had to rewrite everything again.
Just in case your wondering … the requirements were clear in the beginning (stick to the standard! [configuration/functinonality]). Well, then the customer found out that this is shit … and well …

So some months later, next big thing. I was appointed technical sublead (is that a word)/sub pm for the topics‚delivery service‘ (cart, delivery time calculation, u name it) and customerregistration - a reward for my great work with the b2b approval process???

Customer's office: 20+ people, mostly SAP related, a few c# guys, and drumrole .... the main (external) overall superhero ‚im the greates and ur shit‘ architect.
Aberage age 45+, me - the ‚hybris guy’ (he really just called me that all the time), age 32.
He powerpoints his „ tables" and other weird out of this world stuff on the wall, talks and talks. Everyone is in awe (or fear?). Everything he says is just bullshit and I see it in the eyes of the others. Finally the hybris guy interrups him, as he explains the overall architecture (which is just wrong) and points out how it should be (according to my docs which very more up to date. From now on he didn't just "not like" me anymore. (good first day)
I remember the looks of the other guys - they were releaved that someone pointed that out - saved the weeks of useless work ...

Instead of talking the customer's tongue he just spoke gibberish SAP … arg (common in SAP land as I had to learn the hard way).
Outcome of about (useless) 5 meetings later: we are going to blow out data from informatica to sap to azure to datahub to hybris ... hmpf needless to say its fucking super slow.
But who cares, I‘ll get my own rest endpoint that‘ll do all I need.

First try: error 500, 2. try: 20 seconds later, error message in html, content type json, a few days later the c# guy manages to deliver a kinda working still slow service, only the results are wrong, customer blames the hybris team, hmm we r just using their fucking results ...
The sap guys (customer service) just don't seem to be able to activate/configure the OOTB odata service, so I was told)
Several email rounds, meetings later, about 2 months, still no working hybris integration (all my emails with detailed checklists for every participent and deadlines were unanswered/ignored or answered with unrelated stuff). Customer pissed at us (god knows why, I tried, I really did!). So I decide to fly up there to handle it all by myself

I got a dayjob in a company. I got an error. I cannot solve it and I am so desperate. So I go to stackoverflow, nobody answers. I post on git issue, but nobody solves the problem. So, I pay someone to solve it, like Hackhands.com to find a mentor. There is no mentor that can help. So I pay more, hired a peer, and finally a development team just to help me. They get paid only if they solved it.

But each of my folks repeat my same steps, asking on stackoverflows or github, and none of these help. So, they end up hiring their own friends and mentors. Their friends also end up paying (pay before problem solved) someone to help them.

their friends pay for friends of friends, then friends of friends of friends

And all of a sudden it becomes a giant MLM scheme.

And those people they paid for actually work for a company behind the scene which I am a founder of 😁

Multi billions startup idea, is it?

In today's episode of kidding on SystemD, we have a surprise guest star appearance - Apache Foundation HTTPD server, or as we in the Debian ecosystem call it, the Apache webserver!

So, imagine a situation like this - Its friday afternoon, you have just migrated a bunch of web domains under a new, up to date, system. Everything works just fine, until... You try to generate SSL certificates from Lets Encrypt.

Such a mundane task, done more than a thousand times already... Yet... No matter what you do, nothing works. Apache just returns a HTTP status code 403 - Forbidden.

Of course, what many folk would think of first when it came to a 403 error is - Ooooh, a permission issue somewhere in the directory structure!

So you check it... And re-check it to make sure... And even switch over to the user the webserver runs under, yet... You can access the challenge just fine, what the hell!

So you go deeper... And enable the most verbose level of logging apache is capable of - Trace8. That tells you... Not a whole lot more... Apparently, the webserver was unable to find file specified? But... Its right there, you can see it!

So you go another step deeper and start tracing the process' system calls to see exactly where it calls stat/lstat on the file, and you see that it... Calls lstat and... It... Returns -1? What the hell#2!

So, you compile a custom binary that calls lstat on the first argument given and prints out everything it returns... And... It works fine!

Until now, I chose to omit one important detail that might have given away the issue to the more knowledgeable right away. Our webservers have the URL /.well-known/acme-challenge/, used for ACME challenges, aliased somewhere else on the filesystem - To /tmp/challenges.

See the issue already?

Some *bleep* over at the Debian Package Maintainer group decided that Apache could save very sensitive data into /tmp, so, it would be for the best if they changed something that worked for decades, and enabled a SystemD service unit option "PrivateTmp" for the webserver, by default.

What it does is that, anytime a process started with this option enabled writes to /tmp/*, the call gets hijacked or something, and actually makes the write to a private /tmp/something/tmp/ directory, where something... Appeared as a completely random name, with the "apache2.service" glued at the end.

That was also the only reason why I managed fix this issue - On the umpteenth time of checking the directory structure, I noticed a "systemd-private-foobarbas-apache2.service-cookie42" directory there... That contained nothing but a "tmp" directory with 777 as its permission, owned by the process' user and group.

Overriding that unit file option finally fixed the issue completely.

I have just one question - Why? Why change something that worked for decades? I understand that, in case you save something into /tmp, it may be read by 3rd parties or programs, but I am of the opinion that, if you did that, its only and only your fault if you wrote sensitive data into the temporary directory.

And as far as I am aware, by default, Apache does not actually write anything even remotely sensitive into /tmp, so...

Why. WHY!
I wasted 4 hours of my life debugging this! Only to find out its just another SystemD-enabled "feature" now!

And as much as I love kidding on SystemD, this time, I see it more as a fault of the package maintainers, because... I found no default apache2/httpd service file in the apache repo mirror... So...

The $customer gets a device from us, with th wifi connected as specified in the order. $customer connects it to the mains and monitor, puts in the dongle and the connection is established.
Fast forward 3 weeks, now everything went south. The device does not connect to the network, the service is offline. Our first question: "Has someone modified the WiFi name or password?"
$customer: "No, there were no changes in the WiFi"

So the full arsenal of debugging the connection over LAN starts, interrupted by $customer unplugging the device "because he needs LAN now"

After sometime, we figured out, everything is fine with the device, and ask $customer once again, if the config $ssid and $password is correct.

$customer: "Oh, we changed the name to $ssid2 because it looks nicer, is that a problem?"

Internal: "Are you f*kin kidding me? I asked you exactly that"
Me: "Alright, that explains the issues. Please tell us in advance if you want to change something with the WiFi."

It was the last year of high school.

We had to submit our final CS homework, so it gets reviewed by someone from the ministry of education and grade it. (think of it as GPA or whatever that is in your country).

Now being me, I really didn’t do much during the whole year, All I did was learning more about C#, more about SQL, and learn from the OGs like thenewboston, derek banas, and of course kudvenkat. (Plus more)

The homework was a C# webform website of whatever theme you like (mostly a web store) that uses MS Access as DB and a C# web service in SOAP. (Don’t ask.)

Part 1/2:

Months have passed, and only had 2 days left to deadline, with nothing on my hand but website sketches, sample projects for ideas, and table schematics.

I went ahead and started to work on it, for 48 hours STRAIGHT.

No breaks, barely ate, family visited and I barely noticed, I was just disconnected from reality.

48 hours passed and finished the project, I was quite satisfied with my it, I followed the right standards from encrypting passwords to verifying emails to implementing SQL queries without the risk of SQL injection, while everyone else followed foot as the teacher taught with plain text passwords and… do I need to continue? You know what I mean here.

Anyway, I went ahead and was like, Ok, lets do one last test run, And proceeded into deleting an Item from my webstore (it was something similar to shopify).

I refreshed. Nothing. Blank page. Just nothing. Nothing is working, at all.

Went ahead to debug almost everywhere, nothing, I’ve gone mad, like REALLY mad and almost lose it, then an hour later of failed debugging attempts I decided to rewrite the whole project from scratch from rebuilding the db, to rewriting the client/backend code and ui, and whatever works just go with it.

Then I noticed a loop block that was going infinite.

NEVER WAIT FOR A DATABASE TO HAVE MINIMUM NUMBER OF ROWS, ALWAYS ASSUME THAT IT HAS NO VALUES. (and if your CPU is 100%, its an infinite loop, a hard lesson learned)

The issue was that I requested 4 or more items from a table, and if it was less it would just loop.

So I went ahead, fixed that and went to sleep.

Part 2/2:

The day has come, the guy from the ministry came in and started reviewing each one of the students homeworks, and of course, some of the projects crashed last minute and straight up stopped working, it's like watching people burning alive.

My turn was up, he came and sat next to me and was like:

Him: Alright make me an account with an email of asd@123.com with a password 123456

Me: … that won't work, got a real email?

Him: What do you mean?

Me: I implemented an email verification system.

Him: … ok … just show me the website.

Me: Alright as you can see here first of all I used mailgun service on a .tk domain in order to send verification emails you know like every single website does, encrypted passwords etc… As you can see this website allows you to sign up as a customer or as a merc…

Him: Good job.

He stood up and moved on.

YOU MOTHERFUCKER.

I WENT THROUGH HELL IN THE PAST 48 HOURS.

AND YOU JUST SAT THERE FOR A MINUTE AND GAVE UP ON REVIEWING MY ENTIRE MASTERPIECE? GO SWIM IN A POOL FULL OF BURNING OIL YOU COUNTLESS PIECE OF SHIT

I got 100/100 in the end, and I kinda feel like shit for going thought all that trouble for just one minute of project review, but hey at least it helped me practice common standards.

Spent 4 hours debugging a script. Running the commands manually in cli worked fine but not when run as a web service. I've done similar things before so I was confident it should work. I tried literary everything I was going crazy. Turns out that Atom added some garbage characters only visible in other editors.. And this was not the first time! Fml..

Having a lot of bad experiences while working as intern in startups and about to join a MNC, i wanted to share my work life balance and technical demands that i expect from a company. These are going to be my list of checkpoints that i look forward , let me know which of them are way too unrealistic. also add some of yours if i missed anything :

Work life balance demands ( As a fresher, i am just looking forward for 1a, 2a and 8, but as my experience and expertise grows, i am looking forward for all 10. Would i be right to expect them? ):

1a 8 hr/day. 1b 9h/day
2a 5days/week. 2b 6 days/week
3 work from home (if am not working on something that requires my office presence)
4 get out of office whenever i feel like i am done for the day
5 near to home/ office cab service
6 office food/gym service
7 mac book for working
8 2-4 paid leaves/month
9 paid overtime/work on a holiday
10.. visa sponsorship if outside india

Tech Demands (most of them would be gone when i am ready to loose my "fresher " tag, but during my time in internship, training i always wished if things happened this way):

1. I want to work as a fresher first, and fresher means a guy who will be doing more non tech works at first than going straight for code. For eg, if someone hires me in the app dev team, my first week task should be documenting the whole app code / piece of it and making the test cases, so that i can understand the environment/ the knowledge needed to work on it

2. Again before coding the real meaningful stuff for the main product, i feel i should be made to prepare for the libraries ,frameworks,etc used in the product. For eg if i don't know how a particular library ( say data binding) used in the app, i should be asked to make a mini project in 1-2 days using all the important aspects of data binding used in the project, to learn about it. The number of mini tasks and time to complete them should be given adequately , as it is only going to benefit the company once am proficient in that tech

3. Be specific in your tasks for the fresher. You don't want a half knowledgeable fresher/intern think on its own diverging from your main vision and coding it wrong. And the fresher is definitely not wrong for doing so , if you were vague on the first place.

4. most important. even when am saying am proficient , don't just take my word for it. FUCKIN REVIEW MY CODE!! Personally, I am a person who does a lot of testing on his code. Once i gave it to you, i believe that it has no possible issues and it would work in all possible cases. But if it isn't working then you should sit with me and we 2 should be looking, disccussing and debugging code, and not just me looking at the code repeatedly.

4. Don't be too hard on fresher for not doing it right. Sometimes the fresher might haven't researched so much , or you didn't told him the exact instructions but that doesn't mean you have the right to humiliate him or pressurize him

5. Let multiple people work on a same project. Sometimes its just not possible but whenever it is, as a senior one must let multiple freshers work on the same project. This gives a sense of mutual understanding and responsibility to them, they learn how to collaborate. Plus it reduces the burden/stress on a single guy and you will be eventually getting a better product faster

Am i wrong to demand those things? Would any company ever provide a learning and working environment the way i fantasize?