Talking to my angry girlfriend is the hardest debugging process.
I can't even find out what went wrong

!rant

My gf loves rubber ducks so much, I'm just waiting for her to learn coding just for the debugging process :D

Fixed a project for a collegue.
Him: How did you fix things, what's the process?
Me: Click stuff and press Buttons randomly until it works.
Him: ... That's all you did?
Me: Yup, the rest is just experience.

(Written March 13th at 2am.)

This morning (yesterday), my computer decided not to boot again: it halts on "cannot find firmware rtl-whatever" every time. (it has booted just fine several times since removing the firmware.) I've had quite the ordeal today trying to fix it, and every freaking step along the way has thrown errors and/or required workarounds and a lot of research.

Let's make a list of everything that went wrong!

1) Live CD: 2yo had been playing with it, and lost it. Not easy to find, and super smudgy.

2) Unencrypt volume: Dolphin reports errors when decrypting the volume. Research reveals the Live CD doesn't incude the cryptsetup packages. First attempts at installing them mysteriously fail.

3) Break for Lunch: automatic powersaving features turned off the displays, and also killed my session.

4) Live CD redux: 25min phonecall from work! yay, more things added to my six-month backlog.

5) Mount encrypted volume: Dolphin doesn't know how, and neither do I. Research ensues. Missing LVM2 package; lvmetad connection failure ad nauseam; had to look up commands to unlock, clone, open, and mount encrypted Luks volume, and how to perform these actions on Debian instead of Ubuntu/Kali. This group of steps took four hours.

6) Chroot into mounted volume group: No DNS! Research reveals how to share the host's resolv with the chroot.

7) `# apt install firmware-realtek`: /boot/initrd.img does not exist. Cannot update.

8) Find and mount /boot, then reinstall firmware: Apt cannot write to its log (minor), listed three install warnings, and initially refused to write to /boot/initrd.img-[...]

9) Reboot!: Volume group not found. Cannot process volume group. Dropping to a shell! oh no..

(Not listed: much research, many repeated attempts with various changes.)

At this point it's been 9 hours. I'm exhausted and frustrated and running out of ideas, so I ask @perfectasshole for help.
He walks me through some debugging steps (most of which i've already done), and we both get frustrated because everything looks correct but isn't working.

10) Thirteenth coming of the Live CD: `update-initramfs -u` within chroot throws warnings about /etc/crypttab and fsck, but everything looks fine with both. Still won't boot. Editing grub config manually to use the new volume group name likewise produces no boots. Nothing is making sense.

11) Rename volume group: doubles -'s for whatever reason; Rebooting gives the same dreaded "dropping to a shell" result.

A huge thank-you to @perfectasshole for spending three hours fighting with this issue with me! I finally fixed it about half an hour after he went to bed.

After renaming the volume group to what it was originally, one of the three recovery modes managed to actually boot and load the volume. From there I was able to run `update-initramfs -u` from the system proper (which completed without issue) and was able to boot normally thereafter.

I've run updates and rebooted twice now.
After twelve+ hours... yay, I have my Debian back!

oof.

So yesterday I deployed a build on our release environment and i had added a new rest api end-point which I needed to test.. A heads up though, its written in java spring and the entire flow consisted of too many calls/returns from various other java & python services.. Also to make things worse, the entire deployment is a really cumbersome process as you need to copy the build from one box to another..

After like almost 4-5 hours of debugging, adding logs left right & center, crazy upload speeds (yaa this is sarcastic) and frustation at its peak, I found the issue..

There was an if condition that was checking for equality between an enum constant & an enum in a request aaaannnnnddd

*Drum roll

THE CONSTANT ENUM BELONGED TO THE WRONG PACKAGE HENCE ALWAYS EVALUATING TO FALSE... ALSO, BOTH THE ENUMS IN THE DIFFERENT PACKAGES ARE IDENTICAL... FUCCKKKKKKK MY LIFE

😑🔫

Ok friends let's try to compile Flownet2 with Torch. It's made by NVIDIA themselves so there won't be any problem at all with dependencies right?????? /s

Let's use Deep Learning AMI with a K80 on AWS, totally updated and ready to go super great always works with everything else.

> CUDA error
> CuDNN version mismatch
> CUDA versions overwrite
> Library paths not updated ever
> Torch 0.4.1 doesn't work so have to go back to Torch 0.4
> Flownet doesn't compile, get bunch of CUDA errors piece of shit code
> online forums have lots of questions and 0 answers
> Decide to skip straight to vid2vid
> More cuda errors
> Can't compile the fucking 2d kernel
> Through some act of God reinstalling cuda and CuDNN, manage to finally compile Flownet2
> Try running
> "Kernel image" error
> excusemewhatthefuck.jpg
> Try without a label map because fuck it the instructions and flags they gave are basically guaranteed not to work, it's fucking Nvidia amirite
> Enormous fucking CUDA error and Torch error, makes no sense, online no one agrees and 0 answers again
> Try again but this time on a clean machine
> Still no go
> Last resort, use the docker image they themselves provided of flownet
> Same fucking error
> While in the process of debugging, realize my training image set is also bound to have bad results because "directly concatenating" images together as they claim in the paper actually has horrible results, and the network doesn't accept 6 channel input no matter what, so the only way to get around this is to make 2 images (3 * 2 = 6 quick maths)
> Fix my training data, fuck Nvidia dude who gave me wrong info
> Try again
> Same fucking errors
> Doesn't give nay helpful information, just spits out a bunch of fucking memory addresses and long function names from the CUDA core
> Try reinstalling and then making a basic torch network, works perfectly fine
> FINALLY.png
> Setup vid2vid and flownet again
> SAME FUCKING ERROR

> Try to build the entire network in tensorflow
> CUDA error
> CuDNN version mismatch
> Doesn't work with TF
> HAVE TO FUCKING DOWNGEADE DRIVERS TOO
> TF doesn't support latest cuda because no one in the ML community can be bothered to support anything other than their own machine
> After setting up everything again, realize have no space left on 75gb machine
> Try torch again, hoping that the entire change will fix things

At this point I'll leave a space so you can try to guess what happened next before seeing the result.

Ready?

3

2

1

> SAME FUCKING ERROR

In conclusion, NVIDIA is a fucking piece of shit that can't make their own libraries compatible with themselves, and can't be fucked to write instructions that actually work.

If anyone has vid2vid working or has gotten around the kernel image error for AWS K80s please throw me a lifeline, in exchange you can have my soul or what little is left of it

I started my internship at the end of the year..
Fuck my ass!!! This code I have to work with is a huge pile of shit.
The code base I need to work with is around 40k LOC. It is a mixture of C++, C, Java, Python, Bash and I think I saw some lonely js files around.

A list of awesome parts:
- Paths are hard coded.
- Redundant code everywhere
- No documentation or inline comments available

Most of the comments in the code are just old code that is not used anymore. But the cherry on the turd is the class that should provide all kind of useful functions in my daily routine. About ninety percent of the functions have the same description or nothing. Sometimes a function name says "readSomethingFromSomewhere" but instead it writes something to a file. It is really confusing and I need to check everything twice instead of rely on what the function name promises.
I have also learned why copy paste isn't that good. The brief descriptions of every method in a files are always the same.

getName() - Description: Fork child process
getIp() - Description: Fork child process
getIpv6() - Description: Fork child process.

Surprise: None of these functions forks a child process. :D

Another awesome feature is the thing that they store up to five different versions of libraries. Everyone with slight modifications but no hint which one you need to use. Sometimes it is the newest, sometimes the oldest which is running in production. Another case of try and error.
Oh and my dev machine is a potato with a power supply and a fan. I started with NetBeans and every time I compiled the code it sounds like the machine wants to lift off and leave for a better place. (At this point I switched to Emacs and everything runs smoothly now)

At first I thought that I'm just not that good at coding and understanding a big project from scratch but some colleagues have the same problem. The whole system is very inflexible and it is all about "std::cout"-debugging to check if your changes do what you want them to do.

Currently I'm just trying to fix this mess to make the life for the next student or employee easier. The first month was just frustrating as hell. I need to ask so many questions and most of the time the answer was "I don't know, haven't touched this code in years". Needless to say that my progress isn't that awesome but at least I get a nice payment for 20 hours of work a week.

So ok here it is, as asked in the comments.

Setting: customer (huge electronics chain) wants a huge migration from custom software to SAP erp, hybris commere for b2b and ... azure cloud
Timeframe: ~10 months….

My colleague and me had the glorious task to make the evaluation result of the B2B approval process (like you can only buy up till € 1000, then someone has to approve) available in the cart view, not just the end of the checkout. Well I though, easy, we have the results, just put them in the cart … hmm :-\
The whole thing is that the the storefront - called accelerator (although it should rather be called decelerator) is a 10-year old (looking) buggy interface, that promises to the customers, that it solves all their problems and just needs some minor customization. Fact is, it’s an abomination, which makes us spend 2 months in every project to „ripp it apart“ and fix/repair/rebuild major functionality (which changes every 6 months because of „updates“.

After a week of reading the scarce (aka non-existing) docs and decompiling and debugging hybris code, we found out (besides dozends of bugs) that this is not going to be easy. The domain model is fucked up - both CartModel and OrderModel extend AbstractOrderModel. Though we only need functionality that is in the AbstractOrderModel, the hybris guys decided (for an unknown reason) to use OrderModel in every single fucking method (about 30 nested calls ….). So what shall we do, we don’t have an order yet, only a cart. Fuck lets fake an order, push it through use the results and dismiss the order … good idea!? BAD IDEA (don’t ask …). So after a week or two we changed our strategy: create duplicate interface for nearly all (spring) services with changed method signatures that override the hybris beans and allow to use CartModels (which is possible, because within the super methods, they actually „cast" it to AbstractOrderModel *facepalm*).
After about 2 months (2 people full time) we have a working „prototype“. It works with the default-sample-accelerator data. Unfortunately the customer wanted to have it’s own dateset in the system (what a shock). Well you guess it … everything collapsed. The way the customer wanted to "have it working“ was just incompatible with the way hybris wants it (yeah yeah SAP, hybris is sooo customizable …). Well we basically had to rewrite everything again.
Just in case your wondering … the requirements were clear in the beginning (stick to the standard! [configuration/functinonality]). Well, then the customer found out that this is shit … and well …

So some months later, next big thing. I was appointed technical sublead (is that a word)/sub pm for the topics‚delivery service‘ (cart, delivery time calculation, u name it) and customerregistration - a reward for my great work with the b2b approval process???

Customer's office: 20+ people, mostly SAP related, a few c# guys, and drumrole .... the main (external) overall superhero ‚im the greates and ur shit‘ architect.
Aberage age 45+, me - the ‚hybris guy’ (he really just called me that all the time), age 32.
He powerpoints his „ tables" and other weird out of this world stuff on the wall, talks and talks. Everyone is in awe (or fear?). Everything he says is just bullshit and I see it in the eyes of the others. Finally the hybris guy interrups him, as he explains the overall architecture (which is just wrong) and points out how it should be (according to my docs which very more up to date. From now on he didn't just "not like" me anymore. (good first day)
I remember the looks of the other guys - they were releaved that someone pointed that out - saved the weeks of useless work ...

Instead of talking the customer's tongue he just spoke gibberish SAP … arg (common in SAP land as I had to learn the hard way).
Outcome of about (useless) 5 meetings later: we are going to blow out data from informatica to sap to azure to datahub to hybris ... hmpf needless to say its fucking super slow.
But who cares, I‘ll get my own rest endpoint that‘ll do all I need.

First try: error 500, 2. try: 20 seconds later, error message in html, content type json, a few days later the c# guy manages to deliver a kinda working still slow service, only the results are wrong, customer blames the hybris team, hmm we r just using their fucking results ...
The sap guys (customer service) just don't seem to be able to activate/configure the OOTB odata service, so I was told)
Several email rounds, meetings later, about 2 months, still no working hybris integration (all my emails with detailed checklists for every participent and deadlines were unanswered/ignored or answered with unrelated stuff). Customer pissed at us (god knows why, I tried, I really did!). So I decide to fly up there to handle it all by myself

In fact I'm a sinful dev, so that I can't easily decide which one is worst. From indenting with tabs, or using nano instead of vim/emacs, to hardcoding database credentials on server, to many hacks and workarounds I use as actual "fixes" when the deadline is upon me and I've tried all I could. But it always led only to my own regret. For instance, my latest sin was that I prefered Debian over Arch and used proprietary graphic drivers to speed up my new setup. But ended up with a curse from St. Ignucius. (check my last rant)

But my worst sin probably goes to when I was "printf-debugging" some issue for a GSM controller on a raspberry pi. I forgot to remove one little print line and deployed the new "fixed" version. I didn't follow that project after that for like a month or so, when the client posted back the device and said that "it just doesn't work anymore". It seemed that raspbian didn't boot beacause the sd card was curroptted. I dd'ed through the card and I noticed that there are billions of lines of "DEBUG:: reading stream from 192.some.shitty.ip", took almost all over the 32G sdcard. Just as I suddenly remembered the cursed line I just added a month ago, I declared the sd card dead with no hesitation, dunce-commented the line (so the history would remember), implemented a time out for the thread containing it, setup a journald unit for my service and removed the redirection of process output to a log file, found a new sd card and installed everything again, and finally posted back the new "fix" to the client.

Moral: Never comfort yourself for the sins you have commited in the past kids, they certainly will come back to you. And also not to do any io especially write to a file on an SD card with ext fs, in a potentially infinite loop with no timeout.

P.S: I'd posted my last rant just before the new week rant last nigh. I really liked the St. Ignucius meme so decided to create a new one. He's very adorable :)

The debugging process is often complicated.

There are often many factors involved in troubleshooting an error, but usually I have a hunch.

Sometimes I'll ignore my hunch and try other, smaller fixes first. This will irritate me for a while until I actually go after my hunch, run the program, and INSTANTLY everything works again.

Found myself relating that to real life today when I had a slight belly ache and eventually decided to take a massive shit and INSTANTLY felt better. The relief was so quick that I actually looked up and paused in surprise.

So lets start here, as i have been preparing myself for a while for that rant. I have been putting it off for a while, but today I had enough.
Fuck react-native and fuck facebook react-native team. Bunch of lazy incompetent twats.
The all amazing framework that suppose to be speed up your development process, since you don't have to compile your code after each change. SO FUCKING WHAT if the god damned framework is so fucking buggy and so fucking shit that you constantly have to fix build, dependancies etc issues. Every day since I work on this project that is using react-native I have to deal with some of the react fucked up behaviour. You got an issue ? don't worry google it just to find out that 100 other people had the same issue. Scroll through down the bottom of the page just to find out that facebook devs have closed the issue as resolved (without fucking fixing it) because there wasnt recent replies to the post. Are you fucking kidding me? It's ok thou, create a new issue just to get an automatic reply from the bot that locks the thread and keeps it locked till you update your React-native version to the newest one. You do that and guess fucking what? Their newest version fucks up remote debugging on iOS(fucking android been broke for over a year) so say good bye to debugging your js code. Documentation is fucking trash. You found a nice function like autoCaptialise on your text input? Great! Ah wait, its not fucking working, what is wrong? You google this just to fucking found out it, function never worked on android, so why the fuck you still have it exposed and still have it in your docs? You want to add package? So fucking ez, just type npm install <name of the package>. Ha! fuck you, you still have to go and add them fucking manually in gradle in android and in pod in xcode, because obviously react-native is a one big fucking bullshit. Oh and a scroll view is a fucking glorious highlight of that framework, try add some styling to it, you gonna have loads of fun. Fuck react-native. And fuck the fucking idiot who convinced my boss that framework is so fucking great and now I have to work on this shit. Sincerely Xamarin Developer.

I love software. Seriously, I love it. /s

Transmission is given a bad torrent (which, given that it's a torrent service, you'd expect it handles quite robustly) and completely fucks up. Like, really badly. It doesn't respond to RPC anymore, systemd has to resort to sending it a SIGKILL to get it off the process tree, and the web interface.. yeah. Nothing.

It doesn't log by default, so fine I'll add that to the systemd unit and restart it with debugging options enabled.

# systemctl daemon-reload && systemctl daemon-reexec

Turns out that /var/log/transmission.log can't be written to by my Transmission user. Well shit. Change that to /home/condor/transmission.log.

# systemctl daemon-reload && systemctl daemon-reexec
# systemctl restart transmission-daemon

*blood starts to reach its boiling point*

Still logs in the wrong fucking location. Systemd, I told you to log over there. I did everything I could to make you steaming pile of shit reload that fucking config. What's the fucking problem!?

*about 15 minutes of fighting systemd*
Finally! It spits out a log in the right location! Thank you Transmission and systemd for finally doing your fucking jobs. So a bad torrent it is, hmm...

*removes torrent from .config/transmission/torrents*
Transmission: *still fucking shits itself on that ostensibly removed torrent*

That's it. BEGONE!!!

Oh and don't get me started on the fact that apparently a service needs some 400MB of memory. Channeling your inner Chrome Transmission?

In today's episode of kidding on SystemD, we have a surprise guest star appearance - Apache Foundation HTTPD server, or as we in the Debian ecosystem call it, the Apache webserver!

So, imagine a situation like this - Its friday afternoon, you have just migrated a bunch of web domains under a new, up to date, system. Everything works just fine, until... You try to generate SSL certificates from Lets Encrypt.

Such a mundane task, done more than a thousand times already... Yet... No matter what you do, nothing works. Apache just returns a HTTP status code 403 - Forbidden.

Of course, what many folk would think of first when it came to a 403 error is - Ooooh, a permission issue somewhere in the directory structure!

So you check it... And re-check it to make sure... And even switch over to the user the webserver runs under, yet... You can access the challenge just fine, what the hell!

So you go deeper... And enable the most verbose level of logging apache is capable of - Trace8. That tells you... Not a whole lot more... Apparently, the webserver was unable to find file specified? But... Its right there, you can see it!

So you go another step deeper and start tracing the process' system calls to see exactly where it calls stat/lstat on the file, and you see that it... Calls lstat and... It... Returns -1? What the hell#2!

So, you compile a custom binary that calls lstat on the first argument given and prints out everything it returns... And... It works fine!

Until now, I chose to omit one important detail that might have given away the issue to the more knowledgeable right away. Our webservers have the URL /.well-known/acme-challenge/, used for ACME challenges, aliased somewhere else on the filesystem - To /tmp/challenges.

See the issue already?

Some *bleep* over at the Debian Package Maintainer group decided that Apache could save very sensitive data into /tmp, so, it would be for the best if they changed something that worked for decades, and enabled a SystemD service unit option "PrivateTmp" for the webserver, by default.

What it does is that, anytime a process started with this option enabled writes to /tmp/*, the call gets hijacked or something, and actually makes the write to a private /tmp/something/tmp/ directory, where something... Appeared as a completely random name, with the "apache2.service" glued at the end.

That was also the only reason why I managed fix this issue - On the umpteenth time of checking the directory structure, I noticed a "systemd-private-foobarbas-apache2.service-cookie42" directory there... That contained nothing but a "tmp" directory with 777 as its permission, owned by the process' user and group.

Overriding that unit file option finally fixed the issue completely.

I have just one question - Why? Why change something that worked for decades? I understand that, in case you save something into /tmp, it may be read by 3rd parties or programs, but I am of the opinion that, if you did that, its only and only your fault if you wrote sensitive data into the temporary directory.

And as far as I am aware, by default, Apache does not actually write anything even remotely sensitive into /tmp, so...

Why. WHY!
I wasted 4 hours of my life debugging this! Only to find out its just another SystemD-enabled "feature" now!

And as much as I love kidding on SystemD, this time, I see it more as a fault of the package maintainers, because... I found no default apache2/httpd service file in the apache repo mirror... So...

>Discovers a new low level profiling tool that could help us at work with stuck process debugging and gets all hyped
>Installs on test machine, tool doesn't work
>Wonders why. Oh. Needs a kernel module to work, compiled and loaded
>"Well, its my test machine... Guess that's no problem..." but... my hype died down a bit. Kernel module installation just for a new tool that aggregates all other commonly used tools? eh... Maybe it will blow me out of my shoes still

>Installs and loads the module
>Tool works. Turns out its just a htop-like tool, with shortcuts to launch specific other profiling tools like strace/ltrace/lsof/netstat/ss etc...
"Oh... That's boring. Maybe it has all those tools built in at least?"
>Tries to run ltrace - tool exits as ltrace is not installed

Lol
>Installs ltrace and launches tool again. Tries to ltrace a process and
>Nothing. Nothing happens. For seconds... Then kicks me off of SSH
WTF?
>Tries to ping machine... silence
Did... our net go down again? (Having issues due to a storm going over our area these few days)
>Pings google and... gets instant reply
More wtf
>Pings the hypervisor the machine was running on
Works like normal

Oh... Oh no. Please tell me it didn't!
>Logs into the hypervisor UI, checks machine state
Running OK
>Opens machine console aaaaand... Yep. Stacktrace as well as a lot of kernel mumbo-jumbo... It took the machine down to kernel panic.

I never went so quick from "We need this tool deployed everywhere" to "Omg I need to get rid of this crap as soon as possible" lol.

And just for those wondering, it was sysdig.

Below is a transcript from work Slack today. Only the names and some code are changed. It ended up causing a bit of drama. DevRanters, what do you take from this?

---

Delivery Lead:
Hey Gang. What's the blocker for FEATURE-123?

Dev1:
FEATURE-122 crashed on iOS app when viewing Feature Introduction page.

Teach Lead:
I've talked about this with Dev1 on a side channel.
And diagnosed the stack trace.
It looks like there is/was some bad handling of a List in the Feature Introduction view logic.
But this is confined to changes that Dev2 is still working on.
(It's not present in master)
Dev2, what's your current position on this?

Dev2:
I have tested at my end with Dev1 but it seems to be working fine

Tech Lead:
There is a race condition related to the use of someList.first()
My guess is that theres a Flow of those lists defined, with an initial value of emptyList
And that on your machine, that Flow is updating with a new value quickly enough that it doesn't matter.
But on Dev1's, for whatever reason, it doesn't get there in time, hits the empty list and falls over.
The logic that's performing the first() needs to gracefully handle empty lists as well.

Dev2:
Where is that logic called?

Tech Lead:
Here's the stack trace Dev1 provided in our conversation earlier:
Caused by: kotlin.NoSuchElementException: List is empty.
...
at 3 iosApp 0x00000000 kfun:kotlin.NoSuchElementException#<init>(kotlin.String?){} + 00
at 4 iosApp 0x0000000 kfun:kotlin.collections#first@kotlin.collections.List<0:0>(){0§<kotlin.Any?>}0:0 + 000
...
at 9 iosApp 0x0000000 kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} + 0000

This line:
kfun:kotlin.collections#first@kotlin.collections.List<0:0>()
...says that it's first() being called on an empty list.

Dev1:
FYI: Dev3/Dev4/myself are seeing the same issue with the same stack-trace above.

Tech Lead:
So Dev2, have you introduced such a call?
Because I checked master branch and there isn't one, in that version of the file.
Ok, I'll check your working branch Dev2
...
Yes you have here:
var processed1 = someList.first()
var processed2 = someList.first()
...
Lines 123, 124.
Solution looks really straightforward guys.

Dev2:
Okay, I will fix that and push the change

Tech Lead:
Check if someList is empty and allow for generating / handling null processedValues in the view.

Now; I'm going to be straight with you here.
This issue has been discussed over several hours today.
I expect that either one of you could have gone through the process I did in the last 10 minutes above, and resolved it in the same way :point_up:

Dev2:
I went on a break and it's not reproducible on my machine

Tech Lead:
I didn't reproduce it on mine either.

Dev1:
Dev2 and myself are now on sharing screen to sort this issue out. Hope to update back later.

Tech Lead:
<Screen shot of diff with changed code>
:point_up: That change should do it.

Dev2:
Already have pushed the change.

Tech Lead:
...just seen it, is good - same approach :ok_hand:
Dev1 please let us know when tested on your machine.

Dev1:
That does it. It fixes the issues. Thank you, Dev2. I will pick it off from here.

Tech Lead:
Glad to hear it guys.

Dev1:
I have to say this that it is not because we are not working on the issue - Dev2 and myself (together with Dev3/Dev4) have been on this issue all this morning. It just difficult to connect the dot when it wasn't reproducable on Dev2's machine. I brought the issue up because I wanted to switch to working on other tickets while waiting for this to resolve. Still thank you largely for Dev2's work and your keen eyes that spot and resolve the issue quickly.

Tech Lead:
Noted Dev1.
I think the take-away has to be to read the stack-trace carefully... don't worry - we've all been guilty of not reading the error in full, at some point.
The stack trace said that the 'first' element is being referenced from an empty list - that's just logically impossible, right?
Looking for that call to first, we saw it wasn't in the code before, and is after (two of them, in fact).
So then we ask ourselves, how can we deal with an empty list - and then solution almost presents itself.
It didn't really take reproduction of the error to resolve.
Maybe working with a new tech stack creates an anxiety that every issue faced will have a complex solution related to that stack; but I think you'll agree, this particular issue really just required a deep breath and your trusty 'debugging skills 101'... don't lose them! :smiling_face:

A CASE AGAINST BLUE PRISM
Let's review one of the worst weeks I had with Blue Prism

Monday: Yay! Solved one of the problems we've been carrying around for a week before.

One of the robots suddenly became slow. Like, REAL slow. A process that would take 3 minutes per record now takes 45, and that broke apart all the following schedule.

There were no updates on the application server, the production machine, the robot, it just became slow. And not always slow; a process manually run from console room would work, a process in debug room would work, it's just the scheduled part that caused problems.

It turned out, BP didn't seem to like that particular combination of schedulation + process + machine. Moving the process to a different machine seemingly fixed that. IDK why.

Tuesday: One of our processes waits for a code to appear in the page, and when that happens, it memorizes this code. However, now it is always returning blank. Worked for months, now it breaks every single time.
After half a day of debugging a bug which DIDN'T HAPPEN IN DEBUG MODE YET AGAIN, at 11pm I decided to just place a nonsensical timeout in page before reading and call it a day.

WEDNESDAY: a scheduled process didn't start. "No sessions created". Thanks Blue Prism, very cool.

THURSTAY: This time, schedulation did start, but the process is "waiting". As in: it's 9:30 am, the process has been stuck in the same step since 6:00 am. Turns out, it blocked during a navigate stage; you need to send a string to clipboard using the standard BP action for that, then paste and click "enter", but for some reason the standard BP object sent "ORRCO" instead of "ORRICO" to clipboard, which obviously returned no results and then... the process just didn't feel like doing things anymore. No errors, no logs, nothing: just sitting on its ass. Because fuck you that's why.

Friday: another process uses a very moderate amount of scripts to work. Nothing really fancy, just a couple of lines of code to place in page some IDs and selector to help BP do its thing, otherwise selecting these elements would be a nightmare.

But

Failed while invoking javascript method:Exception from HRESULT: 0x80020101-> at mshtml.HTMLWindow2Class.IHTMLWindow2_execScript(String code, String language)

The same script -it's not dynamically generated-worked yesterday, the day before and the day after. But sometimes it will not. Why? The answer, my friend, is blowin'' in the wind

SIM 800L
The fucking nail to my coffin. This thing is so unreliable. I fixed on issue get the next one. Then get an error trying to make a http request, with little information on the web. Eventually switch to FTP which is working for a while. Then suddenly nothing is working properly. Even the serial communication has errors. This process took over 6 months. Constant debugging and headscraching involved. After hundreds of hours I give up. I'm going to switch to a Raspberry Pi Zero with an UMTS Stick attached. This is going to cost way more battery time but my project needs to be finished by july and I'm tired of this shitty little module.

I am stuck in a loop where I lose confidence in a project and leave it 75% finished (without debugging) while also being doubtful of my skills every time I code and being so demotivated. I have been learning but its a slow process and I keep being lazy about it. I honestly wish I had less stress externally and was able to focus on software engineering more.

Spent 3 hours yesterday debugging a process in the business that's tied together with my software. Only to find out they actually meant they were printing to the wrong printer 😥🤔

Visual Studio is the worst. Ever. I was about a half hour digging into debugging my code, and I was about 8 layers deep into the API, with breakpoints to anchor me to each level. With no warning, Visual Studio crashed and I lost all my breakpoints, and I didn't know which file I left off in. I had to completely restart the debugging process. Visual Studio deserves to burn...

Bloody fucking Android! Updates, updates and more updates! My development Nexus 5X won't allow me to sideload apps since it updated... Hello, printf debugging! Goodbye, profiler and debugger!

My hate for Android grows with each version after 4.0.$something... 2 was shit, I missed 3, 4 was OK, and since then it's going steeply down.

And don't get me started on Material Design...! Good luck figuring out what's a button and what's a label...

And what's up with the "let's keep all apps running all the time to save a few ms on start" philosophy!? Who thought that is a good idea!? Yeah, System.exit(0) works, but... Is it so hard to determine when it's not needed anymore (has no services running etc.)? Why should a web browser (for example) stay in memory after I quit? Minimize is a thing (Home button), why make it so confusing?

Another thing - feedback-less async tasks - why? I like to know when it is working in the background... How the hell am I supposed to find out if it is supposed to do this or if it is frozen?

And Android deciding to kill your process whenever it pleases without any callback... Happened to me once with an Activity in the foreground (no exceptions anywhere in my app, it just quit). How do you do IO properly? It seems you can't guarantee some file or socket or something that must be closed doesn't stay open (requiring to restart Bluetooth 'cause the socket wasn't closed, for example)...

Today I spent almost a full hour after office hours debugging my code for an issue only to realise that the local process responsible for live reloading my code did not pick up changes after git branch switch.
So in retrospect, if I had left the bug for the day it would have automatically got fixed tomorrow once my laptop restarted.
But no...I just had to figure out the issue on my own today which wasted a complete hour and I won't get it back 🤦🏽‍♂️

[CONCEITED RANT]
I'm frustrated than I'm better tha 99% programmers I ever worked with.

Yes, it might sound so conceited.
I Work mainly with C#/.NET Ecosystem as fullstack dev (so also sql, backend, frontend etc), but I'm also forced to use that abhorrent horror that is js and angular.
I write readable code, I write easy code that works and rarely, RARELY causes any problem, The only fancy stuff I do is using new language features that come up with new C# versions, that in latest version were mostly syntactic sugar to make code shorter/more readable/easier.

People I have ever worked with (lot of) mostly try to overdo, overengineer, overcomplicate code, subdivide into methods when not needed fragmenting code and putting tons of variables.

People only needed me to explain my code when the codebase was huge (200K+ lines mostly written by me) of big so they don't have to spend hours to understand what's going on, or, if the customer requested a new technology to explain such new technology so they don't have to study it (which is perfectly understandable). (for example it happened that I was forced to use Devexpress package because they wanted to port a huge application from .NET 4.5 to .NET 8 and rewriting the whole devexpress logic had a HUGE impact on costs so I explained thoroughly and supported during developement because they didn't knew devexpress).

I don't write genius code or clevel tricks and patterns. My code works, doesn't create memory leaks or slowness and mostly works when doing unit tests at first run. Of course I also put bugs and everything, but that's part of the process.

THe point is that other people makes unreadable code, and when they pass code around you hear rising chaos, people cursing "WTF this even means, why he put that here, what the heck this is even supposed to do", you got the drill. And this happens when I read everyone code too.

But it doesn't happens the opposite. My code is often readable because I do code triple backflips only on personal projects because I don't have to explain anyone and I can learn new things and new coding styles.

Instead, people want to impress at work, and this results in unintelligible, chaotic code, full of bugs and that people can't read. They want to mix in the coolest technologies because they feel their virtual penis growing to showoff that they are latest bleeding edge technology experts and all.

They want to experiment on business code at the expense of all the other poor devils who will have to manage it.

Heck, I even worked with a few Microsoft MVPs.
Those are deadly. They're superfast code throughput people that combine lot of stuff.
THen they leave at you the problems once they leave.

This MVP guy on a big project for paperworks digital acquisiton for a big company did this huge project I got called to work in, which consited in a backend and a frontend web portal, and pushed at all costs to put in the middle another CDN web project and another Identity Server project to both do Caching with the cdn "to make it faster" and identity server for SSO (Single sign on).

We had to deal with gruesome work to deal with browser poor caching management and when he left, the SSO server started to loop after authentication at random intervals and I had to solve that stuff he put in with days of debugging that nasty stuff he did.

People definitely can't code, except me.
They have this "first of the class syndrome" which goes to the extent that their skill allows them to and try to do code backflips when they can't even do code pushups, to put them in a physical exercise parallelism.

And most people is like this. They will deny and won't admit, they believe they're good at it, but in reality they aren't.

There is some genius out there that does revoluitionary code and maybe needs to do horrible code to do amazing stuff, and that's ok. And there is also few people like me, with which you can work and produce great stuff.

I found one colleague like this and we had a $800.000 (yes, 800k) project in .NET Technology, which consisted in the renewal of 56 webservices and 3 web portals and 2 Winforms applications for our country main railway transport system. We worked in 2 on it, with a PM from the railway company.

It was estimated 14 months of work and we took 11 and all was working wonders. We had ton of fun doing it because also their PM was a cool guy and we did an awesome project and codebase was a jewel. The difficult thing you couldn't grasp if you read the code is if you don't know how railway systems work and that's the only difficult thing.

Sight, there people is macking me sick of this job

For a little background on the sort of stuff I'm dealing with, check out my last rant.

Anyways, I'm testing this pipeline at work and was just reminded of the fucktarded way a "software engineer", who had a bachelor's biology degree, decided to handle a json file.

The script is question is loading a json file containing an array of objects. The script is written in perl. There's a JSON module. Use that? Fuck no! Let's rather perform an in-place sed command on the file substituting the commas separating objects in the array with newlines, then proceed to read the file line-by-line and parse out the tokens manually. Mind you, in the process of adding the newlines he didn't keep the commas, so now all of these json files his bullshit handled are invalid json that cannot be parsed.

The dumb ass was lucky the data in the file is always output upstream as a single line and the tokens for each object are always in the same order, so that never led to problems. But now, months later after I fixed his stupidity I am being reminded of it again as I'm testing and debugging some old projects as part of regression testing new changes I'm making.

TL;DR Fuck dumbwit motherfuckers who can't even google search "parsing a json file" and doing literally anything that is less fucktarded than manually parsing a json file