WHY THE FUCKIDY FUCK DO PEOPLE THINK THAT JUST BECAUSE YOU ARE A DEVELOPER THEN YOU CAN DO EVERY FUCKING THING IN A SNAP?!

THIS ENTIRE SUMMER I WORKED FOR THIS MIDGET OF A FUCK AND THE IDEA WAS TO MAINLY DEVELOP AN IOS APP FOR THEIR PRODUCT. THAT ONE APP TRANSLATED TO THREE SEPARATE APPS AND KEEP IN MIND THAT I AM THE ONLY DEVELOPER THEY HAVE SO I HAVE TO DEVELOP IT, TEST IT AND DOCUMENT IT!! AND WHAT'S EVEN WORSE IS HE KEEPS GIVING ME DESIGNS AT THE LAST MINUTE SO I ALWAYS END UP HAVING TO CREATE NEW PROJECTS ALL THE TIME. WHEN A WEEK PASSES BY WITHOUT ANY UPDATE HE AND HIS FUCKED UP BRAIN CELLS GATHER A MEETING WITH HIS 'DONALD TRUMP HANDS-SIZED' BALLS TO ASK ME, 'WHY ARE YOU TAKING SO LONG? THESE ARE JUST THREE BUTTONS ON THIS VIEW?' MOTHERFUCKING COCKSUCKER!! GO GET YOUR MOM TO DO THIS WORK THEN IF IT'S JUST THREE BUTTONS. TO HIM EVERYTHING IN IOS WHICH INCORPORATES A TAP IS A FUCKING BUTTON! BUTTON THIS! BUTTON THAT! AND IT'S NOT LIKE HE HAS SIMPLE DESIGNS..NO.NO.NO.NO. THIS ASSHOLE-SHAPED-HEAD MUGGET DESIGNS SHIT WHICH REQUIRES ME TO HAVE TO DRAW A UIVIEW AS THE SHAPE OF A HUMAN BODY AND HEART. THEN ASIDE FROM THAT, JUST BECAUSE MY RESUME SAYS I MINORED IN MATH AND APPLIED MATH, HE SENDS ME A PAPER THAT EXPLAINS MATHEMATICAL CONCEPTS OF DATA ANALYSIS THEN WANTS TO MEET WITH ME TO DISCUSS THE SUGGESTED METHODS THEN IMPLEMENT THEM. AT THIS POINT I HAD ALREADY IMPLEMENTED AN ALGO FOR COUNTING THE NUMBER OF STEPS THAT WAS WAY WAY WAY MORE ACCURATE THAN THE SHIT THEY HAD IN THEIR CONTROLLER..

SO AS I AM ABOUT TO FINISH IMPLEMENTING JUST THE INITIAL 5 VIEWS OF THIS 'FINAL' APP, HE SERIOUSLY WALKS IN AND TELLS ME, SO I'M STARTING TO WORRY THAT WE'LL NOT MEET THE AUGUST DEADLINE SO I'M THINKING MAYBE YOU SHOULD START SWITCHING BETWEEN DEVELOPMENT. WORK ON IOS FOR 4 HOURS THEN SWITCH TO ANDROID FOR THE REMAINDER OF THE TIME. OH AND UHH IF YOU FEEL LIKE TAKING THE IMAC WITH YOU TO WORK FROM HOME, THAT'S FINE AS WELL AND I'LL BE AVAILABLE IN THE WEEKENDS AS WELL. IT WAS AT THIS MOMENT THAT I REALLY REALLY HOPED I WERE GAY! LIKE 'NO LUBE' STYLE KINDA GAY!! OH BTW AT SOME POINT HE HAD ME PROGRAM ONE OF THEIR CONTROLLERS, AND WAS ASKING IF I COULD START RESEARCHING MEANS OF WRITING AN SDK FOR THEM.

KEEP IN MIND THAT I'M AN INTERN WITH ONE YEAR IOS DEV EXPERIENCE.. THEN HE WANTS ME TO ENTIRELY START LEARNING ANDROID AND GIVE HIM TWO VERSIONS OF THIS THIRD APP IN TWO WEEKS.

HE CAN GO SUCK HIS OWN DICK WHILE GETTING FINGER FUCKED BY A FORK WITH A BLUE WHALE'S 6FT LONG DICK UP HIS MOUTH.

*** that felt good ****

Each month my department compiles a 4M row 150 column data table for compliance with a federal agency. Before submitting, we check it against about 400 rules.

The existing system was simply 400 queries that ran in sequence, table-scanning 4M rows each time, taking upwards of 6 hours, which is a huge bottleneck, especially if you have to make changes and rerun. Plus the output was rather one-dimensional.

I built a proper normalized database and created a sort of rules engine, running all 400 rules in one table scan. Not only does it complete in 30 minutes, but the reports generate automatically, and the results can be filtered on several dimensions to aid with root-cause analysis.

Management was pleased.

No it's not AI. YOU ARE RUNNING FUCKING SQL QUERIES AND CALLING IT AI!

No it's not AI. YOU ARE RUNNING SIMPLE DATA ANALYSIS MACROS AND FUNCTIONS IN EXCEL!

Stop labelling everything as AI, you attention and investment seeking morons! @&£$¢×xo##!

Husband wanted to not go out after dinner to "go cuddle in bed." This apparently is code for him browsing mathmatica stack exchange and reading books on data analysis with R... 😂

Smart India Hackathon: Horrible experience
Background:- Our task was to do load forecasting for a given area. Hourly energy consumption data for past 5 years was given to us.

One government official asks the following questions:-
1. Why are you using deep learning for the project? Why are you not doing data analysis?
2. Which neural network "algorithm" you are using? He wanted to ask which model we are using, but he didn't have a single clue about Neural Networks.

3. Why are you using libraries? Why not your own code?

Here comes the biggest one,

4. Why haven't you developed your own "algorithm" (again, he meant model)? All you have done is used sone library. Where is "novelty" in your project?

I just want to say that if you don't know anything about ML/AI, then don't comment anything about it. And worst thing was, he was not ready to accept the fact that for capturing temporal dependencies where underlying probability distribution ia unknown, deep learning performs much better than traditional data analysis techniques.
After hearing his first question, second one was not a surprise for us. We were expecting something like that. For a few moments, we were speechless. Then one of us started by showing neural network architecture. But after some time, he rudely repeated the same question, "where is the algorithm". We told him every fucking thing used in the project, ranging from RMSprop optimizer to Backpropagation through time algorithm to mean squared loss error function.

Then very calmly, he asked third question, why are you using libraries? That moron wanted us to write a whole fucking optimized library. We were speechless at this question. Finally, one of us told him the "obvious" answer. We were completely demotivated. But it didnt end here. The real question was waiting. At the end, after listening to all of us, he dropped the final bomb, WHY HAVE YOU USED A NEURAL NETWORK "ALGORITHM" WHICH HAS ALREADY BEEN IMPLEMENTED? WHY DIDN'T YOU MAKE YOU OWN "ALGORITHM"? We again stated the obvious answer that it takes atleast an year or two of continuous hardwork to develop a state of art algorithm, that too when gou build it on top of some existing "algorithm". After listening to this, he left. His final response was "Try to make a new "algorithm"".

Needless to say, we were completely demotivated after this evaluation. We all had worked too hard for this. And we had ability to explain each and every part of the project intuitively and mathematically, but he was not even ready to listen.
Now, all of us are sitting aimlessly, waiting for Hackathon to end.😢😢😢😢😢

Perhaps not "best", but certainly most amusing, so what the heck!

Years ago as an intern, I applied to a large pharmaceutical company. On part of the application form, you had to enter the code of the department you were applying to.

What I *should have* put down was "IT", which is the department that houses all their devs. However, I didn't actually read any of what the codes meant, assumed that was the department for helping people with how to mail merge, and put down "COMPSCI" instead. This was computational sciences - loosely summarised as computational data analysis on various druggable molecules.

I do *not* have any sort of biology or chemistry background, so the interview was rather... interesting, and I muddled through on the basis of getting some more interview practice assuming it was a no go.

To my amazement, got a phone call saying that they'd been thinking they wanted someone more technical on the team, and despite my lack of scientific experience they thought I'd be a good fit. I was unsure as to whether I should accept for a while, but then decided to just go for it - and had a fantastic internship there, working on a great variety of stuff, and learning tons all under a supervisor who I'm still in touch with to this day.

tl;dr - Applied for the wrong job. Coincidentally got it anyway, and miraculously had a fantastic year working there.

One comment from @Fast-Nop made me remember something I had promised myself not to. Specifically the USB thing.
So there I was, Lieutenant Jr at a warship (not the one my previous rants refer to), my main duties as navigation officer, and secondary (and unofficial) tech support and all-around "computer guy".
Those of you who don't know what horrors this demonic brand pertains to, I envy you. But I digress. In the ship, we had Ethernet cabling and switches, but no DHCP, no server, not a thing. My proposition was shot down by the CO within 2 minutes. Yet, we had a curious "network". As my fellow... colleagues had invented, we had something akin to token ring, but instead of tokens, we had low-rank personnel running around with USB sticks, and as for "rings", well, anyone could snatch up a USB-carrier and load his data and instructions to the "token". What on earth could go wrong with that system?
What indeed.
We got 1 USB infected with a malware from a nearby ship - I still don't know how. Said malware did the following observable actions(yes, I did some malware analysis - As I said before, I am not paid enough):
- Move the contents on any writeable media to a folder with empty (or space) name on that medium. Windows didn't show that folder, so it became "invisible" - linux/mac showed it just fine
- It created a shortcut on the root folder of said medium, right to the malware. Executing the shortcut executed the malware and opened a new window with the "hidden" folder.
Childishly simple, right? If only you knew. If only you knew the horrors, the loss of faith in humanity (which is really bad when you have access to munitions, explosives and heavy weaponry).
People executed the malware ON PURPOSE. Some actually DISABLED their AV to "access their files". I ran amok for an entire WEEK to try to keep this contained. But... I underestimated the USB-token-ring-whatever protocol's speed and the strength of a user's stupidity. PCs that I cleaned got infected AGAIN within HOURS.
I had to address the CO to order total shutdown, USB and PC turnover to me. I spent the most fun weekend cleaning 20-30 PCs and 9 USBs. What fun!
What fun, morons. Now I'll have nightmares of those days again.

I’m so mad I’m fighting back anger tears. This is a long rant and I apologize but I’m so freaking mad.

So a few weeks ago I was asked by my lead staff person to do a data analysis project for the director of our dept. It was a pretty big project, spanning thousands of users. I was excited because I love this sort of thing and I really don’t have anything else to do. Well I don’t have access to the dataset, so I had to get it from my lead and he said he’d do it when he had a chance. Three days later he hadn’t given it to me yet. I approach him and he follows me to my desk, gives me his login and password to login to the secure freaking database, then has me clone it and put it on my computer.

So, I start working on it. It took me about six hours to clean the database, 2 to set up the parameters and plan of attack, and two or three to visualize the data. I realized about halfway through that my lead wasn’t sure about the parameters of the analysis, and I mentioned to him that the director had asked for more information than what he was having me do. He tells me he will speak with director.

So, our director is never there, so I give my lead about a week to speak with her, in the mean time I finish the project to the specifications that the director gave. I even included notes about information that I would need to make more accurate predictions, to draw conclusions, etc. It was really well documented.

Finally, exasperated, and with the project finished but just sitting on my computer for a week, I approached my director on a Saturday when I was working overtime. She confirmed that I needed to what she said in the project specs (duh), and also mentioned she needed a bigger data set than what I was working with if we had one. She told me to speak to my lead on Monday about this, but said that my work looked great.

Monday came and my lead wasn’t there so I spoke with my supervisor and she said that what I was using was the entire dataset, and that my work looked great and I could just send it off. So, at this point 2/3 of my bosses have seen the project, reviewed it, told me it was great, and confirmed that I was doing the right thing.

I sent it off to the director to disseminate to the appropriate people. Again, she looked at it and said it was great.

A week later (today) one of the people that the project was sent to approaches me and tells me that i did a great job and thank you so much for blah blah blah. She then asks me if the dataset I used included blahblah, and I said no, that I used what was given to me but that I’d be happy to go in and fix it if given the necessary data.

She tells me, “yeah the director was under the impression that these numbers were all about blahblah, so I think there was some kind of misunderstanding.” And then implied that I would not be the one fixing the mistake.

I’m being taken off of the project for two reasons: 1. it took to long to get the project out in the first place,
2. It didn’t even answer the questions that they needed answered.

I fucking told them in the notes and ALL THROUGH THE VISUALIZATIONS that I needed additional data to compare these things I’m so fucking mad. I’m so mad.

UPDATE: devRant Trans-Oceanic Journey Community Project

It was a mere 12 days ago that I asked the question; 'Could devRanters, as a community, build a 21st Century Technology-Laden ‘devRant devie-Stressball-in-a-Bottle’ and send it on a journey across the Atlantic ocean?

I am thrilled to report that devRanters enthusiastically accepted this difficult challenge. A core team quickly formed and a tremendous amount of research and progress has been made in a short period of time. I want to give you a high level-flavor of what we are doing. Please keep in mind we still need your help. We welcome all develops to take part in this journey.

I want to give appreciation to the devRant Founders @dfox and @trogus. Without your support and sponsorship this project would not have been possible. devRant brought us together and it a reality. Devie journeying across the Ocean the Columbus sailed will stir the imagination of children and adults worldwide when we launch on May 1, 2017.

Some of the research and action items in progress:

- Slack and trello environments were created to capture research and foster discussion.

- A Stony Brook University Oceanography Professor suggested the Gulf Stream would be a good pathway across the ocean. We researched it very and agree. The Gulf Stream has been a trans-Atlantic conduit for hundreds of years. We are deciding whether to launch from Cape Hatteras, NC or the Virginia coast. Both have easy access to the rapid currents in the Gulf Stream.

- We are researching every detail of the Gulf Stream to make the journey easier and faster for devie. We have maps and a team member gathered valuable ideas reading a thorough book – ‘The Gulf Stream’.

- We decided on using a highly resilient plastic rather than glass for the bottle material. Plastic is much lighter, faster and glass breaks down more easily. The lightweight enclosure will allow us to take full advantage of waves and ample trade winds. We are still discussing the final design as we want to minimize friction and mimic the non-locomotion fish that migrate thousands of miles riding the Gulf Stream.

-The enclosure might be 3D printed unless we can locate a commercial solution. We have 3D specs and are speaking with some experts. There are advantages and dis-advantages to each solution.

- We will be using Iridiums' RockBLOCK two-way satellite technology to bounce lat-long coordinate pings off their 36 low-orbit satellites. The data will be analyzed by our devRant devie analysis software. IOS and Android public apps being built by the team will display devie's location throughout the journey in.

- Arduino will be used as the brains

- Multiple sensors including temperature and depth are being considered

-A project plan will be published to the team Friday 12/9. Sorry I am a few days late but adding some new ideas.

There are still a lot of challenges we must overcome and we will.

That’s all for now. I will send updates and all ideas / comments are valued.

Two years ago I moved to Dublin with my wife (we met on tour while we were both working in music) as visa laws in the UK didn’t allow me to support the visa of a Russian national on a freelance artists salary.

After we came to Dublin I was playing a lot to pay rent (major rental crisis here), I play(ed) Double Bass which is a physically intensive instrument and through overworking caused a long term injury to my forearm which prevents me playing.

Luckily my wife was able to start working in Community Operations for the big tech companies here (not an amazing job and I want her to be able to stop).

Anyway, I was a bit stuck with what step to take next as my entire career had been driven by the passion to master an art that I was very committed to. It gave me joy and meaning.

I was working as hard as I could with a clear vision but no clear path available to get there, then by chance the opportunity came to study a Higher Diploma qualification in Data Science/Analysis (I have some experience handling music licensing for tech startups and an MA with components in music analysis, which I spun into a narrative). Seemed like a ‘smart’ thing to do to do pick up a ‘respectable’ qualification, if I can’t play any more.

The programme had a strong programming element and I really enjoyed that part. The heavy statistics/algebra element was difficult but as my Python programming improved, I was able to write and utilise codebase to streamline the work, and I started to pull ahead of the class. I put in more and more time to programming and studied personally far beyond the requirements of the programme (scored some of the highest academic grades I’ve ever achieved). I picked up a confident level of Bash, SQL, Cypher (Neo4j), proficiency with libraries like pandas, scikit-learn as well as R things like ggplot. I’m almost at the end of the course now and I’m currently lecturing evening classes at the university as a paid professional, teaching Graph Database theory and implementation of Neo4j using Python. I’m co-writing a thesis on Machine Learning in The Creative Process (with faculty members) to be published by the institute. My confidence in programming grew and grew and with that platform to lift me, I pulled away from the class further and further.

I felt lost for a while, but I’ve found my new passion. I feel the drive to master the craft, the desire to create, to refine and to explore.

I’m going to write a Thesis with a strong focus on programmatic implementation and then try and take a programming related position and build from there. I’m excited to become a professional in this field. It might take time and not be easy, but I’ve already mastered one craft in life to the highest levels of expertise (and tutored it for almost 10 years). I’m 30 now and no expert (yet), but am well beyond beginner. I know how to learn and self study effectively.

The future is exciting and I’ve discovered my new art! (I’m also performing live these days with ‘TidalCycles’! (Haskell pattern syntax for music performance).

Hey all! I’m new on devRant!

Unintentionally Hilarious joke at work yesterday.

We were doing some data analysis, and I had to dump some stuff into a table for my colleague. So I ran the script and went to the bathroom (no.2).

When I came back, they asked me if the dump is done. And I said without thinking: "I just went." 😂

Using ml for most data analysis jobs is like delivering a single letter in a Boeing-747

FUCK THE RECRUITERS WHO ASK US TO MAKE AN ENTIRE PROJECT AS A CODE TEST.

Oh you need to scrape this website and then store the data in some DB. Apply sentimental analysis on the data set. On the UI, the user should be able to search the fields that were scraped from the website. Upon clicking it should consume a REST API which you have to create as well. Oh and also deploy it somewhere... Oh I almost forgot, make the UI look good. If you could submit it in one week, we will move towards further rounds if we find you fit enough.

YOU KNOW WHAT, FUCK YOU!

I can apply to 10 others companies in one week and get hired in half the effort than making this whole project for you which you are going to use it on your website YOU SADIST MOTHERFUCK

I CURSE YOUR COMPANY WITH THE ETERNITY OF JS CALLBACK HELL 😡😤😣

When scrolling through free Udemy courses for fun, my gf says:
"Whaaaat? Data analysis with Panda and Python? Isn't that shit dangerous? Like they go in the jungle and stuff?"
Me:"oh..."

(Disclaimer: She has no background in CS)

Biggest challenge I overcame as dev? One of many.

Avoiding a life sentence when the 'powers that be' targeted one of my libraries for the root cause of system performance issues and I didn't correct that accusation with a flame thrower.

What the accusation? What I named the library. Yep. The *name* was causing every single problem in the system.

Panorama (very, very expensive APM system at the time) identified my library in it's analysis, the calls to/from SQLServer was the bottleneck

We had one of Panorama's engineers on-site and he asked what (not the actual name) MyLibrary was and (I'll preface I did not know or involved in any of the so-called 'research') a crack team of developers+managers researched the system thoroughly and found MyLibrary was used in just about every project. I wrote the .Net 1.1 MyLibrary as a mini-ORM to simplify the execution of database code (stored procs, etc) and gracefully handle+log database exceptions (auto-logged details such as the target db, stored procedure name, parameter values, etc, everything you'd need to troubleshoot database errors). This was before Dapper and the other fancy tools used by kids these days.

By the time the news got to me, there was a team cobbled together who's only focus was to remove any/every trace of MyLibrary from the code base. Using Waterfall, they calculated it would take at least a year to remove+replace MyLibrary with the equivalent ADO.Net plumbing.

In a department wide meeting:

DeptMgr: "This day forward, no one is to use MyLibrary to access the database! It's slow, unprofessionally named, and the root cause of all the database issues."
Me: "What about MyLibrary is slow? It's excecuting standard the ADO.Net code. Only extra bit of code is the exception handling to capture the details when the exception is logged."
DeptMgr: "We've spent the last 6 weeks with the Panorama engineer and he's identified MyLibrary as the cause. Company has spent over $100,000 on this software and we have to make fact based decisions. Look at this slide ... "
<DeptMgr shows a histogram of the stacktrace, showing MyLibrary as the slowest>
Me: "You do realize that the execution time is the database call itself, not the code. In that example, the invoice call, it's the stored procedure that taking 5 seconds, not MyLibrary."
<at this point, DeptMgr is getting red-face mad>
AreaMgr: "Yes...yes...but if we stopped using MyLibrary, removing the unnecessary layers, will make the code run faster."
<typical headknodd-ers knod their heads in agreement>
Dev01: "The loading of MyLibrary takes CPU cycles away from code that supports our customers. Every CPU cycle counts."
<headknod-ding continues>
Me: "I'm really confused. Maybe I'm looking at the data wrong. On the slide where you highlighted all the bottlenecks, the histogram shows the latency is the database, I mean...it's right there, in red. Am I looking at it wrong?"
<this was meeting with 20+ other devs, mgrs, a VP, the Panorama engineer>
DeptMgr: "Yes you are! I know MyLibrary is your baby. You need to check your ego at the door and face the facts. Your MyLibrary is a failed experiment and needs to be exterminated from this system!"

Fast forward 9 months, maybe 50% of the projects updated, come across the documentation left from the Panorama. Even after the removal of MyLibrary, there was zero increases in performance. The engineer recommended DBAs start optimizing their indexes and other N+1 problems discovered. I decide to ask the developer who lead the re-write.

Me: "I see that removing MyLibrary did nothing to improve performance."
Dev: "Yes, DeptMgr was pissed. He was ready to throw the Panorama engineer out a window when he said the problems were in the database all along. Didn't you say that?"
Me: "Um, so is this re-write project dead?"
Dev: "No. Removing MyLibrary introduced all kinds of bugs. All the boilerplate ADO.Net code caused a lot of unhandled exceptions, then we had to go back and write exception handling code."
Me: "What a failure. What dipshit would think writing more code leads to less bugs?"
Dev: "I know, I know. We're so far behind schedule. We had to come up with something. I ended up writing a library to make replacing MyLibrary easier. I called it KnightRider. Like the TV show. Everyone is excited to speed up their code with KnightRider. Same method names, same exception handling. All we have to do is replace MyLibrary with KnightRider and we're done."
Me: "Won't the bottlenecks then point to KnightRider?"
Dev: "Meh, not my problem. Panorama meets primarily with the DBAs and the networking team now. I doubt we ever use Panorama to look at our C# code."

Needless to say, I was (still) pissed that they had used MyLibrary as dirty word and a scapegoat for months when they *knew* where the problems were. Pissed enough for a flamethrower? Maybe.

Currently, a classmate and I are working on our technical thesis.
It is all about industry 4.0, IIoT, big data and stuff.
This week, we presented our interim results to our supervisor. He is very pleased with our work and made the following suggestion:
He thinks it would be awesome to publish our work on our own GitHub repository and make it open source because he is convinced that this thesis is able to kind of "set a new standard" in some specific fields of using big data analysis in production processes.

I guess I'm kind of proud :)

Man, most memorable has to be the lead devops engineer from the first startup I worked at. My immediate team/friends called him Mr. DW - DW being short for Done and Working.

You see, Mr. DW was a brilliant devops engineer. He came up with excellent solutions to a lot of release, deployment, and data storage problems faced at the company (small genetics firm that ships servers with our analysis software on them). I am still very impressed by some of the solutions he came up with, and wish I had more time to study and learn about them before I left that company.

BUT - despite his brilliance, Mr. DW ALWAYS shipped broken stuff. For some reason this guy thinks that only testing a single happiest of happy path scenarios for whatever he is developing constitutes "everything will work as expected!" As soon as he said it was "done", but golly for him was it "done". By fucking God was that never the truth.

So, let me provide a basic example of how things would go:
my team: "Hey DW, we have a problem with X, can you fix this?"
DW: "Oh, sure. I bet it's a problem with <insert long explanations we don't care about we just want it fixed>"
my team: "....uhh, cool! Looking forward to the fix!"
... however long later...
DW: "OK, it's done. Here you go!"
my team: "Thanks! We'll get the fix into the processing pipelines"
... another short time later...
my team: "DW, this thing is broken. Look at all these failures"
DW: "How can that be? It was done! I tested it and it worked!"
my team: "Well, the failures say otherwise. How did you test?"
DW: "I just did <insert super basic thing>"
my team: "...... you know that's, like, not how things actually work for this part of the pipeline. right?"
DW: "..... But I thought it was XYZ?"
my team: "uhhhh, no, not even close. Can you please fix and let us know when it's done and working?"
DW: "... I'll fix it..."

And rinse and repeat the "it's done.. oh wait, it's broken" a good half dozen times on average. But, anyways, the birth of Mr. Done and Working - very often stuff was done, but rarely did it ever work!

I'm still friends with my team mates, and whenever we're talking and someone says something is done, we just have to ask if it's done AND working. We always get a laugh, sadly at the excuse of Mr. DW, but he dug his own hole in this regard.

Little cherry on top: So, the above happened with one of my friends. Mr. DW created installation media for one of our servers that was deployed in China. He tested it and "it was done!" Well, my friend flies out to China for on-site installation. He plugs the install medium in and goes for the install and it crashes and burns in a fire. Thankfully my friend knew the system well enough to be able to get everything installed and configured correctly minus the broken install media, but definitely the most insane example of "it's done!" but sure as he'll "it doesn't work!" we had from Mr. DW.

Inmates are trying to take over the asylum again.

Got a message from the web team manager deeply concerned because since switching to the new logging framework, the site is significantly slower.

She provided no proof or any data to what 'significantly slower' means.

#1 The 'new logging' has been in place and logging for 5 years. We only recently depreciated the ILogger interface ('new' ILogger interface only has 1 method instead of 5)
#2 The 'old logging' was modified 5 years ago, so even if you were using the 'old' interface, the underlying implementation is still the same.

She tried to push the 'it wasn't this slow before' argument, so I decided to do some fact based analysis.
Knowing they deployed their logging changes couple of weeks ago, I opened up AppDynamics, looked at the average call time to Splunk (along with a few other http calls they are doing)
- caching services - 5ms
- splunk - 30ms
- Order Service - 350ms
- Product Data Service -525ms

Then I look at the data they are logging, for the month of June, over 5 million messages. At 30ms each, that's almost 42 hours spent logging errors...yes errors. Null reference exceptions, Argument exceptions, easily fixable stuff.

So far for the month of July (using the 'new' logging), almost 2.5 million errors. Pretty close so far with June's numbers.

My only suggestion was to fix the bugs in their code so they don't log so many errors.

Her response.."Can we have one of our developers review your logging code? We believe we can find ways to optimize the http requests"

Oh good Lord. I'm not a drinking man..but ...I might start.

CTO: Research, problem analysis, customer need validations, and data based prioritisation is stupid.

Me: So, then why should we solve this problem?

CTO: Because my team invests a lot of time in here (read "because we build a shitty system in past without thinking and we are doing it again").

Me: I don't see this as a good idea.

CTO: I become emotional when I request product to align and they don't. We must solve this problem and not what customers want.

Me: I am not participating here.

CTO: And I want you to work on weekends to support my team.

Me: *disconnects*

Best code performance incr. I made?

Many, many years ago our scaling strategy was to throw hardware at performance problems. Hardware consisted of dedicated web server and backing SQL server box, so each site instance had two servers (and data replication processes in place)

Two servers turned into 4, 4 to 8, 8 to around 16 (don't remember exactly what we ended up with). With Window's server and SQL Server licenses getting into the hundreds of thousands of dollars, the 'powers-that-be' were becoming very concerned with our IT budget. With our IT-VP and other web mgrs being hardware-centric, they simply shrugged and told the company that's just the way it is.

Taking it upon myself, started looking into utilizing web services, caching data (Microsoft's Velocity at the time), and a service that returned product data, the bottleneck for most of the performance issues. Description, price, simple stuff. Testing the scaling with our dev environment, single web server and single backing sql server, the service was able to handle 10x the traffic with much better performance.

Since the majority of the IT mgmt were hardware centric, they blew off the results saying my tests were contrived and my solution wouldn't work in 'the real world'. Not 100% wrong, I had no idea what would happen when real traffic would hit the site.

With our other hardware guys concerned the web hardware budget was tearing into everything else, they helped convince the 'powers-that-be' to give my idea a shot.

Fast forward a couple of months (lots of web code changes), early one morning we started slowly turning on the new framework (3 load balanced web service servers, 3 web servers, one sql server). 5 minutes...no issues, 10 minutes...no issues,an hour...everything is looking great. Then (A is a network admin)...
A: "Umm...guys...hardly any of the other web servers are being hit. The new servers are handling almost 100% of the traffic."
VP: "That can't be right. Something must be wrong with the load balancers. Rollback!"
A:"No, everything is fine. Load balancer is working and the performance spikes are coming from the old servers, not the new ones. Wow!, this is awesome!"
<Web manager 'Stacey'>
Stacey: "We probably still need to rollback. We'll need to do a full analysis to why the performance improved and apply it the current hardware setup."
A: "Page load times are now under 100 milliseconds from almost 3 seconds. Lets not rollback and see what happens."
Stacey:"I don't know, customers aren't used to such fast load times. They'll think something is wrong and go to a competitor. Rollback."
VP: "Agreed. We don't why this so fast. We'll need to replicate what is going on to the current architecture. Good try guys."
<later that day>
VP: "We've received hundreds of emails complementing us on the web site performance this morning and upset that the site suddenly slowed down again. CEO got wind of these emails and instructed us to move forward with the new framework."

After full implementation, we were able to scale back to only a few web servers and a single sql server, saving an initial $300,000 and a potential future savings of over $500,000. Budget analysis considering other factors, over the next 7 years, this would save the company over a million dollars.

At the semi-annual company wide meeting, our VP made a speech.

VP: "I'd like to thank everyone for this hard fought journey to get our web site up to industry standards for the benefit of our customers and stakeholders. Most of all, I'd like to thank Stacey for all her effort in designing and implementation of the scaling solution. Great job Stacy!"
<hands her a blank white envelope, hmmm...wonder what was in it?>

A few devs who sat in front of me turn around, network guys to the right, all look at me with puzzled looks with one mouth-ing "WTF?"

So according to Pfizer, ~39,000 people received a second dose of vaccine/placebo. So about 20,000 got the placebo, and out of that group only 85 got infected in the span of 3.5 months, during the second wave. That's 0.425%.
So they either:
* Proved Covid is not infectious
* Manipulated the shit out of their data
How that fuck is this considered valid analysis?!
If I tried to publish a paper based on such unreliable, statistically meaningless data, I would be rejected without a blink.

Spec'ing out a data analysis tool for several billion transactions and I got this when sending the final spec to the client.

Client: "I don't see why it takes so long, I downloaded this template from theme forest and edited it in dreamweaver, it only took 3 hours"

If anyone has been keeping up with my data warehouse from hell stories, we're reaching the climax. Today I reached my breaking point and wrote a strongly worder email about the situation. I detailed 3 separate cases of violated referential integrity (this warehouse has no constraints) and a field pulling from THE WRONG FLIPPING TABLE. Each instance was detailed with the lying ER diagram, highlighted the violating key pairs, the dangers they posed, and how to fix it. Note that this is a financial document; a financial document with nondeterministic behavior because the previous contractors' laziness. I feel like the flipping harbinger of doom with a cardboard sign saying "the end is near" and keep having to self-validate that if I was to change anything about this code, **financial numbers would change**, names would swap, description codes would change, and because they're edge cases in a giant dataset, they'll be hard to find. My email included SQL queries returning values where integrity is violated 15+ times. There's legacy data just shoved in ignoring all constraints. There are misspellings where a new one was made instead of updating, leaving the pk the same.

Now I'd just put sorting and other algos, but the data is processed by a crystal report. It has no debugger. No analysis tools. 11 subreports. The thing takes an hour to run and 77k queries to the oracle backend. It's one of the most disgusting infrastructures I've ever seen. There's no other solution to this but to either move to a general programming language or get the contractor to fix the data warehouse. I feel like I've gotten nowhere trying to debug this for 2 months. Now that I've reached what's probably the root issue, the office beaucracy is resisting the idea of throwing out the fire hazard and keeping the good parts. The upper management wants to just install sprinklers, and I'm losing it.

First year: intro to programming, basic data structures and algos, parallel programming, databases and a project to finish it. Homework should be kept track of via some version control. Should also be some calculus and linear algebra.

Second year:
Introduce more complex subjects such as programming paradigms, compilers and language theory, low level programming + logic design + basic processor design, logic for system verification, statistics and graph theory. Should also be a project with a company.

Year three:
Advanced algos, datastructures and algorithm analysis. Intro to Computer and data security. Optional courses in graphics programming, machine learning, compilers and automata, embedded systems etc. ends with a big project that goes in depth into a CS subject, not a regular software project in java basically.

What you are expected to learn in 3 years:

power electronics,
analogue signal,
digital signal processing,
VDHL development,
VLSI debelopment,
antenna design,
optical communication,
networking,
digital storage,
electromagnetic,
ARM ISA,
x86 ISA,
signal and control system,
robotics,
computer vision,
NLP, data algorithm,
Java, C++, Python,
javascript frameworks,
ASP.NET web development,
cloud computing,
computer security ,
Information coding,
ethical hacking,
statistics,
machine learning,
data mining,
data analysis,
cloud computing,
Matlab,
Android app development,
IOS app development,
Computer architecture,
Computer network,
discrete structure,
3D game development,
operating system,
introduction to DevOps,
how-to -fix- computer,
system administration,
Project of being entrepreneur,

and 24 random unrelated subjects of your choices

This is a major called "computer engineering"

More adventures of working with data scientists:

Doing combinatorial analysis.

One of the post processing scripts throws away combinatorial information, analyses significance of individual data points instead.

Code was so complicated, nobody noticed.

Full stack developer.

I know what it's supposed to mean, but I feel like it gives discredit to the devs who perfect their area (frontend, backend, db, infrastructure). It's, to me, like calling myself a chef because I can cook dinner..

The depth, analysis and customization of the domain to shape an api to a website is never appreciated. The finicle tweaks on the frontend to make those final touches. Then comes a brat who say they are full stack, and can do all those things. Bullshit. 99.9% of them have never done anything but move data through layers and present it.

Throw these wannabes an enterprise system with monoliths and microservices willy nelly, orchestrate that shit with a vertical slice nginx ssi with disaster recovery, horizontal scaling, domain modeling, version management, a busy little bus and events flowing all decimal points of 2pi. Then, if you fully master everything going on there, I believe you are full stack.

Otherwise you just scraped the surface of what complexities software development is about. Everyone who can read a tutorial can scrape together an "in-out" website. But if your db is looking the same as your api, your highest complexity is the alignment of an infobox, I will laugh loud at your full stack.

And if you told me in an interview that you are full stack, you'd better have 10+ years experience and a good list of failed and successful projects before I'd let you stay the next two minutes..

Fuck you Intel.
Fucking admit that you're Hardware has a problem!

"Intel and other technology companies have been made aware of new security research describing software analysis methods that, when used for malicious purposes, have the potential to improperly gather sensitive data from computing devices that are operating as designed. Intel believes these exploits do not have the potential to corrupt, modify or delete data"

With Meltdown one process can fucking read everything that is in memory. Every password and every other sensible bit. Of course you can't change sensible data directly. You have to use the sensible data you gathered... Big fucking difference you dumb shits.

Meltown occurs because of hardware implemented speculative execution.
The solution is to fucking separate kernel- and user-adress space.
And you're saying that your hardware works how it should.
Shame on you.

I'm not saying that I don't tolerate mistakes like this. Shit happens.
But not having the balls to admit that it is because of the hardware makes me fucking angry.

Been at a startup for 2-3 weeks, hood some presentations about Linux and Foss stuff but they didn't have any work for anyone and as such didn't pay anything.

Pretty shitty.

So a colleague was like "hey man, I seed you making those presentations and stuff but not getting anything back, I think I might got a job for you".

And now I do small data analysis at another company. And get money.

Indian outsourcing web companies on their websites be like:
"We provide services in IOT, Machine Learning, Data Analysis, Web security analysis, blah blah"

Me excitingly joins the company and finds out, they all are PHP developers developing unsecured websites.

i want to get my own social network up and running.

so far ive got -
login 100% securely
register (1000% securely)
view someone’s profile (10^7% securely)

to add -
scrypt (maybe bcrypt, however scrypt looks like the better option)
friend a user
track their every move (ill use facebooks and googles apis for that)

to describe my product -
ai
blockchain
iot
big data
machine learning
secure
empower
analysis

call me when im a gazillionaire

but seriously, im making a social network and i hope its done by wk105 tbh

After you automate away task into a tool or program, do you tell your boss or keep it to yourself so they remain amazed at your efficiency and productivity?

And for data analysis ones, that you have godly data processing skills as you can somehow read a 1,000,000 line log file and find the root cause and other insights in an hour or minutes?

Data Disinformation: the Next Big Problem

Automatic code generation LLMs like ChatGPT are capable of producing SQL snippets. Regardless of quality, those are capable of retrieving data (from prepared datasets) based on user prompts.
That data may, however, be garbage. This will lead to garbage decisions by lowly literate stakeholders.
Like with network neutrality and pii/psi ownership, we must act now to avoid yet another calamity.

Imagine a scenario where a middle-manager level illiterate barks some prompts to the corporate AI and it writes and runs an SQL query in company databases.
The AI outputs some interactive charts that show that the average worker spends 92.4 minutes on lunch daily.
The middle manager gets furious and enacts an Orwellian policy of facial recognition punch clock in the office.
Two months and millions of dollars in contractors later, and the middle manager checks the same prompt again... and the average lunch time is now 107.2 minutes!
Finally the middle manager gets a literate person to check the data... and the piece of shit SQL behind the number is sourcing from the "off-site scheduled meetings" database.
Why? because the dataset that does have the data for lunch breaks is labeled "labour board compliance 3", and the LLM thought that the metadata for the wrong dataset better matched the user's prompt.

This, given the very real world scenario of mislabeled data and LLMs' inability to understand what they are saying or accessing, and the average manager's complete data illiteracy, we might have to wrangle some actions to prepare for this type of tomfoolery.

I don't think that access restriction will save our souls here, decision-flumberers usually have the authority to overrule RACI/ACL restrictions anyway.
Making "data analysis" an AI-GMO-Free zone is laughable, that is simply not how the tech market works. Auto tools are coming to make our jobs harder and less productive, tech people!
I thought about detecting new automation-enhanced data access and visualization, and enacting awareness policies. But it would be of poor help, after a shithead middle manager gets hooked on a surreal indicator value it is nigh impossible to yank them out of it.

Gotta get this snowball rolling, we must have some idea of future AI housetraining best practices if we are to avoid a complete social-media style meltdown of data-driven processes.
Someone cares to pitch in?

Startup-ing 101, from Fitbit:
- spy on users
- sell data
- cut production costs
- mutilate people's bodies, leaving burn scars that will never heal
- announce the recall, get PR, and make the refund process impossibly convoluted
- never give actual refunds
- claim that yes, fitbit catches fire, but only the old discontinued device, just to mess with search results and make the actual info (that all devices catch fire) hard to find
- try hard to obtain the devices in question, so people who suffered have no evidence
- give bogus word salad replies to the press

This is what one of the people burned has to say:
"I do not have feeling in parts of my wrist due to nerve damage and I will have a large scar that will be with me the rest of my life. This was a traumatic experience and I hope no one else has to go through it. So, if you own a Fitbit, please reconsider using it."

Ladies and gentlemen, cringefest starts. One of fitbit replies:
"Fitbit products are designed and produced in accordance with strict standards and undergo extensive internal and external testing to ensure the safety of our users. Based on our internal and independent third party testing and analysis, we do not believe this type of injury could occur from normal use. We are committed to conducting a full investigation. With Google's resources and global platform, Fitbit will be able to accelerate innovation in the wearables category, scale faster, and make health even more accessible to everyone. I could not be more excited for what lies ahead".

In the future, corporate speech will be autogenerated.

(if you wear fitbit, just be aware of this.)

I always dreamed of doing data analysis while playing my favorite game.

2020 seems to be the year of the "dev who has never seen scale."

TypeA -> "Here's a reasoned explanation for a change I think we should make. Here is the current deficiency analysis, here is the desired resolution, here is the course of action and all calculations leading to the resolution + data. This will have x,y,z beneficial result according to our operational metrics."

TypeD -> "Those were words. Why do you need that? Change is bad, learning is worse. This will just slow me down, development speed is all that matters; there is no chance that a poorly considered/factored/checked design could ever require a ground up rewrite or fuck us utterly in the long term. Why do you make my life harder? We could x -> y -> zBUTI haven't done the math and I really don't see the benefit in x, so z is pointless. What even is scale?"

The consequences of the war caused by the ever-widening gap between engineers and developers is low key terrifying.

When someone doesn't send raw data with their analysis, I feel like they have something to hide ...

Dev: We have a data quality problem. Combined with data silos, we can't trust the numbers. We need to agree on metric definitions as a company.

CTO: (Points a PM)Help him on the metrics he has defined. We'll discard any other alternate definitions inside the company.

PM: You can start with these. (Throws a bunch of metrics without their definitions. Doesn't respond to questions regarding definitions.)

After a week of generating aggregates,

PM: The numbers seem wrong.

Fuck I know the numbers will be wrong. Because you didn't give me the definitions and I'll have to deduce them myself.

Seriously guys, how do you deal with PMs who don't cooperate in the requirement analysis?

For starters this is my first post, found devrant the other day you're all hilarious.

I hate math, I hate proofs. I'm in a class "Analysis of Algorithms" and I have understood and do understand the importance of optimizing algorithms and data structures and I understand the algorithms and data structures themselves. That being said, I'm fucking sick of math and proofs and all this bullshit that is probably pretty important but ugggghh, I guess I just have to push through, but writing this out helped.

I fixed my big data processing code, I think. If all works as planned, I'll wake up to some processed data that I can do some statistical analysis on... I hope...

So I joined a course for big data analysis. And they setup a lab specifically for us. Pulling us away from the usual computer labs
AND GUESS WHAT THEY DON'T EVEN BLOODY HAVE MYSQL INSTALLED. THEY'VE CHARGED ME A FORTUNE AND THEY DIDN'T EVEN CARE TO INSTALL THE BASIC SOFTWARES AND ITS ALREADY BEEN 2 WEEKS WITH THE START OF THE COURSE AND NOTHING.
F**king hate this man!!!!

Last week I received an invitation to lead the development of a e-commerce redesign, replatforming and data migration. I was excited to work on it, and started the analysis and planning, glad to spend time focusing on quality. But Murphy's law is never asleep - this Monday, I was asked to speed things up and reduce a 4 month project to 1 week.

Small company, sole engineer. Non-tech management. Increasingly fancy job titles despite working alone most of the time, with the promise of hiring someone (again) I can actually manage soon.

Backlog of projects/tasks is truly a mindfuck, with new things being added each week. This backlog will never ever get done, and nothing matters anyway because the next idea is "the future", all the time.

While I have influence on some aspects of decision making, it usually ends up being what the boss wants. Actively opposed a project because it's just too big of an undertaking, it was forced through anyway. I'm trying to keep the scope manageable as I'm building it now, and it's hard.

"It's the future, we absolutely have to do this. It will be the biggest thing we've ever done."

Boss's excitement then quickly faded since it's actually in development, now nobody really seems to want to know where it's at, or how it will all work. I need to scope it out, with the knowledge that many decisions boss signed off will be questioned when he actually looks at it. We now have even more "exciting" ideas of utter grandeur. Stuff that I can't even begin to comprehend the complexity of, while struggling to keep a self imposed deadline on the current one.

Every single morning we sit on Zoom for a "valuable" "catch-up". This is absolutely perfect for one thing: Completely destroying whatever drive and focus I have going into the day. Unrelated topics, marketing conversations, even more ideas, ideas for ideas sake, small problems blown out of proportion, the list goes on. I recently argued in detail why it should be scrapped or at least be optional to attend. No luck, it's "valuable".

Today a new idea was announced, and we absolutely have to do it ASAP because it can only be better than the current solution. I raise my concerns, saying it's not as easy as you make it out to be, we should properly think about it. Nope! We'll botch something to prove that it works... So you'll base your decision whether it's good on some half ass botch job that nobody really has the mental capacity to actually pay attention to. What a reliable way to measure!

"Our analytics data isn't useful enough to tell us the impact of things we do. We (you) have to fix this." Over the last 2 or so years, I've been pushing for an overhaul and expansion of our data analysis capabilities for exactly this reason. Integrating different data sources into a unified solution so we can easily see what we're doing, etc. Nope, never happened.

The new project idea which is based on wild assumptions is ALWAYS more important than the groundwork.

Now when I mentioned that this is what I wanted to do all along, it got brushed aside. "We don't need to do anything complicated, just fix this, add that, and it's done. It should be an easy thing to do. This is very important for our decision making." Fine, have it your way.

I'm officially burned out. It's so fucking hard to get myself to focus on my work for more than an hour or two. I started a side project, and even that effort is falling victim to my day-job-induced apathy.

I'm tempted to hand in my resignation without another offer on the table. I just need time to rediscover my passion, and go job hunting from that position, instead of the utter desperation of right now.

If you've read through all this rambling, kudos to you!

QA personal voice assistant that runs locally without cloud, it’s like never ending project. I look at it from time to time and time pass by. Chat bots arrived, some decent voice algorithms appeared. There is less and less stuff to code since people progress in that area a lot.

I want to save notes using voice, search trough them, hear them, find some stuff in public data sources like wikipedia and also hear that stuff without using hands, read news articles and stuff like that.

I want to spend, more time for math and core algorithms related to machine learning and deep learning.

Problem is once I remember how basic network layers, error correction algorithms work or how particular deep learning algorithm is constructed and why is that, it’s already a week passed and I don’t remember where I started.

I did it couple of times already and every time I remember more then before but understanding core requires me sitting down with pen and paper and math problems and I don’t have time for that.

Now when I’m thinking about it - maybe I should write it somewhere in organized way. Get back to blogging and write articles about what I learned. This would require two times the time but maybe it would help to not forget.

I’m mostly interested in nlp, tts, stt. Wavenet, tacotron, bert, roberta, sentiment analysis, graphs and qa stuff. And now crystallography cause crystals are just organized graphs in 3d.

Well maybe if I’m lucky I retire in the next decade or at least take a year or two years off to have plenty of time to finish this project.

During one of our visits at Konza City, Machakos county in Kenya, my team and I encountered a big problem accessing to viable water. Most times we enquired for water, we were handed a bottle of bought water. This for a day or few days would be affordable for some, but for a lifetime of a middle income person, it will be way too much expensive. Of ten people we encountered 8 complained of a proper mechanism to access to viable water. This to us was a very demanding problem, that needed to be sorted out immediately. Majority of the people were unable to conduct income generating activities such as farming because of the nature of the kind of water and its scarcity as well.
Such a scenario demands for an immediate way to solve this problem. Various ways have been put into practice to ensure sustainability of water conservation and management. However most of them have been futile on the aspect of sustainability. As part of our research we also considered to check out of the formal mechanisms put in place to ensure proper acquisition of water, and one of them we saw was tree planting, which was not sustainable at all, also some few piped water was being transported very long distances from the destinations, this however did not solve the immediate needs of the people.We found out that the area has a large body mass of salty water which was not viable for them to conduct any constructive activity. This was hint enough to help us find a way to curb this demanding challenge. Presence of salty water was the first step of our solution.

SOLUTION

We came up with an IOT based system to help curb this problem. Our system entails purification of the salty water through electrolysis, the device is places at an area where the body mass of water is located, it drills for a suitable depth and allow the salty water to flow into it. Various sets of tanks and valves are situated next to it, these tanks acts as to contain the salty water temporarily. A high power source is then connected to each tank, this enable the separation of Chlorine ions from Hydrogen Ions by electrolysis through electrolysis, salt is then separated and allowed to flow from the lower chamber of the tanks, allowing clean water to from to the preceding tanks, the preceding tanks contains various chemicals to remove any remaining impurities. The whole entire process is managed by the action of sensors. Water alkalinity, turbidity and ph are monitored and relayed onto a mobile phone, this then follows a predictive analysis of the data history stored then makes up a decision to increase flow of water in the valves or to decrease its flow. This being a hot prone area, we opted to maximize harnessing of power through solar power, this power availability is almost perfect to provide us with at least 440V constant supply to facilitate faster electrolysis of the salty water.
Being a drought prone area, it was key that the outlet water should be cold and comfortable for consumers to use, so we also coupled our output chamber with cooling tanks, these tanks are managed via our mobile application, the information relayed from it in terms of temperature and humidity are sent to it. This information is key in helping us produce water at optimum states, enabling us to fully manage supply and input of the water from the water bodies.
By the use of natural language processing, we are able to automatically control flow and feeing of the valves to and fro using Voice, one could say “The output water is too hot”, and the system would respond by increasing the speed of the fans and making the tanks provide very cold water. Additional to this system, we have prepared short video tutorials and documents enlighting people on how to conserve water and maintain the optimum state of the green economy.
IBM/OPEN SOURCE TECHNOLOGIES
For a start, we have implemented our project using esp8266 microcontrollers, sensors, transducers and low payload containers to demonstrate our project. Previously we have used Google’s firebase cloud platform to ensure realtimeness of data to-and-fro relay to the mobile. This has proven workable for most cases, whether on a small scale or large scale, however we meet challenges such as change in the fingerprint keys that renders our device not workable, we intend to overcome this problem by moving to IBM bluemix platform.
We use C++ Programming language for our microcontrollers and sensor communication, in some cases we use Python programming language to process neuro-networks for our microcontrollers.

Any feedback conserning this project please?

There was this project where a bunch a scripts had been running for three weeks analysing a bunch of fileshares. The project was in overrun and the analysis wasn't anywhere near done.

I was given complete isolation and a team of people who were instructed to do anything I needed. I had them replicate the data to as many machines as possible and I started scripting the analysis with some sample data. After half a day of collecting laptops, desktops and severs I transfered my scripts to those machines and ran the analysis in 5 hours.

I felt like I saved the project.

[story of your first dev project] - i really think there should be a headline like that for wk rants

Anyways, it was a while back, le college teacher approached my friend and me asking if we wanted to do a project. We said sure, it was a medium sized data analysis project. We got the specs with a lot of formulas, basically implement them all and make a web frontend, thats it. Took like a year but we did it. Few months later teacher is furious because the calculations didnt give him data that he expected (by expect i mean he thought that a distribution formula would accurately yield 200+ data fields from around 4) and blamed it on us. Not the retard other professor who fucked up half of the formulas. Ok.

Best: learned to code, started writing smart-home scripts for home automation and developed biologic and quantitative data analysis scripts in Python and R.
Worst: didn't get paid to develop them and haven't got enough experience for it to be more than a hobby.

A time I (almost) screamed at co-worker?
Too many times to keep up with.

Majority of time its code like ..
try
{
using (var connection = new SqlConnection(connectionString))
{
// data access code that does stuff
}
catch (Exception e)
{
// Various ways of dealing with the error such as ..
Console.WriteLine("Here");
ShowMessage("An error occured.");
return false;
// or do nothing.
}
}

Range of excuses

- Users can't do anything about the error, so why do or show them anything?
- I'll fix the errors later
- Handling the errors were not in the end-user specification. If you want it, you'll have to perform a cost/benefit analysis, get the changes approved by the board in writing, placed in the project priority queue ...etc..etc
- I don't know.
- Users were tired of seeing database timeout errors, deadlocks, primary key violations, etc, so I fixed the problem.

On my tip of my tongue are rages of ..

"I'm going to trade you for a donkey, and shoot the donkey!"
or
"You are about as useful as a sack full of possum heads."

I haven't cast those stones (yet). I'll eventually run across my code that looks exactly like that.

One inappropriate experience I can think of is during an internship at a multinational company that made networking and electrical components. My mandate was to do an analysis on the electrical performance of the company's products versus the competitors. It took me something close four months to measure, to compile data, do an analysis and create a report.

Then came the time to show the results to the engineering director. Let's say the news were not good, so I figured people should have their hears opened. Anyways, my supervisor and I made a presentation to abstract the main points so the information was not too difficult to process.

10 minutes into the presentation, the director of engineering just fell asleep and was snoring quite loudly. I asked my supervisor "should we wake him up?" and so we did. When he woke up, he asked us to wrap it up and pretty much gave no fucks about the results or the presentation. Nobody really cared about the results after that. Talk about wasted time lol.

Luckily I don't work at this shithole of a company today.

So we’ve taken over from a project team that disbanded... read: “cut their contracts because fuck this, I can earn more working for better people”.

Me and one other guy have been tasked with saving this heap of shit.

Obviously the project guys left saying “it’s nearly done, just this one feature”. Because cut contracts are easier to deal with if “everything is almost done”.

We jump on and find that’s not the case at all... this thing, is a beast, a big old stats analysis program... so we’re like “cool, let’s see what’s going o...OH MY GOD”.

The “recalculation” function was core to this POS. The contractors had done it in C# through entity framework... it took 24 hours to run, over a reasonably small data set that was due to double every 2-5 years.

So... here’s the deal, it ran over night.... then failed. And no cunt had noticed. Entity framework “can’t commit because I’m muddled up as fuck, did you really just put the whole db in EF in memory to work with it?” Exception.

Que 6 months of me and my lead doing the job properly.

Anyway, the failure: I ended up in Hospital again with a Crohn’s flare up... about 5 months in.

Fuckall to do with all this nonsense I just wanted to tell a story. it was an interesting/fun project to fix and my lead was a legend... so happy days.

Similar story, different set of contracted devs... they’d been defining requirements with the business users using the term “Risk” which the business users knew as a group of risks.

The domain model had been written RiskGroup<>—

i was learning neural networks, started with keras and was on the first tutorial where they started by importing pandas
so i switched to learning data analysis using pandas in Python where they started by importing matplotlib and i realized data visualization is also important and now I'm reading matplotlib docs...🙄

I need to estimate how much ram and CPUs my team will need next year for our apps... That have yet to be built.

We load a lot of data feeds with batch processes running on a few large machines, some can use like 30 GB RAM at times...) which should be a lot less if we get the data real time I hope...

But wondering how to estimate well... I sorta did a worse case analysis where I just multiply and sum # CPUs/memory* nodes*approx apps...

Comes out to be like 600 CPUs and 800 RAM... So wondering if that's ok...

RAM is ok but # of CPUs is way higher bc now all the apps basically run on their own machines...

One problem for CS education is that the salaries for academia are so low compared to industry that if someone is even vaguely competent, they can at least double their income by working a 'real' job. Now this may be different at higher levels of colleges but generally those folks are such bullshitters they wouldn't last outside of academia.

As what to improve?
Depends if it's a research or vocational course.

For vocational; heavy on group projects, common tools, methodologies and architectures. All demonstrated in something like c#/java/python. And one project must have a web app (db, app layer and JavaScript from end)
Basic knowledge of algorithms, runtime analysis (O notation) and some data structures and you're an instahire.

For research, go wild. Deep dive into the math, algorithmic side. Read up lots of research papers. Try out different programming paradigms. You would aim for a career in academia, AI, quant finance etc...

never before have I been happy to be asked to work overtime, but for once, fuck yeah...

Bit of back story, I am tech lead on a massive project that has been run like a complete shit show, the PM who also happens to be the brains behind the project seems to think we are miracle workers and for the first 9/10 months of the project would make significant, like delete a weeks worth of code and start over changes, 3-5 times per week. There are features for the v1 release that have been built in excess of 5 times. I have been saying since October that even without all his constant changes, we will NOT make the deadline, and naturally as is part of my job I argued against every unnecessary feature he tried to implement, eventually he pulled me into a meeting to tell me how much he values my opinion, I need to stop arguing with him and he does not want to work with yes men (I have a rant about that convo already).

I believe our CEO finally started smelling a rat as he insisted on joining our daily stand-ups, during which said PM scripted some lovely stories to disguise the fuckup we are in, and since has assigned another PM to take over and do proper project management and risk analysis.

That is where the email comes in, a lot of the work assigned to me will miss the deadline by a month, honestly I am impressed that it is by so little and so few people will not be missing it, but anyway, he probably spun a few stories there too.

So I spent part of the work compiling the most perfect surgical response as not not actively throw him under the bu, but create a quite a few questions that they hopefully as, as himself and the CEO where cc'd into the mail.

And the jist is, the deadline itself was still impossible and 8 of the 10 tasks assigned to be have ZERO back-end whatsoever, and those tasks are about 80/90% integration to said non-existent back-end, some of those services and data structures have not even been planned yet and we are a week past the deadline and 3 weeks from the just as useless extension.

I made a script that performs some analysis on data from a trading card game tournament and I would appreciate your input

I used to be a big security guy, not allowing stuff like most of the social media, not bringing my phone anywhere, carrying a RPi tablet for privacy reasons. Very Stallman stuff.

Recently I noticed that I don't care so much.. I see these things as opportunities, for instance Microsoft products could be benefitial for job opportunities, I have some workout sessions on my phone.

I could restrict myself... but is it worth it just to decline some capitalist/politician's row in a dataset for analysis?

But then again I feel as a society I think we should either do this or request this data to be distributed to us as well.

Should you be playing a game of cards, when the enemy can see your hand? What do u think?

So a team at on-site sent a OOM(Out of Memory) issue in our morning.
Everyone analysed the issue as being code issue since we were bringing too much data into the runtime process. The analysis was done on the heap dumps. The number of records reported by user were 1k. At the end of the day it turns the number of records were actually 100k+.
Why do people jump.to conclusions without thinking about the obvious. :-(

I code. I'm not a network or security admin. Did you even read my resume before hiring me? It says "math degree then data analysis then programming degree" Subnets, firewalls and security certs are not my areas of expertise. Given enough time, could I stumble through? Probably. But I'd probably come up with something worthy of mocking on networkadminrant.

I guess the devs at rdocumentation need to work on how they "analyze data" 🤣

Time to switch to offline and hide in some dark corner to get work done. Tired of all the IM’s and coming over to my desk from 1 person for “critical” work. If they’re all critical then none of them are truly critical. If you sit on the data for 2 months, and then today is the day it becomes critical and the compliance issue is because of your ineptitude then its a you problem not an IT problem. Then on top of that you submit your data to be loaded in the incorrect request form and spreadsheet format you can go fuck yourself asking this be done in an hour. It could be done in 15 minutes if you had it in the correct format as specified in the 20 meetings over the past year which removed all manual analysis and automated the entire process you idiot. Now I have to get it into the correct format in that hour so I don’t have to do the analysis for you.
I have other things to do besides your etl tickets, like finding the actual problems in our actual critical applications. You know the ones where the VP’s of this giant corporation start calling if they go down.
Sorry for the rambling guys.

So I'm in a scenario I'm uncomfortable, need some encouragement from fellow devRanters. (Looong post)

I've been working at this startup for about 10mths (since I graduated). They have been really good to me since the start, and overlooked some fuck ups I did at first.

But now I've been way more experienced , picked it up really quick. And I've basically redesigned several of their admin solutions and data products. Also, I'm basically their entire data analysis team now. I do backend (node, PHP, MySQL) and analysis for them (stats, deep learning, python, big data packaging for clients).

But seeing as I've moved in their company, and have been consulted on several major decisions, as well as built a really good relationship with some of their clients. I still haven't seen a raise, moreover I've been told that I'm expected to work from 8am to 5:30pm (9.5hrs no overtime pay). Which really pisses me off, since I know I'm worth more than what I'm paid (about 40k a year).

My brother (who's also a dev) suggested to tell them that I'm not happy at work due to this. And quit if they don't react well.

How should i bring this up? Should I really quit? This is all new grounds.

I want to do something data-science-y.
Gimme project ideas, and where can I get the data for it?

Also, not looking for machine learning, just basic data analysis stuff.

I'm bored.

(Day 1 of Database Class)
Database Prof - "Your final project is a program or app that deals with big data and showcases data analysis. Make something that can be used by consumers and has real-world application."

(After the Day of Showcasing Projects)
Database Prof -
(What he should have said on Day 1) - "Make something that makes me laugh with added data."

I think another intriguing job asides programming is engineering (*for some*). A week has past and I've been on the hike assisting my beloved brother on his contracted engineering job while I am less occupied. The job is based on 🗼Tower analysis and It's quite risky as you'd have to climb up to 56 meters high just to take readings of antennas, and fix some other stuffs. The only thing I find intriguing about this job is his love for it, funny enough he also thinks I love the job too and I guess I'm guilty for his thoughts (*Sorry bro, I love the job for you not me*).
With my little experience so far on my *new brotherly job* I noticed the most hectic task isn't going up and down the tower taking readings but at the end of all operations, he'll have to gather the values and snapshots he took while on the tower to prepare reports on msword & excel for the other buttwags at the office (or home I guess)
then archive and sends via mail. Seeing this lengthy process I was forced to ask why he wasn't using any reporting tool like Jotforms or any other equivalent and I was willing to look up some recommendations for him, his reply was: "I'm already used to this form of reporting, its what I was trained with and what the company provided, nevertheless a friend of mine suggested something of such weeks back but I would have to pay monthly fee for its usage which is quite on the high side and I don't think I'd prefer that."
Sounds convincing but not enough, okay here is another deal: You use an android phone right? and at my office we work on system automation (*basically does not know what I do for a living probably thinks I'm a hacker the illegal one*), how about i design you an android app for you to capture the tower data and a PC software for you to auto generate the msword & excel reports, I can get this ready for you in less than 5 nights (*I've got less task on my desk, and was willing to take the timeout to prepare the solution that he needed, all I needed to hear for a kick start was an "Okay" just to be sure he wants it*) I suggested and re-assured but up to this point he still declined my offer and is willing to stick with his current reporting pattern (*Me died*).

I work with statistics/data analysis and web development. I study these subjects for almost a decade and now I have 4 years of practical experience.

This information is on my LinkedIn profile and from time to time tech recruiters contact me wanting to have an interview. I always accept because I find it a great way to practice interviews and talking in English, as it isn't my native language.

A remark that I always make to my colleagues wanting to start doing data analysis related work is that it may seem similar to development, but it's not. When you develop, your code work or not. It may be ugly, it may be full of security problems, but you almost always have a clear indication if things are functioning. It's possible to more or less correlate experience using a programming language with knowing how to develop.

Data science is different. You have to know what you are doing because the code will run even if you are doing something totally wrong. You have to know how to interpret the results and judge if they make sense. For this the mathematics and theory behind is as important as the programming language you use.

Ok, so I go to my first interview for a data science position. Then I discover that I will be interview by... a psychologist. A particularly old one. Yeah. Great start.

She proceeds to go through the most boring checklist of questions I ever saw. The first one? "Do you know Python?". At this point I'm questioning myself why I agreed to be interviewed. A few minutes later, a super cringy one: "Can you tell me an example of your amazing analytics skills?". I then proceed to explain what I wrote in the last two paragraphs to her. At this point is clear that she has no idea of what data science is and the company probably googled what they should expect from a candidate.

20 minutes later and the interview is over. A few days later I receive an email saying that I was not selected to continue with the recruitment process because I don't have enough experience.

In summary: an old psychologist with no idea on how data science works says I don't have experience on the subject based on a checklist that they probably google. The interview lasted less than 30 minutes.

Two weeks later another company interviews me, I gave basically the same answers and they absolutely liked what they heard. Since that day I stopped trying to understand what is expected from you on interviews.

My boss is being a stupid cunt. To give you a background we were facing issues with our Collections system. First week December 2019, I and a colleague of mine came up with a new efficient collections architecture. My colleague and I started to Code and create automation scripts mid December and completed it in First week of Jan 2020. This PoC version was supposed to be just between the Dev team(App Dev and Back end, also one from the Ops side to verify the data). I did not receive any feedback on the actual collections system and the data integrity but during this time all they’ve done is take meetings with no real outcome. I raised this and the only email I got is data is looking fine when I know it is not.Now in First week of Feb, he is stressing us to go ahead and deploy the architecture in Production and we have not done any Code Review, Static Code analysis, any real tests on Code and deployment scripts. Have not discussed any metrics for our dashboard and alerting. I have no idea how to handle this cunt. I have even asked for resources to atleast productionalize the code and move ahead the deployment and still no out come. I’ll go in a meeting with him in an hour, I will be very blunt and tell him that whatever he is doing is a foolish way and maybe resign in couple of weeks

Bootcamps get you up and running in coding quickly. If you are a programmer, companies are only interested on how quickly, error free and cheaply you produce marketable output. Bootcamps enable this.

More or less you are not more than a former assembly line worker putting parts on a car platform. Your value is not very high as you may be exchanged at any time at their will.

Nevertheless, you can earn money quickly. You trade in your youth and time which might be a dead end in the long-term. Trends go to machine learning, artificial intelligence. They will not need Bootcamp people and code workers.

It is better you set up Bootcamps and sell them versus absolving this. Like selling shovels during the gold rush, but not working in the mud of Alaska by yourself.

Your choice is: Making quick money, which fades anyway; or striving for the long-term future proof career.

C/S degrees from Technical Universities of reputation give to you the right direction under a strategic consideration. Companies which pay well, or freelancing with a solid acknowledged background, will always look for top graduates. People from Bootcamps are just OK for hammering assembly line coding. Even worse with SCRUM in one noisy room under enormous team server pressure controls, counting your lines of code per minute, with pale people all around. And groups of controllers never acknowledging nor trusting your work.

To acquire a serious degree, a Bachelor is nothing. Here, in INDIA, Bachelor now is what a former high school grade was. You must carry a diploma or Masters degree combined with internships at big companies with high brand recognition. This will require 4–6 years of your lifetime. You can support this financially by working part-time freelancing as making some projects front- or back-end web, data analysis and else.

Bootcamp people will lose in the long-term. They are the modern cannon fudder of software production.

It is your choice. Personally, I would never do Bootcamps. Quality and sustainability require time, deep studies and devotion.

When your IT VP starts speaking blasphemy:

"Team,
We all know what’s going on with the API. Next week we may see 6x order volumes.

We need to do everything possible to minimize the load on our prod database server.

Here are some guidelines we’re implementing immediately:

· I’m revoking most direct production SQL access. (even read only). You should be running analysis queries and data pulls out of the replication server anyway.
· No User Management activities are allowed between 9AM and 9PM EST. If you’re going to run a large amount of updates, please coordinate with a DBA to have someone monitoring.
· No checklist setup/maintenance activities are allowed at all. If this causes business impact please let me know.
· If you see are doing anything in [App Name] that’s running long, kill it and get a DBA involved.

Please keep the communication level high and stay vigilant in protecting our prod environment!"

RIP most of what I do at work.

I was studying a lot the last year, i learned a lot about Machine Learning/Deep Learning, Data Gathering, Data Analysis, ETL, Model Architecture Design, Training, Fine Tuning, Backend Development, DataBases, API Development, ORMs, Rest, GraphQL, OAuth, CI/CD, Docker, Deployment to Production environments like Heroku, Git and more stuff i dont remember while writing this. I built and keep adding stuff to my Github Portafolio.

Im not able to get a job. I started looking for jobs as Data Scientists, no response never. I take a look at freelancer sites, nothing seems to fit my skills. And when there is a minimal fit, they always want a Full Stack Web Developer, i dont know Frontend Development, i dont like do it.

Dont know what to do or how to land any job.

My options aeems to be:

1.Learn Frontend Dev and work as Full Stack in underpaying freelance jobs

2.Keep applying to Remote-Only startups, but they still wants people with 3+ years of experience.

i cant work in my city, here are not any company startup hiring no one, we are 30 years in the past here.

What you do in my place?

I don't know what to choose...

A no-brainer job in data input with security, stability and a chance of promotion or transfer to another area IT related far away in the future...

Or

An intership of one year, in systems analysis, with a 30% better salary but no guarantees after that...

How 2018 brings an aww bug.

At my company, we've a reporting application. Which kind of provides analysis of client's weekly business. When you open the application it shows their business trend starting from Sunday to day of week.

As usual their is an endpoint to get the data basis on start date. As soon as date changed to 1st Jan, it stopped showing data. Given that it was a long weekend, no one was available from the tech. Support team got tremendous amount of tickets for this. Later on Tuesday while debugging we got to know while forming the date in the application the logic was like this

- get current year
- get the date and month on start of week

Combine these 2 and request the data. All the time it was fetching data starting from 31 Dec 2018.

😒😒😒

Today I had a full-day job interview for a junior data scientist position.

First I met the team which was only like half of everyone because apparently everyone was gone on Fridays. However the few there were really nice.

First task is to do some basic data analysis stuff even though I already spent a week on the coding challenge and sent them all my code/tasks. I log into my machine and create a new virtual environment but can't for the life of me figure out how to use the command line in windows to install packages. Turns out there is some problem with their proxy and they have to log me in on that. Then I am struggling on the keyboard because it's for a language different that my mother tongue and it takes me 3x as long to so the most simple things. All my shortcuts are out the window. Haven't a hard time typing parentheses and brackets. Start freaking out and have a panic attack mid task. I'm sweating bullets. I didn't even make it to the simple visualization tasks much less the models at the end. Time gets called and we all go to lunch and I'm freaking out on the inside the entire time. Angry at myself because I know I am better and just couldn't think.

After lunch I present my code and results from a coding challenge I did weeks prior. People from other teams get invited and I end up getting grilled for 2 hours by 15 people. Questions are flying in from all sides. They ask me almost everything I know about machine learning and some more. Under stress I forgot the name of the optimizer I used and couldn't answer some easy stuff because my mind was racing.

Right now I am on the train home and my body physically hurts. I am disappointed with myself and wish I could have shown up better. Never really froze up like this before.

I have found that when companies say or think that they do not have large amounts of data for using even the smallest bit of data acience analysis in terms of generating a model that can assist them that they do have data. No matter the size of the organization, the data is there.

LABELED data, well now that is a other story 🥴

!rant

I just made my API in my laravel and I understand how it works! It may seem like not a lot, but I got from far.

Just came two years ago in this industry as I worked as a customer service agent for a hostingcompany. I entered a whole new area what I immediatly got into at the time. Mind I already was studying Biomedical labresearch at the same time and was the IT guy in the family. Well, think back then I was just googling and fixing shit most of the time.

I was 21 at the time and began to learn everything I could learn in my position and soon it was not enough and wanted to learn more by working parttime(study already asks a lot of time). I soon applied as Junior System Engineer within the same company without prior education and got the job! And I'm back feeling I entered a new area where you feel you can do so much by just learning how it works. Now I want to learn to develop in PHP so I may make another step further.

Not a rant, but I want to share my experience as labrat starting to someting programming(did some bio-informatics, which was really interesting but with less emphasis on programming but more on data analysis). Still got a gigantic of list I want to learn from languages and frameworks to orchestration systems.

I currently work in data analysis.

Yesterday one of the colleagues working in a different department came to me asking if I could fix his malfunctioning headphone set :/

Stop calling data analysis tool AI... Stop calling droping notifications by % of new events on occurance calculation - AI!

Me, working hard on a SQL project with a deadline that is half what it should be with no support from the other people on the project and was mostly made with with data I imagined would be in there cos no one could get me any fucking shit done (i.e. effectively designed, built, tested, fixed, upgraded, documented on my own for an entire weekly/monthly/ad-hoc analysis process that would output various reports for internal/external/management)

Manager - man who is a known waste of space but for some reason is in charge of the smallest part of the project, shouldn't have been fucking involved fucking management guzzling stain magnet...

Manager: Hey, do I need to refresh the database?

Me: .................

Me: .................

Me: ............I dunno, do you think we should refresh the database that this entire project is reliant? I mean...why do we need up to date transactions to analyse? Wait....you telling me it's not been being refreshed this ENTIRE time?

Manager: No....you never said I should. So should I?

Me: ..................I never said you should!?!? Are you not in the meetings talking about dependencies?????? Do you think i should have up to date trans or just run this with old stuff????? Why would you not update it!??!!?!?!? Its transactions...... (Desperately trying not to punch through my screen, through his, into his throat)

Manager: ..............

Me: .................

Manager: I think i'll refresh it and add it to the job?

Me: ....................(goes back to work cursing with music in so I think its quiet but who knows).

Tard, don't know how he even gets to work without someone holding his fucking hand.

Happy ending, I don't work there anymore :p

Sad ending, his spirit of tard follows me to my new jobs and possesses someone (or three sometimes)

Big Data Analysis

Worked entire day on an ML to predict train ticket status for Indian Railways.

While doing the analysis of the data, and trying to check/research which parameters to use and which not, I'm feeling like racist -_-.

I was looking for busy period of the year(read festive seasons) and I removed those festivals which are celebrated by minor groups.
There's more to it, but the results are better now.

So I have question about my resume.
During my college time, I have done two projects related to politics:
One is to analyze the bias of media. What I did is scrape news covers for Trump and Hillary during election year and get sentiment analysis. The result is not surprising that among NY Times, NBC, Fox, Eashington Post, and CNN, Fox news is clearly favoring Trump, since Fox news is a republican news site.
The other project I did was to analyze the speech complexity and sentiment of the election. One of the observation we made was that Hillary and Trump are almost at the same level regarding speech complexity. However, Trump has a more positive sentiment in the speech, which is true consider how much he loves to say make America great again.

Now the question is, when I gave my advisor my resume, she said that I'd better not put those two projects on my resume since they are related to politics.

But, I am applying for a data science master degree. Seriously, I was just collecting the data and the data speaks for himself, why should I take those projects off my resume? I'm very proud of those projects I did as a matter of fact.

So here is the question. Shall I take off those two projects on my resume because they were political or I should leave it thereawarreally need some professional views. Please.

personal projects, of course, but let's count the only one that could actually be considered finished and released.

which was a local social network site. i was making and running it for about three years as a replacement for a site that its original admin took down without warning because he got fed up with the community. i loved the community and missed it, so that was my motivation to learn web stack (html, css, php, mysql, js).

first version was done and up in a week, single flat php file, no oop, just ifs. was about 5k lines long and was missing 90% of features, but i got it out and by word of mouth/mail is started gathering the community back.

right as i put it up, i learned about include directive, so i started re-coding it from scratch, and "this time properly", separated into one file per page.

that took about a month, got to about 10k lines of code, with about 30% of planned functionality.

i put it up, and then i learned that php can do objects, so i started another rewrite from scratch. two or three months later, about 15k lines of code, and 60% of the intended functionality.

i put it up, and learned about ajax (which was a pretty new thing since this was 2006), so i started another rewrite, this time not completely from scratch i think.

three months later, final length about 30k lines of code, and 120% of originally intended functionality (since i got some new features ideas along the way).

put it up, was very happy with it, and since i gathered quite a lot of user-generated data already through all of that time, i started seeing patterns, and started to think about some crazy stuff like auto-tagging posts based on their content (tags like positive, negative, angry, sad, family issues, health issues, etc), rewarding users based on auto-detection whether their comments stirred more (and good) discussion, or stifled it, tracking user's mental health and life situation (scale of great to horrible, something like that) based on the analysis of the texts of their posts...

... never got around to that though, missed two months hosting payments and in that time the admin of the original site put it back up, so i just told people to move back there.

awesome experience, though. worth every second.

to this day probably the project i'm most proud of (which is sad, i suppose) - the final version had its own builtin forum section with proper topics, reply threads, wysiwyg post editor, personal diaries where people could set per-post visibility (everyone, only logged in users, only my friends), mental health questionnaires that tracked user's results in time and showed them in a cool flash charts, questionnaire editor where users could make their own tests/quizzes, article section, like/dislike voting on everything, page-global ajax chat of all users that would stay open in bottom right corner, hangouts-style, private messages, even a "pointer" system where sending special commands to the chat aimed at a specific user would cause page elements to highlight on their client, meaning if someone asked "how do i do this thing on the page?", i could send that command and the button to the subpage would get highlighted, after they clicked it and the subpage loaded, the next step in the process would get highlighted, with a custom explanation text, etc...

dammit, now i got seriously nostalgic. it was an awesome piece of work, if i may say so. and i wasn't the only one thinking that, since showing the page off landed me my first two or three programming jobs, right out of highschool. 10 minutes of smalltalk, then they asked about my knowledge, i whipped up that site and gave a short walkthrough talking a bit about how the most interesting pieces were implemented, done, hired XD

those were good times, when I still felt like the programmer whiz kid =D

as i said, worth every second, every drop of sweat, every torn hair, several times over, even though "actual net financial profit" was around minus two hundred euro paid for those two or three years of hosting.

#justathought

Since a country is considered to be the strongest definition of a group of unified people, consider the following scenario:

Every country makes it mandatory that any foreign company can only sell their software products as free to use softwares, or one time cost software, or a membership software.

These foreign companies are NOT allowed to generate revenue via Advertisement services or data collection/sharing/analysis AT ALL.

The sole right to share/collect data and use Advertisments as a revenue model remains in the hands of domestic companies born, registered and working in that country only.

This would generate an equal chance for both domestic and international companies to grow(domestic companies getting a chance to grow with a better revenue model, while international companies getting a chance to grow by monetising their tools/algorithms and investing in domestic companies) , resulting in countries getting a chance to grow themselves.
Open source will still win, as open source majorly contains tools and technologies for general public use.
Premium tools and frameworks would become even more valuable, and would he shared among countries like the way they share space researches and resources on nuclear technology today.

Privacy will win, as the data of a particular country stayed within a country. Domestic Companies sharing data with other companies (or even their foreign parent companies) would be held against the respective country's laws only, and government would be more involved in protecting its citizens from data theft.

Is it feasible?

Is a masters in statistics worth it?

A bit of background:
I got my bachelor in actuarial math (statistics for insurance risk), then found machine learning and got a couple of gigs in software development and data engineering. I became my previous employers the go to guy for questions about data integrity and structure.

Now I am heading to a new job that specializes in ML for gambling. And while I love the math, I really see myself doing more software development and system architecture work (with some analysis). I already started this masters program, so I got less than a year to finish, but starting to feel like its a waste of my time, but also, I dont want to just quit it.

I have last few months left out of graduation and i don't know what should i learn. There's so much things (web dev , ai/ml, blockchain, android , cloud, ,hybrid apps, gaming, ar/vr, data analysis, security,etc) and as a cs student, i feel i should be knowing them all.

In last 6 years ,

Techs that i liked or got success in :
java, Android,python, data analysis, hybrid apps(flutter)

Techs that i didn't liked or failed in : ai/ml, cloud computing , webdev(css/js) ,hybrid apps(react/angular/ionic/...)

Techs that i didn't tried : security, cryptography, blockchain, open cv , ar/vr, gaming

I am not bound by my likeness or success.

My failures was mainly because i didn't liked those techs and continued further in them. And my success comprises of just launching a few apps, passing in some certification or grabbing an internship opp because of those skills.

But if you think a particular skill is necessary to have as a cs professional then let me know. I just want to earn a lot of money and get out of this mess asap

Not sure what I wanna do ---

Backend Development or Data Analyst

Even in backend development confused between nodejs, PHP, Python though I don't have proper knowledge of all of them.

For Data Analyst I am looking the way from where I should start.

I am in 3rd year of my college
Can anyone please help me with this?

There's a team where the leader has some real weird/bad ideas here, it's like, ugh, no. I can't say no because I'm not in position to contest, but geez I so want to.

Like, we wanted to load data with pagination because there could be a lot of them (could reach thousands objects of data easily).
Team Leader: No, no pagination because that ask to call those services several times. Only one call, you load all the data in the same single page and you don't call the service again, so stock them in cache.

So the idea at first looks bad, and after analysis and research, it is real bad, of course.

just receive a refurbished 4 years old laptop.... then deep clean it.... now having dilemma on picking a distro......

mainly use to run data analysis (r, Python, Java, C++, mySQL, MongoDB and some cloud servers...)

my thinking about a good distro to me, comfortable appearances, customize freedom, community support and constant security update.

any suggestion people??

want to do software for my final year project but the lecturer's not in office and didnt reply my email during choosing topics, so end up with data analysis. 😴

Which is better .. R or PYTHON
for data analysis ?

A numerical and data analysis platform for individuals and teams

"Dear TitanLannister : You are in the final year. A lot of shit is happening around u. its now time to make a career and take tough decisions. What would you do?"

CHOICE 1: COMPETITIVE
>>>>background : "a lot of super companies like wallmart, fb, amazon, ms, google,.. etc simply takes a straight coding test for fresher placement. They ask tough bad ass level questions, but with right guidance, a hell ton of dedicated hours of coding, and making it to the top of various coding tests could make you a potential candidate"
>>>>+ve points :
- "You got the teachers and professionals with great experience to guide you"
- "a dream job come true.you can go there and join teams that interests you"
- "it was your first exposure to computer world. maybe you would like doing it again, after 4 years"
>>>> -ve points:
- "You have always been an average 70 percentile guy. The task requires 2000-3000 hours of coding an year. it will be hard and you always grow bored out of this pretty quickly"
- "Even If you did that , you stand a lesser chance because your maths is shitty.There are millions running in this race with brains faster than your IDE"
- "your college will riot with you because they expect 75% attendance"
- "You are virtually out of college placements, in which , even though shitty companies come and offer even shittier 4LPA packages($6000 per annum), would take a tough logical/aptitude based test for which you won't be able to prepare"

CHOICE 2: PROFESSIONAL WORK
>>>>background: "you always wanted to create something , and therefore you started taking android based courses. you have been doing android for over 2 years and today you know a lot of things in android. you might be good in other professional lines like web dev, data analytics, ml,ai, etc too if you give time to that"
>>>>+ve points :
- "you will love doing this, you always did"
- "With the support of a good team, you will always be able to complete tasks and build new things quickly"
- "Start ups might offer you the placement, they always need students with some good exposure"
>>>>-ve points :
- "Every established company which provides interesting dev work takes their first round as coding, and do not considers your extra curricular dev work. So you are placing your all hopes in 1 good start up with super offerings that would somehow be amazed by your average profile and offer you a position"
- "start ups are well, startups and may not offer a job security as strong as est. companies"
- "You are probably not as awesome dev as you think you are. for 2 years, you have only learned the concepts , and not launched more than 1 shitty app and a few open source work"

CHOICE 3: NON CODING
>>>>background: "companies coming in college placements have 1-2 rounds of aptitude,logical reasoning , analysis based questions and other non tech tests. There are also online tests available like elitmus,AMCAT, etc which, when cleared with good marks help receive placements from decent established companies like TCS, infosys, accenture,etc"
>>>>+ve points :
- "you will eventually get placed from college, or online tests"
- "there will be a job security, as most of these companies bonds the person for 2-3 years"
>>>> -ve points:
- "You really don't like this. These companies are low profile consultant/services based companies which would put you in any area: from testing to sales, and job offers are again $5000-6000 per annum at max"
- "Since it includes college, the other factors like your average cgpa and 1 backlog will play an opposing role"
- "Again, you are a 70 percentile avg guy. who knows you might not able to crack even these simple tests"

Ugh... I am fucking confused. Please be me, and help.The things that i wrote about myself are true, but the things that i assumed about super companies, start ups or low profile companies might not be correct, these points comes from my limited knowledge ,terrified and confused brain, after all.
:(

What's your favourite Data analytics/ visualisation tool? and why ?

Probably working for a scientific research firm and building in house tools for collection and analysis of data...

Any recommendations for books on statistical analysis and data science?

The more survival guide & short manual types, the better.

I had a pretty good year! I've gone from being a totally unknown passionate web dev to a respected full stack dev. This will be a bit lengthy rant...
Best:
- Got my first full time employment dev role at a company after being self-taught for 8+ years at the start of the year. Finally got someone to take the risk of hiring someone who's "untested" and only done small and odd jobs professionally. This kickstarted my career, super grateful for that!
- Started my own programming consulting company.
- Gained enough confidence to apply to other jobs, snatched a few consulting jobs, nailed the interviews even though I never practiced any leet code.
- Currently work as a 99% remote dev (only meet up in person during the initialization of some projects.) I never thought working remotely could actually work this well. I am able to stay productive and actually focus on the work instead of living up to the 9-5 standard. If I want to go for a walk to think I can do that, I can be as social and asocial as I want. I like to sleep in and work during the night with a cup of tea in the dark and it's not an issue! I really like the freedom and I feel like I've never been more productive.
- Ended up with very happy customers and now got a steady amount of jobs rolling in and contracts are being extended.
- I learned a lot, specialized in graph databases, no more db modelling hell. Loving it!
- Got a job where I can use my favorite tools and actually create something from scratch which includes a lot of different fields. I am really happy I can use all my skills and learn new things along the way, like data analysis, databricks, hadoop, data ingesting, centralised auth like promerium and centralised logging.
- I also learned how important softskills are, I've learned to understand my clients needs and how to both communicate both as a developer and an entrepeneur.
Worst:
- First job had a manager which just gave me the specifications solo project and didn't check in or meet me for 8 weeks with vague specifications. Turns out the manager was super biased on how to write code and wanted to micromanage every aspect while still being totally absent. They got mad that I had used AJAX for requests as that was a "waste of time".
- I learned the harsh reality of working as a contractor in the US from a foreign country. Worked on an "indefinite" contract, suddenly got a 2 day notification to sum up my work (not related to my performance) after being there for 7+ months.
- I really don't like the current industry standard when it comes to developing websites (I mostly work in node.js), I like working with static websites (with static website generators like what the Svelte.js driver) and use a REST API for dynamic content. When working on the backend there's a library for everything and I've wasted so many hours this year to fix bugs and create workarounds related to dependencies. You need to dive into a rabbit hole for every tool and do something which may work or break something later. I've had so many issues with CICD and deployment to the cloud. There's a library for everything but there's so many that it's impossible to learn about the edge cases of everything. Doesn't help that everything is abstracted away, which works 90% of the time but I use 15 times the time to debug things when a bug appears. I work against a black box which may or may not have an up to date documentation and it's so complex that it will require you to yell incantations from the F#$K
era and sacrifice a goat for it to work properly.
- Learned that a lot of companies call their complex services "microservices". Ah yes, the microservice with 20 endpoints which all do completely unrelated tasks?

I'm taking data structures and algorithm analysis class at college but don't know how to study... I need advice, by the way the language we're using is c++

I pitched this crazy sentiment analysis story and it got approved.
Which is great... only now, I have to start learning regular expressions to tokenize my data. Regular Expressions are the worst. 😒

Wanted to know what you guys think about "Dev" jobs that also include making slide shows and presentations?

Eg. Imagine working on an analysis engine. And just when the core functionality is working, your boss wants your team to make presentation for every client that it's going to be used for - using the raw data given by the engine.

Instead of maybe adding that function to the engine itself.

Suddenly your work is now 12+hours of MS office instead of 8 hours of coding.

And a year later. You have 10 unfinished skeleton code architectures, poorly documented and 90% of the test cases never written.

And most of the devs who were on the initial project have either left out of frustration or have been fired because apparently fresher's who can not code with a senior coder level proficiency is not performing well.

Sales guys in my company are unbelievable. They told client that my analysis algorithm runs for about one hour, so that is the time they have to wait for results. I asked if he actually asked what hardware do they have, and if they want all analysis done (its a complex algo) paralell, or just one by one.
Ofcoursenot.
Then I suggested to better ask these things, otherwise we end up reckt. Annnnd he said they will question our professionalism if we ask too much questions.
Ok so let me summarize. My algo needs to run maximum 1 hour long, regardless the actual functionality and input data, on any fcking hw you can imagine, or else i am not a pro. Mmmkay.

Rant:
My jupyter notebook has outgrown itself on some real world trading data analysis and its becoming a pain to add to (further) and share.
Need to find better alternatives, web apps where are you?

But i know nothing about it. Learning curve ahead!

Requirements:
I've 7 interactive dashboard plots (from some data) in jupyter-notebook.
- It'd be nicer to have a web app to use them without running notebook from a different location.
- Or running notebooks remotely (running as daemons on host machine).
Any suggestions for a starter ?
rant before requirements, coz rants lead to better requirements.

if rant++:
make_requirements(what_something)
do_work(that_something)

So, our lab professor in university for data structures was evaluating my friend's group assignment next to me and my colleague.

Professor: So, what does this method do?

Friend: I feel it would be best to explain our structure first and then how our methods work.

P: No, no. It's okay, I'm seeing it here in my laptop. *Looking at their report which included nothing about the structure of the assignment since she asked specifically only to place the methods and their complexity analysis due to not having time to look and evaluate a full report for each group*

F: Okay, *proceeds to explain*.

P: Okay, and why is your code commented?

F: Because it's good practice and we learned in subject x last semester.

P: Impossible! I teach that subject and I did not teach that in my classes.

F: Okay, but our professor did.

P: *Says nothing*.

They got a 3.25/5 as a grade for it and got really pissed off. I mean, they spent a lot of time making the assignment's structure perfect, optimizing their code and the professor did not care for the structure, just the methods, on the data structures subject.
FYI, they are workaholic, dedicate a lot of time improving their skills and their normal grades are usually >= 4.25.

If I wanted good feedback for my products (especially if they were geared towards developers) I would scrap devRant and do some data analysis on how our products are perceived in raw form. Would be very raw and informative insight indeed because you are at the heart of raging innovation (raging innovation: when a developer is so pissed at a flaw in a piece of software they highlight or fabricate an ingenious feature or solution) and will help not only iron the kinks out but make a better product all together. Also, of course the good aspects would be lauded.

In nearby county to mine, coder was arrested for 3 months for nickname matching leaked terrorist nickname "grower" by coincidence. His coding education was enough reason for arrest.

All hdd/ssd/usb/mobile devices were confiscated for thorough analysis right from the morning by police.

Feeding my security paranoia. Encrypting fully filesystem(LUKS) and my internet traffic (self raised open vpn), wiping fully usbs. I ll be protected from my ISP recording my traffic, and from unauthorised access to my data.

Sometimes I have to do some data analysis a I use R Studio.

Today I tried to do the same thins using KNIME.

It just like comparing playing with Lego Duplo after having tried Lego Technic.

Has anyone (who does either Data Analysis, ML/DL, NLP) had issues using AMD GPUs?
I'm wondering if it's even worth considering or if it's too early to think about investing in computers with such GPUs.

I'm starting to look at how to get devices to send data to an endpoint for storage and analysis. I'm looking at AWS iot stuff like core and green grass but then I'm thinking that a REST API could also do the job. I don't need to connect devices to each other (in first iteration). Dont think I need any edge stuff either. Anybody have any experience with this?

Data structure and Analysis

For experienced ones, add "System Design".

...path for big fat packages.

My client wants me to "help increase sales" by applying seo voodoo. For over two years they didn't even have Analytics. Now, after 12 days they're asking me what do I make of the results. Like that's a meaningful amount of data for analysis.

Does anyone on here use the 3D software blender. What kind of Python do I need to learn (learned data analysis for internship but 80% sure won't be same as animation Python)

Hey guys is there anything we as devs could do regarding this covid-19 situation in terms of our technical skills? Like any project or something for making awareness at global or local level, or some status update regarding various things/items that are affected by this pandemic?

I am looking for ideas or currently ongoing projects that i could contribute to. Currently my tech stack is limited to java, python , android and a little bit of basic data analysis, but i am fluid enough to learn and contribute.

everyone needs to be a data scientist/evangelist/superhero or AI enthusiast/developer/super-coder or project head for critical business needs/ or doing analytical analysis for business processes...even school students who are learning just to write English sentences, think they can code easily
AI folks, who think you can code automatically by thinking with no typing..

to them i say

Hi So I need some solid advice from you all wonderful people.
I think i am now ready to look into job side of this world, but have lots of doubts , read my story.
I have been learning android for last 2 years. Most of the time i have been trying to understand how stuff works in android , but i have also gained a few other skills ( python programming, kotlin/flutter basics data analysis basics, testing, some graphic designing, aweful web dev ,etc). But i really want to work with Android. I don't have any specific Salary figure in mind, but i guess my knowledge is better or atleast par with most of the good android developers.

So i want to know how is this fresher/placement thingy work?
1.) GETTING KNOWN? : How can i make some good android based company aware that I am available for hiring? Should i start emailing every android related company that i know of? Should i start listing my profile on recruitment sites like linkedin or internshala? This year it is being said that companies will come for placements. From the status of my college, they are going to give me way to less $ , nd i know am not going to like any of them, but i guess i have to sit for them too.

2.INTERVIEW OR DIRECT PLACEMENTS? A little pre-context: i am currently starting my 4th year in clg. Afaik , 4th year isnt that strict and their can be leniency in terms of attendance. But my college is a place full of political cun*s in the name of directors and HODs and I don't know if they are again going to enforce the old 75% mandatory criteria. Plus if the company is from a different state/country , then my attendance would definitely not suffice.
So mainly i am unsure if somehow a company hires me, i would be able to start immediately. I heard that there are interviews for job recruitment after which the candidate is binded with an agreement to do some months training followed by permanent working after college completion.
This type of agreement is very much suitable for me, since from what my friend tells me, trainings can be lenient and understanding regarding exam preparations nd stuff.

So what do company usually chooses? Binding a fresher on immediate working basis or do they consider graduate completion?

Also, i suck at competitive coding. Do i need to polish myself on that or some company is willing to give me chance on the basis of my other skills 🙈(okay, no kidding , that's a serious question. I need to either work on getting better in competitive or build more apps based on that)

3.) ANDROID OR EVERYTHING? From what i have heard, working as a professional fresher is more like being an allrounder than being a domain specialist. But as i already stated, i really dig android and that's no small framework. I may di other stuff too, but won't interest me nd my output might be less efficient than expected.
So freshers can really be asked to do any stuff? Or can i still be in the area i like being into?

4.) COMPANY OR START-UP? Yeah, this is a general debate starter. Ignoring the business side of the conversation ( job safety vs more salary, experience, etc) the thing that's most important for me is the presence of a team. I want someone to assign me a task, whose vision i could follow, from whom i could learn, and some other people who are supportive and doing the same amount / similar work that am doing . This is so much import8 for me that i can easily ignore other factors for a better team. I once took a call from a startup ceo who hired me, a 2 month old android beginner at that time, as the "lead android developer"
But if am being on a team where i am supposed to do any random stuff that is assigned, then obviously this whole point of "visionary, helpful leader, guiding team, "etc goes moot

Twitter api restricts me to fetch only 3202 tweets and if I had to fetch the retweeters count is limited to 100 .. is there any open source which will help me get through this problem because the data being collected is not sufficient for analysis

Question: what's your opinion on Julia? It seems promising, the macro syntax is neat. But would you use it for data analysis/management or general purpose development?

!rant

Does ayone knows of a

+ Good
+ Open Source
+ Free

Data test generator tool?

I'm on my first work with a development that will go to production, and I'd like to test performance, UI and all of that with random generated data. I know about dbSchema but it's trial pulls me back

do i have to learn advanced statistics to jump into data analysis and machine learning?

It seems like it never came to the mind of the developers of Apache Spark that people might want to use two of their libraries together to perform graph analysis on streamed Twitter data.

Why can't I simply use the streamed batches to create and extend a graph and perform continuous graph Algorithms on it??