TL;DR: disaster averted!

Story time!

About a year ago, the company I work for merged with another that offered complementary services. As is always the case, both companies had different ways of doing things, and that was true for the keeping of the financial records and history.

As the other company had a much larger financial database, after the merger we moved all the data of both companies on their software.

The said software is closed source, and was deployed on premises on a small server.

Even tho it has a lot of restrictions and missing features, it gets the job done and was stable enough for years.

But here comes the fun part: last week there was a power outage. We had no failsafe, no UPS, no recent backups and of course both the OS and the working database from the server broke.

Everyone was in panic mode, as our whole company needs the software for day to day activity!

Now, don't ask me how, but today we managed to recover all the data, got a new server with 2 RAID HDDs for the working copy of the DB, another pair for backups, and another machine with another dual HDD setup for secondary backups!

We still need a new UPS and another off site backup storage, but for now...disaster averted!

Time for a beer! Or 20...

That is all :)

This is not an interview test just an awkward experience in general regarding interview.

This happened two years ago when I was a fresh university graduate looking for a job in UK as an immigrant (Im EU national).

Went to an interview for a web dev+tech support position. Two fat guys with tshirts met me and started interviewing me for a sysadmin position. Started asking me about disaster recovery and stuff.

Turns out recruiter messed up not only companies but positions as well. Also these two guys didnt bother to check anything.

I pulled out the job ad for which I applied originally, interviewers had a look at it and still proceeded questioning me while knowing that I prepared for completely different position interview.

Needless to say, it went terrible and I didnt get the job. I dont know if its just me or Im unlucky, but I had a lot of encounters in UK with so many incompetent recruiters.

Investigating a "bug"(turned out to be user error) on the production servers, accidently wiped out the permissions for 1,800 users.
Thankfully was able to recover them in under 10min. God bless a solid disaster recovery policy.

"UPDATE table SET field = 1"
Neglected to include "WHERE ClientId = XXX"

1800 users loose access to thier software in the middle of the day

Restored in 10 minutes thanks to disaster recovery policy.

Full stack developer.

I know what it's supposed to mean, but I feel like it gives discredit to the devs who perfect their area (frontend, backend, db, infrastructure). It's, to me, like calling myself a chef because I can cook dinner..

The depth, analysis and customization of the domain to shape an api to a website is never appreciated. The finicle tweaks on the frontend to make those final touches. Then comes a brat who say they are full stack, and can do all those things. Bullshit. 99.9% of them have never done anything but move data through layers and present it.

Throw these wannabes an enterprise system with monoliths and microservices willy nelly, orchestrate that shit with a vertical slice nginx ssi with disaster recovery, horizontal scaling, domain modeling, version management, a busy little bus and events flowing all decimal points of 2pi. Then, if you fully master everything going on there, I believe you are full stack.

Otherwise you just scraped the surface of what complexities software development is about. Everyone who can read a tutorial can scrape together an "in-out" website. But if your db is looking the same as your api, your highest complexity is the alignment of an infobox, I will laugh loud at your full stack.

And if you told me in an interview that you are full stack, you'd better have 10+ years experience and a good list of failed and successful projects before I'd let you stay the next two minutes..

Blizzard coming this weekend. I get to spend the night at work because our disaster recovery plan isn’t in full swing (Yes it’s paid, but I am going to lose my mind). Awesome. Suck my ass, winter.

Don't use your senior software engineer title or years of experience as reasons in a debate or argument about software

My manager was asking me what steps needs to be done to perform a disaster recovery for our cluster( on production). I will be honest here, I have not maintained this type of cluster(kubernetes) in production before. However, I have enough understand of the system to answer my boss question. I basically told him there are A, B, C you have to do.

My senior developer jumped in and said "No you should do A,C, B because C is more critical than B. " I then replied to him: "I understand your point, I notice that too, but .." Before I can even finish my sentence, this dude has already rolled his eyes and interrupted me very loudly: "Have you worked with these systems on production before? I did". The asshole knows I haven't maintained Kubernetes on production yet of course.

I got super pissed at him and pretty much shouted back to him and my manager: "Just because I haven't worked on this system on production yet, does not mean my argument is wrong" .

I then dragged my managers, that asshole, and other engineers in a room and settle this out. In the end, people agreed with my steps over that asshole senior engineer dude because I gave them rational reasons.

The conclusion is: Your senior title is given by the company, It doesn't mean anything to me. Also, it doesn't make you more right than another person just because he has a "lower" title than you.

TL;DR - (almost) childhood trauma due to Wesrern Digital crap products lead to lot of data loss and a plege to not trust or purchase their products for the rest of my life.

....

So, I got my first ever Wester Digital 2TB Mybook, back when 2TB was a really big thing. While in the midst of moving (not copying) a LOT of data to it, the damn disk just.. died. There was no fall, no power outage, no damage, it just stopped working. I was out of words and out of options. Tried yanking out the disk and connecting it directly to a system, but no luck because it looks like it's the HDD mobo that died.

Also stupid young me did not realise back then that, even if a "moved" the data, the original data is still most likely in their original location, and so, never bothered a recovery.

Lots of good stuff lost that day.

And as with a lot of you, my disaster recovery system kicked up 10 fold. Now I got redundant local and cloud backup copies of all critical and otherwise unattainable data.

As you may have guessed, I never bought another Wester Digital product ever again. My internal HDDs are Segate, and external is a suprisingly long lived Toshiba Canvio.

A few months ago I applied for an IT Support role managing computer systems for a smaller manufacturing corporation. Now some back story, I'm a recent college grad looking for work and this hit my radar. I did well in the phone interview and really enjoyed the in person interview as well.

However, if I was offered the role I'd be the only person working on their infrastructure. The person who I interviewed with was leaving and thus his position was available. It was kinda strange to interview with the person you'd be replacing.

I started asking questions about their critical infrastructure and how they manage it. Short answer is they don't know.

I asked about off-site disaster recovery. "Oh we back everything up to a 2TB disk and I take it home every day."
I asked "What if that backup fails?"
Their response was "That would suck."

The company decided to go with a managed IT solution instead of me as I don't have the required experience in their eyes. The previous guy left because they we're stuck in their ways.
Yah, no thank you.

Serious question guys.

How do you deal with stress of f-ing up at work?

I had to upgrade a whole postgresql stack today. Most of the upgrade went fine, but... Restoring a backup by pg_basebackup lead to an unusable database (would not replicate), had to apologize to the client and make last second modifications as disaster recovery, and all the while, ever since the DB didn't start up, to when I eventually went back to work and was no longer alone on the task, I was going through a crippling anxiety...

I... Love the job, but incidents like this... Make me doubt my future as anything more than a mediocre sysadmin...

Who the fick asks for BPMN diagrams, Disaster Recovery diagrams and Business Continuity Plan for a SaaS product that is deployed in cloud (Azure) ??

This is a simple app/dashboard that just showcases some data in fancy charts. It shows the data YOU feed. Why do you need BPMN diagram? What am I supposed to put? "Client loads data" and "user sees data"?

I already linked Azure's page on DR. Do yiu want me to copy-paste the contents from that link?

Clients are too much PITA!!!

No idea what the fuck just happened, but my home router just dropped the internet connection and started demanding that I change the admin (default) password.
Now, I know that default passwords are bad and all that, but why the fuck now? This thing has been sitting there for over a year, and it only decided to complain now?

There have been some weird things going on lately, and I'm starting to worry that some of my systems may have been compromised in some way... but I'm not sure what/how, nor how to look for it...

Any tips for identifying a breach and disaster recovery?

Been asking the devops guy to schedule database backups for some of our side services for weeks! Just waiting for the day it all goes to hell and have no disaster recovery plan to fall back on.

So I decided to have some funzies over the weekend. Started XCOM 2. Built a team for a mission, spent an hour customising. Then I spent 3 hours going through and not failing miserably even once. The last fucking part of the mission, my favourite sniper dies after getting hammered in the face with a fucking laser beam. Then I lose control of the battlefield and everyone dies. Start again because I'm NOT gonna lose my awesome sniper!

This game is like designing software, planning deployment strategies, and disaster recovery at the same time. Recommended for all devs!

Is disaster recovery capability to meet business requirements and testing it part of your new system implementation roadmap?

Hey if your company refreshing / retiring laptops for the latest greatest - you might want to think about using for disaster recovery. Store them in another location or home. Push out updated image automatically. Recently saved a company 400k recurring expense with this strategy.

When you roll out a new system is disaster recovery testing on your roadmap? For example, do you do failover and have users work through a comprehensive test script and have them sign off before going into production?

Two new paragraphs in our Contingency and Disaster Recovery Plan.

When you identify your most critical process and do insufficient mitigation against loss. Bad.

Try leveraging retired laptops for disaster recovery. Companies can realize resilience and save a fortune doing this. Hey, if everyone evacuated your office right now could you continue to keep your critical processes going?

If you had to evacuate your office due to a fire without your laptop and the office was destroyed could you continue to work? Do you have offsite recovery laptops?

Developers usually have flexibility where they can work from but sometimes do not have backup machines available and configured.

As a side question - would your other critical processes - accounting, HR, sales be able to continue to keep your business going in a disaster - or would they be like deer in headlights?

Microsoft is responsible for protecting the Office 365 physical and virtual infrastructure and ensuring availability. Although Microsoft addresses certain security threats, it cannot prevent all malicious threats. Businesses are responsible for protecting their data. This means that if a business’s Office 365 data is compromised or corrupted, it is not Microsoft’s job to restore the data outside of the Software Licensing Terms. To protect data, businesses need to make sure they have office 365 disaster recovery and recovery plans in place

All day developing a solution for a major incident to be told 'thanks but don't deploy, we only wanted to show we had the capability' FFS I have a project deploying on Monday!! I have shit to do!