Learn More
Resend Email
136
Your most emotional/intense day as a dev?
Week 53 Group Rant
T3hbeowulf
7y

Call comes down from the CEO and through his "Yes Man" that some investors are coming by to visit and he want to show off the data center and test servers. There are four full racks of storage servers filled with HDDs and each server has 4 to 16 HDDs a piece.
I got told to "make all of the lights blink", which can be epic seeing it in action but my test cycles rarely aligned that way.

All morning I was striping RAID arrays and building short mixed I/O tests to maximize "LED blinking" for the boss's henchman.

Investors apparently live/die by blinking light progress and it was all on me to get everything working.

undefined

wasabi

storage

investors

iometer

blinky

iscsi

wk53
Favorite
Ranter
Comments
  • 1
    7y
    You're the hero 😄
  • 1
    7y
    Hahaha this is amazing!!
  • 10
    7y
    Put some RGB in there for extreme blinking
  • 2
    7y
    Should have had this playing when they came down https://m.youtube.com/watch/...
  • 12
    7y
    Sadly I find this very believable
  • 0
    7y
    All hail the blinkenlights!
  • 5
    7y
    I would have bought led stripes and a couple of Arduinos 😅
  • 27
    7y
    When you're the front end dev of a data server
  • 1
    7y
    @lo98be This guy knows. Just order a reel of LEDs and go nuts!
  • 0
    7y
    @SpencerBeige nice pun!
  • 2
    7y
    Thanks for confirming that my prejudices about upper management types have a solid footing in reality.
  • 0
    7y
    Blinking lights apparently give the higher ups orgasms... Who knew...
  • 0
    7y
    Endless while loops on all, DONE
  • 2
    7y
    @lo98be Given that this task was given the same day as the visit, the fastest​ way to cause excessive but real disk activity was striping arrays and running IOmeter scripts. The blink patterns are mesmerizing.

    If I had more time, I would have considered building an LED matrix connected to the backplanes of each server and driven over the network. We were building our own iSCSI storage server software at the time and slipping a "test the access lights" endpoint would have been possible.
  • 1
    7y
    @T3hbeowulf the same day... 😂 It's incredible what managers think
  • 1
    7y
    @T3hbeowulf is it good for the drives to be working that hard?
  • 2
    7y
    I don't think it's a problem, disks should be able to run for​ decades, 2 hours of activity shouldn't do much
    Also testing them is routinely done, he just aligned all the tests
  • 0
    7y
    @tisaconundrum Vibration and electrical issues were the biggest enemies of drives in our data center. Heat and mechanical hardship didn't seem to affect much.

    The torture those test drives saw regularly​ would have caused very frequent premature death if "heavy load" adversely ​affected the life span.
  • 1
    7y
    @T3hbeowulf what do you mean by electrical issues?
  • 1
    7y
    @lo98be I use "data center" liberally when I describe that particular data center. It had racks, that is about all I can say that made it qualify.

    Electrically, we couldn't turn everything on at the same time or PDU circuits would trip and take random servers offline in the middle of heavy I/O. Drives usually didn't come back from hard power cycles like that.

    Later in that job tenure, my coworker and I took it upon ourselves to completely refactor the wiring and layout and solved many of the electrical issues in the racks.
  • 1
    7y
    Sounds like the servers in my school 😅 it's so broke that my networks teacher and I did it
Related Rants
devRant © 2021 Hexical Labs LLC
Privacy Policy  |  Terms of Service
Add Comment