Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
-
@NoMad
I also need to do data warehousing. It is a new thing for me. Any advise on what to look out for or any book you can recommend it to me? -
@mr-user yah. Don't take the data you don't need. Any data you gather should be useful to some report and have a reasonable life cycle. Don't gather mini-crap. Also, don't even think of data mining while doing warehousing. They're different activities. Books, not sure. I sat through lectures so maybe you could see some in YouTube or something.
-
@NoMad Thank you for your info.
How about the data which does not have any usage but they may be used in the future?
What do you mean by mini-crap?
I am trying but every resource I found is just abstract. I cannot seem to link back the abstract concept into "the implementation side"
Daghhhhhhhh Kafka.
Set it up, seems to work fine.
Oh no...! Take a broker down, then messages go missing - hmm, that's not right. Fine, I'll just look into... Ah, bad replication factor, my fault. So then it's all fixed! Woop. Wait, no. Some messages still going missing occasionally. Oh, only set to "at most once" delivery. My bad, fix that, and... now everything is out of order. Oh, ok, partitions setup wrongly. Wtf, now the whole thing stalls when there's a network blip until a restart. Right, ok, looks like commits have to receive acks in the library I'm using before continuing. Switch to a library that uses CommitWithoutReply. Brilliant....
Apart from said library seems to have commits failing all over the place because it keeps trying to commit during a rebalance 🙄😒😤
The frustrating thing is I KNOW for a fact that Kafka is a fault tolerant, resilient, horizontally scalable thing capable of handling stupid amounts more than I'm throwing at it without missing a beat. But damn,configuring it, and checking you've configured it sanely is a royal, monumental PITA.
rant