Ranter
Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Comments
-
addvilz4428y@fuckfuckityfuck any proven, robust, ACID compliant database and any language with static types and good concurrency support, depending on what are the requirements.
-
brod100338y@addvilz nope 😶 I've got mongo and node to >50,000 inserts per second on a single database. These are very hot documents and have never ran into atomic errors.
(My use case is more like 2,000/sec but peaked at 18,000 with no issue)
I should also note concurrency / locking isn't great - but I'm sure any of those issues could be resolved.
What read/upsert/insert numbers are you aiming for? -
addvilz4428y@brod No they can not, because Mongo is not ACID by design. Multi collection updates? Rollbacks? It is not normal that I need to handle concurrent modification exceptions on upserts in app level, not to mention that with flush assurance on a cluster your 50k ops per second is perhaps possible only with aws xlarge memory machines - and even with all that, operations are still not guaranteed to succeed because some official drivers are garbage at handling errors and atomicity is guaranteed for single process only.
Im talking about millions of upserts per second for quite reasonable mongo cluster. And I am not aiming - long there and way past.
My rant isn't something I imagined. It has a real implementations and years with mongo to it, handling datasets with billions and trillions of documents in them. -
brod100338y@addvilz well damn, I hear what you're saying, especially in terms of ACID compliant transactions.. and a million/sec upsert does blow my mind a bit.
I have a couple ideas to solve isolation for multi doc transactions which afaik is the only non-acid part of mongo. Tomorrow I'm gonna sit down with the db guys at work and see what we can pull off and at what cost. -
addvilz4428y@brod well there are two ways - use a meta document for locking, use versions like in Java Morphia, or use ACID db for transactional data and mongo for the rest.
-
brod100338y@addvilz I lost this post until now, but we got this to work, no cluster, atomic and throttled to 50k write/s. Easily 1m writes with a better environment.
I don't fully agree with the validation methods of atomicity, I'll dig up the project file and put it on GitHub.
Related Rants
You used Node and Mongo to handle highly concurrent financial transactions. What where you thinking.
undefined
node
reliability