21
addvilz
7y

You used Node and Mongo to handle highly concurrent financial transactions. What where you thinking.

Comments
  • 1
    Have you ever have a bad experience like this?
  • 0
    What would you have used?
  • 0
    I don't see the issue, mongo and node are reliable af if you know how to use them
  • 1
    @brod you are kidding, right? That was a sarcasm? RIGHT?!...
  • 0
    @fuckfuckityfuck any proven, robust, ACID compliant database and any language with static types and good concurrency support, depending on what are the requirements.
  • 0
    @addvilz nope 😶 I've got mongo and node to >50,000 inserts per second on a single database. These are very hot documents and have never ran into atomic errors.

    (My use case is more like 2,000/sec but peaked at 18,000 with no issue)

    I should also note concurrency / locking isn't great - but I'm sure any of those issues could be resolved.

    What read/upsert/insert numbers are you aiming for?
  • 0
    @brod No they can not, because Mongo is not ACID by design. Multi collection updates? Rollbacks? It is not normal that I need to handle concurrent modification exceptions on upserts in app level, not to mention that with flush assurance on a cluster your 50k ops per second is perhaps possible only with aws xlarge memory machines - and even with all that, operations are still not guaranteed to succeed because some official drivers are garbage at handling errors and atomicity is guaranteed for single process only.

    Im talking about millions of upserts per second for quite reasonable mongo cluster. And I am not aiming - long there and way past.

    My rant isn't something I imagined. It has a real implementations and years with mongo to it, handling datasets with billions and trillions of documents in them.
  • 0
    @addvilz well damn, I hear what you're saying, especially in terms of ACID compliant transactions.. and a million/sec upsert does blow my mind a bit.

    I have a couple ideas to solve isolation for multi doc transactions which afaik is the only non-acid part of mongo. Tomorrow I'm gonna sit down with the db guys at work and see what we can pull off and at what cost.
  • 0
    @brod well there are two ways - use a meta document for locking, use versions like in Java Morphia, or use ACID db for transactional data and mongo for the rest.
  • 0
    @addvilz I lost this post until now, but we got this to work, no cluster, atomic and throttled to 50k write/s. Easily 1m writes with a better environment.

    I don't fully agree with the validation methods of atomicity, I'll dig up the project file and put it on GitHub.
Add Comment