Home > Blogs > VMware vFabric Blog

Banks Are Breaking Away From Mainframes to Big, Fast Data Grids

The world’s largest banks have historically relied on mainframes to manage all their transactions and the related cash and profit. In mainframe terms, hundreds of thousands of MIPS are used to keep the mainframe running these transactions, and the cost per MIP can make mainframes extremely expensive to operate. For example, Sears was seeing the overall cost per MIP at $3000-$7000 per year and didn’t see that as a cost-effective way to compete with Amazon. While the price of MIPS has continued to improve, mainframes can also face pure capacity issues.

In today’s world of financial regulations, risk, and compliance, the entire history of all transactions must be captured, stored, and available to report on or search both immediately and over time. This way, banks can meet audit requirements and allow for scenarios like a customer service call that results in an agent search for the transaction history leading up to a customer’s current account balance. The volume of information created across checking, savings, credit card, insurance, and other financial products is tremendous—it’s large enough to bring a mainframe to its knees.

Mainframe Jam can be Sticky

No, this is not some type of jelly you find in a mason jar at a farmer’s market in Armonk, NY.

Just like other data stores, a spike in resource requests can bring a system to a halt. When mainframes get jammed and go offline for a few minutes—it costs banks millions of dollars per minute. Mainframe jams eventually cause the executive team to drop what they are doing until they know the CIO and IT organization are all focused 100% on removing the jam and working to avoid it in the future.

What causes a mainframe jam? In the case of one of Latin America’s largest banks, it is simply the volume of data. Their mainframe runs transactions and creates logs for compliance, risk, and audit purposes. These logs are sent to a message queue and database that also runs on the mainframe hardware. If this database is overloaded, the messages queue up, and the system begins to slow down until it begins to deny transactions.

Making Big, Fast Data Scale for Financial Services Mainframes

What the bank needed is a way to ingest a large number of transaction log messages and never deny transactions. Valued at over $60 billion, their client’s transactions generate a message volume of around 200-300 million per day with peaks of up to 15,000 messages per second. With each message being about 4KB of data, this totals up to be about 1.2 terabytes per day of financial transactions.

This is big data for sure, but it was not yet fast data.

Architecture Approaches with Pivotal’s vFabric GemFire and Greenplum

The VMware vFabric team partnered with the bank to show them how this data jam could be avoided in the future by building out a proof of concept.

Below you’ll find an architecture diagram of the proposed solution. We used vFabric GemFire as the fast ingest layer to consume MQSeries messages right as they hit the server. This prevents messages from queueing up and also allows them to be asynchronously persisted to a Greenplum appliance for analytics. For the customer, this deployment worked like a black box because GemFire is embedded inside the Greenplum hardware appliance, so setup was minimal and easy.  vFabric tc Server, Spring Integration, and RabbitMQ could also be used to host various web services. .

Basically, this solution allowed the high volume set of log events and messages need to be stored on a persistence layer in GemFire outside the mainframe that could scale elastically as needed by simply adding more nodes.  Since the data is placed in-memory with GemFire, it also allows the data to available for search or reporting as soon as it leaves the MQSeries queue.

The bank saw several additional advantages to running a big, fast data grid outside the mainframe:

  • Running the messages outside of the mainframe would save MIPS and money for a set of data that was not considered core to the business operations.
  • The mainframe would be able to handle a much higher transaction throughput, avoid jams and have a greater ability to scale on it’s current hardware.
  • Business intelligence analyses could be run on the transaction logs. Once this data could be placed on a big data grid outside the mainframe, it would allow various departments to do analysis on customer profiles, usage characteristics, fraud, security, customer service, and marketing.

For more information on vFabric GemFire and Greenplum, see:

About the Author: Frederico Melo (a.k.a. Fred Melo) has a degree in Computer Science and has been working with Software Engineering for the last 14 years. His areas of expertise include Grid Computing, Highly Scalable Architectures, Big Data, Fast Data and Legacy Modernization. He is currently based in Sao Paulo, Brazil working as Field Engineer for Pivotal.

50 thoughts on “Banks Are Breaking Away From Mainframes to Big, Fast Data Grids

  1. Rick Wagner

    Thanks for this interesting post. I used to work in bank data processing in the 90s, but have since gone on to work in other parts of the IT world.

    I am surprised by a few things in this article. Probably the biggest is the size of each transaction– 4k! The mainframe banking systems I worked on were very good about keeping small, fixed-layout records passed. Data was squeezed as much as possible– fields might be re-used depending on a code in a header, for example. A record of a few hundred bytes would be a large one, small onces might be in the dozens of bytes.

    I’ll be interested to see if more front-end systems like the one you describe become popular. IBM doesn’t miss many tricks, I would imagine they will not be caught off guard. But I admire you for entering this work space, it is a worthy prize.

    Good luck, and please post again later to let us know how things develop over time.


  2. get ex back no contact

    Others might not see this do get your ex back coming? Be sure these
    rules and boundaries are understood ahead of time before the
    distance becomes an issue. Thhe absencee or presence of it can be so open,
    honest discussion, you get to openly do get your ex back and directly
    address the feelings behind the action or situation.
    If yoou hhave some support. Who [exactly] contacted you before,
    during and after you completed this segment of words?

    Here is my webpage get ex back no contact

  3. bangaiza.bernsoft.com

    Today, while I was at work, my cousin stole my iphone
    and tested to see if it can survive a 40 foot drop, just so she
    can be a youtube sensation. My iPad is now broken and she has 83 views.
    I know this is totally off topic but I had to share it with someone!

  4. سرور مجازی ایران

    Excellent post

  5. دانلود فیلم

    its good . and please visit my website : http://stop-koch.com

  6. b5he3vb027


  7. goodkandom.in


  8. goodkandom.in


  9. epsfi.in


  10. goodkandom.in


  11. i7networks.in

    opopioii vgvfggfgghygyujkl

  12. goodkandom.in

    sadad mvmvbnm pp[uku;kiurfoir kos madaret ryfiuyh54jklfuoriucvhrjgckuctre

  13. tnewslive.in

    ,nmbvnbm, oktgohthik re8turt89r8u9t8rt

  14. empbank.in

    fklgfgjk nfnvfefewuf

  15. goodkandom.in

    iortioretore jnfgjkgnjd laldasdksa

  16. خرید بک لینک


  17. یو پی اس

    it is great

  18. اخبار ایران و جهان

    🙂 very good.

  19. یو پی اس

    thanks vmware

  20. Snapback Caps

    A record of a few hundred bytes would be a large one, small onces might be in the dozens of bytes.

  21. پاور بانک

    خرید انواع پاوربانک و شارژ همراه

  22. خرید کاندوم

    خرید کاندوم

  23. باربری تهران

    باربری تهران با سرویس های اتوبار ویژه
    باربری سعادت آباد

  24. پرگولای چوبی

    Actually the era of mainframes is going to end.

  25. دانلود فیلم

    On-Premise and in the Cloud

  26. افزایش ممبر

    very nice

  27. Farsiapk.ir

    دانلود نرم افزار و بازی اندروید

  28. زیرنویس فارسی

    Thanks it was useful.

  29. ترجمه متن فارسی به انگلیسی

    cool content

  30. یو پی اس

    thanks vmware. so good

  31. mrsaeed

    Tnx vmware.

    see us in : https://ssa.co.ir/urban-tv/

  32. اتوبار


  33. اتوبار


  34. 3005 aluminum coil

    Good idea ,I like it!

  35. بازی انلاین

    Fast Data Grids is a important topic.

  36. بازی انلاین

    Mainframe Jam can be Sticky is a usefull part.

  37. لیزر کیست مویی

    Nice Your email

  38. تهویه مطبوع

    Air conditioning is also known as a delightful or pleasant airline. Air conditioning is a branch of mechanical engineering science, and its role is to create conditions in which human beings can feel comfortable in the working and living environment, and can easily process day activities or industrial activities in this space or process the product. Or keep them so that these activities take place regardless of weather conditions.

  39. آهنگ هندی جدید

    دانلود آهنگ هندی جدید

    1. embassy appointments safarito


  40. pamela

    owsome information

  41. آموزش ساخت فروشگاه با ووکامرس

    آیا می خواهید ساخت فروشگاه آنلاین خود را شروع کنید؟ ممکنه فکرکنید ساخت فروشگاه آنلاین می تواند یک فکر وحشتناک باشد. اما در آموزش ساخت فروشگاه با ووکامرس
    ، میبینید که این کار ساده است اما باید دقت داشته باشید و مراحل را بدرستی انجام دهید و مثل هر کار دیگری نیاز به تمرین و تجربه هم است. در اموزش ساخت فروشگاه انلاین، از ساده ترین نحوه ساخت فروشگاه اینترنتی رایگان استفاده شده و آن هم ساخت فروشگاه انلاین با ووکامرس است.

  42. ساندویچ پنل

    very good.

  43. فروشگاه لوله و اتصالات فولادستان

    Thanks for your site

  44. توسعه اپلیکیشن


  45. sinnoty.com


  46. Mayara

    Very nice

  47. خرید اینترنتی موکت

    thanks vmware

  48. فرش سجاده ای

    nice site

  49. Raphblog

    Thanks for sharing this information. Great Blog


Leave a Reply

Your email address will not be published. Required fields are marked *