Home > Blogs > VMware vFabric Blog

3 Key Stages to Evolve from Legacy Databases to a Modern Cloud Data Grid

How do you plan a roadmap for moving from a legacy data architecture to a cloud-enabled data grid? In this article, we will offer a pragmatic, three-stage approach. At SpringOne-2012, the “Effective design patterns with NewSQL” session (see presentation embedded below) generated a lot of interest. (Thank you to everyone who joined us!) Jags Ramnarayan and I discussed problems with legacy RDBMS systems, NewSQL driving principles, SQLFire architecture, application design patterns as well as data consistency and reliability.

We went deep into vFabric SQLFire which is a pragmatic solution that addresses these data challenges:

  • How do I architect my data tier for very high concurrent workloads?
  • How do I achieve predictability both for data access response time and availability?
  • How do I distribute data efficiently and real time to multiple data centers (and to external clouds)? 
  • How do I process these large quantities of data in an efficient manner to allow for better real-time decision-making? 

At VMware, we understand that adopting a new technology can be a daunting task.  With SQLFire, one can take an evolutionary approach. You can start with 1) an embedded or distributed cache, 2) move to or add a full-fledged OLTP data store, and 3) then enable a global, cloud database or distributed compute grid (even for map-reduce style computing)… at your own pace. This approach is both popular with customers and based on quite a few years of experience working in this space with hundreds of mission-critical systems and their respective companies.


As an Embedded, Clustered Java Database
To get started, you can embed SQLFire into Java applications by including the required SQLFire libraries. When the application initiates a connection to SQLFire, it starts a peer server that joins other peers in the same cluster. Unlike other embedded databases such as H2 or Derby, SQLFire allows several servers to store replicated and partitioned tables, persist data to disk, communicate directly with other servers and participate in distributed queries. In this diagram, we show how the SQLFire database is embedded in a tcServer application server and is still part of the overall data fabric.

With Write-thru Distributed Caching
When you need to boost application performance and reduce network traffic, you can use SQLFire as a distributed cache to offload your existing DBs. Applications can use a familiar SQL syntax to read from and write to the distributed cache.

As depicted in the diagram below, whenever your application looks for a record that is not in the cache, SQLFire will use a RowLoader which is the callback component associated with the table you are querying. Then, the RowLoader transparently reads the record from the external database using its primary key, updates the cache and returns that record to your application.

With the write-through pattern, all SQLFire changes can be synchronously written to the external database before the cache is changed via a Database Writer. Because SQLFire can participate in the container transaction, if the write-through succeeds then the data becomes available in the cache.  In this diagram, an InsertWriter is registered on the Flights table, an ‘Insert flight 747’ will trigger the callback event handler of the InsertWriter to synchronize with the external database first and then insert into SQLFire Flights data partition.

Distributed Caching with Asynchronous Writes to DB
If synchronous writes to the backend are too costly, your application can asynchronously write to the external database using a cache listener.  For high availability purposes, you can designate a primary listener with one or more stand-by instances to ensure that the solution continues to run even under failure conditions. Whenever the external database is not available, your application will continue to run as SQLFire queues up events until the external database comes back up.  Once that happens, the listener will forward data changes to keep the external databases up to date.  In this diagram, we show SQLFire persistent, redundant queues with the primary listener writing in asynchronous batches to a legacy database.

Step 2: Full-Fledged, Scalable OLTP Data Store (or In-Memory Data Grid)

Because of SQLFire’s performance, availability, and reliability guarantees, you may no longer need legacy databases behind SQLFire.  Given the fact that SQLFire relies on standard SQL, JDBC, and ADO.NET, applications designed for traditional databases can easily migrate to SQLFire. This means you can add a sophisticated distributed data platform to applications whose requirements are thousands of transactions per second, sub-millisecond response time, and linear scalability. This pattern can greatly simplify your data architecture and reduce maintenance costs as well as operational overhead.

In this diagram, we show colored boxes as representing replicated or partitioned data across SQLFire servers. SQLFire’s shared-nothing architecture is designed to prevent any single point of failure as it maintains data availability and consistency even when servers go offline unexpectedly.

Step 3: Global, Cloud Database or Distributed Compute Grid

SQLFire as Global, Cloud Database
SQLFire supports global WAN connectivity and gives you the option of replicating data across data centers and cloud providers. This multi-site topology makes geographically-distributed clusters appear as one global system. If one of the SQLFire clusters fails or is taken offline for maintenance, clients in that geographical region can failover to the next available cluster and continue to operate.

This makes SQLFire the ideal solution for disaster recovery and business continuity purposes.

The figure below shows three globally distributed sites (New York, Tokyo, and London), each site hosting a SQLFire cluster. Within each site, a gateway is configured to provide data distribution between sites in case of a failure event, or for consistent global views of data around the world.

SQLFire as Distributed Compute Grid  (Real-time, in-database Map-Reduce)
Using SQLFire stored procedures, you can implement business logic at the server level that runs on the same process space as your data and in parallel on multiple SQLFire servers, thus significantly improving application performance and scalability. The capability here is basically a real-time, in-database map-reduce.

Because the SQLFire clusters consist of multiple servers, the stored procedure execution in SQLFire is parallelized to run on multiple servers, concurrently. SQLFire makes the whole map-reduce process in real-time and transparent to your application. As well, the ability to write functions and procedures in Java brings the complete set of Java APIs and the power of Spring into your SQL environment as server side logic. There is a tremendous amount of power, speed, and scale in this architecture pattern as we have seen in Hadoop examples. In this diagram, we show how an analysis job can be executed in parallel across SQLFire servers as Java stored procedures.

One thing is clear. If data is locked up in heavyweight and legacy RDBMSs, organizations can’t realize the benefits promised by cloud computing.

Now you have a very good idea of how to incorporate SQLFire into your architecture at your own pace to continue your journey towards cloud computing. Go ahead and test-drive SQLFire. It will be worth your time.

Thank you for reading!

> > For more on SQLFire:

Spring One 2012 Presentation – Effective design patterns with NewSQL

About the Author: Guillermo is an award-winning Enterprise Architect with 17+ years of progressive experience in different industries. As a Regional Senior Systems Engineer for VMware’s Cloud Application Platform division, Guillermo works with customers to understand their business needs and challenges and helps them seize new opportunities by leveraging vFabric to modernize their IT architecture. Guillermo is passionate about his family, business, technology and soccer.

28 thoughts on “3 Key Stages to Evolve from Legacy Databases to a Modern Cloud Data Grid

  1. Girish Gurudutt

    Is RDBMS still essential in the cloud era?how can we solve some if not all problems without RDBMS and reduce the cost while increasing the performance.

    1. Chris DeLashmutt

      For new development, I would agree with you Girish. It is possible that your new application might not even need an RDBMS. However, there is a lot of legacy and momentum out there that will require a lot of folks to live in a world split between that legacy data living in an RDBMS (and all the legacy still pointing at it), and more modern data architectures.

      I think the great news is that technologies like SQLFire and GemFire give you the option to co-exist relatively peacefully with legacy, while allowing you a transition to modern data fabric architectures that are more suited to high scale applications.

  2. Pingback: 3 Signs Your Relational Database Must Go | VMware vFabric Blog - VMware Blogs

  3. Pingback: Scaling and Modernizing .NET and Java: SQLFire Performance Test Blows Away Traditional RDBMS | VMware vFabric Blog - VMware Blogs

  4. Pingback: VMware vFabric Blog: Scaling and Modernizing .NET and Java: SQLFire Performance Test Blows Away Traditional RDBMS | Virtualization

  5. Pingback: 3 Key Stages to Evolve from Legacy Databases to a Modern Cloud Data Grid | VMware vFabric Blog – VMware Blogs | The Daily Kebab

  6. دانلود نرم افزار

    very nice

  7. عکس هنرمندان

    very well

  8. evolvecorp.in

    sadsadopasokdop opepodfoefod

  9. طراحی سایت

    Though withdrawal from your lover can be as powerful as withdrawal from cocaine, there are ways to deal with heart-breakers other than by kissing your existence goodbye:

  10. کاندوم خاردار


  11. جهیزیه عروس

    اگر به دنبال خرید جهیزیه عروس هستید می توانید از نوین جهاز که مرجع خرید لوازم آشپزخانه است خریداری کنید

  12. azmoon.org

    Thank you ever so for you article post.Really looking forward to read more.
    pls see my site

  13. Office installation 365

    Installation begins downloading into a spot after affirming SharePoint. The document location is still already stored. It is going to enter your browser folder for those who have not chosen anywhere to get into the installation file by default.

  14. james Smith

    Elucidating the Cloud Enterprise Architecture (CEA) for Smarter Enterprises … Finally, standards-compliant data analytics platforms and appliances are to extract … In short, technologies for enabling minimization, integration, federation, … to create a workable and worthwhile plans and roadmap to reach their business vision .

  15. Jasmin Samule

    I am ready for VMware exam when i am find Latest VMware certification exam then i find this site Crack4sure .com My Friend Already told me for Crack4sure is the best certification site and very used to couponcod
    today i am finale decide i purchased VMwares certification exam in this site
    i really very impressd for reviewed for good

  16. خرید موکت

    خرید موکت عمده

  17. roku.com/link

    roku.com/link this is simplest way to stream entertainment to your TV. On your terms. With thousands of available channels to choose from for more detils.

  18. roku activation link

    Roku gives the least troublesome way to deal with stream redirection to your TV. On your terms With countless channels to peruse.

  19. Office.com/Setup

    General process of Office.com/Setup is already mentioned above, however from the starting till the last process of activating you may be face some technical issues.

  20. Norton.com/Setup

    Norton setup downloaded from the norton.com/setup page helps to work and explore the ever-changing digital landscape. Norton makes it as easy to protect and manage your devices while installing an application on your smartphone.

  21. webroot.com/safe

    Webroot.com/safe is the only security solution which can protect your system from viruses, worms, malware threats. Get Webroot safe today & install on your system and say goodbye to viruses.

  22. www.avg.com/retail

    AVG is the one of best antivirus which protects your computer and network from bad infection and threats. AVG now includes real-time security updates, scans for both malware and performance issues, and even catches malicious downloads before they reach your PC. User can Install AVG Antivirus on their computer to protect thier computer and network…

  23. www.avg.com/retail

    AVG AntiVirus is a family of antivirus software developed by AVG Technologies, a subsidiary of Avast. It is available for Windows, macOS and Android. AVG ensuring your safety and security online. In AVG features, most of the common functions available in modern antivirus and Internet security programs, including periodic scans. AVG Online Shield also ensures the safety of exchanging files via instant messengers and VoIP clients.

  24. www.avg.com/retail

    AVG line-up combines effective malware eradication with fast scans and just about every security feature you could wish for. Simply activate AVG license and install AVG on your system to keep your system protected.

  25. webroot.com/safe

    It is very helpful to secure your device and it is very light weighted antivirus product.
    Webroot secure anywhere offers a user-friendly interface which provides several functions & they all work together to make a system protected. To activate Webroot go to webroot.com/safe
    Webroot security is a great antivirus program, which amazingly lightweight tool with accurate URL blocking and powerful bonus features.

  26. www.webroot.com/secure

    Webroot, a Carbonite company, harnesses the cloud and machine learning to reliably predict and protect against cyberattacks in real time.

  27. www.trendmicro.com/downloadme

    Trend micro has become a superior antivirus program over many antiviruses due to its advanced protection features. trend micro download process becomes easy when you purchase the subscription.


Leave a Reply

Your email address will not be published. Required fields are marked *