By using RabbitMQ, SockJS, Cowboy, Erlang, and more, his team built a highly scalable, customizable solution for real-time comments based on websockets. His talk, Realtime Web @ HuffingtonPost, was for developers of real-time, Erlang-based solutions—he covered architecture, lessons learned, pitfalls, and future improvements. The presentation objectives included:
Why they went with Erlang and SockJS for the commenting platform
How they integrated RabbitMQ routing to power the subscription architecture
How they managed subscriptions for real time channels
Their plans to extend the framework into an open source solution
Where they want other mechanisms for publishing comments outside of RabbitMQ Continue reading →
In our last post, we 1) covered how geographic data can release value in mobile and machine-based applications, 2) explained how technology is used to overcome barriers to these types of big data scenarios, and 3) detailed the architecture for a data fabric or grid (like vFabric GemFire) that works with geographic data and specialized or alternative indexes. There were also code examples to explain the object model, the spatial index, and data changes.
Now, we will continue the examples, show you how to make the index highly available, and use a function to access the data via the index.
The Scenario for a Highly Available Index
In some cases, a piece of data may be added to a node, or become primary on a node without a clean method call. This happens in the cases of both failover and rebalancing. In the case of failover, a bucket that is on a node (that was also a redundant copy) may suddenly become the primary copy if the node that held the primary failed.
In the case of rebalancing, an entire bucket can be moved to a new node that was added to the system without the benefit of capturing the “put” call on each piece of data. Continue reading →
Apache Derby is used for its RDBMS components, JDBC driver, query engine, and network server.
The partitioning technology of GemFire is used to implement horizontal partitioning features of vFabric SQLFire.
vFabric SQLFire specifically enhances the Apache Derby components, such as the query engine, the SQL interface, data persistence, and data eviction, as well as adding additional components like SQL commands, stored procedures, system tables, functions, persistence disk stores, listeners, and locators, to operate a highly distributed and fault tolerant data management cluster.
Application and operations teams sometimes reach a point where they must upgrade the database. Whether it’s due to data growth, lack of throughput, too much downtime, the need to share data globally, adding ETLs, or otherwise, it’s never a small project. Since these projects are expensive, any recommendation requires a solid justification. This article a) characterizes 3 signs where traditional databases hit a wall, b) explains how vFabric SQLFire provides an advantage over traditional databases in each case, and c) should help you make a case for moving towards an in-memory, distributed data grid based on SQL.
For those of us tasked with upgrading (or architecting) the data layer, we all go through similar steps. We build a project plan, make projections and sizing estimates, perform architecture and code reviews, create configuration checklists, provide hardware budgets and plans, talk to vendors about options, and more. Then, we work to plan the deployment with the least downtime, procure hardware and software, test different data load times, evaluate project risks, develop back-up plans, prepare communications to users about downtime, etc. You know the drill. These projects can take months and consume a fair amount of internal resources or consulting dollars. If you are starting or working on one of these types of projects with a traditional database architecture in mind, are you considering these 3 signs as you consider your options? Continue reading →
The next release of Hyperic is coming up soon and the biggest change is to the backend. In the next release, we will only support one database, namely PostgreSQL. Those of you who have been with Hyperic for a while as long as I have may be surprised considering our history with PostgreSQL, but, as you read though this blog, it will start to make sense.
History of PostgreSQL and Hyperic
For the last few years Hyperic has supported only two databases for production use at scale—Oracle and MySQL. This in itself was a big change since at one point, PostgreSQL was our bread and butter. Hyperic was originally designed on PostgreSQL 7.x. As an open source project, PostgreSQL has a very easy license for distribution. As a startup company we had to get our product out into the marketplace quickly and affordably, so therefore PostgreSQL made sense.
How do you plan a roadmap for moving from a legacy data architecture to a cloud-enabled data grid? In this article, we will offer a pragmatic, three-stage approach. At SpringOne-2012, the “Effective design patterns with NewSQL” session (see presentation embedded below) generated a lot of interest. (Thank you to everyone who joined us!) Jags Ramnarayan and I discussed problems with legacy RDBMS systems, NewSQL driving principles, SQLFire architecture, application design patterns as well as data consistency and reliability.
We went deep into vFabric SQLFire which is a pragmatic solution that addresses these data challenges:
How do I architect my data tier for very high concurrent workloads?
How do I achieve predictability both for data access response time and availability?
How do I distribute data efficiently and real time to multiple data centers (and to external clouds)?
How do I process these large quantities of data in an efficient manner to allow for better real-time decision-making?
If you haven’t heard, the SpringTrader reference architecture is used to help Java-based application architects, developers, infrastructure, and operations teams advance their application roadmaps and provide reusable patterns. Some might also consider how vFabric Application Director can be used with the SpringTrader app to enable continuous deployment or automatically provision and scale the app in a completely virtual data center (i.e. a software defined data center). As well, vFabric Application Performance Manager can be used to monitor the entire stack and trigger automated scaling events like adding a new JVM and tc Server to the SpringTrader app’s production environment. Continue reading →
vFabric GemFire is a sophisticated product in a complex problem space: data management in distributed systems. In order to help our users get the most out of GemFire, we are starting a “cookbook” series, which will provide tried and tested recipes that we hope every GemFire user will find useful.
Our first topic is the Visual Statistics Display (VSD). VSD is a visual tool for analyzing GemFire statistics. It reads GemFire statistics from special statistics archive files created by GemFire, and renders their graphs for analysis. It is not a real-time online monitoring tool, such as vFabric Hyperic, so it does not have the real-time monitoring and alerting capabilities that they have. On the other hand, it is the most powerful tool for examining the state of a vFabric GemFire system, as it provides access to all the statistics collected by GemFire. No real-time monitoring tool can do that, as the amount of statistics that GemFire collects is prohibitive for real-time collection in a distributed system.
We’ve talked to dozens of people, and the theme we keep hearing is simplicity, simplicity, simplicity.
Many amazingly bright application architects have stopped by to understand and learn more about the vFabric application architecture, and these folks hail from a number of industries – giant telecom manufacturers, government ministries of defense, and multi-industry service companies to name a few.
These conversations with architects have tended to fall into one of the falling categories: Continue reading →
Today, VMware announced the release of VMware vFabric Suite 5.1, an integrated family of components that streamline how companies build, run and manage Java Spring applications whether on-premise or in the cloud. At VMware, we understand that smart, agile and cost-effective cloud infrastructure services are important. But ultimately, it’s about the applications that run atop that cloud infrastructure.
This latest release of vFabric Suite puts the largest set of modern cloud services within easy reach for companies that need to build cloud-ready Java applications that are deployed easily, scale quickly, and use popular open source application infrastructure technologies.