In a previous post, we explored a new class of products called in-memory data grids, that provide scalability and performance for data access in today’s cloud applications.  In this post, we’ll walk through using one these products, Hazelcast, as an in-memory data grid backed by a relational database as a persistent store.

It should be noted that this is not an endorsement for Hazelcast.  Hazelcast is a very good open-source product in this space, but there are other products, both commercial and open-source, that offer the same features or more.  We chose Hazelcast for this post because its a lightweight, open-source solution with an easy to understand API.

Two best of breed commercial products in this space are VMware vFabric Gem Fire and Oracle Coherence.  Give both of these products a look (or let me know and I’ll help you) if you’re interested in enterprise-grade in-memory data grids.  Our example uses Hazelcast, but the concepts are applicable to other in-memory data grids.

Prerequisites

There are a few assumptions and prerequisites for this example.  For this example, we’re going to create a Java main class that starts the Hazelcast datagrid.  The project is created in Spring Source Tool Suite, uses Maven 2 for build and dependency management, and uses Spring 3.x for dependency injection, and MySQL for persistence.  If you’re not comfortable with any of these technologies, it’s probably worthwhile to become familiar with them.

Maven Configuration

The first step is to add the dependencies for Hazelcast, Spring Framework, and MySQL to your Maven POM.  Of course, adjust the versions and Spring and MySQL dependencies accordingly:

<dependencies>
  <dependency>
    <groupId>com.hazelcast</groupId>
    <artifactId>hazelcast</artifactId>
    <version>1.9.3.3</version>
  </dependency>
  <dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-core</artifactId>
    <version>3.0.5.RELEASE</version>
  </dependency>
  <dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-jdbc</artifactId>
    <version>3.0.5.RELEASE</version>
  </dependency>
  <dependency>
    <groupId>commons-dbcp</groupId>
    <artifactId>commons-dbcp</artifactId>
    <version>1.4</version>
  </dependency>
  <dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <version>5.1.16</version>
  </dependency>
</dependencies>

Domain and DAO Classes

The next step is to create the domain and data access classes for accessing our database and back the Hazelcast cache.  For this example, we’re going to use a music artist whose information is already stored in the database.  The domain model is simple, but can be easily extended.  We’re also using Spring’s JDBC template for data access; any data access technology can be used, including JPA and JDO.  There’s nothing very interesting about these classes.

The domain class is:

package net.avantia.datagrid.domain;

/**
 * Domain object for artist.
 * @author Brian Jimerson
 *
 */
public class Artist {

	private Integer id;
	private String name;
	private String url;

	/**
	 * @return the id
	 */
	public Integer getId() {
		return id;
	}
	/**
	 * @param id the id to set
	 */
	public void setId(Integer id) {
		this.id = id;
	}
	/**
	 * @return the name
	 */
	public String getName() {
		return name;
	}
	/**
	 * @param name the name to set
	 */
	public void setName(String name) {
		this.name = name;
	}
	/**
	 * @return the url
	 */
	public String getUrl() {
		return url;
	}
	/**
	 * @param url the url to set
	 */
	public void setUrl(String url) {
		this.url = url;
	}
}

And the DAO is:

package net.avantia.datagrid.dao;

import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.List;

import javax.sql.DataSource;

import net.avantia.datagrid.domain.Artist;

import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.RowMapper;

/**
 * JDBC Template DAO for artists.
 * 
 * @author Brian Jimerson
 *
 */
public class ArtistJdbcDao {

	private JdbcTemplate jdbcTemplate;
	
	/**
	 * Constructs a new ArtistJdbcDao with the specified data source.
	 * @param dataSource The data source to use.
	 */
	public ArtistJdbcDao(DataSource dataSource) {
		this.jdbcTemplate = new JdbcTemplate(dataSource);
	}
	
	/**
	 * Gets all artists.
	 * @return All artists
	 */
	public List<Artist> getAllArtists() {
		List<Artist> allArtists = this.jdbcTemplate.query(
				"select * from artist", new ArtistMapper());
		return allArtists;
	}
	
	/**
	 * Finds an artist by id.
	 * @param id The id of the artist to find.
	 * @return The artist if found.
	 */
	public Artist findArtistById(Integer id) {
		Artist artist = this.jdbcTemplate.queryForObject(
				"select * from artist a where a.id = ?", new Object[]{id}, new ArtistMapper());
		return artist;
	}
	
	/**
	 * Artist mapper implementation of RowMapper
	 * @author Brian Jimerson
	 *
	 */
	final class ArtistMapper implements RowMapper<Artist> {
		
		/**
		 * Maps a result set row to an Artist object.
		 */
		public Artist mapRow(ResultSet rs, int rowNum) throws SQLException {
			Artist artist = new Artist();
			artist.setId(rs.getInt("id"));
			artist.setUrl(rs.getString("url"));
			return artist;
		}
	}
}

Map Loader

Our implementation of MapLoader is what tells Hazelcast how to back it’s Map cache entries with our database entries.  Our example only needs read access to data (not write access to the database) so we only have to implement a MapLoader.  The code for our MapLoader is:

package net.avantia.datagrid.util;

import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;

import org.springframework.context.support.ClassPathXmlApplicationContext;

import net.avantia.datagrid.dao.ArtistJdbcDao;
import net.avantia.datagrid.domain.Artist;

import com.hazelcast.core.MapLoader;

/**
 * MapLoader implementation for load artists into Hazelcast.
 * 
 * @author Brian Jimerson
 *
 */
public class ArtistMapLoader implements MapLoader<String, String> {

	private ArtistJdbcDao dao;
	
	/**
	 * Default constructor for ArtistMapLoader
	 */
	public ArtistMapLoader() {
		ClassPathXmlApplicationContext ctx = new ClassPathXmlApplicationContext("avantia-data-grid-context.xml");
		dao = ctx.getBean("artistJdbcDao", ArtistJdbcDao.class);
	}
	

	public String load(String id) {
		Integer intId = Integer.parseInt(id);
		Artist a = dao.findArtistById(intId);
		if (a != null) {
			return a.getName();
		} else {
			StringBuilder sb = new StringBuilder();
			sb.append("Artist for id [");
			sb.append(id);
			sb.append(" not found.");
			return sb.toString();
		}
	}

	public Map<String, String> loadAll(Collection<String> ids) {
		Map<String, String> artistMap = new HashMap<String, String>();
		List<Artist> artists = dao.getAllArtists();
		for (Artist a : artists) {
			artistMap.put(a.getId().toString(), a.getName());
		}
		return artistMap;
	}

	public Set<String> loadAllKeys() {
		Set<String> allArtistIds = new HashSet<String>();
		List<Artist> artists = dao.getAllArtists();
		for (Artist a : artists) {
			allArtistIds.add(a.getId().toString());
		}
		return allArtistIds;
	}

}

This MapLoader returns a Map to be cached; the key of the Map entry is the artist id, and the value is the artist name.  You could easily make the the value of the entry be an Artist bean with all of the attributes of the artist by changing the implementation and method signatures.

Configuration

The next step (almost done!) is to configure Hazelcast and our Spring beans.  The only thing special for our Hazelcast configuration is the class name for our map called 'artists'.  Everything else is the standard Hazelcast configuration:

<!-- Regular Hazelcast configuration omitted -->

<map name="artists">
  <map-store enabled="true">
   <class-name>net.avantia.datagrid.util.ArtistMapLoader</class-name>
   <write-delay-seconds>0</write-delay-seconds>
  </map-store>
</map>

And our Spring configuration is standard too; it just configures our data source and DAO:

<bean id="dataSource" destroy-method="close">
 <property name="url" value="jdbc:mysql://localhost:3306/artists" />
 <property name="username" value="root" />
 <property name="password" value="password" />
 <property name="driverClassName" value="org.hibernate.dialect.MySQLDialect" />
</bean>

<bean id="artistJdbcDao">
 <constructor-arg ref="dataSource"/>
</bean>

<context:component-scan base-package="net.avantia" />

Main Class

The final class for the data grid server is the Java main to start Hazelcast. This could also be a Servlet in an application server or other ways of starting a listener.

Our Main class is:

package net.avantia.datagrid;

import com.hazelcast.config.ClasspathXmlConfig;
import com.hazelcast.config.Config;
import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;

/**
 * Starts the cache server.
 * @author Brian Jimerson
 *
 */
public class DataGridServer {

	private HazelcastInstance instance;

	/**
	 * Main entry point for the memory data grid server.
	 * @param args
	 */
	public static void main(String[] args) {
		Config cfg = new ClasspathXmlConfig("hazelcast.xml");
		HazelcastInstance instance = Hazelcast.newHazelcastInstance(cfg);
		DataGridServer server = new DataGridServer(instance);
		int status = server.run();
	}

	/**
	 * Creates a new CacheServer instance.
	 * @param instance The underlying HazelcastInstance
	 */
	private DataGridServer(HazelcastInstance instance) {
		this.instance = instance;
	}

	/**
	 * Runs the server.
	 *
	 * @return 0 if run completes successfully; a negative number if there was an error.
	 */
	private int run() {
		System.out.println("Started data grid server with instance [" + instance.getName() + "].");
		return 0;
	}

}

Data Grid Client

Once you start the data grid server, there are a number of ways to access the data grid. Hazelcast provides Java, memcached (yes, memcached client access, but it doesn’t seem to support named maps), and REST clients out of the box, and these should cover most of your needs.

For Java client access, you can either wire up a Spring bean like this:

	<bean id="hazelcastClient" 
		class="com.hazelcast.client.HazelcastClient"
		factory-method="newHazelcastClient">
		<constructor-arg value="artist"/>
		<constructor-arg value="artist"/>
		<constructor-arg value="localhost"/>
	</bean>

or in Java like this:

  HazelcastClient hazelcastClient = HazelcastClient.newHazelcastClient("artist", "artist", "localhost");

And once you have a client instance you can get all of the grid’s artists like this:

  Map<String, String> allArtists = hazelcastClient.getMap("artist");
  System.out.println("All artists = " + allArtists);

Conclusion

In this post, we’ve walked through the process of using Hazelcast as an in-memory data grid, providing a map cache backed by a persistent store. This cache is read-only, but could easily support massive reads with little latency in a cloud-based infrastructure.

There are many other use cases for in-memory data grids in the cloud: writing to persistence-backed caches, name-value entries, and map-reduce strategies. This is just one use case that shows how to leverage a traditional database to support large-scale data access for persistent data.