Small Talks/2012/May/Handling huge data with Biglistbox and Hibernate

Author
Vincent Jian, Engineer, Potix Corporation

Date
May 04, 2012

Version
ZK 6.0.1, Hibernate 3.6.8

Introduction

In the previous article, Jumper Chen introduced a new component in ZK 6.0.1 called Biglistbox that can handle huge data with great performance. In this article, using the same Biglistbox component I will demonstrate how to present huge data from database with Hibernate.

Huge data in Biglistbox

In Jumper's article, the demo uses static data which has a data size of one trillion. However in most of the real cases, the data size is much larger than what a database server can handle. Therefore, in this article I will create a sample, saving data into a database and retrieving such data via Hibernate instead of using the static data. The database used here is MySQL and the BigTable here used contains 100 columns and 22,000 data rows.

Create a ZK project and integrate with Hibernate

There are two ways to do so. You can either refer to the following documents to create a new project:

• Create and Run Your First ZK Application with Eclipse and ZK_Studio
• Integration with Hibernate

Or you can download the demo project in Jumper's article and modify based on that project.

• Jumper's demo project

Here I will go with the 2nd approach - modifying Jumper's demo project.

Prepare bean object and data access object

In Jumper's article, the demo uses static data. To change it to Hibernate + DB we need to create two classes to handle data loading:

The BigTable bean:

public class BigTable implements Serializable {
    private int _id;
    private String _column1, _column2, _column3, _column4, _column5, _column6, _column7, _column8, _column9, _column10;
    // omitted other members...

    // getters and setters
}

The BigTable Data Access Object:

public class BigTableDAO {
    private static final Log log = Log.lookup(BigTableDAO.class);
    
    public List<BigTable> findBetweenIds(int startId, int endId, boolean sortDir) {
		Session session = HibernateUtil.currentSession();
		List<BigTable> list = null;
		try {
			session.beginTransaction();
			String query = "SELECT b FROM BigTable b WHERE b.id BETWEEN :start AND :end ORDER BY b.id "
					+ (sortDir ? "asc" : "desc");
			Query q = session.createQuery(query);
			q.setInteger("start", startId);
			q.setInteger("end", endId);
			list = q.list();
			session.getTransaction().commit();
		} catch (HibernateException he) {
			log.error("get failed", he);
		}
		return list;
	}
}

Implement MatrixModel interface

To render the view port data dynamically, we have to load a fixed data size and cache them in the memory. Here is the code snippet of MyMatrixModel:

public class MyMatrixModel<Row extends List, Head extends List, Cell, Header>
		extends AbstractListModel<Row> implements MatrixModel<Row, Head, Cell, Header>, Sortable<Cell> {
	
	// omitted...
	
	private int _beginOffset; //first row number in _rowCache
	private int _cacheSize = 1000; // cache data size load from database
	private Map<String, MyKeyList<String>> _rowCache; // cache data
	
	public Row getElementAt(int index) {
		final String key = String.valueOf(index);
		if(_rowCache == null || index < _beginOffset || index >= _beginOffset + _cacheSize) {
			_beginOffset = index - _cacheSize/2 < 0 ? 0 : index - _cacheSize/2;
			setRowCache(_beginOffset, _beginOffset + _cacheSize);
		}
		return (Row) _rowCache.get(key);
	}
	
	private void setRowCache(int start, int end) {
		BigTableDAO dao = new BigTableDAO();
		int startId = _sortDir ? start : (_rowSize - start),
			endId   = _sortDir ? end   : (_rowSize - end);
		
		List<BigTable> list = _sortDir ? dao.findBetweenIds(startId, endId, _sortDir) : dao.findBetweenIds(endId, startId, _sortDir);
		final List<String> fieldNames = new BigTable().getFieldsName();
		
		_rowCache = new HashMap<String, MyKeyList<String>>();
		for (Iterator<BigTable> iterator = list.iterator(); iterator.hasNext();) {
			final BigTable bigTable = iterator.next();
			MyKeyList<String> cells = new MyKeyList<String>(_colSize, startId, new Fun() {
				public Object apply(int index) throws Exception {
					Field field = bigTable.getClass().getDeclaredField(fieldNames.get(index));
					field.setAccessible(true);
					return (String) field.get(bigTable);
				}
			});
			_rowCache.put(String.valueOf(startId), cells);
			startId++;
		}
	}
	
	// omitted...
}

In line 6 and 7, we define two variables, the first one is _beginOffset which represents the first row index of current cached data. The second one is _cacheSize representing the data size that is to be loaded from the database. You can modify the cache size to reduce the load of the database server. Also one thing to note is that in line 12 we can see that if the cache is empty or the first row index of view port is out of the cached data, we need to load data from database again by setRowCache method defined in line 19.

Other modifications

In Jumper's demo page, each cell contains an icon image that is rendered within MatrixRenderer. Since we do not need icons here, the implementation of MatrixRenderer is simplified as follows:

public class MyMatrixRenderer implements MatrixRenderer<List<String>> {

	public String renderCell(Component owner, List<String> data, int rowIndex, int colIndex) throws Exception {
		return data.get(colIndex);
	}

	public String renderHeader(Component owner, List<String> data, int rowIndex, int colIndex) throws Exception {
		return data.get(colIndex);
	}
}

Now, you should be able to run the project to see the result.

Conclusion

With this new Biglistbox, we can easily present huge data to end-users with great performance, either using the static data or by loading data from the database.

Download

Download the complete sample project here.

Comments

Copyright © Potix Corporation. This article is licensed under GNU Free Documentation License.