Handling huge data with Biglistbox and Hibernate

This page is under construction, so we cannot guarantee the accuracy of the content!

Author
Vincent Jian, Engineer, Potix Corporation

Date
May ?, 2012

Version
ZK 6.0.1, Hibernate 3.6.8

Introduction

In the previous article, Jumper Chen introduced a new component in ZK 6.0.1 called Biglistbox that can handle huge data with great performance. In this article, using the same Biglistbox component I will demonstrate how to present huge data from database with Hibernate.

Huge data in Biglistbox

In Jumper's article, the demo is using static data which size is one trillion. In most case, the data size is quite large beyond database server can handle. Therefore, I will use a BigTable in MySQL database with 100 columns and 22,000 data rows to create a sample project as following steps:

Create a ZK project and integrate with Hibernate

Please refer to the following documents to create a new project:

• Create and Run Your First ZK Application with Eclipse and ZK_Studio
• Integration with Hibernate

Or you can download demo project in Jumper's article and do some modification.

• Jumper's demo project

Here I will modify the code from Jumper's demo project.

Prepare bean object and data access object

In Jumper's article, the demo is using static data, therefore we need to create two class to handle data loaded from database. The BigTable bean:

public class BigTable implements Serializable {
    private int _id;
    private String _column1, _column2, _column3, _column4, _column5, _column6, _column7, _column8, _column9, _column10;
    // omitted other members...

    // getters and setters
}

The BigTable Data Access Object:

public class BigTableDAO {
    private static final Log log = Log.lookup(BigTableDAO.class);
    
    public List<BigTable> findBetweenIds(int startId, int endId, boolean sortDir) {
		Session session = HibernateUtil.currentSession();
		List<BigTable> list = null;
		try {
			session.beginTransaction();
			String query = "SELECT b FROM BigTable b WHERE b.id BETWEEN :start AND :end ORDER BY b.id "
					+ (sortDir ? "asc" : "desc");
			Query q = session.createQuery(query);
			q.setInteger("start", startId);
			q.setInteger("end", endId);
			list = q.list();
			session.getTransaction().commit();
		} catch (HibernateException he) {
			log.error("get failed", he);
		}
		return list;
	}
}

Implement MatrixModel interface

To render the view port data dynamically, we have to load a fix data size and cache them in memory. Here is fragment code of MyMatrixModel:

public class MyMatrixModel<Row extends List, Head extends List, Cell, Header>
		extends AbstractListModel<Row> implements MatrixModel<Row, Head, Cell, Header>, Sortable<Cell> {
	
	// omitted...
	
	private int _beginOffset; //first row number in _rowCache
	private int _cacheSize = 1000; // cache data size load from database
	private Map<String, MyKeyList<String>> _rowCache; // cache data
	
	public Row getElementAt(int index) {
		final String key = String.valueOf(index);
		if(_rowCache == null || index < _beginOffset || index >= _beginOffset + _cacheSize) {
			_beginOffset = index;
			setRowCache(index, index + _cacheSize);
		}
		return (Row) _rowCache.get(key);
	}
	
	private void setRowCache(int start, int end) {
		BigTableDAO dao = new BigTableDAO();
		int startId = _sortDir ? start : (_rowSize - start),
			endId   = _sortDir ? end   : (_rowSize - end);
		
		List<BigTable> list = _sortDir ? dao.findBetweenIds(startId, endId, _sortDir) : dao.findBetweenIds(endId, startId, _sortDir);
		final List<String> fieldNames = new BigTable().getFieldsName();
		
		_rowCache = new HashMap<String, MyKeyList<String>>();
		for (Iterator<BigTable> iterator = list.iterator(); iterator.hasNext();) {
			final BigTable bigTable = iterator.next();
			MyKeyList<String> cells = new MyKeyList<String>(_colSize, start, new Fun() {
				public Object apply(int index) throws Exception {
					Field field = bigTable.getClass().getDeclaredField(fieldNames.get(index));
					field.setAccessible(true);
					return (String) field.get(bigTable);
				}
			});
			_rowCache.put(String.valueOf(start), cells);
			start++;
		}
	}
	
	// omitted...
}

In line 6 and 7, we define two variables, first one is _beginOffset to represent the first row index of current cached data and second one is _cacheSize to represent data size load from database where you can modify the number to reduce loading of database server. And in line 12 we can see if cache is empty or the first row index of view port is outside the cache data, we need to load data from database again by setRowCache method defined in line 19.

Other modifications

Since we do not need icons here, the implementation of MatrixRenderer is simplified as follows:

public class MyMatrixRenderer implements MatrixRenderer<List<String>> {

	public String renderCell(Component owner, List<String> data, int rowIndex, int colIndex) throws Exception {
		return data.get(colIndex);
	}

	public String renderHeader(Component owner, List<String> data, int rowIndex, int colIndex) throws Exception {
		return data.get(colIndex);
	}
}

Now, you should be able to run the project to see the result.

Conclusion

Before Biglistbox component, we can only show huge data in browser with listbox which has poor performance if the data size is large. Now, with the help of Biglistbox, we can easily present huge data to end-users with great improvement of performance.

Download

Download Sample project here.

Comments

Copyright © Potix Corporation. This article is licensed under GNU Free Documentation License.