One major challenge presented by extreme-scale scientific computing is the huge amount of data that the simulation is capable of generating. Since each run of a state-of-the-art simulation can output data at the petascale, storing all of the data is no longer an option. Aside from simply dropping selected time steps, as has been the practice, additional data reduction methods must be considered in order to meet data movement and storage requirements, while maintaining the accuracy and integrity of the data. This is particularly important as scientific supercomputing moves toward exascale.
Firstly, we proposed a proper orthogonal decomposition (POD) based parallel compression method. A binary load-distributed approach is also proposed for fully utilizing all parallel nodes [1]. However, this method can only deal with the case that both the number of time steps and the number of parallel nodes are power-of-two. In order to resolve this problem, we proposed a 2-3-4 combination method [2]. Furthermore, in order to minimize the error, we proposed an m-swap method [3]. This method allows users to control the compression scale in one node. Recently, we are trying to apply these methods for in-situ visualization.
Besides of the POD-based compression methods mentioned above, a run-length method [4] was also proposed for lossless compression. Now we are proposing a compression method for reducing the size of streamlines and pathlines from large-scale simulations.
These approaches allow us to effectively use all of the processors and to reduce the interprocessor communication cost throughout the parallel compression calculations. The results of tests using the K computer indicate the superior performance of our design and implementation.

参考文献:
[1]. Chongke Bi, Kenji Ono, Kwan-Liu Ma, Haiyuan Wu, and Toshiyuki Imamura, “Proper Orthogonal Decomposition Based Parallel Compression for Visualizing Big Data on the K Computer,” in Proceedings of Eurographics Symposium on Parallel Graphics and Visualization, pp. 1-8, June, 2014.
[2]. Chongke Bi and Kenji Ono, “2-3-4 Combination for Parallel Compression on the K Computer,” in Proceedings of IEEE Pacific Visualization Symposium, pp. 281-285, March, 2014. doi: 10.1109/PacificVis.2014.28
[3]. Chongke Bi, Kenji Ono, and Lu Yang, “Parallel POD Compression of Time-Varying Big Datasets Using m-Swap on the K Computer,” in Proceedings of IEEE International Congress on Big Data, pp. 438-445, June, 2014. doi: 10.1109/BigData.Congress.2014.70
[4]. Shota Ishikawa, Haiyuan Wu, Chongke Bi, Qian Chen, Hirokazu Taki, and Kenji Ono, “Fluid Data Compression and ROI Detection Using Run Length Method,” in Procedia Computer Science, Vol. 35, pp. 1284-1291, 2014. doi: 10.1016/j.procs.2014.08.228