IgnisHPC is a framework whose main objective is to unify the execution of Big Data and HPC workloads in the same computing engine. IgnisHPC has native support for multi-language applications using JVM and non-JVM-based languages. Currently it supports C, C++, Python, Java and Go.
Since MPI was used as its backbone technology, IgnisHPC allows MPI applications and libraries to be directly executed in an efficient way in the framework. In this way, most of the HPC scientific applications, which in many cases contain tens of thousands of lines of code, do not have to be ported to a new API or programming model. The main consequence is that users could combine in the same multi-language code HPC tasks (using MPI) with Big Data tasks (using MapReduce operations).
IgnisHPC outperforms the state-of-the-art framework Spark in terms of performance and scalability running applications that represent the most typical algorithmic patterns in Big Data and scientific computing. For example, considering a 12-node cluster with 2 × Intel Xeon E5-2630v4 (2.2Ghz, 10 cores) per node, the experimental results show that:
| Application | No. times faster than Spark |
|---|---|
| Minebench | 3.9x [Python/C++] 1.3x [Python] |
| Terasort | 1.8x [C++] 1.4x [Python] |
| K-Means | 1.9x [Python/C++] |
| PageRank | 1.1x [Python] |
| Transitive Closure | 1.12x [Python] |
Documentation:
- IgnisHPC is a containerized framework, so all the system modules are executed inside Docker containers. More information about the installation can be found in Getting started.
- To facilitate the adoption from the Big Data community, the IgnisHPC API was inspired by the Spark API in such a way that IgnisHPC codes are easily understandable by users who are familiar with Spark.
- Example of how to run an MPI application from IgnisHPC. More code examples can be found in our papers (see below).
- The complete documentation is here.
If you use IgnisHPC, please cite:
César Piñeiro and Juan C. Pichel. A Unified Framework to Improve the Interoperability between HPC and Big Data Languages and Programming Models. Future Generation Computer Systems, Vol. 134, pages 123-139, 2022.
César Piñeiro, Rodrigo Martínez-Castaño and Juan C. Pichel. Ignis: An efficient and scalable multi-language Big Data framework. Future Generation Computer Systems, Vol. 150, pages 705-716, 2020.
|
|
|
|
| Python | C/Cpp | Java (under revision) | Go (beta) |