Tag Archives: distributed computing

On Overcoming HPC Challenges of Trillion-Scale Real-World Graph Datasets – BigData’23 (Short Paper)

Posted on by

2023 IEEE International Conference on Big Data (BigData’23)
December 15-18, 2023, Sorrento, Italia

DOI: 10.1109/BigData59044.2023.10386309
PDF (Authors Copy)

Progress in High-Performance Computing in general, and High-Performance Graph Processing in particular, is highly dependent on the availability of publicly-accessible, relevant, and realistic data sets.

To ensure continuation of this progress, we (i) investigate and optimize the process of generating large sequence similarity graphs as an HPC challenge and (ii) demonstrate this process in creating MS-BioGraphs, a new family of publicly available real-world edge-weighted graph datasets with up to 2.5 trillion edges, that is, 6.6 times greater than the largest graph published recently. The largest graph is created by matching (i.e., all-to-all similarity aligning) 1.7 billion protein sequences. The MS-BioGraphs family includes also seven subgraphs with different sizes and direction types.

We describe two main challenges we faced in generating large graph datasets and our solutions, that are, (i) optimizing data structures and algorithms for this multi-step process and (ii) WebGraph parallel compression technique.

The datasets are available online on https://blogs.qub.ac.uk/DIPSA/MS-BioGraphs.

BibTex

@INPROCEEDINGS{10.1109/BigData59044.2023.10386309,
   author = {Koohi Esfahani, Mohsen and Boldi, Paolo and Vandierendonck, Hans and Kilpatrick,  Peter and  Vigna, Sebastiano},  
  booktitle={2023 IEEE International Conference on Big Data (BigData'23)},  
  title={On Overcoming {HPC} Challenges of  Trillion-Scale Real-World Graph Datasets}, 
  year={2023},
  volume={},
  number={},
  pages={},
  location={Italia, Sorrento},
  publisher={IEEE Computer Society},
  doi={10.1109/BigData59044.2023.10386309}
}

MS-BioGraphs

Related Posts

Sustainable Wearable EdgE inTelligence (SWEET)

Posted on by

The SWEET project will investigate the efficient deployment and sustainability issues of wearable sensors in particular for health analytics. Real-time remote monitoring of physiological indicators can support early detection and intervention in heart diseases and save lives. These services however require wearable technologies with strong predictive abilities, fast networks and fast servers to extract insights from the collected data. Unfortunately, these technology components are often inaccessible to hundreds of millions of people, specifically, people who live in areas with limited broadband connectivity and limited means to invest in local computing and communication infrastructure.

The project will focus on three components (i) energy-efficient wearable hardware accelerators using custom instruction set acceleration, (ii) distributed scheduling and machine learning model serving to account for performance variability of the systems and networks, (iii) technologies for efficient and portable deployment of web services and approximate key caching.

The project will support one PhD student and one post-doctoral researcher in our group.

The project is a collaboration between Deepu John (University College Dublin), Dimitrios S. Nikolopoulos (Virginia Tech), Bo Ji (Virginia Tech) and ourselves in DIPSA (Queen’s University Belfast), and is funded through the tripartite US-Ireland funding scheme.

We graciously acknowledge the support by the Department for the Economy, NI (contracts to be finalised).

TOD: Transprecise Object Detection

Posted on by

Real-time video analytics on the edge is challenging as the computationally constrained resources typically cannot analyse video streams at full fidelity and frame rate, which results in loss of accuracy. We propose a Transprecise Object Detector (TOD) which maximises the real-time object detection accuracy on an edge device by selecting an appropriate Deep Neural Network (DNN) on the fly with negligible computational overhead.

TOD makes two key contributions over the state of the art: (1) TOD leverages characteristics of the video stream such as object size and speed of movement to identify networks with high prediction accuracy for the current frames; (2) it selects the best-performing network based on projected accuracy and computational demand using an effective and low-overhead decision mechanism.

Experimental evaluation on a Jetson Nano demonstrates that TOD improves the average object detection precision by 34.7 % over the YOLOv4-tiny-288 model on average over the MOT17Det dataset. In the MOT17-05 test dataset, TOD utilises only 45.1 % of GPU resource and 62.7 % of the GPU board power without losing accuracy, compared to YOLOv4-416 model. We expect that TOD will maximise the application of edge devices to real-time object detection, since TOD maximises real-time object detection accuracy given edge devices according to dynamic input features without increasing inference latency in practice.

TOD was presented at the 5th International Conference on Fog and Edge Computing (ICFEC). Read the paper on arXiv.

RAPID: ReAl-time Process ModellIng and Diagnostics: Powering Digital Factories

Posted on by

This projects aims to develop algorithms for real-time processing for analytics in digital factories. A particular use case is the design of hardware circuits, where silicon can easily be damaged during manufacturing. The production defects that arise negatively affect yield and/or quality of the devices. Uncovering these defects, and how they may be mitigated through tunable process parameters, is a demanding process, especially considering the high production rate and voluminous metrics that are collected.

The project considers the design of sketching algorithms, transprecise computing and their efficient implementation on modern high-throughput hardware such as graphics processing units.

RAPID is sponsored by EPSRC.

Project members: