{"id":3046,"date":"2024-05-01T05:44:14","date_gmt":"2024-05-01T04:44:14","guid":{"rendered":"https:\/\/blogs.qub.ac.uk\/dipsa\/?p=3046"},"modified":"2025-11-09T10:42:54","modified_gmt":"2025-11-09T10:42:54","slug":"selective-parallel-loading-of-large-scale-compressed-graphs-with-paragrapher","status":"publish","type":"post","link":"https:\/\/blogs.qub.ac.uk\/dipsa\/selective-parallel-loading-of-large-scale-compressed-graphs-with-paragrapher\/","title":{"rendered":"Selective Parallel Loading of Large-Scale Compressed Graphs with ParaGrapher &#8211; arXiv Version"},"content":{"rendered":"\n<p><strong><a href=\"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-content\/uploads\/sites\/14\/2024\/06\/ParaGrapher-arXiv-v2.pdf\" target=\"_blank\" rel=\"noreferrer noopener\">PDF version<\/a><\/strong><br><strong>DOI: <a href=\"https:\/\/doi.org\/10.48550\/arXiv.2404.19735\" target=\"_blank\" rel=\"noreferrer noopener\">10.48550\/arXiv.2404.19735<\/a><\/strong><\/p>\n\n\n\n<p class=\"has-text-align-justify\">Comprehensive evaluation is one of the basis of experimental science. In High-Performance Graph Processing, a thorough evaluation of contributions becomes more achievable by supporting common input formats over different frameworks. However, each framework creates its specific format, which may not support reading large-scale real-world graph datasets. This shows a demand for high-performance libraries capable of loading graphs to (i) accelerate designing new graph algorithms, (ii) to evaluate the contributions on a wide range of graph algorithms, and (iii) to facilitate easy and fast comparison over different graph frameworks.<br><br>To that end, we present <strong>ParaGrapher<\/strong>, a high-performance API and library for loading large-scale and compressed graphs. ParaGrapher supports different types of requests for accessing graphs in shared- and distributed-memory and out-of-core graph processing. We explain the design of ParaGrapher and present a performance model of graph decompression, which is used for evaluation of ParaGrapher over three storage types.<br><br>Our evaluation shows that by decompressing compressed graphs in WebGraph format, ParaGrapher delivers up to 3.2 times speedup in loading and up to 5.2 times speedup in end-to-end execution in comparison to the binary and textual formats.<br><br>ParaGrapher is available online on <a href=\"https:\/\/blogs.qub.ac.uk\/DIPSA\/ParaGrapher\/\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/blogs.qub.ac.uk\/DIPSA\/ParaGrapher\/<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading has-medium-font-size\"><strong>Source Code<\/strong><\/h2>\n\n\n\n<p style=\"font-size:16px\"><strong><a href=\"https:\/\/github.com\/DIPSA-QUB\/ParaGrapher\">https:\/\/github.com\/DIPSA-QUB\/ParaGrapher<\/a><\/strong><\/p>\n\n\n\n<h2 class=\"wp-block-heading has-medium-font-size\"><strong>API Documentation<\/strong><\/h2>\n\n\n\n<p>Please refer to the Wiki, <a href=\"https:\/\/github.com\/DIPSA-QUB\/ParaGrapher\/wiki\/API-Documentation\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/github.com\/DIPSA-QUB\/ParaGrapher\/wiki\/API-Documentation<\/a>, or download the PDF file using <a href=\"https:\/\/github.com\/DIPSA-QUB\/ParaGrapher\/raw\/main\/doc\/api.pdf\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/github.com\/DIPSA-QUB\/ParaGrapher\/raw\/main\/doc\/api.pdf<\/a> .<\/p>\n\n\n\n<p class=\"has-medium-font-size\"><strong>BibTex<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>@misc{paragrapher-arxiv,\n  title = { Selective Parallel Loading of Large-Scale \n            Compressed Graphs with {ParaGrapher}}, \n  author = { {Mohsen} {Koohi Esfahani} and Marco D'Antonio and \n             Syed Ibtisam Tauhidi and Thai Son Mai and \n             Hans Vandierendonck},\n  year = {2024},\n  eprint = {2404.19735},\n  archivePrefix = {arXiv},\n  primaryClass = {cs.AR},\n  doi = {10.48550\/arXiv.2404.19735},\n  url={https:\/\/arxiv.org\/abs\/2404.19735}, \n}<\/code><\/pre>\n\n\n\n<p class=\"has-medium-font-size\"><strong>Related Posts &amp; Source Code<\/strong><\/p>\n\n\n\n<p style=\"font-size:16px\"><a href=\"https:\/\/blogs.qub.ac.uk\/DIPSA\/ParaGrapher\"><strong>ParaGrapher Web Page<\/strong><\/a> <\/p>\n\n\n<ul class=\"wp-block-latest-posts__list has-dates wp-block-latest-posts\"><li><div class=\"wp-block-latest-posts__featured-image alignleft\"><img decoding=\"async\" width=\"150\" height=\"150\" src=\"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-content\/uploads\/sites\/14\/2025\/11\/1-150x150.jpg\" class=\"attachment-thumbnail size-thumbnail wp-post-image\" alt=\"\" style=\"max-width:75px;max-height:75px;\" \/><\/div><a class=\"wp-block-latest-posts__post-title\" href=\"https:\/\/blogs.qub.ac.uk\/dipsa\/paragrapher-a-parallel-and-distributed-graph-loading-library-for-large-scale-compressed-graphs-bigdata25\/\">ParaGrapher: A Parallel and Distributed Graph Loading Library for Large-Scale Compressed Graphs &#8211; BigData&#8217;25 (Short Paper)<\/a><time datetime=\"2025-11-08T12:02:51+00:00\" class=\"wp-block-latest-posts__post-date\">8 November 2025<\/time><\/li>\n<li><div class=\"wp-block-latest-posts__featured-image alignleft\"><img decoding=\"async\" width=\"150\" height=\"150\" src=\"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-content\/uploads\/sites\/14\/2024\/12\/1-150x150.jpg\" class=\"attachment-thumbnail size-thumbnail wp-post-image\" alt=\"\" style=\"max-width:75px;max-height:75px;\" \/><\/div><a class=\"wp-block-latest-posts__post-title\" href=\"https:\/\/blogs.qub.ac.uk\/dipsa\/accelerating-loading-webgraphs-in-paragrapher\/\">Accelerating Loading WebGraphs in ParaGrapher<\/a><time datetime=\"2024-12-02T05:51:29+00:00\" class=\"wp-block-latest-posts__post-date\">2 December 2024<\/time><\/li>\n<li><div class=\"wp-block-latest-posts__featured-image alignleft\"><img decoding=\"async\" width=\"150\" height=\"150\" src=\"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-content\/uploads\/sites\/14\/2024\/05\/fern-150x150.jpg\" class=\"attachment-thumbnail size-thumbnail wp-post-image\" alt=\"\" style=\"max-width:75px;max-height:75px;\" \/><\/div><a class=\"wp-block-latest-posts__post-title\" href=\"https:\/\/blogs.qub.ac.uk\/dipsa\/selective-parallel-loading-of-large-scale-compressed-graphs-with-paragrapher\/\">Selective Parallel Loading of Large-Scale Compressed Graphs with ParaGrapher &#8211; arXiv Version<\/a><time datetime=\"2024-05-01T05:44:14+01:00\" class=\"wp-block-latest-posts__post-date\">1 May 2024<\/time><\/li>\n<li><div class=\"wp-block-latest-posts__featured-image alignleft\"><img loading=\"lazy\" decoding=\"async\" width=\"150\" height=\"150\" src=\"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-content\/uploads\/sites\/14\/2024\/04\/passerine-150x150.jpg\" class=\"attachment-thumbnail size-thumbnail wp-post-image\" alt=\"\" style=\"max-width:75px;max-height:75px;\" \/><\/div><a class=\"wp-block-latest-posts__post-title\" href=\"https:\/\/blogs.qub.ac.uk\/dipsa\/an-evaluation-of-bandwidth-of-different-storage-types-hdd-vs-ssd-vs-lustrefs-for-different-block-sizes-and-different-read-methods-mmap-vs-pread-vs-read\/\">An Evaluation of Bandwidth of Different Storage Types (HDD vs. SSD vs. LustreFS) for Different Block Sizes and Different Parallel Read Methods (mmap vs pread vs read)<\/a><time datetime=\"2024-04-20T09:48:10+01:00\" class=\"wp-block-latest-posts__post-date\">20 April 2024<\/time><\/li>\n<li><div class=\"wp-block-latest-posts__featured-image alignleft\"><img loading=\"lazy\" decoding=\"async\" width=\"150\" height=\"150\" src=\"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-content\/uploads\/sites\/14\/2024\/02\/loriini-150x150.jpg\" class=\"attachment-thumbnail size-thumbnail wp-post-image\" alt=\"\" style=\"max-width:75px;max-height:75px;\" \/><\/div><a class=\"wp-block-latest-posts__post-title\" href=\"https:\/\/blogs.qub.ac.uk\/dipsa\/paragrapher-integrated-to-laganlighter\/\">ParaGrapher Integrated to LaganLighter<\/a><time datetime=\"2024-02-16T08:29:26+00:00\" class=\"wp-block-latest-posts__post-date\">16 February 2024<\/time><\/li>\n<li><div class=\"wp-block-latest-posts__featured-image alignleft\"><img loading=\"lazy\" decoding=\"async\" width=\"150\" height=\"150\" src=\"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-content\/uploads\/sites\/14\/2024\/02\/poplar2-150x150.jpg\" class=\"attachment-thumbnail size-thumbnail wp-post-image\" alt=\"\" style=\"max-width:75px;max-height:75px;\" \/><\/div><a class=\"wp-block-latest-posts__post-title\" href=\"https:\/\/blogs.qub.ac.uk\/dipsa\/paragrapher-source-code-for-webgraph-types\/\">ParaGrapher Source Code For WebGraph Types<\/a><time datetime=\"2024-02-16T08:13:13+00:00\" class=\"wp-block-latest-posts__post-date\">16 February 2024<\/time><\/li>\n<\/ul>","protected":false},"excerpt":{"rendered":"<p>PDF versionDOI: 10.48550\/arXiv.2404.19735 Comprehensive evaluation is one of the basis of experimental science. In High-Performance Graph Processing, a thorough evaluation of contributions becomes more achievable by supporting common input formats over different frameworks. However, each framework creates its specific format, which may not support reading large-scale real-world graph datasets. This shows a demand for high-performance [&hellip;]<\/p>\n","protected":false},"author":1315,"featured_media":3048,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[119],"tags":[34,35,38,64,122,66,65,19,123,124],"class_list":{"0":"post-3046","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-paragrapher","8":"tag-algorithm-design-and-engineering","9":"tag-graph-processing","10":"tag-high-performance-computing","11":"tag-high-performance-graph-processing","12":"tag-parallel-io","13":"tag-real-world-graphs","14":"tag-sequence-similarity-graphs","15":"tag-source-code","16":"tag-storage","17":"tag-trillion-scale-graph-datasets","18":"czr-hentry"},"jetpack_featured_media_url":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-content\/uploads\/sites\/14\/2024\/05\/fern.jpg","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/posts\/3046","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/users\/1315"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/comments?post=3046"}],"version-history":[{"count":19,"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/posts\/3046\/revisions"}],"predecessor-version":[{"id":3631,"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/posts\/3046\/revisions\/3631"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/media\/3048"}],"wp:attachment":[{"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/media?parent=3046"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/categories?post=3046"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/tags?post=3046"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}