Congratulations to Brian Dandurand who has received notification that his Individual Fellowship proposal entitled “Scaling Parallelism and Convexity Hurdles in Bi-Level Machine Learning” has been proposed for funding.
More details will follow in due time.
Poplar source code has been integrated to LaganLighter and access to different WebGraph formats are available in LaganLighter:
PARAGRAPHER_CSX_WG_400_APPARAGRAPHER_CSX_WG_404_APPARAGRAPHERCSX_WG_800_APFor further details, please refer to
– LaganLighter source coder Repository: https://github.com/DIPSA-QUB/LaganLighter, particularly, the graph.c file.
– ParaGrapher source code repository: https://github.com/DIPSA-QUB/ParaGrapher particularly, the src/webgraph.c and src/WG*.java files.
Read more about ParaGrapher and LaganLighter.
Related Posts
ParaGrapher source code for accessing WebGraphs have been published. The supported graph types are:
PARAGRAPHER_CSX_WG_400_AP: graphs compressed in WebGraph format with 4 Bytes ID per vertex. Graphs in this category: LAW web graphs (https://law.di.unimi.it/datasets.php) .PARAGRAPHER_CSX_WG_404_AP: graphs compressed in WebGraph format with 4 Bytes ID per vertex and 4 Bytes integer weights per edge. Graphs in this category: MS-BioGraphs (https://blogs.qub.ac.uk/DIPSA/MS-BioGraphs/).PARAGRAPHER_CSX_WG_800_AP: graphs compressed in Big WebGraph format with 8 Bytes ID per vertex. Graphs in this category: (i) WDC Hyper Link 2012 (https://webdatacommons.org/hyperlinkgraph/) and (ii) SWH graphs (https://docs.softwareheritage.org/devel/swh-dataset/graph/dataset.html)ParaGrapher uses its asynchronous and parallel API to implement these graph types. The user needs to implement a callback function that is called by the API upon completion of reading a block of edges. Poplar uses a shared memory for interaction between its C library and the Java library that deploys the WebGraph framework.
For further details, please refer to Poplar source code repository: https://github.com/DIPSA-QUB/ParaGrapher, particularly, src/webgraph.c and src/WG*.java files.
Related Posts
2023 IEEE International Conference on Big Data (BigData’23)
December 15-18, 2023, Sorrento, Italia
DOI: 10.1109/BigData59044.2023.10386309
PDF (Authors Copy)
Progress in High-Performance Computing in general, and High-Performance Graph Processing in particular, is highly dependent on the availability of publicly-accessible, relevant, and realistic data sets.
To ensure continuation of this progress, we (i) investigate and optimize the process of generating large sequence similarity graphs as an HPC challenge and (ii) demonstrate this process in creating MS-BioGraphs, a new family of publicly available real-world edge-weighted graph datasets with up to 2.5 trillion edges, that is, 6.6 times greater than the largest graph published recently. The largest graph is created by matching (i.e., all-to-all similarity aligning) 1.7 billion protein sequences. The MS-BioGraphs family includes also seven subgraphs with different sizes and direction types.
We describe two main challenges we faced in generating large graph datasets and our solutions, that are, (i) optimizing data structures and algorithms for this multi-step process and (ii) WebGraph parallel compression technique.
The datasets are available online on https://blogs.qub.ac.uk/DIPSA/MS-BioGraphs.
BibTex
@INPROCEEDINGS{10.1109/BigData59044.2023.10386309,
author = {Koohi Esfahani, Mohsen and Boldi, Paolo and Vandierendonck, Hans and Kilpatrick, Peter and Vigna, Sebastiano},
booktitle={2023 IEEE International Conference on Big Data (BigData'23)},
title={On Overcoming {HPC} Challenges of Trillion-Scale Real-World Graph Datasets},
year={2023},
volume={},
number={},
pages={},
location={Italy, Sorrento},
publisher={IEEE Computer Society},
doi={10.1109/BigData59044.2023.10386309}
}MS-BioGraphs
Related Posts
2023 IEEE International Symposium on Workload Characterization (IISWC’23)
October 1-3, 2023, Ghent, Belgium
DOI: 10.1109/IISWC59245.2023.00029
PDF Version
Progress in High-Performance Computing in general, and High-Performance Graph Processing in particular, is highly dependent on the availability of publicly-accessible, relevant, and realistic data sets.
In this paper, we announce publication of MS-BioGraphs, a new family of publicly-available real-world edge-weighted graph datasets with up to 2.5 trillion edges, that is, 6.6 times greater than the largest graph published recently.
We briefly review the two main challenges we faced in generating large graph datasets and our solutions, that are, (i) optimizing data structures and algorithms for this multi-step process and (ii) WebGraph parallel compression technique. We also study some characteristics of MS-BioGraphs.
The datasets are available on https://blogs.qub.ac.uk/DIPSA/MS-BioGraphs .
Please visit https://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-Sequence-Similarity-Graph-Datasets/ for a complete version of this paper.
Bibtex
@INPROCEEDINGS{10.1109/IISWC59245.2023.00029,
author = {Koohi Esfahani, Mohsen and Boldi, Paolo and Vandierendonck, Hans and Kilpatrick, Peter and Vigna, Sebastiano},
booktitle={2023 IEEE International Symposium on Workload Characterization (IISWC'23)},
title={Dataset Announcement: {MS-BioGraphs}, Trillion-Scale Public Real-World Sequence Similarity Graphs},
year={2023},
volume={},
number={},
pages={},
location={Belgium, Ghent},
publisher={IEEE Computer Society},
doi={10.1109/IISWC59245.2023.00029}
}MS-BioGraphs
Related Posts
DOI: 10.48550/arXiv.2308.16744
PDF Version
arXiv Link
Progress in High-Performance Computing in general, and High-Performance Graph Processing in particular, is highly dependent on the availability of publicly-accessible, relevant, and realistic data sets.
To ensure continuation of this progress, we (i) investigate and optimize the process of generating large sequence similarity graphs as an HPC challenge and (ii) demonstrate this process in creating MS-BioGraphs, a new family of publicly available real-world edge-weighted graph datasets with up to 2.5 trillion edges, that is, 6.6 times greater than the largest graph published recently. The largest graph is created by matching (i.e., all-to-all similarity aligning) 1.7 billion protein sequences. The MS-BioGraphs family includes also seven subgraphs with different sizes and direction types.
We describe two main challenges we faced in generating large graph datasets and our solutions, that are, (i) optimizing data structures and algorithms for this multi-step process and (ii) WebGraph parallel compression technique. We present a comparative study of structural characteristics of MS-BioGraphs.
The datasets are available online on https://blogs.qub.ac.uk/DIPSA/MS-BioGraphs .
BibTex
@article{MS-BioGraphs-arxiv,
title = {{MS-BioGraphs}: Sequence Similarity Graph Datasets},
author = {Koohi Esfahani, Mohsen and Boldi, Paolo and Vandierendonck, Hans and Kilpatrick, Peter and Vigna, Sebastiano},
year = 2023,
journal = {CoRR},
volume = {abs/2308.16744},
doi = {10.48550/arXiv.2308.16744},
url = {https://doi.org/10.48550/arXiv.2308.16744},
archiveprefix = {arXiv},
eprint = {2308.16744}
}
MS-BioGraphs
Related Posts
| Name | MS-BioGraphs – MS |
| URL | https://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-MS |
| Graph Explanation | Vertices represent proteins and each edge represents the sequence similarity between its two endpoints |
| Edge Weighted | Yes |
| Directed | No |
| Number of Vertices | 1,757,323,526 |
| Number of Edges | 2,488,069,027,875 |
| Maximum Degree | 814,957 |
| Minimum Weight | 98 |
| Maximum Weight | 634,925 |
| Number of Zero-Degree Vertices | 6,437,984 |
| Average Degree | 1,415.8 |
| Size of The Largest WCC | 2,486,890,448,664 |
| Number of WCC | 148,861,367 |
| Creation Details | MS-BioGraphs: Sequency Similarity Graph Datasets |
| Format | WebGraph |
| License | CC BY-NC-SA |
| QUB IDF | 2223-052 |
| DOI | 10.5281/zenodo.7820808 |
| Citation | Koohi Esfahani, Mohsen, Boldi, Paolo, Vandierendonck, Hans, Kilpatrick, Peter, Vigna,Sebastiano. (2023). MS-BioGraphs - MS. https://doi.org/10.5281/zenodo.7820808. http://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-MS. |
| Bibtex | @misc{MS-BioGraphs-MS,
year = {2023},
author = {Mohsen Koohi Esfahani and Paolo Boldi and
Hans Vandierendonck and Peter Kilpatrick and
Sebastiano Vigna},
title = {{MS-BioGraphs - MS}},
doi = {10.5281/zenodo.7820808},
url = {http://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-MS},
howpublished= {\url{http://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-MS}}} |
| Underlying Graph |
The underlying graph in WebGraph format:
These files are validated using ‘Edge Blocks SHAs File’ as follows. |
| Weights (Labels) |
The weights of the graph in WebGraph format:
These files are validated using ‘Edge Blocks SHAs File’ as follows. |
| Edge Blocks SHAs File (Text) |
This file contains the shasums of edge blocks where each block contains
64 Million continuous edges and has one shasum for its 64M endpoints and
one for its 64M edge weights. The file is used to validate the underlying graph and the weights. For further explanation about validation process, please visit the https://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-Validation.
|
| Offsets (Binary) |
The offsets array of the CSX (Compressed Sparse Rows/Columns) graph in binary
format and little endian order. It consists of |V|+1 8-Bytes elements. The first and last values are 0 and |E|, respectively. This array helps converting the graph (or parts of it) from WebGraph format to binary format by one pass over (related) edges.
|
| WCC (Binary) |
The Weekly-Connected Compontent (WCC) array in binary format and little endian order. This array consists of |V| 4-Bytes elements The vertices in the same component have the same values in the WCC array.
|
| Names (tar.gz) |
This compressed file contains 120 files in CSV format using ‘;’ as the separator.
Each row has two columns: ID of vertex and name of the sequence. Note: If the graph has a ‘N2O Reordering’ file, the n2o array should be used to convert the vertex ID to old vertex ID which is used for identifying name of the protein in the `names.tar.gz` file.
|
| OJSON |
The charactersitics of the graph and shasums of the files. It is in the open json format and needs a closing brace (}) to be appended before being passed to a json parser.
|
For the explanation about the plots, please refer to the MS-BioGraphs paper.
To have a better resolution, please click on the images.
| Degree Distribution |  |
| Weight Distribution |  |
| Vertex-Relative Weight Distribution |  |
| Degree Decomposition |  |
| Cell-Binned Average Weight Degree Distribution |  |
| Weekly-Connected Components Size Distribution |  |
Related Posts
| Name | MS-BioGraphs – MSA500 |
| URL | https://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-MSA500 |
| Graph Explanation | Vertices represent proteins and each edge represents the sequence similarity between its two endpoints |
| Edge Weighted | Yes |
| Directed | Yes |
| Number of Vertices | 1,757,323,526 |
| Number of Edges | 1,244,904,754,157 |
| Maximum In-Degree | 229,442 |
| Maximum Out-Degree | 814,461 |
| Minimum Weight | 98 |
| Maximum Weight | 634,925 |
| Number of Zero In-Degree Vertices | 6,437,984 |
| Number of Zero Out-Degree Vertices | 16,843,087 |
| Average In-Degree | 711.0 |
| Average Out-Degree | 715.3 |
| Size of The Largest Weakly Connected Component | 1,244,203,865,823 |
| Number of Weakly Connected Components | 148,861,367 |
| Creation Details | MS-BioGraphs: Sequency Similarity Graph Datasets |
| Format | WebGraph |
| License | CC BY-NC-SA |
| QUB IDF | 2223-052 |
| DOI | 10.5281/zenodo.7820810 |
| Citation | Koohi Esfahani, Mohsen, Boldi, Paolo, Vandierendonck, Hans, Kilpatrick, Peter, Vigna,Sebastiano. (2023). MS-BioGraphs - MSA500. https://doi.org/10.5281/zenodo.7820810. http://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-MSA500. |
| Bibtex | @misc{MS-BioGraphs-MSA500,
year = {2023},
author = {Mohsen Koohi Esfahani and Paolo Boldi and
Hans Vandierendonck and Peter Kilpatrick and
Sebastiano Vigna},
title = {{MS-BioGraphs - MSA500}},
doi = {10.5281/zenodo.7820810},
url = {http://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-MSA500},
howpublished= {\url{http://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-MSA500}}} |
| Underlying Graph |
The underlying graph in WebGraph format:
These files are validated using ‘Edge Blocks SHAs File’ as follows. |
| Weights (Labels) |
The weights of the graph in WebGraph format:
These files are validated using ‘Edge Blocks SHAs File’ as follows. |
| Edge Blocks SHAs File (Text) |
This file contains the shasums of edge blocks where each block contains
64 Million continuous edges and has one shasum for its 64M endpoints and
one for its 64M edge weights. The file is used to validate the underlying graph and the weights. For further explanation about validation process, please visit the https://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-Validation.
|
| Offsets (Binary) |
The offsets array of the CSX (Compressed Sparse Rows/Columns) graph in binary
format and little endian order. It consists of |V|+1 8-Bytes elements. The first and last values are 0 and |E|, respectively. This array helps converting the graph (or parts of it) from WebGraph format to binary format by one pass over (related) edges.
|
| WCC (Binary) |
The Weekly-Connected Compontent (WCC) array in binary format and little endian order. This array consists of |V| 4-Bytes elements The vertices in the same component have the same values in the WCC array.
|
| Transposed’s Offsets (Binary) |
The offsets array of the transposed graph in binary format and little endian order.
It consists of |V|+1 8-Bytes elements. The first and last values are 0 and |E|, respectively. It helps to transpose the graph by performing one pass over edges.
|
| Names (tar.gz) |
This compressed file contains 120 files in CSV format using ‘;’ as the separator.
Each row has two columns: ID of vertex and name of the sequence. Note: If the graph has a ‘N2O Reordering’ file, the n2o array should be used to convert the vertex ID to old vertex ID which is used for identifying name of the protein in the `names.tar.gz` file.
|
| OJSON |
The charactersitics of the graph and shasums of the files. It is in the open json format and needs a closing brace (}) to be appended before being passed to a json parser.
|
For the explanation about the plots, please refer to the MS-BioGraphs paper.
To have a better resolution, please click on the images.
| In-Degree Distribution |  |
| Out-Degree Distribution |  |
| Weight Distribution |  |
| Vertex-Relative Weight Distribution |  |
| Degree Decomposition |  |
| Push and Pull Locality |  |
| Cell-Binned Average Weight Degree Distribution |  |
| Weekly-Connected Components Size Distribution |  |
Related Posts
| Name | MS-BioGraphs – MS200 |
| URL | https://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-MS200 |
| Graph Explanation | Vertices represent proteins and each edge represents the sequence similarity between its two endpoints |
| Edge Weighted | Yes |
| Directed | No |
| Number of Vertices | 1,414,493,449 |
| Number of Edges | 502,930,788,612 |
| Maximum Degree | 745,735 |
| Minimum Weight | 460 |
| Maximum Weight | 634,925 |
| Number of Zero-Degree Vertices | 0 |
| Average Degree | 355.6 |
| Size of The Largest WCC | 485,867,547,569 |
| Number of WCC | 338,348,495 |
| Creation Details | MS-BioGraphs: Sequency Similarity Graph Datasets |
| Format | WebGraph |
| License | CC BY-NC-SA |
| QUB IDF | 2223-052 |
| DOI | 10.5281/zenodo.7820812 |
| Citation | Koohi Esfahani, Mohsen, Boldi, Paolo, Vandierendonck, Hans, Kilpatrick, Peter, Vigna,Sebastiano. (2023). MS-BioGraphs - MS200. https://doi.org/10.5281/zenodo.7820812. http://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-MS200. |
| Bibtex | @misc{MS-BioGraphs-MS200,
year = {2023},
author = {Mohsen Koohi Esfahani and Paolo Boldi and
Hans Vandierendonck and Peter Kilpatrick and
Sebastiano Vigna},
title = {{MS-BioGraphs - MS200}},
doi = {10.5281/zenodo.7820812},
url = {http://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-MS200},
howpublished= {\url{http://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-MS200}}} |
| Underlying Graph |
The underlying graph in WebGraph format:
These files are validated using ‘Edge Blocks SHAs File’ as follows. |
| Weights (Labels) |
The weights of the graph in WebGraph format:
These files are validated using ‘Edge Blocks SHAs File’ as follows. |
| Edge Blocks SHAs File (Text) |
This file contains the shasums of edge blocks where each block contains
64 Million continuous edges and has one shasum for its 64M endpoints and
one for its 64M edge weights. The file is used to validate the underlying graph and the weights. For further explanation about validation process, please visit the https://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-Validation.
|
| Offsets (Binary) |
The offsets array of the CSX (Compressed Sparse Rows/Columns) graph in binary
format and little endian order. It consists of |V|+1 8-Bytes elements. The first and last values are 0 and |E|, respectively. This array helps converting the graph (or parts of it) from WebGraph format to binary format by one pass over (related) edges.
|
| WCC (Binary) |
The Weekly-Connected Compontent (WCC) array in binary format and little endian order. This array consists of |V| 4-Bytes elements The vertices in the same component have the same values in the WCC array.
|
| Names (tar.gz) |
This compressed file contains 120 files in CSV format using ‘;’ as the separator.
Each row has two columns: ID of vertex and name of the sequence. Note: If the graph has a ‘N2O Reordering’ file, the n2o array should be used to convert the vertex ID to old vertex ID which is used for identifying name of the protein in the `names.tar.gz` file.
|
| N2O Reordering (Binary) |
The New to Old (N2O) reordering array of the graph in binary format and little endian order. It consists of |V| 4-Bytes elements and identifies the old ID of each vertex which is used in searching the name of vertex (protein) in the names.tar.gz’ file .
|
| OJSON |
The charactersitics of the graph and shasums of the files. It is in the open json format and needs a closing brace (}) to be appended before being passed to a json parser.
|
For the explanation about the plots, please refer to the MS-BioGraphs paper.
To have a better resolution, please click on the images.
| Degree Distribution |  |
| Weight Distribution |  |
| Vertex-Relative Weight Distribution |  |
| Degree Decomposition |  |
| Cell-Binned Average Weight Degree Distribution |  |
| Weekly-Connected Components Size Distribution | |
Related Posts
| Name | MS-BioGraphs – MSA200 |
| URL | https://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-MSA200 |
| Graph Explanation | Vertices represent proteins and each edge represents the sequence similarity between its two endpoints |
| Edge Weighted | Yes |
| Directed | Yes |
| Number of Vertices | 1,757,323,526 |
| Number of Edges | 500,444,322,597 |
| Maximum In-Degree | 658,879 |
| Maximum Out-Degree | 709,176 |
| Minimum Weight | 98 |
| Maximum Weight | 634,925 |
| Number of Zero In-Degree Vertices | 6,437,984 |
| Number of Zero Out-Degree Vertices | 7,471,315 |
| Average In-Degree | 285.8 |
| Average Out-Degree | 286.0 |
| Size of The Largest Weakly Connected Component | 496,880,685,957 |
| Number of Weakly Connected Components | 221,467,156 |
| Creation Details | MS-BioGraphs: Sequency Similarity Graph Datasets |
| Format | WebGraph |
| License | CC BY-NC-SA |
| QUB IDF | 2223-052 |
| DOI | 10.5281/zenodo.7820815 |
| Citation | Koohi Esfahani, Mohsen, Boldi, Paolo, Vandierendonck, Hans, Kilpatrick, Peter, Vigna,Sebastiano. (2023). MS-BioGraphs - MSA200. https://doi.org/10.5281/zenodo.7820815. http://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-MSA200. |
| Bibtex | @misc{MS-BioGraphs-MSA200,
year = {2023},
author = {Mohsen Koohi Esfahani and Paolo Boldi and
Hans Vandierendonck and Peter Kilpatrick and
Sebastiano Vigna},
title = {{MS-BioGraphs - MSA200}},
doi = {10.5281/zenodo.7820815},
url = {http://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-MSA200},
howpublished= {\url{http://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-MSA200}}} |
| Underlying Graph |
The underlying graph in WebGraph format:
These files are validated using ‘Edge Blocks SHAs File’ as follows. |
| Weights (Labels) |
The weights of the graph in WebGraph format:
These files are validated using ‘Edge Blocks SHAs File’ as follows. |
| Edge Blocks SHAs File (Text) |
This file contains the shasums of edge blocks where each block contains
64 Million continuous edges and has one shasum for its 64M endpoints and
one for its 64M edge weights. The file is used to validate the underlying graph and the weights. For further explanation about validation process, please visit the https://blogs.qub.ac.uk/DIPSA/MS-BioGraphs-Validation.
|
| Offsets (Binary) |
The offsets array of the CSX (Compressed Sparse Rows/Columns) graph in binary
format and little endian order. It consists of |V|+1 8-Bytes elements. The first and last values are 0 and |E|, respectively. This array helps converting the graph (or parts of it) from WebGraph format to binary format by one pass over (related) edges.
|
| WCC (Binary) |
The Weekly-Connected Compontent (WCC) array in binary format and little endian order. This array consists of |V| 4-Bytes elements The vertices in the same component have the same values in the WCC array.
|
| Transposed’s Offsets (Binary) |
The offsets array of the transposed graph in binary format and little endian order.
It consists of |V|+1 8-Bytes elements. The first and last values are 0 and |E|, respectively. It helps to transpose the graph by performing one pass over edges.
|
| Names (tar.gz) |
This compressed file contains 120 files in CSV format using ‘;’ as the separator.
Each row has two columns: ID of vertex and name of the sequence. Note: If the graph has a ‘N2O Reordering’ file, the n2o array should be used to convert the vertex ID to old vertex ID which is used for identifying name of the protein in the `names.tar.gz` file.
|
| OJSON |
The charactersitics of the graph and shasums of the files. It is in the open json format and needs a closing brace (}) to be appended before being passed to a json parser.
|
For the explanation about the plots, please refer to the MS-BioGraphs paper.
To have a better resolution, please click on the images.
| In-Degree Distribution |  |
| Out-Degree Distribution |  |
| Weight Distribution |  |
| Vertex-Relative Weight Distribution |  |
| Degree Decomposition |  |
| Push and Pull Locality |  |
| Cell-Binned Average Weight Degree Distribution |  |
| Weekly-Connected Components Size Distribution | |
Related Posts