{"id":3889,"date":"2026-05-01T05:11:51","date_gmt":"2026-05-01T04:11:51","guid":{"rendered":"https:\/\/blogs.qub.ac.uk\/dipsa\/?p=3889"},"modified":"2026-05-01T05:12:39","modified_gmt":"2026-05-01T04:12:39","slug":"user-behaviour-on-kelvin2-hpc","status":"publish","type":"post","link":"https:\/\/blogs.qub.ac.uk\/dipsa\/user-behaviour-on-kelvin2-hpc\/","title":{"rendered":"User behaviour on Kelvin2 HPC"},"content":{"rendered":"\n

1. User Job Distribution: A Heavy-Tailed Workload<\/strong><\/h3>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n

The cluster’s workload is heavily concentrated among a very small fraction of the total user base. The data exhibits an extreme version of the Pareto principle (80\/20 rule)<\/sup>.<\/p>\n\n\n\n

    \n
  • The Top 2%:<\/strong> Just 2.12%<\/strong> of the heaviest users are responsible for 70%<\/strong> of the total workload.<\/li>\n\n\n\n
  • The Top 3%:<\/strong> Only 3.75%<\/strong> of users account for 80%<\/strong> of all jobs.<\/li>\n\n\n\n
  • The 99th Percentile:<\/strong> To reach 99% of the total cluster workload, you only need to account for the top 29.32%<\/strong> of users.<\/li>\n<\/ul>\n\n\n\n

    This indicates that a vast majority of users submit relatively few jobs, while a core group of “power users” drives almost all cluster activity.<\/p>\n\n\n\n

    \n\n\n\n

    2. Job Type Distribution: The Dominance of Normal and Array Jobs<\/strong><\/h3>\n\n\n\n
    \"\"<\/figure>\n\n\n\n

    The vast majority of the tens of millions of jobs run on the cluster fall into a few primary categories, while complex step-based or interactive jobs are statistically rare<\/sup>.<\/p>\n\n\n\n

      \n
    • Top Job Types (Log Scale Order of Magnitude):<\/strong>\n
        \n
      • X (Normal)<\/code>:<\/strong> The most common job type, with ~2.93 million<\/strong> instances.<\/li>\n\n\n\n
      • X.batch (Normal Batch)<\/code>:<\/strong> Close behind with ~2.68 million<\/strong> jobs.<\/li>\n\n\n\n
      • X_Y (Array Job)<\/code>:<\/strong> Accounting for ~1.98 million<\/strong> jobs.<\/li>\n\n\n\n
      • X_Y.batch (Array Batch)<\/code>:<\/strong> Representing ~1.90 million<\/strong> jobs.<\/li>\n\n\n\n
      • X.Y (Job Step)<\/code>:<\/strong> Representing ~1.35 million<\/strong> jobs.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
      • Rare Job Types:<\/strong> Conversely, standard interactive jobs (X.interactive<\/code>) barely register, with only 2 counts in the entire analysed dataset. Complex mixed array types like X_[Mixed%S]<\/code> and X_[Mixed]<\/code> only have 3 and 12 counts respectively.<\/li>\n<\/ul>\n\n\n\n
        \n\n\n\n

        3. Partition Distribution & Behaviours<\/strong><\/h3>\n\n\n\n
        \"\"<\/figure>\n\n\n\n

        Job volume is radically different depending on the partition, and specific partitions attract entirely different shapes of workloads<\/sup>.<\/p>\n\n\n\n

        Overall Volume:<\/strong><\/p>\n\n\n\n

          \n
        • k2-hipri<\/code><\/strong> (High Priority) is the absolute behemoth of the cluster, processing an astonishing 8,384,627 jobs<\/strong>.<\/li>\n\n\n\n
        • k2-medpri<\/code><\/strong> (Medium Priority) is a distant second, processing 1,214,761 jobs<\/strong>.<\/li>\n\n\n\n
        • Other notable general partitions include k2-himem<\/code> (259k), k2-living-labs<\/code> (176k), and k2-lowpri<\/code> (142k).<\/li>\n\n\n\n
        • Among GPU partitions, k2-gpu-v100<\/code> sees the highest job count (65k), followed by k2-gpu-a100<\/code> (32k) and k2-gpu-a100mig<\/code> (20k).<\/li>\n<\/ul>\n\n\n\n

          Job Types by Partition (The Heatmap Analysis):<\/strong> The way users interact with different partitions varies greatly based on the queue<\/sup>:<\/p>\n\n\n\n

            \n
          • General Queues (k2-lowpri<\/code> & k2-living-labs<\/code>):<\/strong> These partitions see a relatively balanced mix of job types. For example, k2-living-labs<\/code> processed ~42k Normal jobs, ~40k Array jobs, and ~31k Normal Batch jobs.<\/li>\n\n\n\n
          • The MIXED<\/code> Partition Anomaly:<\/strong> The MIXED<\/code> partition is almost exclusively used for massive array workloads. It processed virtually zero “Normal” jobs, but handled roughly 69k Array jobs (X_Y<\/code>)<\/strong> and 68k Array Batch jobs (X_Y.batch<\/code>)<\/strong>.<\/li>\n\n\n\n
          • k2-gpu<\/code>:<\/strong> The standard GPU partition sees a much lower volume overall compared to CPU nodes, with its highest count being standard X (Normal)<\/code> jobs (~4.3k) and X.batch<\/code> (~3.4k), indicating less use of arrays for basic GPU tasks.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"

            1. User Job Distribution: A Heavy-Tailed Workload The cluster’s workload is heavily concentrated among a very small fraction of the total user base. The data exhibits an extreme version of the Pareto principle (80\/20 rule). This indicates that a vast majority of users submit relatively few jobs, while a core group of “power users” drives […]<\/p>\n","protected":false},"author":1149,"featured_media":3550,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[152],"tags":[],"class_list":{"0":"post-3889","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-kelvin-living-lab","8":"czr-hentry"},"jetpack_featured_media_url":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-content\/uploads\/sites\/14\/2025\/08\/cf973c88-02be-4493-b13c-3face3e98b73.jpg","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/posts\/3889","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/users\/1149"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/comments?post=3889"}],"version-history":[{"count":1,"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/posts\/3889\/revisions"}],"predecessor-version":[{"id":3893,"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/posts\/3889\/revisions\/3893"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/media\/3550"}],"wp:attachment":[{"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/media?parent=3889"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/categories?post=3889"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.qub.ac.uk\/dipsa\/wp-json\/wp\/v2\/tags?post=3889"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}