Files that are both large and changed often are where bugs tend to cluster and where refactoring pays off first. Each file in the current tree is placed by its size and by the number of commits that changed it; its hotspot score is changes × √lines. Files without counted lines, such as binary files, are left out.
Each dot is a file; a bigger dot is a higher score. The dashed lines mark the median file size and the median number of changes, so the large files that change often are in the top right. Both axes are logarithmic. Showing the 2,000 highest-scoring of 16,701 files.
The files ranked by hotspot score: the larger a file is and the more often it changes, the higher it ranks. Authors is how many people have changed the file, bots aside, as on the Code Ownership page: a hotspot only one person has changed is also a bus-factor risk.
| File | Lines | Changes | Authors | Score | |
|---|---|---|---|---|---|
| pom.xml | 3,810 | 2,105 | 247 | 129,931.5 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 10,778 | 1,242 | 270 | 128,940.9 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala | 4,898 | 1,073 | 196 | 75,094.7 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala | 8,558 | 749 | 161 | 69,289.6 | |
| common/utils/src/main/resources/error/error-conditions.json | 12,508 | 594 | 173 | 66,432.5 | |
| python/pyspark/sql/functions/builtin.py | 34,952 | 264 | 66 | 49,356.0 | |
| project/SparkBuild.scala | 2,171 | 967 | 177 | 45,056.4 | |
| sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala | 5,377 | 584 | 145 | 42,823.6 | |
| python/pyspark/sql/dataframe.py | 7,345 | 479 | 141 | 41,051.7 | |
| core/src/main/scala/org/apache/spark/SparkContext.scala | 3,983 | 637 | 205 | 40,201.7 |
| File | Lines | Changes | Authors | Score | |
|---|---|---|---|---|---|
| pom.xml | 3,810 | 2,105 | 247 | 129,931.5 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 10,778 | 1,242 | 270 | 128,940.9 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala | 4,898 | 1,073 | 196 | 75,094.7 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala | 8,558 | 749 | 161 | 69,289.6 | |
| common/utils/src/main/resources/error/error-conditions.json | 12,508 | 594 | 173 | 66,432.5 | |
| python/pyspark/sql/functions/builtin.py | 34,952 | 264 | 66 | 49,356.0 | |
| project/SparkBuild.scala | 2,171 | 967 | 177 | 45,056.4 | |
| sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala | 5,377 | 584 | 145 | 42,823.6 | |
| python/pyspark/sql/dataframe.py | 7,345 | 479 | 141 | 41,051.7 | |
| core/src/main/scala/org/apache/spark/SparkContext.scala | 3,983 | 637 | 205 | 40,201.7 | |
| docs/configuration.md | 4,420 | 589 | 203 | 39,158.5 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala | 3,688 | 616 | 143 | 37,409.0 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala | 5,218 | 490 | 122 | 35,395.5 | |
| core/src/main/scala/org/apache/spark/util/Utils.scala | 3,455 | 510 | 175 | 29,977.4 | |
| core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala | 5,598 | 386 | 128 | 28,880.4 | |
| python/pyspark/pandas/frame.py | 14,569 | 226 | 36 | 27,278.7 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala | 3,694 | 433 | 133 | 26,317.0 | |
| sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala | 5,729 | 340 | 69 | 25,734.7 | |
| core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala | 10,310 | 248 | 97 | 25,181.5 | |
| sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala | 2,932 | 407 | 115 | 22,038.2 | |
| core/src/main/scala/org/apache/spark/internal/config/package.scala | 3,389 | 367 | 133 | 21,364.9 | |
| dev/sparktestsupport/modules.py | 1,886 | 486 | 104 | 21,106.1 | |
| .github/workflows/build_and_test.yml | 1,415 | 560 | 50 | 21,065.2 | |
| sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala | 6,720 | 253 | 88 | 20,739.8 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala | 5,947 | 262 | 86 | 20,204.6 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala | 5,450 | 265 | 75 | 19,563.4 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala | 1,544 | 492 | 142 | 19,332.5 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala | 3,174 | 336 | 107 | 18,929.7 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala | 2,967 | 338 | 84 | 18,410.9 | |
| python/pyspark/ml/feature.py | 7,834 | 203 | 62 | 17,967.5 | |
| sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala | 2,749 | 330 | 85 | 17,302.2 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala | 1,453 | 443 | 114 | 16,886.4 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala | 3,130 | 295 | 70 | 16,504.2 | |
| python/pyspark/sql/connect/dataframe.py | 2,649 | 317 | 40 | 16,315.5 | |
| sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala | 1,359 | 433 | 117 | 15,962.4 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala | 4,366 | 238 | 81 | 15,726.0 | |
| sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala | 3,298 | 272 | 104 | 15,620.5 | |
| python/pyspark/sql/types.py | 4,263 | 238 | 74 | 15,539.4 | |
| sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala | 4,045 | 242 | 64 | 15,391.3 | |
| sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala | 5,516 | 207 | 60 | 15,373.8 | |
| sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala | 3,510 | 257 | 76 | 15,226.0 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala | 2,978 | 279 | 85 | 15,225.3 | |
| sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala | 3,410 | 258 | 110 | 15,066.0 | |
| sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala | 2,684 | 285 | 86 | 14,765.1 | |
| core/src/main/scala/org/apache/spark/storage/BlockManager.scala | 2,600 | 287 | 104 | 14,634.2 | |
| sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala | 5,501 | 194 | 84 | 14,388.7 | |
| python/pyspark/pandas/series.py | 7,583 | 165 | 25 | 14,368.3 | |
| sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala | 4,969 | 202 | 70 | 14,239.2 | |
| sql/api/src/main/scala/org/apache/spark/sql/functions.scala | 18,775 | 103 | 39 | 14,113.3 | |
| python/pyspark/ml/classification.py | 4,383 | 213 | 47 | 14,101.5 | |
| sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala | 4,426 | 207 | 85 | 13,771.3 | |
| sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala | 1,772 | 321 | 91 | 13,512.5 | |
| R/pkg/R/DataFrame.R | 4,335 | 202 | 57 | 13,299.8 | |
| python/pyspark/sql/tests/test_functions.py | 4,321 | 195 | 62 | 12,818.2 | |
| docs/sql-migration-guide.md | 1,318 | 344 | 97 | 12,488.7 | |
| R/pkg/R/functions.R | 5,105 | 174 | 57 | 12,432.2 | |
| core/src/main/scala/org/apache/spark/rdd/RDD.scala | 2,291 | 250 | 110 | 11,966.1 | |
| python/pyspark/sql/session.py | 2,607 | 232 | 67 | 11,845.6 | |
| dev/deps/spark-deps-hadoop-3-hive-2.3 | 289 | 696 | 49 | 11,832.0 | |
| python/pyspark/sql/connect/plan.py | 3,252 | 203 | 46 | 11,576.3 | |
| core/src/main/scala/org/apache/spark/executor/Executor.scala | 1,668 | 283 | 109 | 11,558.0 | |
| python/pyspark/sql/readwriter.py | 2,650 | 223 | 81 | 11,479.6 | |
| python/pyspark/sql/tests/connect/test_connect_basic.py | 1,830 | 268 | 38 | 11,464.6 | |
| core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala | 3,237 | 190 | 82 | 10,810.0 | |
| python/pyspark/sql/connect/functions/builtin.py | 5,980 | 138 | 40 | 10,671.6 | |
| sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala | 2,083 | 229 | 86 | 10,451.5 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala | 2,418 | 212 | 58 | 10,424.7 | |
| sql/core/src/test/scala/org/apache/spark/sql/connector/KeyGroupedPartitioningSuite.scala | 11,170 | 98 | 20 | 10,357.4 | |
| sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala | 3,575 | 169 | 67 | 10,104.7 | |
| sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala | 4,011 | 159 | 48 | 10,069.9 | |
| R/pkg/tests/fulltests/test_sparkSQL.R | 4,336 | 151 | 46 | 9,943.1 | |
| core/pom.xml | 752 | 357 | 100 | 9,789.9 | |
| sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala | 3,713 | 157 | 51 | 9,566.7 | |
| sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala | 1,553 | 239 | 75 | 9,418.5 | |
| docs/running-on-kubernetes.md | 2,531 | 187 | 58 | 9,407.8 | |
| sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala | 1,677 | 229 | 72 | 9,377.8 | |
| core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala | 1,318 | 258 | 102 | 9,366.5 | |
| R/pkg/R/generics.R | 1,747 | 223 | 51 | 9,320.8 | |
| sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4 | 3,270 | 162 | 73 | 9,263.8 | |
| python/pyspark/pandas/groupby.py | 5,065 | 129 | 23 | 9,180.8 | |
| sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala | 1,500 | 234 | 70 | 9,062.8 | |
| sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala | 1,057 | 277 | 68 | 9,005.7 | |
| python/pyspark/sql/connect/proto/relations_pb2.pyi | 4,926 | 125 | 29 | 8,773.2 | |
| python/pyspark/ml/regression.py | 3,341 | 151 | 40 | 8,728.0 | |
| sql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite.scala | 3,246 | 153 | 60 | 8,717.0 | |
| project/MimaExcludes.scala | 231 | 569 | 164 | 8,648.1 | |
| sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala | 2,902 | 160 | 59 | 8,619.2 | |
| core/src/main/scala/org/apache/spark/deploy/master/Master.scala | 1,506 | 221 | 71 | 8,576.4 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala | 1,517 | 217 | 68 | 8,451.9 | |
| core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala | 3,349 | 146 | 86 | 8,449.1 | |
| sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala | 3,966 | 133 | 56 | 8,375.8 | |
| sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala | 3,230 | 147 | 60 | 8,354.5 | |
| sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala | 4,688 | 121 | 53 | 8,284.7 | |
| sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala | 5,336 | 111 | 30 | 8,108.3 | |
| sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala | 1,513 | 206 | 69 | 8,012.8 | |
| python/pyspark/sql/connect/client/core.py | 2,861 | 149 | 46 | 7,969.8 | |
| python/pyspark/sql/tests/test_types.py | 4,177 | 123 | 46 | 7,949.5 | |
| core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala | 3,270 | 139 | 63 | 7,948.6 | |
| core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala | 2,039 | 175 | 84 | 7,902.2 | |
| mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala | 1,691 | 192 | 46 | 7,895.4 |
How Apache Spark compares with other Apache projects →
How Apache Spark ranks among Apache projects by AI-assisted commits →
Generate this report for your own repository with gitstats-action · All Gallery reports
Publishes the report to GitHub Pages on every push to main: turn on Pages for the gh-pages branch.
# .github/workflows/gitstats.yml
name: GitStats Report
on:
push:
branches: [main]
permissions:
contents: write
jobs:
gitstats:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
- uses: shenxianpeng/gitstats-action@v1
with:
deploy-to-pages: true