PATTERN Can Be Solved from Memory: Testing Graph Neural Networks on Unseen Patterns

PATTERN asks graph neural networks to find a planted subgraph in a new host graph. The hosts change, but the benchmark draws just 100 patterns and reuses them across training and test splits. A lookup that extracts these patterns from the training labels achieves 96.6% class-weighted accuracy on the official test split, above every published result, without fitting a model or consulting the generator. On unseen patterns, the same lookup scores 50.6%.

A matcher that also uses the generator reaches 99.2% on the official split. Among five graph networks we retrained, GRASS and PPGT go from the top two on the official split to the bottom two on unseen patterns. Each loses 8.1 percentage points even when familiar and unseen patterns are matched in size. Logistic regression on two-hop node statistics scores above every run of the five networks on unseen patterns.

For CKGConv, the advantage follows the training patterns: it reappears with each of three independently drawn pattern libraries and disappears when every training and validation graph gets a new pattern. Generator-based inference also scores 82.3% on CLUSTER, compared with a published best of 81.6%. On Minesweeper's released board, four of eight planned chains produced samples within the time budget; none converged, but their pooled predictions reached 99.86% ROC AUC, above the published best of 99.55%. PATTERN's official split tests its intended task of finding familiar patterns in new graphs.

We recommend reporting the same checkpoint's score on unseen patterns alongside it.

Source: arXiv cs.LG (Machine Learning) — Read the full article on the original site ↗