Self-Supervised Learning for Patent Representations Using Section Structure

A new paper on arXiv explores self-supervised learning methods for creating effective patent representations using contrastive learning objectives.

The Challenge with Patent Documents#

Patents present a unique challenge for representation learning: they are long, highly structured documents with distinct sections (titles, abstracts, claims, summaries, backgrounds, drawings, and descriptions). Standard self-supervised approaches—which typically create positive examples by encoding the same text with different dropout masks—require careful tuning when applied to this domain.

The Approach#

The researchers propose mixed dropout–section positives, a patent-specific strategy that:

• Uses the title–abstract view as the anchor
• Constructs positive examples either through dropout re-encoding or by sampling from different sections of the same patent
• Leverages patent-internal structure as a training signal without requiring IPC labels, citations, or manual relevance annotations

This approach treats different sections of a single patent as naturally related views, effectively encoding the semantic alignment between a patent's abstract and its claims, summary, background, or technical description.

Results#

When evaluated on multiple benchmarks—graded EPO search-report retrieval, DAPFAM (a family-level patent retrieval benchmark), and IPC subclass classification—section-based positives outperformed calibrated dropout-only baselines and matched citation-informed encoders and general-purpose embedding models. The method showed particularly strong performance on out-of-domain splits.

The authors also found that section-pair training improved cross-section alignment, indicating better compatibility among different parts of the same patent document in the learned representation space.