New Framework Adapts Language Models for E-Commerce Click Prediction

Click-through rate prediction—estimating whether a user will click on a product or ad—plays a vital role in recommendation engines and e-commerce platforms. Recent work has explored treating user behavior histories as sequences of text and leveraging pre-trained language models to analyze them, capitalizing on their ability to grasp context and meaning.

Yet a fundamental problem undermines this approach: unlike natural language that language models were trained on, user behavior sequences are made up of discrete actions separated by tokens that carry no semantic weight. This structural mismatch causes the model's attention mechanism to scatter inefficiently, focusing on meaningless separators rather than the actual behavioral patterns that matter.

The Problem: Attention Fragmentation#

Language models rely on attention mechanisms to identify which parts of input text are most important for making predictions. When applied to user behavior, these mechanisms struggle because the sequences lack the linguistic coherence present in typical training data. The model's attention becomes fragmented, bouncing between irrelevant separators instead of understanding relationships between actual user actions. This degradation in focus translates directly into weaker predictions about whether users will engage.

CTR-Sink: A Targeted Solution#

A team of researchers has developed CTR-Sink, a framework designed specifically to address this mismatch. The method introduces "sink tokens"—special tokens placed between consecutive user actions that serve as stable focal points for the model's attention. These tokens are enriched with recommendation-specific information, such as the time elapsed between actions, making them semantically meaningful anchors.

The approach combines two key components:

• Behavior-level attention sinks: Special tokens inserted between user actions that guide the model's focus toward meaningful behavioral boundaries
• Two-stage training strategy: An explicit training process that teaches the language model to concentrate attention on sink tokens and strengthens dependencies between them

Validation and Scope#

The researchers validated CTR-Sink across multiple datasets, including an industrial platform dataset as well as open public benchmarks like MovieLens and Kuairec. Visualization analysis confirmed that the framework successfully redirects attention patterns to improve behavioral understanding. The source code has been made publicly available, enabling broader adoption and extension of the work.