Framework Extracts Reusable AI Agent Skills from Videos, Code, and Tutorials

Researchers have proposed a new approach to building reusable procedural knowledge for software agents by automatically extracting information from the kinds of resources humans already create—tutorial videos, code repositories, articles, and reference materials.

The system, called RESOURCE2SKILL, addresses a significant limitation in how agent skills are currently developed. Most existing skill libraries rely on hand-written documentation, text-only formats, or traces of agent behavior, which means valuable information locked in videos and other multimodal materials goes largely untapped.

How It Works#

The framework transforms diverse resources into a structured knowledge base called a Skill Wiki. Each entry in this wiki combines multiple complementary forms of information:

• Video content captures the temporal flow and visual appearance of operations
• Code snippets document concrete tool patterns that can be executed directly
• Articles and artifacts provide conceptual background and stylistic guidance
• Metadata and provenance track the source and context of each skill

At runtime, agents search this wiki for relevant skills and combine them to complete tasks. When existing skills don't cover what's needed, the same system can acquire new skills on demand by learning from online sources.

Performance and Testing#

Testing across seven different authoring domains showed that agents using RESOURCE2SKILL improved their overall performance by 11.9 percentage points compared to agents without skill libraries. The system outperformed baseline approaches in 26 out of 28 test configurations spanning different models and domains.

The researchers also tested each component of their approach through ablation studies, confirming that the multimodal format, hierarchical organization, diversity of sources, skill selection strategy, and ability to learn new skills online all contribute meaningfully to the framework's effectiveness.