Gen AI-Based Conversational Metadata Intelligence for OCI Lakehouse & FDI ADW

User_5FRCQ

Description

Metadata exploration in OCI Lakehouse & FDI ADW is manual and fragmented, unlike Fusion ETL dictionaries, which are publicly available. While OCI Data Catalog provides metadata management, it lacks tenant-specific insights, real-time data quality tracking, and a conversational AI interface.

Additionally, the Run Analyser in FAW DevOps provides load details only at a job level, not at a table level. For data modeling, development, and reporting, a holistic view of tables in a functional area is essential from both business and technical perspectives.

This solution extends existing ETL dictionaries to OCI Lakehouse & FDI ADW, acting as a smart add-on to OCI Data Catalog. It will serve as a centralized AI-powered metadata repository for Data Model & Content Team Developers and customers, enhancing metadata accessibility.

By leveraging Data Catalog and in-house tools like Run Analyser, it delivers:

• Technical Insights – Refresh frequency, sort distribution, good/bad data %, duplicates
• Business Insights – Primary/foreign keys, available metrics, functional relevance

A Generative AI agent enables natural language interaction, reducing friction between users and metadata systems.

Use Case & Business Need

Developers and analysts need:

• Quick access to table definitions, update schedules, and relationships
• Real-time data profiling without SQL queries
• Contextual business insights – Key metrics and relationships
• A centralized metadata hub – Similar to Fusion’s ETL dictionaries

Without this, metadata discovery remains slow and manual.

How we aim to solve this for our warehouse and lakehouse customers:

Gen AI-Powered Solution

• Conversational Metadata Chatbot – Query tables via natural language
• Automated Data Profiling – Tracks refresh times, duplicates, and sort order dynamically
• Tenant-Specific Metadata Summaries – Custom insights per customer
• Business Metadata Mapping – Identifies primary/foreign keys, functional areas, dependencies

How This Enhances OCI Data Catalog

Unlike OCI Data Catalog, which focuses on static metadata and lineage tracking, this solution provides:

• Conversational AI Access – Natural language queries ("When was customer_data last refreshed? What is the table health for DW_AP_INVOICE_CF? What metrics can be sourced from here?")
• Tenant-Specific Insights – Custom metadata views based on activated functional areas
• Automated Data Quality Checks – Good/bad data %, duplicates, sort order tracking
• Business Metadata Mapping – Primary/foreign keys, metrics, and functional areas
• Metadata Intelligence as a Service – Deploy as a REST API or an OCI-native service, allowing seamless integration with FDI ADW and OCI Lakehouse based on tenant-specific extensions. This would enable external applications, developers, and data pipelines to programmatically query metadata insights, making it more versatile beyond just a conversational interface.

This solution enhances OCI Data Catalog with real-time, AI-driven metadata intelligence, bridging the gap between metadata management and practical business insights.