Oracle AI and Analytics

Jon_M

Feedback from using Claude to do the guided lab:

Resulting agent:

Oracle AIDP - CLI and Claude Code Gaps

Feedback for the Oracle AI Data Platform product manager, captured while running the AIW AIDP Hackathon guided build ("Shipment Routing Advisor") from Claude Code instead of the Workbench UI. All 8 guided-build steps were completed. Where Claude couldn't do a step, a UI step was used and the result read back through the API to learn the correct format.

Setup

• Instance: LL241580-AIDP (ca-toronto-1), tenancy oaccommunity3, workshop user LL241580-USER
• Tooling: OCI CLI 3.95.1 · aidp-cli 5.1.1 (venv ~/.venvs/aidp) · Claude Code plugin oracle-ai-data-platform-workbench-engineer-agent 0.5.0 (community marketplace) · session-token auth
• Approach: control plane via aidp CLI / oci raw-request; Spark via the plugin's aidp_sql.py WebSocket helper

Top asks

1. Show the instance OCID, region and API endpoint in the Workbench, and let aidp configure resolve them from the Workbench URL (gaps 1–2).
2. Add CLI groups for knowledge bases, AI Registry and Agent Hub, plus CLI equivalents of the SQL and RAG tool Test buttons (gaps 5, 24, 26).
3. Publish typed models or a JSON Schema for the agent diagram (tool parameters, guardrail policies, KB references). Today the API accepts shapes the UI can't render (gaps 19–22).
4. Make the API apply the same defaults as the UI and reject create bodies that would leave orphaned objects (gap 6).
5. Use one key convention: catalog.schema.name versus catalog.schema versus bare schema (gap 23).

Coverage by guided-build step

Step	Claude / CLI	UI needed	Notes
1. Compute + AI Compute	Yes (status check)	No	AI Compute only shows with ?type=AI_COMPUTE
2. Catalog, schema, tables	Yes	No	CSVs loaded through Spark saveAsTable, not the CLI upload flow
3. Volume + PDFs	Yes	No	aidp volume create / upload-file, MD5-verified
4. Knowledge base	No	Yes	API create left an orphan; add-source and ingest have no reachable API
5. Notebook	Yes	No	Cells run through aidp_sql.py; notebook saved with workspace-object create
6. Agent flow	Partly	Skeleton only	UI places the nodes; Claude sets every config through aidp agent update
7. Test / Playground	Partly	Tool tests	Deployed endpoint testable through REST; tool Test buttons are UI-only
8. Deploy / register / Agent Hub	Deploy only	Register + Hub	No registry or hub API

Gaps

Discovery and onboarding

1. No way to find the instance OCID or region from the Workbench. The URL is <id>.datalake.oci.oraclecloud.com with no OCID. Settings → Profile shows tenancy and home region (IAD), but the instance is in Toronto. I had to open browser DevTools and copy a dataLakes/<ocid> request URL. Ask: show instance OCID, region and API endpoint in Settings, with a "copy CLI config" button.
2. Listing instances needs tenancy IAM that workshop users don't have. oci ai-data-platform ... list-ai-data-platforms and GET /dataLakes?compartmentId= return 404 NotAuthorizedOrNotFound in every region and compartment tried. A user with full rights inside AIDP can't discover it from the CLI. Ask: a "list instances I can access" call, or aidp configure that resolves from the Workbench hostname.
3. Four hostnames for one service. The Workbench is on *.datalake.oci.oraclecloud.com, the OCI control plane on datalake.<region>, the data plane on aidp.<region>, and the deployed agents on gateway.aidp.<region>. The plugin docs also mention datahub-dp.<region> returning 404.
4. Session tokens last about 1 hour, and workshop users have no in-Workbench way to issue an API key. Settings → Profile has an "API key fingerprint" field but nothing to generate a key.

Knowledge bases

1. No knowledge-base command group in aidp-cli 5.1.1, and no KB client in the SDK (only models). Create, add-source, ingest and job-run paths are undocumented.
2. An API-created KB becomes an orphan. POST /knowledgeBases with only the required fields returned 202, and the async op SUCCEEDED. The KB lists as ACTIVE but is invisible in the UI ("No available items"). GET, PUT and DELETE /knowledgeBases/<key> fail with "Invalid volumeKey value. Format .." (the error even says "volumeKey"), and the catalog.schema.name form returns 404. It can't be seen, fixed or deleted, and it's still in the instance.
   • Root cause: the UI-created KB stores schemaKey: "gold_schema" (bare). /tables and /volumes and the KB list query all require catalog.schema, so I sent that, and the create silently produced a broken record. The UI also fills in embeddingModelName: ALL_MINILM_L12_V2, embeddingModelSourceType: ADW_26_AI, chunk 500/100, and workspaceKey/clusterKey for a system workspace that the workspaces API doesn't list. None of these defaults are documented.
   • The KB list API returns only the orphan, not the UI-created KB. GET /knowledgeBases/<catalog.schema.name> works for the UI one.
3. Embedding models aren't discoverable. GET /models?modelType=EMBEDDING returns empty, even though the UI uses ALL_MINILM_L12_V2.
4. Ingestion status can't be read through the API. GET .../knowledgeBases/<kb>/jobs lists the job, but .../jobs/<key> and .../jobs/<key>/jobRuns return 404. Status was only visible in the UI's Job runs tab.

Data, volumes and workspace files

1. AI Compute is hidden from the default cluster list. GET /workspaces/<ws>/clusters only returns USER clusters. You need ?type=AI_COMPUTE, and the cluster comes back as AGENT_FLOW_COMPUTE. /aiComputes returns 404.
2. File → table upload is awkward from the CLI. It takes 4 steps (temp upload target → PUT to the PAR → infer → create). create-data-table ignores header=true and needs positional _c0..._cN column names, then a rename. Ask: a single aidp schema create-table --file x.csv --header, equivalent to the UI upload.
3. volume download-file writes a JSON response envelope, not the file bytes, when redirected (an 18 KB PDF came back as about 200 KB). It can't round-trip binary files.
4. Path conventions are inconsistent. volume list-files requires --path / even for the root, but workspace-object list --path rejects a leading / ("should not start with /").
5. The notebook contents API is unreachable. GET .../notebook/api/contents returns 404 at the root and 400 for Shared. workspace-object list works instead.
6. Notebook cells can't be run from the CLI. The SDK and CLI can create a notebook but not execute it. Running code needs the plugin's Jupyter-WebSocket helper (aidp_sql.py), which also leaves a Shared/_aidp_sql_scratch.ipynb behind in the workspace.

CLI ergonomics

1. CLI output isn't pure JSON. aidp prints a Response: prefix before the JSON and has no --output json or --query option, so every script has to strip the prefix before parsing.
2. Global flags are repeated on every call (--instance-id, --region, --auth, --profile). An aidp configure profile would help. The plugin docs mention one, but it wasn't needed or tested here.

Claude Code plugin

1. The SessionStart hook runs python3 but the skills run python. On a machine with conda plus Homebrew Python, the hook reports "deps MISSING" on every session start even though they're installed for the interpreter the skills actually use.
2. The plugin docs are behind the CLI. They say aidp-cli isn't on PyPI yet (5.1.1 is), default to us-ashburn-1 and api_key (the CLI defaults to security_token), and mark many write routes "UNVERIFIED / verify-first". That caution is correct, but Claude ends up reading request models out of the Python SDK source to build valid bodies. Most importantly, the aidp-agent-flows skill says agents have no CLI and points at the LA REST agentFlows path, while aidp-cli 5.1.1 has a full aidp agent group (create / get / update / validate-agent / deploy / sessions). Claude only found it by browsing aidp command-groups.

Agent diagram (configuration formats)

The SDK types these as free-form dict(str, object) or doesn't model them at all. In each case below the server accepted my guessed format and validate-agent passed, but the UI didn't render it or the runtime didn't work. I found the correct formats by making the change in the UI and reading the result back.

1. SQL tool parameters (inputSchema). The working format is {"<name>": {"name": "<name>", "type": "string", "description": "...", "defaultValue": "ALL"}}. The name has to appear twice. With default instead of defaultValue, the UI showed a blank default. Without the inner name, the Test form showed an unlabelled box and failed with "Missing runtime param: shipment_id".
2. RAG tool knowledgeBase takes the bare KB name (supply_chain_knowledge_base), with catalogKey and a bare schemaKey. The fully qualified name is accepted, but the UI then shows it doubled (catalog.schema.catalog.schema.name). Even with the correct value, the first RAG test failed with "knowledge base ... was not found or is not accessible". Re-selecting the KB in the UI saved the identical value and the test then passed, which suggests caching or intermittency (the guide itself says "refresh a few times if you receive an error").
3. Guardrail PII policies. The lake-level defaults (GET /agentFlowGuardrails) list PII categories under categories, but the diagram's guardrailsMap uses piiCategories (category, isEnabled, action, threshold) plus customPiiRules, and the UI sets the policy-level action to INFORM with MASK set per category. Input and output are separate policies (scope USER_REQUEST / AGENT_RESPONSE). The node links to its policies through guardrailsConfigKey = a guardrailsMap key, which isn't documented. The SDK SafetyPolicy model has neither field name.
4. validate-agent checks structure only. It returned isValidAgent: true for all the broken configurations above. Ask: a deeper validation mode that resolves KB references, model IDs and regions, and parameter bindings.
5. Three key styles across the API: catalog.schema.name (volume keys, KB GET), catalog.schema (schemaKey for tables and volumes), and bare schema (schemaKey for KBs, SQL and RAG tools). Sending the wrong style produces silent failures (gaps 6, 20).
6. No CLI for tool tests. The SDK has TestSqlToolDetails / TestRagToolDetails models, but the CLI has no command for them, so the SQL and RAG Test buttons are UI-only.

Deploy, registry and runtime

1. Deploy works from the CLI. aidp agent deploy with {displayName, agentKey, agentComputeKey} returned 201, and the async op SUCCEEDED in about 30s. The guide's "Authorize using AIDP workbench, Run as Yourself" choice has no matching field in the request (the only auth field, oAuthConfig, is for an external IdP), so it appears to be the default. That should be documented.
2. No AI Registry or Agent Hub API. Registering the agent and testing it in Agent Hub can only be done in the UI.
3. The chat endpoint's request format isn't documented. POST https://gateway.aidp.<region>.oci.oraclecloud.com/agentendpoint/<agentKey>/chat takes {"query": "..."}. message and input are rejected (the error for message does name query). The response is LangChain-style response.messages[] including the tool calls, which makes it easy to evaluate from a script.
4. Observation (agent behaviour, not platform): on the deployed endpoint, "Should shipment S1005 be rerouted?" called only the SQL tool. The recommendation was correct (don't dispatch T-014: cooling "Needs Check", not available), but the "Policy Considerations" section wasn't grounded in RAG results, despite instruction 2. A per-tool "required" or "always call" option on the Executor Agent would help.

What worked well

• The UI-first, API-second pattern. Create the agent and drop the nodes in the UI, aidp agent get the skeleton, fill in the configuration, then aidp agent update. That gives correct node keys, edges and tool references for free. A CLI agent init --template would remove the UI step.
• aidp_sql.py Spark execution from the terminal: SELECT 1, CSV → Delta tables, the notebook classification cells, and binary checksums of volume files all worked first time.
• aidp volume create / upload-file worked cleanly, and the uploads were byte-identical to the originals (MD5 verified).
• aidp agent get / update / deploy all worked, and reading the diagram back is the fastest way to learn the formats.
• The REST API is consistent once you have the instance OCID, and async operations are easy to poll with aidp async-operations get.
• The SDK models (aidp_python_client) are the best schema reference available.

Open items

• Orphaned KB 584c040182cf4d6b9494f036a1063247 in supply_chain_catalog.gold_schema can't be deleted. Ask Oracle to clean it up, or confirm it goes when the instance expires.
• Share with the Oracle AIDP PM (name TBC).
• Repeat for the open build (part 2) and add any new gaps.