LLM assisted CI/CD Ontology validation

.github/workflows/ontology-ci.yml

@@ -0,0 +1,29 @@
+name: Ontology CI
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v4
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install dependencies
+        run: |
+          pip install pyshacl rdflib
+
+      - name: Run SHACL validation
+        run: |
+          python -m pyshacl \
+            -s validation/ClassDeclarationsMustHaveCommentShape.ttl \
+            -e ontologies/tbox.ttl \
+            test_fail.ttl

README.md

@@ -9,3 +9,4 @@ See the accompanying paper here: https://arxiv.org/abs/2605.17374
 The following license applies: https://creativecommons.org/licenses/by/4.0/deed.en
 
 FSL is a sibling of FSL: https://github.com/softlang/fex
+trigger CI run

broken.log

@@ -0,0 +1,2 @@
+Validation Report
+Conforms: True

llm-validation/docs/milestone2_experiment.md

@@ -0,0 +1,30 @@
+# Milestone 2 Experiment: SHACL + LLM-style Explanation Layer
+
+## Objective
+Evaluate whether SHACL validation results can be transformed into human-readable explanations.
+
+## Setup
+- FSL ontology (tbox.ttl)
+- pySHACL validation engine
+- Custom SHACL shapes for:
+  - documentation completeness (rdfs:comment)
+  - external reference completeness (foaf links)
+
+## Experiments
+
+### Experiment 1: Missing rdfs:comment
+- Removed rdfs:comment from :Entity (owl:Class)
+- Result: SHACL violation detected successfully
+
+### Experiment 2: Missing FOAF link
+- Removed foaf:page from :Entity (owl:Class)
+- Result: SHACL violation detected successfully
+
+## Result
+Both violations were successfully detected and transformed into structured explanations.
+
+## Observation
+SHACL outputs are machine-readable but not human-friendly. A transformation layer is required.
+
+## Conclusion
+A rule-based interpretation layer can bridge SHACL validation outputs and human-readable explanations.

llm-validation/docs/validation_analysis.md

@@ -0,0 +1,18 @@
+# Validation Coverage Analysis
+
+## ClassDeclarationsMustHaveCommentShape
+
+Purpose:
+Ensures ontology classes contain an rdfs:comment annotation.
+
+Target:
+Resources explicitly declared as owl:Class or rdfs:Class.
+
+Experiment:
+Removed the rdfs:comment annotation from :Entity.
+
+Result:
+SHACL validation failed as expected.
+
+Observation:
+The shape does not apply to entities typed as :ConceptualEntity.

llm-validation/llm_explainer.py

@@ -0,0 +1,244 @@
+import re
+import json
+import ollama
+
+# -----------------------------------
+# Extract JSON from LLM output
+# -----------------------------------
+
+def safe_parse_llm_output(raw):
+    try:
+        return json.loads(raw)
+
+    except:
+        match = re.search(r"\{.*\}", raw, re.DOTALL)
+
+        if match:
+            try:
+                return json.loads(match.group(0))
+            except:
+                return {
+                    "error": "invalid_json",
+                    "raw": raw
+                }
+
+        return {
+            "error": "no_json_found",
+            "raw": raw
+        }
+
+
+# -----------------------------------
+# Files containing SHACL violations
+# -----------------------------------
+
+from pathlib import Path
+
+files = sorted(Path("results").glob("*.txt"))
+
+if not files:
+    raise FileNotFoundError("No validation reports found in results/")
+# -----------------------------------
+# Parse SHACL report
+# -----------------------------------
+
+def parse(report):
+
+    focus = re.search(r"Focus Node:\s*(.*)", report)
+
+    message = re.search(
+        r'sh:message Literal\("(.*?)"\)',
+        report,
+        re.DOTALL
+    )
+
+    shape = re.search(
+        r"Source Shape:\s*(.*)",
+        report
+    )
+
+    return {
+        "focus_node": focus.group(1) if focus else None,
+        "message": message.group(1) if message else None,
+        "shape": shape.group(1) if shape else None
+    }
+
+
+# -----------------------------------
+# Ask LLM
+# -----------------------------------
+
+def explain_with_llm(v):
+
+    prompt = f"""
+You are an ontology validation assistant.
+
+You MUST return ONLY valid JSON.
+
+Focus Node: {v['focus_node']}
+Shape: {v['shape']}
+Message: {v['message']}
+
+Return ONLY valid JSON.
+
+Rules:
+- Do NOT include markdown
+- Do NOT include explanations outside JSON
+- fix_rdf must contain Turtle RDF
+- Do NOT return JSON inside fix_rdf
+
+Format:
+
+{{
+  "focus_node": "...",
+  "shape": "...",
+  "issue": "...",
+  "why_it_matters": "...",
+  "fix_rdf": "..."
+}}
+"""
+
+    response = ollama.chat(
+        model="llama3.1",
+        messages=[
+            {
+                "role": "system",
+                "content": "Return only valid JSON."
+            },
+            {
+                "role": "user",
+                "content": prompt
+            }
+        ]
+    )
+
+    return response["message"]["content"]
+
+
+# -----------------------------------
+# Quality checks
+# -----------------------------------
+
+def evaluate_output(parsed):
+
+    warnings = []
+
+    fix = parsed.get("fix_rdf", "")
+    if fix.startswith("'") or fix.endswith("'"):
+       warnings.append(
+       	   "Fix RDF wrapped in quotes"
+       )
+
+    if "rdf:type ve:Class" in fix:
+        warnings.append(
+          "Fix recreates class instead of modifying existing class"
+        )
+
+    if fix.startswith("http://"):
+       warnings.append(
+           "Subject URI missing angle brackets"
+       )
+
+    if "<:" in fix:
+        warnings.append(
+            "Invalid Turtle syntax detected (<:Entity)"
+        )
+
+    if "example.org" in fix:
+        warnings.append(
+            "Placeholder URI detected"
+        )
+
+    if parsed.get("why_it_matters", "") == "":
+        warnings.append(
+            "Empty explanation"
+        )
+
+    if fix == "":
+        warnings.append(
+            "Missing RDF fix"
+        )
+
+    return warnings
+
+
+# -----------------------------------
+# Main execution
+# -----------------------------------
+
+results = []
+
+for file in files:
+
+    print("\n==============================")
+
+    report = open(file).read()
+
+    v = parse(report)
+
+    print("\nParsed:")
+    print(v)
+
+    # -------------------------------
+    # Skip incomplete SHACL reports
+    # -------------------------------
+    if not all([v["focus_node"], v["shape"], v["message"]]):
+
+        print("\nSkipping incomplete validation report.")
+
+        results.append({
+            "file": str(file),
+            "parsed_violation": v,
+            "llm_output": None,
+            "warnings": ["Incomplete SHACL report"]
+        })
+
+        continue
+
+    # -------------------------------
+    # Call the LLM only for valid reports
+    # -------------------------------
+    # -------------------------------
+    # Call the LLM only for valid reports
+    # -------------------------------
+    raw = explain_with_llm(v)
+
+    parsed = safe_parse_llm_output(raw)
+
+    if "error" in parsed:
+        warnings = [
+            f"LLM parsing failed ({parsed['error']})"
+        ]
+    else:
+        warnings = evaluate_output(parsed)
+
+    print("\nLLM Output:\n")
+    print(json.dumps(parsed, indent=2))
+
+    if warnings:
+
+        print("\nWarnings:")
+
+        for w in warnings:
+            print("-", w)
+
+    else:
+        print("\nNo warnings detected.")
+
+    results.append({
+        "file": str(file),
+        "parsed_violation": v,
+        "llm_output": parsed,
+        "warnings": warnings
+
+    })
+# -----------------------------------
+# Save results
+# -----------------------------------
+
+with open("results_output.json", "w") as f:
+    json.dump(results, f, indent=2)
+
+print("\n====================================")
+print("Results saved to results_output.json")
+print("====================================")

llm-validation/results/broken.txt

@@ -0,0 +1,3 @@
+Validation Report
+
+Conforms: False

llm-validation/results/comment_violation.txt

@@ -0,0 +1,23 @@
+Validation Report
+Conforms: False
+Results (1):
+Constraint Violation in SPARQLConstraintComponent (http://www.w3.org/ns/shacl#SPARQLConstraintComponent):
+	Severity: sh:Violation
+	Source Shape: ve:ClassDeclarationsMustHaveCommentShape
+	Focus Node: :Entity
+	Value Node: :Entity
+	Source Constraint: [ rdf:type sh:SPARQLConstraint ; sh:message Literal("Class declaration is missing an rdfs:comment.") ; sh:select Literal("
+      SELECT $this
+      WHERE {
+        $this rdf:type ?t .
+        FILTER(STRSTARTS(STR($this), "http://www.softlang.org/ontologies/"))
+        FILTER (
+          ?t IN (
+            owl:Class,
+            rdfs:Class
+          )
+        )
+        FILTER NOT EXISTS { $this rdfs:comment ?c . }
+      }
+    ") ]
+	Message: Class declaration is missing an rdfs:comment.

llm-validation/results/foaf_violation.txt

@@ -0,0 +1,24 @@
+Validation Report
+Conforms: False
+Results (1):
+Constraint Violation in SPARQLConstraintComponent (http://www.w3.org/ns/shacl#SPARQLConstraintComponent):
+	Severity: sh:Violation
+	Source Shape: ve:ClassDeclarationsMustHaveFoafLinkShape
+	Focus Node: :Entity
+	Value Node: :Entity
+	Source Constraint: [ rdf:type sh:SPARQLConstraint ; sh:message Literal("Class declaration is missing both foaf:page and foaf:isPrimaryTopicOf.") ; sh:select Literal("
+      SELECT $this
+      WHERE {
+        $this rdf:type ?t .
+        FILTER(STRSTARTS(STR($this), "http://www.softlang.org/ontologies/"))
+        FILTER (
+          ?t IN (
+            owl:Class,
+            rdfs:Class
+          )
+        )
+        FILTER NOT EXISTS { $this foaf:page ?page . }
+        FILTER NOT EXISTS { $this foaf:isPrimaryTopicOf ?topic . }
+      }
+    ") ]
+	Message: Class declaration is missing both foaf:page and foaf:isPrimaryTopicOf.

llm-validation/results/property_comment.txt

@@ -0,0 +1,8 @@
+Validation Report
+Conforms: False
+
+Focus Node: :hasAuthor
+
+Source Shape: ve:PropertyDeclarationsMustHaveCommentShape
+
+sh:message Literal("Property declaration is missing an rdfs:comment.")