Add NetDevOps automation toolkit: Python SDK + CLI, in-app Automation tab, CI gate

.github/workflows/ci.yml

@@ -18,6 +18,7 @@ jobs:
         run: |
           python -m pip install --upgrade pip
           pip install -r requirements.txt
+          pip install -r automation/requirements.txt
           pip install pytest
       - name: Run tests
         run: python -m pytest -q

.github/workflows/netdevops.yml

@@ -0,0 +1,39 @@
+name: NetDevOps
+
+# Treats the network like code: boot NetSim, render configs from templates,
+# push them, and fail the build unless the intended state (interfaces up,
+# routes present, hosts reachable) validates. This is the CI gate a network
+# automation pipeline would run against a staging device pool.
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  provision-and-verify:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+          pip install -r automation/requirements.txt
+      - name: Start NetSim
+        run: |
+          uvicorn app.main:app --host 127.0.0.1 --port 8000 --workers 1 &
+          for i in $(seq 1 30); do
+            curl -sf http://127.0.0.1:8000/api/topology >/dev/null && break
+            sleep 1
+          done
+      - name: Deploy network from code and verify intended state
+        run: |
+          python -m automation.cli deploy automation/labs/two_router_ospf.yaml \
+            --url http://127.0.0.1:8000
+      - name: Run the end-to-end automation example (with drift detection)
+        run: python examples/automation/provision_and_verify.py --url http://127.0.0.1:8000

README.md

@@ -9,11 +9,12 @@ Build a topology, configure routers and switches with real Cisco IOS-style comma
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge)](LICENSE)
 
 ![CI](https://github.com/joxorsayan/netsim/actions/workflows/ci.yml/badge.svg)
+![NetDevOps](https://github.com/joxorsayan/netsim/actions/workflows/netdevops.yml/badge.svg)
 [![Python 3.12](https://img.shields.io/badge/Python-3.12-3776AB?logo=python&logoColor=white)](https://www.python.org/)
 [![FastAPI](https://img.shields.io/badge/FastAPI-009688?logo=fastapi&logoColor=white)](https://fastapi.tiangolo.com/)
 [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)
 
-[**🚀 Try the live demo**](https://netsim-lab.osc-fr1.scalingo.io) · [Features](#-features) · [Quickstart](#-quickstart) · [Architecture](#-architecture) · [AI tutor pipeline](#-ai-tutor--on-device-ml-pipeline) · [Why NetSim?](#-why-netsim)
+[**🚀 Try the live demo**](https://netsim-lab.osc-fr1.scalingo.io) · [Features](#-features) · [Quickstart](#-quickstart) · [Automation](#-network-automation-netdevops) · [Architecture](#-architecture) · [AI tutor pipeline](#-ai-tutor--on-device-ml-pipeline) · [Why NetSim?](#-why-netsim)
 
 ![NetSim topology editor and Cisco-style CLI](docs/img/hero-cli.png)
 
@@ -27,7 +28,7 @@ Build a topology, configure routers and switches with real Cisco IOS-style comma
 
 It covers a **CCNA-level** feature set (VLANs & 802.1Q trunking, STP, OSPF/EIGRP/RIP/BGP routing, DHCP, NAT/PAT, ACLs, IPv6 dual-stack, FHRP, EtherChannel) and ships with three things most free simulators lack: a **packet capture exporter** (`.pcap` for Wireshark), an **on-device AI tutor** that explains misconfigurations, and **labs-as-code with auto-grading**.
 
-> **Keywords:** network simulator, Cisco Packet Tracer alternative, CCNA lab, OSPF/BGP/EIGRP, VLAN, spanning tree, packet capture, Wireshark, FastAPI, on-device LLM, WebGPU, transformers.js.
+> **Keywords:** network simulator, Cisco Packet Tracer alternative, CCNA lab, OSPF/BGP/EIGRP, VLAN, spanning tree, packet capture, Wireshark, FastAPI, on-device LLM, WebGPU, transformers.js, network automation, NetDevOps, Python network automation, Jinja2 config templating, config-as-code, drift detection, intent-based networking, pyATS, Batfish.
 
 ## 🎯 Why NetSim?
 
@@ -50,6 +51,7 @@ It's also a from-scratch demonstration that core networking — packet forwardin
 | 📡 **Packet capture** | Every frame hop is logged, replayable, and exportable as a **`.pcap`** that opens directly in **Wireshark**. |
 | 🤖 **AI tutor** | Ask *"why can't PC1 ping PC2?"* — it scans your live topology (interfaces down, subnet/area mismatch, missing gateway, duplicate IP) and explains the fix. Runs a fine-tuned LLM **on-device in your browser** (no API key), with a deterministic rules engine for ground truth. |
 | 🎓 **Labs-as-code + auto-grading** | Export a lab to JSON, apply a spec (devices + links + config), and grade student configs against checks (`ping`, `route`, `interface_up`) for a 0–100 score. |
+| 🔧 **Network automation (NetDevOps)** | A **Python SDK + CLI** that drives the sim like real gear: **Jinja2 config templating** from per-device vars, config **push**, **drift detection**, structured **intent** (YANG-ish), and **intended-state validation** (routes/reachability) — with a **CI pipeline** that fails on misconfig and an in-app **Automation** tab. |
 
 ### Supported CLI (highlights)
 
@@ -95,12 +97,19 @@ app/
     pcap.py      serialise a capture to a libpcap (.pcap) for Wireshark
     tutor.py     rule-based analyzer + optional Cloudflare Workers AI tutor
     labs.py      lab-as-code auto-grading checks
+    automation.py  config templating, intent↔CLI, snapshot/drift, validation
   tutor_ai/      training-data generators for the on-device tutor model
   workspace.py   in-memory, per-client isolated topologies + lab apply/export
   main.py        FastAPI REST API + static SPA
-  static/        vanilla-JS topology editor, CLI console, capture/tutor/lab tabs
+  static/        vanilla-JS topology editor, CLI console, capture/tutor/lab/automation tabs
+automation/      Python automation toolkit: SDK + CLI + Jinja2 templates + network defs
+  client.py      NetSimClient — typed REST SDK
+  cli.py         `python -m automation.cli` (render/deploy/validate/drift/intent)
+  netdevops.py   render → push → validate pipeline
+  labs/          declarative network definitions (topology + vars + assertions)
+examples/        runnable automation examples (provision_and_verify.py)
 ai/              ML pipeline: dataset → Kaggle T4 training → ONNX → CDN delivery
-tests/           engine + CLI + advanced-feature unit tests
+tests/           engine + CLI + advanced-feature + automation toolkit tests
 ```
 
 **Tech stack:** Python 3.12 · FastAPI · vanilla JS/SVG front-end · Qwen2.5 LoRA fine-tune · ONNX Runtime + transformers.js (WebGPU/WASM) · Scalingo (app) + Cloudflare Pages/Backblaze B2 (model CDN).
@@ -120,6 +129,43 @@ The full, reproducible pipeline lives in [`ai/`](ai/README.md):
 
 To use a hosted LLM instead (Cloudflare Workers AI, free tier), set `CF_ACCOUNT_ID` + `CF_API_TOKEN`; it calls `@cf/meta/llama-3.1-8b-instruct` and falls back to the rules engine on any error.
 
+## 🔧 Network automation (NetDevOps)
+
+NetSim isn't just clickable — its REST API is driven by a small **Python automation toolkit** ([`automation/`](automation/)), so you can treat the network as code the way you would with netmiko/NAPALM + Jinja2 + pyATS:
+
+- **Config as code** — render Cisco configs from **Jinja2 templates** + per-device variables.
+- **Push** — apply config to devices (netmiko `send_config_set` style).
+- **Validate intended state** — assert interfaces up, routes present, hosts reachable (Batfish/pyATS style).
+- **Drift detection** — snapshot a running-config baseline and diff against it.
+- **Structured intent** — a YANG-ish JSON view of a device, and the reverse (intent → CLI).
+
+One declarative file describes the whole network ([`automation/labs/two_router_ospf.yaml`](automation/labs/two_router_ospf.yaml)); a single command deploys **and** verifies it:
+
+```bash
+pip install -r automation/requirements.txt
+uvicorn app.main:app --port 8000 &
+python -m automation.cli deploy automation/labs/two_router_ospf.yaml --url http://localhost:8000
+```
+
+```
+=== NetDevOps deploy: two-router-ospf ===
+[OK] topology: 4 devices, 3 links
+[OK] push R1: 13 lines   [OK] push R2: 13 lines   [OK] push PC1/PC2
+[OK] saved running-config baseline
+[OK] validation: 8/8 passed (score 100/100)
+=== result: PASS ===
+```
+
+The **[NetDevOps CI workflow](.github/workflows/netdevops.yml)** runs this on every push (render → push → validate), failing the build on any misconfiguration. The live app's **Automation** tab exposes the same flow in the browser. More in [`examples/automation/`](examples/automation/).
+
+```python
+from automation import NetSimClient
+
+nc = NetSimClient("http://localhost:8000")
+nc.push_config("R1", ["interface g0/0", "ip address 10.0.0.1 255.255.255.252", "no shutdown"])
+print(nc.validate([{"type": "reachable", "src": "PC1", "dst": "192.168.2.10"}]))
+```
+
 ## ☁️ Deploy (Scalingo / any buildpack host)
 
 The repo is buildpack-ready (`requirements.txt`, `.python-version`, `Procfile`):

app/static/app.js

@@ -1128,5 +1128,203 @@ document.getElementById("topoImportFile").addEventListener("change", (ev) => {
 });
 renderTopoList();
 
+// ---- NetDevOps automation -------------------------------------------------
+// A two-router OSPF backbone used to demo config-as-code, validation and drift.
+const AUTO_DEMO_TOPO = {
+  devices: [
+    { kind: "host", name: "PC1", ports: 1, x: 80, y: 140 },
+    { kind: "router", name: "R1", ports: 2, x: 300, y: 240 },
+    { kind: "router", name: "R2", ports: 2, x: 560, y: 240 },
+    { kind: "host", name: "PC2", ports: 1, x: 780, y: 140 },
+  ],
+  links: [
+    { dev_a: "PC1", if_a: "eth0", dev_b: "R1", if_b: "GigabitEthernet0/1" },
+    { dev_a: "R1", if_a: "GigabitEthernet0/0", dev_b: "R2", if_b: "GigabitEthernet0/0" },
+    { dev_a: "R2", if_a: "GigabitEthernet0/1", dev_b: "PC2", if_b: "eth0" },
+  ],
+  config: {}, checks: [],
+};
+// Per-device config bodies (push wraps enable/configure terminal/end).
+const AUTO_DEMO_CONFIG = {
+  R1: ["hostname R1", "interface GigabitEthernet0/1", "ip address 192.168.1.1 255.255.255.0",
+    "no shutdown", "exit", "interface GigabitEthernet0/0", "ip address 10.0.0.1 255.255.255.252",
+    "no shutdown", "exit", "router ospf 1", "network 192.168.1.0 0.0.0.255 area 0",
+    "network 10.0.0.0 0.0.0.3 area 0", "exit"],
+  R2: ["hostname R2", "interface GigabitEthernet0/1", "ip address 192.168.2.1 255.255.255.0",
+    "no shutdown", "exit", "interface GigabitEthernet0/0", "ip address 10.0.0.2 255.255.255.252",
+    "no shutdown", "exit", "router ospf 1", "network 192.168.2.0 0.0.0.255 area 0",
+    "network 10.0.0.0 0.0.0.3 area 0", "exit"],
+  PC1: ["interface eth0", "ip address 192.168.1.10 255.255.255.0", "no shutdown", "exit",
+    "ip default-gateway 192.168.1.1"],
+  PC2: ["interface eth0", "ip address 192.168.2.10 255.255.255.0", "no shutdown", "exit",
+    "ip default-gateway 192.168.2.1"],
+};
+const AUTO_TPL_DEFAULT = `hostname {{ hostname }}
+{% for i in interfaces %}
+interface {{ i.name }}
+ ip address {{ i.ip }} {{ i.mask }}
+ no shutdown
+ exit
+{% endfor %}
+router ospf {{ ospf.pid }}
+{% for n in ospf.networks %}
+ network {{ n.net }} {{ n.wildcard }} area {{ n.area }}
+{% endfor %}
+ exit`;
+const AUTO_VARS_DEFAULT = JSON.stringify({
+  hostname: "R1",
+  interfaces: [
+    { name: "GigabitEthernet0/1", ip: "192.168.1.1", mask: "255.255.255.0" },
+    { name: "GigabitEthernet0/0", ip: "10.0.0.1", mask: "255.255.255.252" },
+  ],
+  ospf: { pid: 1, networks: [
+    { net: "192.168.1.0", wildcard: "0.0.0.255", area: 0 },
+    { net: "10.0.0.0", wildcard: "0.0.0.3", area: 0 },
+  ] },
+}, null, 2);
+const AUTO_ASSERT_DEFAULT = JSON.stringify([
+  { type: "no_duplicate_ip" },
+  { type: "interface_up", device: "R1", interface: "GigabitEthernet0/0" },
+  { type: "route_present", device: "R1", network: "192.168.2.0/24" },
+  { type: "reachable", src: "PC1", dst: "192.168.2.10" },
+], null, 2);
+
+function autoEl(id) { return document.getElementById(id); }
+
+function refreshAutoDevices() {
+  const sel = autoEl("autoDevice");
+  if (!sel) return;
+  const cur = sel.value;
+  const names = (topo.devices || []).map((d) => d.name);
+  sel.innerHTML = names.map((n) => `<option value="${n}">${n}</option>`).join("");
+  if (names.includes(cur)) sel.value = cur;
+  autoEl("autoDevs").textContent = names.length
+    ? `${names.length} devices: ${names.join(", ")}` : "no devices — load the OSPF demo";
+}
+
+let autoSeeded = false;
+function seedAutoFields() {
+  if (autoSeeded) return;
+  autoEl("autoTpl").value = AUTO_TPL_DEFAULT;
+  autoEl("autoVars").value = AUTO_VARS_DEFAULT;
+  autoEl("autoAssert").value = AUTO_ASSERT_DEFAULT;
+  autoSeeded = true;
+}
+
+autoEl("autoDemoBtn").addEventListener("click", async () => {
+  const out = autoEl("autoRenderOut");
+  try {
+    const res = await api("POST", "/lab/apply", { spec: AUTO_DEMO_TOPO });
+    topo = res.topology;
+    for (const [dev, cfg] of Object.entries(AUTO_DEMO_CONFIG)) {
+      await api("POST", "/automation/push", { device: dev, config: cfg });
+    }
+    topo = await api("GET", "/topology");
+    cliHistory = {}; closeCli(); render(); refreshAutoDevices();
+    out.textContent = "Provisioned PC1–R1=OSPF=R2–PC2 via automation push.\n" +
+      "OSPF area 0 between R1/R2; PC1 and PC2 in different subnets.\n" +
+      "Now Render & push a device, Validate, or check Drift.";
+  } catch (e) { out.textContent = "% " + e.message; }
+});
+
+function parseJson(id, out) {
+  try { return JSON.parse(autoEl(id).value); }
+  catch (e) { out.textContent = "% invalid JSON in " + id + ": " + e.message; return undefined; }
+}
+
+autoEl("autoRenderBtn").addEventListener("click", async () => {
+  const out = autoEl("autoRenderOut");
+  const variables = parseJson("autoVars", out);
+  if (variables === undefined) return;
+  try {
+    const res = await api("POST", "/automation/render", { template: autoEl("autoTpl").value, variables });
+    out.textContent = "Rendered config:\n\n" + res.lines.join("\n");
+  } catch (e) { out.textContent = "% " + e.message; }
+});
+
+autoEl("autoPushBtn").addEventListener("click", async () => {
+  const out = autoEl("autoRenderOut");
+  const variables = parseJson("autoVars", out);
+  if (variables === undefined) return;
+  const device = autoEl("autoDevice").value;
+  if (!device) { out.textContent = "% no target device — load the OSPF demo first."; return; }
+  try {
+    const res = await api("POST", "/automation/render",
+      { template: autoEl("autoTpl").value, variables, device, push: true });
+    topo = await api("GET", "/topology"); render(); refreshAutoDevices();
+    const r = res.result || {};
+    out.textContent = `Pushed ${res.lines.length} lines to ${device}:\n\n` +
+      res.lines.join("\n") +
+      (r.errors && r.errors.length ? "\n\n! errors:\n" + r.errors.join("\n") : "\n\n✓ applied cleanly");
+  } catch (e) { out.textContent = "% " + e.message; }
+});
+
+autoEl("autoPyBtn").addEventListener("click", async () => {
+  const device = autoEl("autoDevice").value || "R1";
+  const tpl = autoEl("autoTpl").value;
+  const vars = autoEl("autoVars").value.trim();
+  const script =
+    `from automation import NetSimClient\n` +
+    `from automation.netdevops import render_configs\n\n` +
+    `nc = NetSimClient("${location.origin}")\n\n` +
+    `template = """${tpl}"""\n` +
+    `variables = ${vars}\n\n` +
+    `# render server-side and push to the device\n` +
+    `result = nc.render(template, variables, device="${device}", push=True)\n` +
+    `print("\\n".join(result["lines"]))\n`;
+  const out = autoEl("autoRenderOut");
+  try { await navigator.clipboard.writeText(script); out.textContent = "# copied Python to clipboard:\n\n" + script; }
+  catch { out.textContent = "# Python SDK script:\n\n" + script; }
+});
+
+autoEl("autoValidateBtn").addEventListener("click", async () => {
+  const out = autoEl("autoValidateOut");
+  const assertions = parseJson("autoAssert", out);
+  if (assertions === undefined) return;
+  try {
+    const res = await api("POST", "/automation/validate", { assertions });
+    const lines = res.results.map((r) => {
+      const a = r.assertion;
+      const label = a.type + (a.device ? ` ${a.device}` : "") + (a.src ? ` ${a.src}->${a.dst}` : "") +
+        (a.network ? ` ${a.network}` : "") + (a.interface ? ` ${a.interface}` : "");
+      return `${r.passed ? "PASS" : "FAIL"}  ${label} — ${r.detail}`;
+    });
+    out.textContent = `${res.compliant ? "✓ COMPLIANT" : "✗ NON-COMPLIANT"} — ` +
+      `${res.passed}/${res.total} passed (score ${res.score}/100)\n\n` + lines.join("\n");
+  } catch (e) { out.textContent = "% " + e.message; }
+});
+
+autoEl("autoSnapBtn").addEventListener("click", async () => {
+  const out = autoEl("autoDriftOut");
+  try {
+    const res = await api("POST", "/automation/snapshot");
+    const n = Object.keys(res.baseline || {}).length;
+    out.textContent = `Saved running-config baseline for ${n} device(s). ` +
+      `Change a config (e.g. shut an interface) then Check drift.`;
+  } catch (e) { out.textContent = "% " + e.message; }
+});
+
+autoEl("autoDriftBtn").addEventListener("click", async () => {
+  const out = autoEl("autoDriftOut");
+  try {
+    const res = await api("GET", "/automation/drift");
+    if (res.in_sync) { out.textContent = "✓ in sync — running-config matches the baseline."; return; }
+    const lines = ["✗ drift detected vs baseline:"];
+    for (const [dev, change] of Object.entries(res.drift)) {
+      (change.removed || []).forEach((l) => lines.push(`  - ${dev}: ${l}`));
+      (change.added || []).forEach((l) => lines.push(`  + ${dev}: ${l}`));
+    }
+    out.textContent = lines.join("\n");
+  } catch (e) {
+    out.textContent = e.message.includes("baseline")
+      ? "% no baseline yet — click Save baseline first." : "% " + e.message;
+  }
+});
+
+// seed fields + refresh device list whenever the Automation tab opens
+document.querySelector('[data-tab="auto"]').addEventListener("click", () => {
+  seedAutoFields(); refreshAutoDevices();
+});
+
 // ---- boot -----------------------------------------------------------------
 (async () => { topo = await api("GET", "/topology"); render(); })();