Cross-Stack AI Infrastructure Operations and Diagnostics

Dependency-Aware Intelligence for Infrastructure Monitoring, Diagnosis, and Recovery

Technology Concept Paper

Author: Christopher Soans
Date: September 2026

Download PDF

Figure 1. Cross-Stack AI Infrastructure Operations and Diagnostics

Executive Summary

Modern infrastructure produces extensive telemetry, but faster telemetry alone does not guarantee faster or more accurate diagnosis. Device severity is local to the reporting component; operational severity depends on the component's role, its dependencies, available redundancy, the services and workloads affected, and the time required to recover.

This paper proposes a Cross-Stack AI Infrastructure Operations and Diagnostics architecture that correlates network, hardware, operating-system, compute, storage, application, security, and environmental information through controlled APIs. The AI uses a dependency-aware model to determine which signals matter, identify plausible root causes, evaluate recovery options, and recommend or execute policy-approved remediation.

The objective is not to replace deterministic network protocols or existing management systems. It is to add an intelligence layer that understands infrastructure-wide context and can answer a more useful operational question: given the current state of the infrastructure, what condition creates the greatest service or resilience exposure, why, and what viable action can restore health within the required timeframe?

1. Problem Definition

Networks are designed to absorb failures. Redundant links, ECMP, routing convergence, link aggregation, dual-homing, multipath storage, redundant supervisors, and workload failover can keep services operating after individual failures. This resilience is desirable, but it can also mask deteriorating conditions.

A failed link may cause traffic to move automatically to an alternate path while applications remain available. From the routing protocol's perspective, recovery succeeded. Operationally, however, the environment may have moved from redundant to non-redundant operation. If the remaining path has limited capacity or a long repair exposure, the risk to the service has materially changed even though there is no outage.

The reverse problem also exists: the network cannot route around an issue that it does not know about. A storage controller, server, application, GPU, or cooling system can degrade while every network interface and routing adjacency remains healthy. Sending network telemetry faster does not close that visibility gap.

1.1 Reachability Is Not Service Health

A path can be reachable while the service behind it is degraded. Traditional routing generally evaluates network reachability and network-derived metrics; it does not inherently know that an endpoint behind a healthy path is overloaded, that a storage queue is saturated, or that an application dependency is timing out.

The proposed architecture therefore distinguishes network path health from end-to-end service health. Cross-stack information gives the AI enough context to determine whether a network symptom is the cause of a problem, a consequence of another failure, or simply correlated with it.

1.2 Device Severity Is Not Operational Severity

Syslog levels, traps, thresholds, and hardware alarms describe conditions from the perspective of the reporting system. They remain valuable and should be preserved, but they should not be treated as the sole measure of operational priority.

A warning on a highly connected dependency with no remaining redundancy and a four-hour replacement time may deserve faster attention than a critical alarm on a redundant component that has already failed over successfully and can be restored in minutes.

2. Design Principles

• Preserve native device and application telemetry rather than redefining vendor severity classifications.
• Evaluate telemetry in the context of topology, dependencies, redundancy, workload importance, current utilization, and recovery exposure.
• Use cross-stack APIs so the AI can query relevant systems instead of relying only on network-device telemetry.
• Prefer open standards and interoperable interfaces where practical, while supporting adapters for vendor-specific systems.
• Keep deterministic forwarding, routing, storage, and application control mechanisms responsible for their normal functions.
• Separate observation privileges from action privileges and require policy validation for remediation.
• Verify the outcome after any remediation rather than treating successful command execution as successful recovery.
• Learn from historical incidents, confirmed root causes, repair actions, recovery duration, and unintended consequences.

3. Cross-Stack Information Domains

Domain	Representative Signals	Diagnostic Value
Hardware	Temperature, power, fans, optics, ECC, ASIC/NIC counters, PCIe errors	Identifies physical degradation and component-level faults.
Network OS	Interfaces, queues, routing adjacencies, MAC/ARP/NDP, drops, congestion, CPU/punt rates	Establishes path, forwarding, control-plane, and congestion state.
Server OS	CPU, memory, processes, sockets, TCP state, kernel and driver errors	Separates host or transport behavior from network causes.
Compute / Accelerators	GPU utilization, HBM use, RDMA activity, stalls, job state	Shows whether workloads are compute-, memory-, or data-delivery constrained.
Storage	IOPS, latency, queue depth, multipath state, controller health, replication	Identifies storage-side conditions that may appear as network or application latency.
Application	Response time, transactions, errors, dependency calls, service health	Provides the service-level impact that infrastructure metrics alone cannot establish.
Security	Policy enforcement, drops, authentication, segmentation, threat events	Identifies intentional policy effects and security-related service disruption.
Facilities	Power, cooling, rack and environmental sensors	Correlates infrastructure behavior with power and thermal constraints.
Topology / Dependencies	Physical/logical topology, service relationships, redundancy, alternate paths	Provides the context needed to assign operational significance.

4. Dependency-Aware Operational Model

The AI maintains or consumes a continuously updated dependency graph. Instead of knowing only that Switch A connects to Switch B, it understands which workloads, storage paths, security services, applications, and physical dependencies traverse or rely on that relationship.

The graph should include current redundancy state. If one of two paths fails, the system records not only the failure but also the loss of redundancy, remaining capacity, dependent services, and expected restoration time.

A useful conceptual model for operational importance is:

Operational Importance = Local Severity + Dependency Impact + Redundancy State + Workload/Service Impact + Recovery Exposure

This is not intended as a fixed arithmetic formula. The factors can be weighted according to organizational policy, service objectives, maintenance state, and real-time infrastructure conditions.

5. API and Integration Architecture

The AI requires controlled machine-readable access beyond the network domain. However, the intelligence layer should not need to implement every vendor's native interface directly. A standardized integration layer can expose common infrastructure concepts while adapters translate them to appropriate platform interfaces.

AI Infrastructure Intelligence Layer
↕
Standardized Infrastructure API / Integration Layer
↕
Network | Compute | Storage | Applications | Security | Facilities

Applicable mechanisms may include REST APIs, gNMI, streaming telemetry, Redfish, SNMP, Kubernetes APIs, cloud APIs, storage-controller APIs, message buses, and vendor-specific integrations. The architecture should normalize identity, health, dependencies, state, capabilities, and available actions without discarding domain-specific detail.

5.1 Observation and Action APIs

Observation APIs answer what is happening. They can expose counters, health state, topology, utilization, errors, dependencies, configuration state, and recent changes.

Action APIs answer what can be done. They may allow traffic steering, service restart, workload migration, storage failover, interface changes, QoS changes, or other remediation. Action access should be more restrictive than observation access and governed by policy, role, scope, risk, and approval requirements.

6. Diagnostic Workflow

1. Observe: Collect relevant events and state from participating systems.
2. Establish Context: Determine the affected component's role, dependencies, current redundancy, and services or workloads involved.
3. Correlate: Query related infrastructure domains for temporally and causally relevant symptoms.
4. Diagnose: Develop and rank technical hypotheses using cross-stack evidence and historical incident knowledge.
5. Assess Impact: Estimate current service effect, resilience loss, propagation risk, and recovery exposure.
6. Recommend: Identify viable remediation paths and their expected time, risk, and operational disruption.
7. Validate: Check the proposed action against policy, capacity, dependencies, change restrictions, and known side effects.
8. Authorize / Act: Request operator approval or execute automatically when policy explicitly permits.
9. Verify: Re-query network, compute, storage, application, and other relevant telemetry to confirm that the original condition improved.
10. Learn: Record confirmed root cause, action, outcome, duration, and residual effects for future incidents.

7. Example: Network Symptom with a Storage Root Cause

An application begins reporting increased response latency. Network telemetry shows higher retransmissions and queue occupancy, but no interface or routing failures. A network-only system might recommend additional capacity or traffic redistribution.

The cross-stack AI follows the dependency graph and queries the storage systems serving the affected application. It finds that storage I/O latency and controller queue depth increased before the network symptoms appeared. Server buffers subsequently increased, application responses slowed, and TCP behavior changed. The network condition is real, but it is downstream of the storage problem.

The recommended response can therefore target the storage condition or move the workload to a healthy storage path instead of making an unnecessary network change.

8. Example: Hidden Physical Network Degradation

A storage workload experiences intermittent delays while all routing adjacencies remain healthy. Cross-stack correlation shows rising GPU RDMA retransmissions, storage latency increases, and FEC corrections on an optical link serving the affected dependency chain.

The AI determines that the optical path is degrading but has not failed hard enough to trigger conventional routing convergence. It can recommend shifting traffic to a validated alternate path, confirm that sufficient capacity exists, and schedule physical repair. The immediate mitigation can occur in seconds or minutes while the hardware repair may require hours.

9. MTTR and Recovery-Aware Decision Making

Mean Time to Repair or Restore should influence operational priority and remediation selection. Two incidents with similar local severity can create very different risk depending on how quickly redundancy or service can be restored.

The system should distinguish immediate mitigation from permanent repair. For example, moving traffic to a redundant path may restore service quickly, while replacing an optic may take two hours. Workload migration may offer another option. The AI should present these as separate recovery choices with their dependencies and expected effects.

Historical incident data can improve estimates by recording how long comparable repairs actually took in the same environment, which actions succeeded, and whether those actions created secondary problems.

10. Security and Operational Guardrails

• Use least-privilege credentials and separate read/observation access from write/action access.
• Authenticate and authorize every integration endpoint and maintain auditable records of AI queries and actions.
• Apply policy boundaries by device, domain, service, maintenance window, risk class, and operator role.
• Require human approval for disruptive or high-impact actions unless a narrowly defined automated policy explicitly authorizes them.
• Validate alternate paths and downstream capacity before rerouting traffic or moving workloads.
• Provide rollback procedures and preserve known-good configuration or state where applicable.
• Treat AI conclusions as evidence-based hypotheses until validated by observed outcomes.

11. Relationship to Existing Network Control

This architecture does not require AI to replace OSPF, IS-IS, BGP, EVPN, ECMP, storage multipathing, cluster schedulers, or other deterministic control systems. Those systems continue providing fast, predictable local behavior.

The AI operates above and across those domains. It supplies context that individual control systems do not possess, detects loss of resilience even when failover succeeds, and recommends coordinated actions when a condition spans multiple infrastructure layers.

An independent management or AI Control/Intelligence network can be used to carry management APIs, telemetry, dependency information, and operator access without relying entirely on the production data path being diagnosed.

12. Implementation Approach

• Phase 1 - Read-only visibility: integrate telemetry and APIs, build dependency maps, and produce contextual alerts and diagnostic recommendations.
• Phase 2 - Historical learning: correlate incidents with confirmed root causes, recovery actions, repair duration, and outcomes.
• Phase 3 - Assisted operations: allow the AI to prepare validated remediation plans and commands for operator approval.
• Phase 4 - Policy-controlled automation: automate narrowly defined low-risk actions with mandatory post-action verification and rollback.
• Phase 5 - Cross-domain optimization: coordinate network, compute, storage, application, security, and facilities actions where policies permit.

13. Expected Benefits

• Reduced time spent investigating symptoms in the wrong infrastructure domain.
• Earlier recognition of degraded redundancy before a second failure becomes service-affecting.
• More meaningful prioritization than device severity or log level alone.
• Improved root-cause analysis through dependency and temporal correlation.
• Recovery recommendations that account for service impact, capacity, risk, and restoration time.
• Better use of existing telemetry by selecting contextually relevant information instead of simply collecting more data.
• A vendor-neutral operational model that can evolve through adapters and open interfaces.
• A controlled path from AI-assisted diagnosis toward safe, verifiable automation.

14. Conclusion

The central challenge in modern infrastructure operations is not simply how quickly devices can report telemetry. It is determining which information matters, understanding what depends on the affected component, recognizing conditions outside the network that influence service health, and selecting a viable response within the required recovery timeframe.

A Cross-Stack AI Infrastructure Operations and Diagnostics architecture addresses this gap by combining telemetry with dependency intelligence, cross-domain APIs, historical operational knowledge, recovery awareness, policy-controlled action, and post-remediation verification.

The resulting system does not treat the network as an isolated collection of devices. It treats networking as part of an interconnected service-delivery system in which compute, storage, applications, security, hardware, and facilities can all influence what appears to be a network problem. This broader context is necessary for AI to progress from faster alarm processing to useful infrastructure diagnosis and recovery.

© 2026 Christopher Soans. All rights reserved.

This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).

Related Posts

Loading related posts...