Experience · Software engineering internship
Two months inside a live data center.
An AIOps monitoring platform built at Al Akhawayn Data Center: async telemetry, hybrid storage, constrained forecasting, and an LLM-assisted investigation flow grounded in the platform’s own data.
- Organisation
- Al Akhawayn Data Center
- Dates
- 04 May — 04 July 2026 Two months, full-time
- Role
- Software engineering intern Design and implementation
- Stack
- Python · FastAPI · Redis · TimescaleDB · ChromaDB · Prophet · Google AI Studio · Docker Compose
- Repository
- Private Architecture shared at a high level
The context
Al Akhawayn Data Center runs critical hosting and cloud infrastructure. Its team wanted to move beyond reactive alerting — the alert that fires after something breaks — toward a platform that could surface trends, detect anomalies, and give an operator a faster way into the data.
This was a two-month software engineering internship inside the Data Center team, working on that platform end to end. The scope ran from an external monitoring API to a live dashboard: polling, normalisation, storage, forecasting, anomaly detection, retrieval, and an LLM-assisted investigation flow. It was my first time working with this many moving services in one system.
The brief
- Ingest infrastructure metrics on a fixed interval without blocking the rest of the application.
- Keep a hot snapshot for the dashboard while persisting history for analysis.
- Forecast resource trends and detect anomalies against that history.
- Ground an LLM in the platform’s own metrics and runbooks instead of letting it answer from general knowledge.
- Make the environment reproducible with containers and explicit service boundaries.
What I built
- An asynchronous FastAPI ingestion pipeline that polls the Centreon API every 10 seconds and normalises the response through Pydantic models.
- Boundary checks and a health score in the normalisation step, so out-of-range values are handled before storage.
- A multi-tier store: Redis for the latest snapshot, TimescaleDB for time-series history, and ChromaDB for incident and runbook embeddings.
- A constrained Prophet forecasting path: a 0–100% cap and floor for utilisation metrics, a 14-day training horizon, and daily seasonality.
- An anomaly check over the time-series data to flag unusual behaviour alongside the forecast.
- A context orchestrator that classifies intent, gathers metrics and documents, and injects a structured context block into the LLM prompt.
- A WebSocket dashboard that receives normalised snapshots and forecast data for live display.
- A Docker Compose environment that separates the services and keeps the local network isolated.
Four layers
Collect
Centreon API metrics polled asynchronously by FastAPI on a 10-second cadence.
Persist
Redis keeps the latest snapshot; TimescaleDB keeps history; ChromaDB keeps incident context.
Reason
Prophet projects load within physical bounds; a z-score check flags anomalies.
Explain
The orchestrator retrieves context, composes a grounded prompt, and streams to a WebSocket dashboard.
Design decisions
Separate the hot path from history
The dashboard needs the latest state immediately; the forecasting work needs a longer record. Redis and TimescaleDB have different jobs, so the design does not make one do both.
Constrain the forecast
A short training window — roughly 21 time steps, or about 90 minutes of telemetry — pushed the model toward a zero-percent collapse. Extending the horizon to 14 days and enforcing 0–100% bounds stabilised the projection.
Ground the language model
The LLM never receives a bare question. The context orchestrator retrieves related metrics and runbook material first, then composes a structured prompt around them.
What I can share
- The repository and infrastructure configuration are private.
- Hostnames, addresses, credentials, and operational data are intentionally omitted.
- This page describes the design and my contribution; it does not publish production performance or availability figures.
The internship ran from 04 May to 04 July 2026. Everything here is limited to the architecture and work I can discuss publicly.
What I took away
Two lessons stayed with me. First, real-time monitoring is mostly a data-shape problem: normalising and bounding metrics correctly matters more than adding another chart. Second, an LLM is only useful when the context around it is engineered — retrieval, storage, and prompt assembly are the actual work.
I left with a much better sense of how asynchronous services fail, and how to design around that failure instead of pretending it will not happen.