Explorer les clusters LLM

Utilisez cette skill lors d'investigations sur les clusters d'analytique LLM — comprendre quels patterns existent dans votre trafic AI/LLM, comparer le comportement des clusters et approfondir les clusters individuels.

Tools

How clustering works

PostHog clusters LLM traces (ou générations individuelles) par similarité d'embedding. Un workflow Temporal s'exécute périodiquement ou à la demande, produisant des événements cluster stockés sous $ai_trace_clusters (niveau trace) ou $ai_generation_clusters (niveau génération).

Chaque événement cluster contient :

• $ai_clustering_run_id — identifiant unique d'exécution (format : <team_id>_<level>_<YYYYMMDD>_<HHMMSS>[_<job_id>])
• $ai_clustering_level — "trace" ou "generation"
• $ai_window_start / $ai_window_end — fenêtre temporelle analysée
• $ai_total_items_analyzed — nombre de traces/générations traitées
• $ai_clusters — tableau JSON d'objets cluster
• $ai_clustering_params — paramètres d'algorithme utilisés

Forme d'objet cluster (à l'intérieur de $ai_clusters)

{
  "cluster_id": 0,
  "size": 42,
  "title": "User authentication flows",
  "description": "Traces involving login, signup, and token refresh operations",
  "traces": {
    "<trace_or_generation_id>": {
      "distance_to_centroid": 0.123,
      "rank": 0,
      "x": -2.34,
      "y": 1.56,
      "timestamp": "2026-03-28T10:00:00Z",
      "trace_id": "abc-123",
      "generation_id": "gen-456"
    }
  },
  "centroid_x": -2.1,
  "centroid_y": 1.4
}

• cluster_id: -1 est le cluster bruit/outliers (éléments qui n'ont pas correspondu à un cluster)
• Les éléments dans traces sont indexés par ID de trace (niveau trace) ou UUID d'événement génération (niveau génération)
• rank ordonne les éléments par proximité au centroïde (0 = le plus proche)
• x, y sont des coordonnées 2D pour la visualisation (UMAP/PCA/t-SNE réduites)

Clustering jobs

Chaque équipe peut avoir jusqu'à 5 clustering jobs. Un job définit :

• name — label lisible
• analysis_level — "trace" ou "generation"
• event_filters — filtres de propriété limitant quelles traces sont incluses
• enabled — si le job s'exécute selon un calendrier

Les jobs par défaut nommés "Default - trace" et "Default - generation" sont auto-créés et désactivés lorsqu'un job personnalisé est créé pour le même niveau.

Workflow : explorer les clusters

Step 1 — List recent clustering runs

posthog:execute-sql
SELECT
    JSONExtractString(properties, '$ai_clustering_run_id') as run_id,
    JSONExtractString(properties, '$ai_clustering_level') as level,
    JSONExtractString(properties, '$ai_window_start') as window_start,
    JSONExtractString(properties, '$ai_window_end') as window_end,
    JSONExtractInt(properties, '$ai_total_items_analyzed') as total_items,
    timestamp
FROM events
WHERE event IN ('$ai_trace_clusters', '$ai_generation_clusters')
    AND timestamp >= now() - INTERVAL 7 DAY
ORDER BY timestamp DESC
LIMIT 10

Step 2 — Get clusters from a specific run

posthog:execute-sql
SELECT
    JSONExtractString(properties, '$ai_clustering_run_id') as run_id,
    JSONExtractString(properties, '$ai_clustering_level') as level,
    JSONExtractString(properties, '$ai_clustering_job_id') as job_id,
    JSONExtractString(properties, '$ai_clustering_job_name') as job_name,
    JSONExtractString(properties, '$ai_window_start') as window_start,
    JSONExtractString(properties, '$ai_window_end') as window_end,
    JSONExtractInt(properties, '$ai_total_items_analyzed') as total_items,
    JSONExtractRaw(properties, '$ai_clusters') as clusters,
    JSONExtractRaw(properties, '$ai_clustering_params') as params
FROM events
WHERE event IN ('$ai_trace_clusters', '$ai_generation_clusters')
    AND JSONExtractString(properties, '$ai_clustering_run_id') = '<run_id>'
LIMIT 1

Le champ clusters est un tableau JSON. Analysez-le pour voir les titres, tailles et descriptions des clusters.

Important : Le JSON clusters peut être très volumineux (des milliers d'ID de trace avec coordonnées). Quand le résultat est trop volumineux pour un affichage en ligne, il s'auto-persiste dans un fichier. Utilisez print_clusters.py depuis scripts/ pour obtenir un résumé lisible.

Step 3 — Compute metrics for clusters

Pour les clusters au niveau trace, calculez les métriques de coût/latence/token :

posthog:execute-sql
SELECT
    JSONExtractString(properties, '$ai_trace_id') as trace_id,
    sum(toFloat(properties.$ai_total_cost_usd)) as total_cost,
    max(toFloat(properties.$ai_latency)) as latency,
    sum(toInt(properties.$ai_input_tokens)) as input_tokens,
    sum(toInt(properties.$ai_output_tokens)) as output_tokens,
    countIf(properties.$ai_is_error = 'true') as error_count
FROM events
WHERE event IN ('$ai_generation', '$ai_embedding', '$ai_span')
    AND timestamp >= parseDateTimeBestEffort('<window_start>')
    AND timestamp <= parseDateTimeBestEffort('<window_end>')
    AND JSONExtractString(properties, '$ai_trace_id') IN ('<trace_id_1>', '<trace_id_2>', ...)
GROUP BY trace_id

Pour les clusters au niveau génération, faites correspondre par UUID d'événement :

posthog:execute-sql
SELECT
    toString(uuid) as generation_id,
    toFloat(properties.$ai_total_cost_usd) as cost,
    toFloat(properties.$ai_latency) as latency,
    toInt(properties.$ai_input_tokens) as input_tokens,
    toInt(properties.$ai_output_tokens) as output_tokens,
    if(properties.$ai_is_error = 'true', 1, 0) as is_error
FROM events
WHERE event = '$ai_generation'
    AND timestamp >= parseDateTimeBestEffort('<window_start>')
    AND timestamp <= parseDateTimeBestEffort('<window_end>')
    AND toString(uuid) IN ('<gen_uuid_1>', '<gen_uuid_2>', ...)

Step 4 — Drill into specific traces

Une fois que vous avez identifié des clusters intéressants, utilisez les outils de trace pour inspecter des traces individuelles :

posthog:query-llm-trace
{
  "traceId": "<trace_id_from_cluster>",
  "dateRange": {"date_from": "<window_start>", "date_to": "<window_end>"}
}

Investigation patterns

"What kinds of LLM usage do we have?"

1. List recent clustering runs (Step 1)
2. Load the latest run's clusters (Step 2)
3. Review cluster titles and descriptions — each represents a distinct usage pattern
4. Compare cluster sizes to understand traffic distribution

"Which cluster is most expensive / slowest?"

1. Load clusters from a run (Step 2)
2. Extract trace IDs from each cluster
3. Compute metrics per cluster (Step 3)
4. Aggregate: avg(cost), avg(latency), sum(cost) per cluster
5. Compare across clusters

"What's in this cluster?"

1. Load the cluster's traces (from the traces field)
2. Sort by rank (closest to centroid = most representative)
3. Inspect the top 3-5 traces via query-llm-trace to understand the pattern
4. Check the cluster title and description for the AI-generated summary

"Are there error-heavy clusters?"

1. Compute metrics (Step 3) with error_count
2. Calculate error rate per cluster: items_with_errors / total_items
3. Focus on clusters with high error rates
4. Drill into errored traces to find root causes

"How do clusters compare across runs?"

1. List multiple runs (Step 1)
2. Load clusters from each run
3. Compare cluster titles — similar titles across runs indicate stable patterns
4. Track cluster size changes to detect shifts in traffic patterns

Constructing UI links

• Clusters overview: https://app.posthog.com/llm-analytics/clusters
• Specific run: https://app.posthog.com/llm-analytics/clusters/<url_encoded_run_id>
• Cluster detail: https://app.posthog.com/llm-analytics/clusters/<url_encoded_run_id>/<cluster_id>

Always surface these links so the user can verify visually in the PostHog UI.

Tips

• Always set a time range in SQL queries — cluster events without time bounds are slow
• Start with run listing to orient, then drill into specific clusters
• Cluster titles and descriptions are AI-generated summaries — verify by inspecting traces
• The noise cluster (cluster_id: -1) contains outliers that didn't fit any pattern
• Use llma-clustering-job-list to understand what clustering configs are active
• Trace IDs in clusters can be used directly with query-llm-trace for deep inspection
• For large clusters, inspect the top-ranked traces (closest to centroid) for representative examples