README.md

metadata

title: NervousSystem Env
emoji: 🧠
colorFrom: indigo
colorTo: blue
sdk: docker
app_port: 7860
pinned: false

🧠 NervousSystem-Env

An AI agent fixing the infrastructure that trains AI.
Every minute of GPU cluster downtime costs $5,000 in wasted compute.

NervousSystem-Env is an OpenEnv-compliant reinforcement learning environment where an agent acts as an SRE operating a distributed GPU training fleet under failure. The agent must diagnose incidents, coordinate specialist workers, and execute recovery actions while minimizing token and coordination waste. The environment models realistic infrastructure failure modes (OOM, topology congestion, NCCL desync, version cascades), includes partial observability, and rewards long-horizon remediation over shortcut behavior. This environment directly models one of the most expensive unsolved problems in modern AI infrastructure.

Links

• Hosted OpenEnv Space: https://huggingface.co/spaces/v4xsh/nervousystem-env
• Mini-blog (Hugging Face): https://huggingface.co/spaces/v4xsh/nervousystem-env/blob/main/Blog.md
• Trained LoRA adapter: https://huggingface.co/v4xsh/nervousystem-sre-agent-lora
• Final training/evaluation job logs: https://huggingface.co/jobs/v4xsh/69eda4d1d2c8bd8662bcf435
• Training reproduction notebook (Colab): https://colab.research.google.com/drive/1twXiRHoAxchy9UUgn7S15a2ac3GrtAlW?usp=sharing
• In-repo project pitch: PITCH.md
• Training evidence: TRAINING_EVIDENCE.md

Why This Matters

• GPU OOM (XID 79): CUDA out-of-memory on one rank stalls the entire AllReduce collective. Training halts fleet-wide.
• Spine Switch Congestion: Ring topology crosses oversubscribed spine links, cutting interconnect bandwidth 40%+.
• Compilation Desync: Different ranks compile different NCCL collectives due to data-dependent branching. Job hangs permanently.
• LD_LIBRARY_PATH Cascade: Wrong NCCL version loaded (2.21.5 vs 2.27.0) triggers message truncation errors that propagate across all communicators. Severity-1 incident.

Architecture

NervousSystem-Env uses a Fleet AI Supervisor-Worker design. The supervisor receives ClusterObservation in a partially observable setting where telemetry can go stale every 3 steps unless diagnostics are refreshed. It delegates specialist sub-tasks through /delegate, and workers return structured output with confidence and uncertainty signals. The supervisor also operates under a delegation budget of 10 per episode, and over-budget delegations reduce coordination reward.

Simulation fidelity is built around realistic infrastructure diagnostics. NCCL logs use exact <hostname>:<pid>:<tid> NCCL INFO formatting, and Flight Recorder payloads follow PyTorch 2.5 v2.5 schema (pg_config, pg_status, circular buffer warnings). Telemetry intentionally includes 30% red-herring alerts and 20% incomplete observability to prevent surface-level pattern matching.

Supervisor Agent
│
├── LogInspectorWorker    (flight recorder, NCCL subsystem logs)
├── PatchAgentWorker      (staged patching: identify→diff→apply)
├── TopoAgentWorker       (topology reorder, bandwidth check)
└── VersionCheckerWorker  (NCCL version, LD_LIBRARY_PATH audit)

Tasks

Task	Difficulty	Max Steps	Failure Type	Key Challenge
easy	easy	50	OOM rank failure	Identify failing rank from noisy telemetry
medium	medium	50	Spine congestion	Confirm fix held under throughput jitter
hard	hard	50	Compilation desync	3-stage patch: identify→diff→apply
cascade	cascade	120	Version cascade	Solve OOM→congestion→desync in order
fleet_coordination	fleet_coordination	50	Multi-agent incident	Delegate to specialists, reach coalition, then remediate

Cascade uses strict phase gating with precondition enforcement. The agent must solve OOM diagnosis, congestion recovery, then desync investigation/patch in sequence; patch attempts without required investigation are blocked, so phases cannot be skipped by direct guess-fixing.

Reward Model

Reward = 0.60 × R_success + 0.30 × R_subgoal − 0.10 × log(total_tokens)

Component	Weight	Description
R_success	0.60	Binary: job_status in {recovered, running}
R_subgoal	0.30	Continuous partial credit per phase/diagnosis
log(total_tokens)	0.10	Efficiency penalty — verbose agents score lower

Final grade applies token efficiency as a multiplier on the raw task score, so solved low-token episodes keep high scores while token stuffing collapses the final score. Additional penalties are active: destructive action penalty (-0.2), delegation over-budget penalty (-0.05 per excess delegation), and anti-shortcut precondition enforcement for diagnosis-before-patch behavior.

Observation Space

NervousSystem-Env is intentionally partially observable:

• Surface logs refresh every 3 steps (stale_telemetry flag)
• 30% of updates contain red-herring node alerts
• 20% of updates mask true degraded node status
• Deep diagnostics only via inspect_flight_recorder or query_nccl_logs
• cumulative_tokens visible to the agent (self-awareness of efficiency)

Quick Start

# Install
pip install -r requirements.txt

# Start environment server
uvicorn app.main:app --host 0.0.0.0 --port 7860

# Start SRE War Room dashboard
python dashboard/war_room.py  # http://localhost:7861

# Run baseline agent (requires HF_TOKEN or API key)
export HF_TOKEN=your_token
python inference.py

# Run seed variance report
python scripts/seed_variance.py

# Train LoRA policy with SFT warmup, then optional GRPO
pip install -r requirements-training.txt
python training/grpo_train.py

# Resume GRPO from existing SFT adapter (continues with environment reward)
RESUME_FROM_ADAPTER=sre_agent_lora \
  GRPO_ADAPTER_DIR=sre_agent_lora_grpo \
  GRPO_MAX_STEPS=120 \
  python training/grpo_train.py

# One-shot overnight pipeline (server + SFT + GRPO + before/after constrained eval)
bash scripts/run_overnight.sh

# Evaluate final trained policy with phase-aware constrained action scoring
EVAL_LENGTH_NORM_POWER=1.0 python scripts/evaluate_model.py --label sft_oraclefix_phaseaware --adapter sre_agent_lora_oraclefix --seeds 3 --tasks easy,medium,hard,cascade --max-steps 8

Judge Reproduction

Run the environment:

pip install -r requirements.txt
uvicorn app.main:app --host 0.0.0.0 --port 7860

In another terminal:

python scripts/reward_audit.py
python scripts/exploit_test.py
python scripts/evaluate.py quick
python scripts/evaluate_multi_agent.py
python scripts/generate_long_horizon_trace.py
EVAL_LENGTH_NORM_POWER=1.0 python scripts/evaluate_model.py --label sft_oraclefix_phaseaware --adapter sre_agent_lora_oraclefix --seeds 3 --tasks easy,medium,hard,cascade --max-steps 8

Hosted Space smoke checks:

curl https://v4xsh-nervousystem-env.hf.space/health
curl -X POST https://v4xsh-nervousystem-env.hf.space/reset -H 'Content-Type: application/json' -d '{"task_id":"easy","seed":42}'

API Endpoints

Endpoint	Method	Description
/health	GET	Health check
/reset	POST	Reset episode (supports challenge_type, difficulty_multiplier)
/step	POST	Apply one SRE action
/state	GET	Current observation
/grade	POST	Grade episode (includes MER + meta-reward)
/tasks	GET	List registered manifest and experimental tasks
/episode	GET	Current episode summary
/delegate	POST	Supervisor delegates to specialist worker
/consensus	GET	Poll all workers for recommended action
/coalition	POST	Attempt 2-worker coalition action
/coalition/options	GET	Available coalition actions
/challenge/generate	POST	Generate novel failure composition
/challenge/list	GET	List generated challenges
/curriculum	GET	Adaptive curriculum state
/curriculum/events	GET	Difficulty escalation/backoff log

Hackathon Theme Coverage

Theme	Coverage	Implementation
Theme 1: Multi-Agent	✅ Strong	fleet_coordination requires specialist delegation and coalition before high score
Theme 2: Long-Horizon	✅ Supported	Cascade has 120-step budget plus a 56-event long-horizon trace artifact
Theme 3.1: Professional Tasks	✅ Strong	NCCL INFO logs, Flight Recorder v2.5, staged patch workflow
Theme 4: Self-Improvement	✅ Moderate	Curriculum changes red herrings, telemetry masking, refresh interval, and secondary failures

Training Results

Oracle Baseline (50 seeds, deterministic optimal agent)

Task	Mean Score	Std	Pass Rate	Mean Steps	CI 95%
easy	0.990	0.000	100.0%	1.00	[0.990, 0.990]
medium	0.990	0.000	100.0%	7.00	[0.990, 0.990]
hard	0.737	0.072	100.0%	5.56	[0.720, 0.753]
cascade	0.857	0.063	100.0%	5.30	[0.840, 0.875]
fleet_coordination	0.990	0.000	100.0%	delegation-only	[0.990, 0.990]
black_swan	0.010	0.000	0%*	—	[0.010, 0.010]

*Black Swan is an experimental ultra-hard stress test in code. The main OpenEnv manifest focuses on easy, medium, hard, cascade, and fleet_coordination tasks for judging.

Random Agent Baseline (50 seeds, lower bound)

Task	Mean Score	Pass Rate
easy	0.299	26.0%
medium	0.926	100.0%
hard	0.360	18.0%
cascade	0.039	0.0%
fleet_coordination	0.050	0.0%

*Medium pass threshold raised to 0.7 after random agent audit

Trained LLM Policy Result

We trained a Qwen2.5-7B LoRA policy on Hugging Face Jobs using an NVIDIA A10G. The policy was trained with corrected oracle SRE trajectories from this OpenEnv environment, then evaluated with phase-aware constrained action scoring: the model ranks valid next-step JSON tool actions for the current workflow phase, and the environment executes the highest-likelihood action.

Training loss fell from 2.53 → ~0.10 over the 40 logged SFT steps preserved in results/sft_warmup_metrics.json (sourced from trainer.state.log_history). The trained policy improved to 0.915 mean score and 100% pass rate over 12 model-policy episodes.

Note on training-evidence provenance. The published sre_agent_lora_oraclefix adapter and the 0.915 evaluation are from a 300-step SFT run on Hugging Face Jobs (terminal proof in docs/proof/final_trained_console.png, job log 69eda4d1d2c8bd8662bcf435). The per-step results/sft_warmup_metrics.json (40 logged entries / ~160 actual steps at logging_steps=4) is from a faithful local re-run with the same script, dataset, and hyperparameters, published so judges can inspect a real trainer.state.log_history directly. Both runs converge to the same loss regime (≈0.08–0.10).

Raw HF Jobs / terminal proof (same evaluator, same base model, only the LoRA adapter changes)

Before — raw unsloth/Qwen2.5-7B-Instruct-bnb-4bit with no adapter, 9 episodes, mean 0.239, pass rate 0%:

After — same base + SFT LoRA sre_agent_lora_oraclefix, 12 episodes, mean 0.915, pass rate 100%:

Task	Trained LoRA phase-aware score
easy	0.990 / 0.990 / 0.990
medium	0.990 / 0.990 / 0.990
hard	0.850 / 0.850 / 0.990
cascade	0.782 / 0.782 / 0.782
overall	0.915 mean, 100% pass rate

Interpretation: the trained policy learns reliable remediation for rank diagnosis, topology congestion, staged desync patching, and cascade recovery when evaluated under the same phase-aware action constraints used by the environment workflow.

Artifacts:

• LoRA adapter: https://huggingface.co/v4xsh/nervousystem-sre-agent-lora
• Final training/evaluation job logs: https://huggingface.co/jobs/v4xsh/69eda4d1d2c8bd8662bcf435
• Plot generator: scripts/plot_final_evidence.py

Multi-Agent Coordination Evaluation

The fleet_coordination task requires supervisor-worker delegation before remediation. A deterministic coordination oracle delegates to log_inspector, delegates to version_checker, polls consensus, and executes a topology_version_fix coalition. It reaches 0.990 mean score / 100% pass rate over 10 seeds. A direct-action baseline that skips specialist evidence scores 0.100 mean score / 0% pass rate.

Evidence: results/multi_agent_eval.json

Long-Horizon Trace

In addition to short optimal solves, results/cascade_long_horizon_trace.json contains a 56-event cascade trace. It demonstrates stale telemetry refresh, phase-gated remediation, sustained monitoring, delayed desync investigation, staged file identification, and final patch recovery. The trace is intentionally non-oracle-like: it shows the environment can exercise long-running state tracking, not only shortest-path scripts.

Reward Integrity

The reward system is independently audited against 4 properties. Final grade applies the MER token-efficiency penalty as a multiplier on the raw task score, so solved low-token episodes keep their score while token stuffing and verbose loops cannot preserve a high final score.

Property	Result	Evidence
Monotonicity	✅ Correct action > wrong > noop	results/reward_audit.json
Anti-Gaming	✅ All 4 exploits blocked	results/reward_audit.json
Token Efficiency	✅ Verbose agents penalized	results/reward_audit.json
Determinism	✅ Same seed = same score	results/reward_audit.json

Exploit resistance verified:

Exploit	Score	Status
Noop spam (50 steps)	0.020	✅ BLOCKED
Wrong rank spam (20 steps)	0.020	✅ BLOCKED
Patch without investigation	0.030	✅ BLOCKED
Destructive action spam	0.010	✅ BLOCKED
Token stuffing (100k tokens)	0.010	✅ BLOCKED
Cascade phase skip	0.010	✅ BLOCKED
Reward farming loop	0.351	✅ BLOCKED

Run audit: python scripts/reward_audit.py
Run exploit test: python scripts/exploit_test.py

Adaptive Curriculum

The environment self-adjusts difficulty based on agent performance:

Level	Pass Rate Trigger	Red Herring %	Telemetry Mask %	Refresh Interval
1 (Base)	< 75%	30%	20%	3 steps
2 (Hard)	> 75% sustained	50%	35%	5 steps
3 (Hardest)	> 75% at level 2	70%	50%	7 steps

• Escalation: triggered when pass_rate > 0.75 AND trend > 0.01 over last 20 episodes, with 10-episode cooldown
• Backoff: triggered when pass_rate < 0.20 AND trend < -0.01, with 5-episode cooldown
• Meta-reward: agents earn bonus reward for improving over their recent rolling window (not just for success)
• Wiring: curriculum levels now change telemetry red-herring probability, telemetry masking, secondary failure probability, and telemetry refresh interval.

Monitor live: GET /curriculum
View events: GET /curriculum/events

Project Structure

nervousystem-env/
├── app/
│   ├── config.py          # Scenarios, black_swan, rack layouts, challenge types
│   ├── env.py             # Core env, Fleet + Curriculum + Challenge wiring
│   ├── main.py            # FastAPI — 15 endpoints
│   └── models.py          # Pydantic v2 models, Coalition/Delegate schemas
├── graders/
│   ├── base.py            # GradeResult, apply_mer(), meta-reward support
│   ├── black_swan_grader.py
│   ├── cascade_grader.py
│   ├── easy_grader.py
│   ├── fleet_coordination_grader.py
│   ├── hard_grader.py
│   └── medium_grader.py
├── simulation/
│   ├── challenge_generator.py  # 5 novel failure compositions
│   ├── cluster.py              # State machine, staged patching, non-linear dynamics
│   ├── curriculum.py           # Adaptive difficulty manager, meta-reward
│   ├── failures.py             # Cascade phase injection, secondary failures
│   ├── fleet.py                # Fleet AI, coalition actions, consensus
│   └── telemetry.py       # NCCL logs, red herrings, staleness
├── tasks/
│   ├── base.py            # EpisodeTracker, preconditions, investigation_count
│   ├── black_swan.py      # 300-step ultra-hard task, false positives, irreversible mistakes
│   ├── cascade.py         # 3-phase chained task, phase gating
│   ├── easy.py
│   ├── fleet_coordination.py
│   ├── hard.py
│   └── medium.py
├── dashboard/
│   └── war_room.py        # Gradio SRE War Room (port 7861)
├── training/
│   └── grpo_train.py      # LoRA SFT warmup + optional GRPO over env rollouts
├── scripts/
│   ├── evaluate.py             # Oracle + random eval harness, 50 seeds, bootstrap CI
│   ├── evaluate_multi_agent.py
│   ├── evaluate_model.py       # Raw/trained model-policy eval with action scoring
│   ├── exploit_test.py         # exploit attack proofs
│   ├── generate_long_horizon_trace.py
│   ├── plot_final_evidence.py  # Rebuilds committed PNG plots from eval JSON + loss log
│   ├── reward_audit.py         # 4 reward property audits
│   ├── run_overnight.sh        # One-shot env+SFT+GRPO+eval pipeline
│   └── seed_variance.py        # Seed variance report
├── results/
│   ├── before_vs_after_scores.png      # Raw 0.239 vs trained 0.915, same evaluator
│   ├── cascade_long_horizon_trace.json
│   ├── exploit_test.json
│   ├── final_phaseaware_model_eval.json
│   ├── final_score_by_task.png         # Trained LoRA mean score per task
│   ├── final_sft_loss_curve.png        # SFT loss 2.53 -> 0.10 over 40 logged steps
│   ├── sft_warmup_metrics.json         # Real per-step loss/lr/grad_norm log_history
│   ├── multi_agent_eval.json
│   └── reward_audit.json
├── docs/
│   └── proof/
│       ├── raw_untrained_console.png   # HF Jobs eval, untrained base model, 0.239 mean
│       └── final_trained_console.png   # Local terminal eval, trained LoRA, 0.915 mean
├── notebooks/
│   ├── README.md
│   └── eval_trained_lora.ipynb         # Colab-ready reproduction of headline numbers
├── tests/
│   ├── test_fleet.py      # 14 fleet + coalition tests
│   └── test_graders.py    # 19 grader + anti-gaming tests
├── inference.py           # Baseline LLM agent runner
├── openenv.yaml           # OpenEnv v2.0.0, 5 manifest tasks, multi-agent
├── requirements.txt
├── requirements-training.txt
└── server/
    └── app.py

OpenEnv Compliance

• openenv validate passes ✅
• Typed Pydantic v2 models with extra="forbid" ✅
• Deterministic seed-based graders ✅
• Docker deployment ready ✅
• 5 manifest tasks with difficulty progression ✅
• Multi-agent /delegate and /coalition evaluation ✅
• Mercor efficiency reward ✅

Known Limitations

The submitted adapter is an SFT warmup policy, not a fully optimized online RL policy. training/grpo_train.py includes an optional GRPO loop wired to the environment reward function, but the submitted quantitative improvement is from supervised LoRA warmup over environment-generated SRE rollouts plus phase-aware constrained evaluation.

The simulator is deterministic under seed for reproducibility and models production-inspired failure signatures rather than connecting to a real GPU cluster.