metadata
tags:
- Pixelcopter-PLE-v0
- reinforce
- reinforcement-learning
- custom-implementation
- deep-rl-class
model-index:
- name: Pixelcopter-RL
results:
- task:
type: reinforcement-learning
name: reinforcement-learning
dataset:
name: Pixelcopter-PLE-v0
type: Pixelcopter-PLE-v0
metrics:
- type: mean_reward
value: 13.10 +/- 6.89
name: mean_reward
verified: false
REINFORCE Agent for Pixelcopter-PLE-v0
Model Description
This repository contains a trained REINFORCE (Policy Gradient) reinforcement learning agent that has learned to play Pixelcopter-PLE-v0, a challenging helicopter navigation game from the PyGame Learning Environment (PLE). The agent uses policy gradient methods to learn optimal flight control strategies through trial and error.
Model Details
- Algorithm: REINFORCE (Monte Carlo Policy Gradient)
- Environment: Pixelcopter-PLE-v0 (PyGame Learning Environment)
- Framework: Custom implementation following Deep RL Course guidelines
- Task Type: Discrete Control (Binary Actions)
- Action Space: Discrete (2 actions: do nothing or thrust up)
- Observation Space: Visual/pixel-based or feature-based state representation
Environment Overview
Pixelcopter-PLE-v0 is a classic helicopter control game where:
- Objective: Navigate a helicopter through obstacles without crashing
- Challenge: Requires precise timing and control to avoid ceiling, floor, and obstacles
- Physics: Gravity constantly pulls the helicopter down; player must apply thrust to maintain altitude
- Scoring: Points are awarded for surviving longer and successfully navigating through gaps
- Difficulty: Requires learning temporal dependencies and precise action timing
Performance
The trained REINFORCE agent achieves the following performance metrics:
- Mean Reward: 13.10 ± 6.89
- Performance Analysis: This represents solid performance for this challenging environment
- Consistency: The standard deviation indicates moderate variability, which is expected for policy gradient methods
Performance Context
The mean reward of 13.10 demonstrates that the agent has successfully learned to:
- Navigate through multiple obstacles before crashing
- Balance altitude control against obstacle avoidance
- Develop timing strategies for thrust application
- Achieve consistent survival beyond random baseline performance
The variability (±6.89) is characteristic of policy gradient methods and reflects the stochastic nature of the learned policy, which can lead to different episode outcomes based on exploration.
Algorithm: REINFORCE
REINFORCE is a foundational policy gradient algorithm that:
- Direct Policy Learning: Learns a parameterized policy directly (no value function)
- Monte Carlo Updates: Uses complete episode returns for policy updates
- Policy Gradient: Updates policy parameters in direction of higher expected returns
- Stochastic Policy: Learns probabilistic action selection for exploration
Key Advantages
- Simple and intuitive policy gradient approach
- Works well with discrete and continuous action spaces
- No need for value function approximation
- Good educational foundation for understanding policy gradients
Usage
Installation Requirements
# Core dependencies
pip install torch torchvision
pip install gymnasium
pip install pygame-learning-environment
pip install numpy matplotlib
# For visualization and analysis
pip install pillow
pip install imageio # for gif creation
Loading and Using the Model
import torch
import gymnasium as gym
from ple import PLE
from ple.games.pixelcopter import Pixelcopter
import numpy as np
# Load the trained model
# Note: Adjust path based on your model file structure
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = torch.load("pixelcopter_reinforce_model.pth", map_location=device)
model.eval()
# Create the environment
def create_pixelcopter_env():
game = Pixelcopter()
env = PLE(game, fps=30, display=True) # Set display=False for headless
return env
# Initialize environment
env = create_pixelcopter_env()
env.init()
# Run trained agent
def run_agent(model, env, episodes=5):
total_rewards = []
for episode in range(episodes):
env.reset_game()
episode_reward = 0
while not env.game_over():
# Get current state
state = env.getScreenRGB() # or env.getGameState() if using features
state = preprocess_state(state) # Apply your preprocessing
# Convert to tensor
state_tensor = torch.FloatTensor(state).unsqueeze(0).to(device)
# Get action probabilities
with torch.no_grad():
action_probs = model(state_tensor)
action = torch.multinomial(action_probs, 1).item()
# Execute action (0: do nothing, 1: thrust)
reward = env.act(action)
episode_reward += reward
total_rewards.append(episode_reward)
print(f"Episode {episode + 1}: Reward = {episode_reward:.2f}")
mean_reward = np.mean(total_rewards)
std_reward = np.std(total_rewards)
print(f"\nAverage Performance: {mean_reward:.2f} ± {std_reward:.2f}")
return total_rewards
# Preprocessing function (adjust based on your model's input requirements)
def preprocess_state(state):
"""
Preprocess the game state for the neural network
This should match the preprocessing used during training
"""
if isinstance(state, np.ndarray) and len(state.shape) == 3:
# If using image input
state = np.transpose(state, (2, 0, 1)) # Channel first
state = state / 255.0 # Normalize pixels
return state.flatten() # or keep as image depending on model
else:
# If using game state features
return np.array(list(state.values()))
# Run the agent
rewards = run_agent(model, env, episodes=10)
Training Your Own Agent
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
from collections import deque
class PolicyNetwork(nn.Module):
def __init__(self, state_size, action_size, hidden_size=64):
super(PolicyNetwork, self).__init__()
self.fc1 = nn.Linear(state_size, hidden_size)
self.fc2 = nn.Linear(hidden_size, hidden_size)
self.fc3 = nn.Linear(hidden_size, action_size)
self.softmax = nn.Softmax(dim=1)
def forward(self, x):
x = torch.relu(self.fc1(x))
x = torch.relu(self.fc2(x))
x = self.fc3(x)
return self.softmax(x)
class REINFORCEAgent:
def __init__(self, state_size, action_size, lr=0.001):
self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
self.policy_net = PolicyNetwork(state_size, action_size).to(self.device)
self.optimizer = optim.Adam(self.policy_net.parameters(), lr=lr)
self.saved_log_probs = []
self.rewards = []
def select_action(self, state):
state = torch.FloatTensor(state).unsqueeze(0).to(self.device)
probs = self.policy_net(state)
action = torch.multinomial(probs, 1)
self.saved_log_probs.append(torch.log(probs.squeeze(0)[action]))
return action.item()
def update_policy(self, gamma=0.99):
# Calculate discounted rewards
discounted_rewards = []
R = 0
for r in reversed(self.rewards):
R = r + gamma * R
discounted_rewards.insert(0, R)
# Normalize rewards
discounted_rewards = torch.FloatTensor(discounted_rewards).to(self.device)
discounted_rewards = (discounted_rewards - discounted_rewards.mean()) / (discounted_rewards.std() + 1e-8)
# Calculate policy loss
policy_loss = []
for log_prob, reward in zip(self.saved_log_probs, discounted_rewards):
policy_loss.append(-log_prob * reward)
# Update policy
self.optimizer.zero_grad()
policy_loss = torch.cat(policy_loss).sum()
policy_loss.backward()
self.optimizer.step()
# Clear episode data
self.saved_log_probs.clear()
self.rewards.clear()
return policy_loss.item()
def train_agent(episodes=2000):
env = create_pixelcopter_env()
env.init()
# Determine state size based on your preprocessing
state_size = len(preprocess_state(env.getScreenRGB())) # Adjust as needed
action_size = 2 # do nothing, thrust
agent = REINFORCEAgent(state_size, action_size)
scores = deque(maxlen=100)
for episode in range(episodes):
env.reset_game()
episode_reward = 0
while not env.game_over():
state = preprocess_state(env.getScreenRGB())
action = agent.select_action(state)
reward = env.act(action)
agent.rewards.append(reward)
episode_reward += reward
# Update policy after each episode
loss = agent.update_policy()
scores.append(episode_reward)
if episode % 100 == 0:
avg_score = np.mean(scores)
print(f"Episode {episode}, Average Score: {avg_score:.2f}, Loss: {loss:.4f}")
# Save the trained model
torch.save(agent.policy_net, "pixelcopter_reinforce_model.pth")
return agent
# Train a new agent
# trained_agent = train_agent()
Evaluation and Analysis
import matplotlib.pyplot as plt
def evaluate_agent_detailed(model, env, episodes=50):
"""Detailed evaluation with statistics and visualization"""
rewards = []
episode_lengths = []
for episode in range(episodes):
env.reset_game()
episode_reward = 0
steps = 0
while not env.game_over():
state = preprocess_state(env.getScreenRGB())
state_tensor = torch.FloatTensor(state).unsqueeze(0)
with torch.no_grad():
action_probs = model(state_tensor)
action = torch.multinomial(action_probs, 1).item()
reward = env.act(action)
episode_reward += reward
steps += 1
rewards.append(episode_reward)
episode_lengths.append(steps)
if (episode + 1) % 10 == 0:
print(f"Episodes {episode + 1}/{episodes} completed")
# Statistical analysis
mean_reward = np.mean(rewards)
std_reward = np.std(rewards)
median_reward = np.median(rewards)
max_reward = np.max(rewards)
min_reward = np.min(rewards)
mean_length = np.mean(episode_lengths)
print(f"\n--- Evaluation Results ---")
print(f"Episodes: {episodes}")
print(f"Mean Reward: {mean_reward:.2f} ± {std_reward:.2f}")
print(f"Median Reward: {median_reward:.2f}")
print(f"Max Reward: {max_reward:.2f}")
print(f"Min Reward: {min_reward:.2f}")
print(f"Mean Episode Length: {mean_length:.1f} steps")
# Visualization
plt.figure(figsize=(12, 4))
plt.subplot(1, 2, 1)
plt.plot(rewards)
plt.axhline(y=mean_reward, color='r', linestyle='--', label=f'Mean: {mean_reward:.2f}')
plt.title('Episode Rewards')
plt.xlabel('Episode')
plt.ylabel('Reward')
plt.legend()
plt.subplot(1, 2, 2)
plt.hist(rewards, bins=20, alpha=0.7)
plt.axvline(x=mean_reward, color='r', linestyle='--', label=f'Mean: {mean_reward:.2f}')
plt.title('Reward Distribution')
plt.xlabel('Reward')
plt.ylabel('Frequency')
plt.legend()
plt.tight_layout()
plt.show()
return {
'rewards': rewards,
'episode_lengths': episode_lengths,
'stats': {
'mean': mean_reward,
'std': std_reward,
'median': median_reward,
'max': max_reward,
'min': min_reward
}
}
# Run detailed evaluation
# results = evaluate_agent_detailed(model, env, episodes=100)
Training Information
Hyperparameters
The REINFORCE agent was trained with carefully tuned hyperparameters:
- Learning Rate: Optimized for stable policy gradient updates
- Discount Factor (γ): Balances immediate vs. future rewards
- Network Architecture: Multi-layer perceptron with appropriate hidden dimensions
- Episode Length: Sufficient episodes to learn temporal patterns
Training Environment
- State Representation: Processed game screen or extracted features
- Action Space: Binary discrete actions (do nothing vs. thrust)
- Reward Signal: Game score progression with survival bonus
- Training Episodes: Extended training to achieve stable performance
Algorithm Characteristics
- Sample Efficiency: Moderate (typical for policy gradient methods)
- Stability: Good convergence with proper hyperparameter tuning
- Exploration: Built-in through stochastic policy
- Interpretability: Clear policy learning through gradient ascent
Limitations and Considerations
- Sample Efficiency: REINFORCE requires many episodes to learn effectively
- Variance: Policy gradient estimates can have high variance
- Environment Specific: Trained specifically for Pixelcopter game mechanics
- Stochastic Performance: Episode rewards vary due to policy stochasticity
- Real-time Performance: Inference speed suitable for real-time game play
Related Work and Extensions
This model serves as an excellent educational example for:
- Policy Gradient Methods: Understanding direct policy optimization
- Deep Reinforcement Learning: Practical implementation of RL algorithms
- Game AI: Learning complex temporal control tasks
- Baseline Comparisons: Foundation for more advanced algorithms (A2C, PPO, etc.)
Citation
If you use this model in your research or educational projects, please cite:
@misc{pixelcopter_reinforce_2024,
title={REINFORCE Agent for Pixelcopter-PLE-v0},
author={Adilbai},
year={2024},
publisher={Hugging Face},
howpublished={\url{https://huggingface.co/Adilbai/Pixelcopter-RL}},
note={Trained following Deep RL Course Unit 4}
}
Educational Resources
This model was developed following the Deep Reinforcement Learning Course Unit 4:
- Course Link: https://huggingface.co/deep-rl-course/unit4/introduction
- Topic: Policy Gradient Methods and REINFORCE
- Learning Objectives: Understanding policy-based RL algorithms
For comprehensive learning about REINFORCE and policy gradient methods, refer to the complete course materials.
License
This model is distributed under the MIT License. The model is intended for educational and research purposes.