Troubleshooting Guide¶

This guide helps you diagnose and resolve common issues with Minato.

Table of Contents¶

Operator Issues
GameServer Issues
Agent Issues
Snapshot Issues
Performance Issues
FAQ

Operator Issues¶

Operator Pod CrashLoopBackOff¶

Symptoms: Operator pod keeps restarting

Diagnosis:

kubectl logs -n minato-system -l app.kubernetes.io/name=minato --previous
kubectl describe pod -n minato-system -l app.kubernetes.io/name=minato

Common Causes:

Missing CRDs: Ensure CRDs are installed

kubectl get crds | grep minato

Fix: make install

RBAC Issues: Check ClusterRole bindings

kubectl auth can-i list gameservers --as=system:serviceaccount:minato-system:minato-operator

Leader Election Failure: Check for network issues between pods

kubectl get leases -n minato-system

Operator Not Reconciling¶

Symptoms: GameServer created but no resources appear

Diagnosis:

# Check operator logs
kubectl logs -n minato-system -l app.kubernetes.io/name=minato -f

# Check GameServer events
kubectl describe gameserver my-server -n default

Common Causes:

Finalizer stuck: Remove stuck finalizer

kubectl patch gameserver my-server -n default --type=merge -p '{"metadata":{"finalizers":[]}}'

Profile not found: Ensure GameProfile exists

kubectl get gameprofile my-profile

GameServer Issues¶

GameServer Stuck in Provisioning¶

Symptoms: GameServer status shows "Provisioning" indefinitely

Diagnosis:

# Check StatefulSet
kubectl describe statefulset my-server -n default

# Check Pod status
kubectl get pods -n default -l minato.io/gameserver=my-server

# Check PVC
kubectl describe pvc my-server -n default

Common Causes:

PVC Pending: No storage class or insufficient resources

kubectl get storageclass
kubectl describe pvc my-server -n default

Fix: Ensure a default StorageClass exists or specify one in the GameServer spec.

Image Pull Errors:

kubectl describe pod my-server-0 -n default

Fix: Check image credentials, network access, or image tag.

Resource Limits: Insufficient CPU/memory

kubectl describe node

Fix: Adjust resource requests/limits or add nodes.

GameServer Stuck in Error¶

Symptoms: GameServer status shows "Error"

Diagnosis:

kubectl describe gameserver my-server -n default
kubectl logs -n default -l minato.io/gameserver=my-server -c minato-game

Common Causes:

Profile Missing: The referenced GameProfile doesn't exist
Invalid Configuration: Environment variables or settings are invalid
Game Container Crash: Check game container logs

Cannot Connect to Game Server¶

Symptoms: Players cannot connect to the game

Diagnosis:

# Check service endpoints
kubectl get endpoints my-server -n default

# Check service type
kubectl get service my-server -n default

# Test connectivity from within cluster
kubectl run -it --rm debug --image=busybox --restart=Never -- wget -O- my-server:25565

Common Causes:

Service Type: ClusterIP only works within the cluster
Use NodePort or LoadBalancer for external access
Or set up an ingress/controller
Port Configuration: Wrong port in GameProfile

kubectl get gameprofile my-profile -o yaml

Firewall Rules: Cloud provider security groups blocking traffic

Agent Issues¶

Agent Unhealthy¶

Symptoms: GameServer shows AgentReachable=False

Diagnosis:

# Check agent container logs
kubectl logs -n default my-server-0 -c minato-agent

# Check agent health endpoint
kubectl exec -n default my-server-0 -c minato-agent -- wget -qO- localhost:8080/healthz

# Check agent metrics endpoint
kubectl exec -n default my-server-0 -c minato-agent -- wget -qO- localhost:9090/metrics

Common Causes:

RCON Not Configured: Agent can't connect to game RCON
Check RCON environment variables
Verify RCON port is exposed in GameProfile
Game Not Ready: Agent starts before game is ready
Add readiness probes to GameProfile
Agent will retry automatically

Actions Failing¶

Symptoms: ActionExecution shows "Failed" status

Diagnosis:

kubectl describe actionexecution my-action -n default
kubectl logs -n default my-server-0 -c minato-agent

Common Causes:

Unknown Action: Action not defined in GameProfile
Missing Parameters: Required params not provided
Agent Error: RCON command failed

Snapshot Issues¶

Snapshots Not Creating¶

Symptoms: GameSnapshot shows error condition

Diagnosis:

kubectl describe gamesnapshot my-snapshot -n default
kubectl logs -n minato-system -l app.kubernetes.io/name=minato

Common Causes:

VolumeSnapshot CRD Not Installed:

kubectl get crd volumesnapshots.snapshot.storage.k8s.io

Fix: Install the CSI snapshotter CRDs.

CSI Driver Not Installed: Your cluster needs a CSI driver that supports snapshots
Check your cloud provider's CSI driver documentation
PVC Not Bound: Ensure the GameServer's PVC is bound before taking snapshots

Performance Issues¶

High CPU/Memory Usage¶

Diagnosis:

kubectl top pods -n minato-system
kubectl top nodes

Solutions:

Adjust operator resource limits
Reduce reconciliation frequency
Scale operator horizontally (if not using leader election)

Slow Reconciliation¶

Symptoms: Changes take a long time to apply

Solutions:

Check etcd performance
Reduce number of watched resources
Check network latency to API server

FAQ¶

Q: How do I restart a game server?¶

A: Create an ActionExecution:

kubectl apply -f - <<EOF
apiVersion: operator.minato.io/v1
kind: ActionExecution
metadata:
  name: restart-my-server
  namespace: default
spec:
  targetRef:
    kind: GameServer
    name: my-server
  actionName: restart
EOF

Q: How do I backup my world?¶

A: Create a GameSnapshot:

kubectl apply -f - <<EOF
apiVersion: operator.minato.io/v1
kind: GameSnapshot
metadata:
  name: manual-backup
  namespace: default
spec:
  gameServerRef: my-server
  retention:
    count: 5
EOF

Q: Can I run multiple game servers on one node?¶

A: Yes, but consider:

Resource limits and requests
Anti-affinity rules for critical servers
Node capacity

Q: How do I update a GameProfile?¶

A: Edit the GameProfile and the operator will roll out changes to affected GameServers:

kubectl edit gameprofile my-profile

Note: Some changes (like image) require GameServer restart.

Q: Where are the logs?¶

A:

Operator: kubectl logs -n minato-system -l app.kubernetes.io/name=minato
Game: kubectl logs -n default my-server-0 -c minato-game
Agent: kubectl logs -n default my-server-0 -c minato-agent

Q: How do I clean up old ActionExecutions?¶

A: The operator automatically cleans up old ActionExecutions based on TTL:

Succeeded: 7 days
Failed/Rejected/TimedOut: 30 days

Q: Can I use my own game container image?¶

A: Yes! Create a GameProfile with your image and configure the agent appropriately.

Getting More Help¶

If your issue isn't covered here:

Check the runbooks for specific scenarios
Search existing GitHub issues
Join our community discussions
Open a new issue with:
Minato version
Kubernetes version
Relevant logs
Steps to reproduce