Troubleshooting

Understanding common errors and their solutions helps maintain a healthy node operation.

Common Error Codes

Here are the most frequent errors you might encounter and their solutions:

Consensus Errors

When you encounter consensus errors, quick and appropriate action is essential:

Error: "Consensus failure - height halted"
Solution: Check for network upgrades or chain halts
Command: paxd status

Error: "Private validator file not found"
Solution: Restore validator key or check file permissions
Location: $HOME/.paxeer/config/priv_validator_key.json

Error: "Duplicate signature"
Solution: IMMEDIATELY STOP NODE - potential double signing risk
Action: Check validator operation on other machines

Network Errors

Network errors can prevent your node from participating in consensus:

Error: "Dial tcp connection refused"
Solution: Check network connectivity and firewall rules
Commands:
  - netstat -tulpn | grep paxd
  - ufw status

Error: "No peers available"
Solution: Verify peer connections and network config
Commands:
  - curl localhost:26657/net_info

Database Errors

Database corruption can require immediate attention:

Error: "Database is corrupted"
Solution: Reset database or restore from backup
Commands:
  - paxd tendermint unsafe-reset-all
  - cp -r backup/data $HOME/.paxeer/

Diagnostic Commands

These commands help you investigate issues and monitor your node:

# Check node synchronization
paxd status

# Check validator status
paxd query staking validator $(paxd tendermint show-validator)

# Monitor real-time logs
journalctl -fu paxd -o cat

# View system resource usage
top -p $(pgrep paxd)

AppHash Mismatch Errors

If you encounter an AppHash mismatch, you'll need to capture the state for comparison with a known good version:

# For state-commit (most non-archive nodes):
paxd debug dump-iavl <latest height>

Always include the app hash, commit hash, and block height from your logs when reporting issues.

Identifying AppHash Errors

AppHash errors typically appear in logs as:

ERR wrong Block.Header.AppHash. Expected [EXPECTED_HASH], got [ACTUAL_HASH]
block_id={"hash":"...","parts":{"hash":"...","total":1}} height=[HEIGHT]

Common Causes:

• Using incorrect node version during sync (ensure you're on the latest version)
• Corrupted or incorrectly applied snapshots
• Database inconsistencies from improper shutdowns
• Syncing with outdated or incompatible peers

Resolution Steps:

1. Stop the node immediately.

2. Try a node rollback first:, see here

3. If rollback fails, restore from a fresh snapshot:

   • Download a recent snapshot from trusted providers
   • Ensure you're using the correct node version
   • Verify peer configurations are up to date

4. Restart the node and monitor logs for continued errors

Peer Connection Issues as AppHash Red Herrings

Important Note: Peer connection failures are often symptoms of underlying AppHash errors, not the root cause.

When you see extensive peer connection errors like:

ERR failed to handshake with peer
ERR failed to send request for peers
ERR peer handshake failed endpoint={} err=EOF

Don't focus solely on fixing peer connections first. Instead:

1. Scan your logs carefully for AppHash errors that may appear intermittently
2. Look for the actual error pattern:

   ERR wrong Block.Header.AppHash. Expected [HASH], got [HASH]

3. Check if your node is stuck at a specific height despite peer connection attempts

Debugging Approach:

1. First, check for AppHash errors in your logs (search for "wrong Block.Header.AppHash")
2. If AppHash errors are found, treat this as the primary issue
3. Only focus on peer connection fixes if no AppHash errors exist

Peer Connection and Handshake Issues

Identifying Peer Issues:

Look for these error patterns in your logs:

ERR failed to handshake with peer err="expected to connect with peer \"[EXPECTED_ID]\", got \"[ACTUAL_ID]\""
ERR failed to send request for peers err="no available peers to send a PEX request to (retrying)"
ERR peer handshake failed endpoint={} err=EOF module=p2p

Common Causes:

• Outdated peer configurations with mismatched node IDs
• Network infrastructure changes on peer side
• Firewall blocking connections on port 26656
• DNS resolution issues

Resolution Steps:

1. Update peer configurations with current node IDs. Use the seed peer:

   e9c56cbadc4a96b67f69dcaaa7b4691851e945ca@31.220.74.140:26656

2. Verify network connectivity:

   # Test connection to peer endpoints
   nc -zv 31.220.74.140 26656
   
   # Check if port 26656 is open for inbound connections
   netstat -tulpn | grep :26656

3. Check current peer status:

   curl http://localhost:26657/net_info | jq '.result.peers | length'
   curl http://localhost:26657/lag_status | jq .

Sync Performance Issues

Identifying Sync Problems:

Monitor these indicators:

# Check sync status and lag
curl http://localhost:26657/lag_status | jq .

# Monitor if height is progressing
curl http://localhost:26657/status | jq '.result.sync_info'

Common Solutions:

1. Increase packet payload size for large block processing:

   # In config.toml [p2p] section
   max-packet-msg-payload-size = 1024000

2. Optimize mempool settings in config.toml:

   # In [mempool] section
   keep-invalid-txs-in-cache = true
   ttl-duration = "5s"
   ttl-num-blocks = 5

3. If node gets stuck at specific height:

   • Try restarting the node
   • If restart doesn't help, perform rollback
   • Consider taking a fresh snapshot

Crash and Panic Debugging

For crashes, panics, or nil pointer exceptions:

• Capture at least 1,000 lines of logs leading up to the crash
• Or collect 15 minutes of log data, whichever provides more context
• Include the full stack trace if available

Logging Configuration

# In config.toml
log_level = "debug"
log_format = "json"

Configure log rotation to manage storage effectively:

sudo tee /etc/logrotate.d/paxd << EOF
/var/log/paxd/*.log {
    daily
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 paxeer paxeer
    sharedscripts
    postrotate
        systemctl reload paxd
    endscript
}
EOF

Other common Issues and Fixes

1. Sync Problems

   • Check available disk space (df -h)
   • Ensure proper peer connections (curl http://localhost:26657/net_info)
   • Verify firewall settings (port 26656 open)

2. Performance Issues

   • Monitor system resources (htop or iotop)
   • Check disk I/O performance (iostat)
   • Analyze network traffic (iftop)

3. Database Issues

   • Run database integrity checks using:

     paxd debug dump-db | grep -i error

     If errors are detected, consider restoring from a recent backup.

   • Consider pruning excessive historical data by adjusting ss-keep-recent in app.toml or running:

     paxd tendermint unsafe-reset-all --home $HOME/.paxeer --keep-addr-book

Node Rollback

To rollback a node from an AppHashed state, you need to stop the node first.

Next, do a rollback with:

paxd rollback

Then, restart the node.

In case you see the following error while trying to rollback:

failed to initialize database: resource temporarily unavailable

This means that you did not shutdown the node properly. Try to shutdown or kill the paxd process directly in that case. If this doesn't help, restart your machine.

Then try the rollback steps again.