Monitoring & Maintenance

This guide covers monitoring SupaScan's health, blockchain events, and performing regular maintenance tasks. SupaScan provides comprehensive monitoring capabilities for all indexed blockchain events with customizable batch processing and real-time analytics.

Health Monitoring

SupaScan System Monitoring

Using PM2

If deployed with PM2, use built-in monitoring:

# View process status
pm2 status

# Real-time monitoring dashboard
pm2 monit

# View SupaScan API logs
pm2 logs supascan-api --lines 100

# View indexing engine logs
pm2 logs supascan-indexer --lines 100

# View detailed process info
pm2 describe supascan-api

Using Systemd

For systemd deployments:

# Check SupaScan API status
sudo systemctl status supascan-api

# Check indexing engine status
sudo systemctl status supascan-indexer

# View recent logs
sudo journalctl -u supascan-api -n 100

# Follow logs in real-time
sudo journalctl -u supascan-indexer -f

Blockchain Event Monitoring

Real-time Indexing Status

Monitor the indexing of blockchain events:

# Check indexing lag
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  https://api.supascan.com/v1/indexing/status

# Monitor slot synchronization
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  https://api.supascan.com/v1/indexing/slot-sync

# Check RPC node health
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  https://api.supascan.com/v1/indexing/rpc-health

Custom Event Monitoring

Set up monitoring for specific blockchain events:

# Monitor token creation events
curl -X POST \
  -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "event_type": "token_creation",
    "filters": {
      "min_liquidity_usd": 10000,
      "dex_protocols": ["raydium", "meteora"]
    },
    "notification_webhook": "https://your-app.com/webhook"
  }' \
  https://api.supascan.com/v1/monitoring/events

# Monitor large transactions
curl -X POST \
  -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "event_type": "large_transaction",
    "filters": {
      "min_amount_usd": 100000,
      "token_types": ["SOL", "USDC", "USDT"]
    },
    "batch_size": 50,
    "notification_frequency": "realtime"
  }' \
  https://api.supascan.com/v1/monitoring/events

Key Metrics to Monitor

1. SupaScan System Health

  • Uptime: API and indexing engine should be continuously running
  • Restarts: Frequent restarts indicate issues
  • Memory Usage: API < 2GB, Indexer < 8GB
  • CPU Usage: Should be optimized for real-time processing

2. Blockchain Indexing Performance

  • Indexing Lag: Should be < 5 seconds behind Solana L1
  • Transaction Throughput: Events processed per second
  • RPC Response Time: < 100ms for healthy nodes
  • Queue Size: Pending events to be indexed

3. ClickHouse Database Health

  • Query Execution Time: < 1 second for standard queries
  • Table Size Growth: Monitor data ingestion rates
  • Replication Lag: < 1 second between replicas
  • Memory Usage: Should stay within limits

4. API Performance

  • Response Time: < 500ms for REST API calls
  • Throughput: Requests per second capacity
  • Error Rate: < 1% for healthy system
  • Rate Limit Usage: Monitor API quota consumption

5. Custom Event Monitoring

  • Webhook Delivery Success Rate: > 99%
  • Event Filter Accuracy: Monitor false positives/negatives
  • Batch Processing Efficiency: Events per batch
  • Notification Latency: < 2 seconds for real-time alerts

Monitoring Tools

SupaScan Application Logs

SupaScan uses Winston logger with daily rotation:

# View today's API logs
tail -f logs/supascan-api-$(date +%Y-%m-%d).log

# View indexing engine logs
tail -f logs/supascan-indexer-$(date +%Y-%m-%d).log

# Search for errors
grep ERROR logs/supascan-*.log

# Monitor specific API requests
grep "GET /v1/transactions" logs/supascan-api-*.log

# Monitor webhook deliveries
grep "webhook_delivery" logs/supascan-api-*.log

Blockchain Event Monitoring

Real-time Event Streams

Monitor specific blockchain events:

# Monitor token creation events
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/events/stream?event_type=token_creation&batch_size=10"

# Monitor DEX swap events
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/events/stream?event_type=dex_swap&dex_protocol=raydium"

# Monitor large wallet movements
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/events/stream?event_type=large_transfer&min_amount_usd=10000"

Custom Event Filters

Create custom monitoring filters:

# Monitor specific token addresses
curl -X POST \
  -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "High Volume Tokens",
    "filters": {
      "event_types": ["token_swap", "token_transfer"],
      "min_volume_24h_usd": 1000000,
      "token_addresses": ["So11111111111111111111111111111111111111112"]
    },
    "batch_config": {
      "batch_size": 25,
      "batch_interval_seconds": 30,
      "max_events_per_batch": 100
    },
    "notification_config": {
      "webhook_url": "https://your-app.com/webhook",
      "notification_frequency": "batch",
      "include_metadata": true
    }
  }' \
  https://api.supascan.com/v1/monitoring/filters

Performance Monitoring

Memory Usage Tracking

# Create SupaScan monitoring script
cat > monitor-supascan.sh << 'EOF'
#!/bin/bash
while true; do
  echo "$(date): API Memory: $(pm2 describe supascan-api | grep memory | awk '{print $4}')"
  echo "$(date): Indexer Memory: $(pm2 describe supascan-indexer | grep memory | awk '{print $4}')"
  echo "$(date): ClickHouse Memory: $(clickhouse-client --query 'SELECT formatReadableSize(sum(memory_usage)) FROM system.processes')"
  sleep 60
done
EOF

chmod +x monitor-supascan.sh
./monitor-supascan.sh > supascan-memory-usage.log &

API Response Time Monitoring

# Extract API response times from logs
grep "API Response time:" logs/supascan-api-*.log | \
  awk '{print $NF}' | \
  awk '{sum+=$1; count++} END {print "Average API Response:", sum/count, "ms"}'

# Monitor indexing performance
grep "Indexing time:" logs/supascan-indexer-*.log | \
  awk '{print $NF}' | \
  awk '{sum+=$1; count++} END {print "Average Indexing Time:", sum/count, "ms"}'

ClickHouse Database Monitoring

# Check ClickHouse cluster health
clickhouse-client --query "SELECT * FROM system.clusters"

# Monitor table sizes
clickhouse-client --query "
SELECT 
    database,
    table,
    formatReadableSize(sum(bytes)) as size,
    sum(rows) as rows
FROM system.parts 
WHERE active = 1 
GROUP BY database, table 
ORDER BY sum(bytes) DESC"

# Check indexing performance
clickhouse-client --query "
SELECT 
    table,
    formatReadableSize(sum(data_compressed_bytes)) as compressed_size,
    formatReadableSize(sum(data_uncompressed_bytes)) as uncompressed_size,
    round(sum(data_uncompressed_bytes) / sum(data_compressed_bytes), 2) as compression_ratio
FROM system.parts 
WHERE active = 1 
GROUP BY table"

# Monitor slow queries
clickhouse-client --query "
SELECT 
    query_duration_ms,
    query,
    user,
    initial_query_start_time
FROM system.query_log 
WHERE query_duration_ms > 1000 
ORDER BY query_duration_ms DESC 
LIMIT 10"

Batch Processing Monitoring

# Monitor batch processing efficiency
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/monitoring/batches/stats"

# Check custom event filter performance
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/monitoring/filters/performance"

# Monitor webhook delivery success rates
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/monitoring/webhooks/stats"

Regular Maintenance Tasks

Daily Tasks

1. SupaScan System Review

# Check for API errors
grep -c ERROR logs/supascan-api-$(date +%Y-%m-%d).log

# Check for indexing errors
grep -c ERROR logs/supascan-indexer-$(date +%Y-%m-%d).log

# Review webhook delivery issues
grep "webhook_failed" logs/supascan-api-$(date +%Y-%m-%d).log

# Check custom event filter performance
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/monitoring/filters/daily-stats"

2. Blockchain Indexing Performance

  • Monitor indexing lag vs Solana L1
  • Check RPC node response times
  • Verify event processing throughput
  • Review batch processing efficiency

3. Custom Event Monitoring Review

# Check active event filters
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/monitoring/filters/active"

# Review webhook delivery success rates
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/monitoring/webhooks/daily-stats"

# Check batch processing metrics
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/monitoring/batches/daily-summary"

Weekly Tasks

1. ClickHouse Database Optimization

# Optimize ClickHouse tables
clickhouse-client --query "OPTIMIZE TABLE transactions FINAL"
clickhouse-client --query "OPTIMIZE TABLE token_transfers FINAL"
clickhouse-client --query "OPTIMIZE TABLE token_swaps FINAL"

# Update table statistics
clickhouse-client --query "ANALYZE TABLE transactions"
clickhouse-client --query "ANALYZE TABLE token_transfers"

# Check table health
clickhouse-client --query "SELECT * FROM system.tables WHERE database = 'supascan'"

2. Log Cleanup

# Compress old SupaScan logs
find logs -name "supascan-*.log" -mtime +7 -exec gzip {} \;

# Remove very old logs
find logs -name "*.gz" -mtime +30 -delete

# Clear ClickHouse query logs older than 30 days
clickhouse-client --query "SYSTEM DROP QUERY LOG"

3. Cache and Performance Cleanup

# Clear Redis cache
redis-cli FLUSHDB

# Clear PM2 logs if too large
pm2 flush

# Restart services for memory cleanup
pm2 restart supascan-api
pm2 restart supascan-indexer

4. Custom Event Filter Maintenance

# Review and optimize custom event filters
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/monitoring/filters/optimize"

# Clean up unused webhook subscriptions
curl -X DELETE \
  -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/monitoring/filters/cleanup"

Monthly Tasks

1. Security Review

  • Review API key usage patterns
  • Check for unusual query patterns
  • Update SupaScan dependencies
  • Audit webhook endpoints

2. Performance Analysis

# Generate monthly SupaScan report
cat > monthly-supascan-report.sh << 'EOF'
#!/bin/bash
echo "=== Monthly SupaScan Report ==="
echo "Date: $(date)"
echo ""
echo "=== API Usage Statistics ==="
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/stats/monthly"
echo ""
echo "=== Database Statistics ==="
clickhouse-client --query "
SELECT 
    table,
    formatReadableSize(sum(bytes)) as size,
    sum(rows) as rows,
    count() as parts
FROM system.parts 
WHERE active = 1 AND database = 'supascan'
GROUP BY table 
ORDER BY sum(bytes) DESC"
echo ""
echo "=== Custom Event Filters Performance ==="
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/monitoring/filters/monthly-stats"
echo ""
echo "=== Error Summary ==="
grep -c ERROR logs/supascan-*.log | tail -30
EOF

chmod +x monthly-supascan-report.sh
./monthly-supascan-report.sh > reports/supascan-monthly-$(date +%Y%m).txt

3. Data Archival and Backup

# Archive old data to cold storage
clickhouse-client --query "
ALTER TABLE transactions 
MOVE PARTITION '202401' TO DISK 'cold_storage'"

# Backup critical tables
clickhouse-client --query "BACKUP TABLE transactions TO Disk('backups', 'transactions_$(date +%Y%m%d)')"
clickhouse-client --query "BACKUP TABLE token_transfers TO Disk('backups', 'token_transfers_$(date +%Y%m%d)')"

# Verify backup integrity
clickhouse-client --query "SELECT count() FROM transactions WHERE toYYYYMM(date) = 202401"

4. Custom Event Filter Optimization

# Analyze filter performance and optimize
curl -X POST \
  -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "action": "optimize_filters",
    "criteria": {
      "min_events_per_day": 1000,
      "max_false_positive_rate": 0.05
    }
  }' \
  https://api.supascan.com/v1/monitoring/filters/optimize

# Review and update batch processing configurations
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/monitoring/batches/monthly-analysis"

Automated Monitoring

SupaScan Health Check Script

Create supascan-health-check.sh:

#!/bin/bash

# Configuration
API_NAME="supascan-api"
INDEXER_NAME="supascan-indexer"
ALERT_EMAIL="admin@example.com"
API_KEY="your-api-key-here"

# Check if SupaScan API is running
if pm2 status | grep -q "$API_NAME.*online"; then
    echo "✓ SupaScan API is running"
else
    echo "✗ SupaScan API is not running!"
    echo "SupaScan API is down!" | mail -s "SupaScan API Alert" $ALERT_EMAIL
    pm2 restart $API_NAME
fi

# Check if indexing engine is running
if pm2 status | grep -q "$INDEXER_NAME.*online"; then
    echo "✓ Indexing engine is running"
else
    echo "✗ Indexing engine is not running!"
    echo "Indexing engine is down!" | mail -s "SupaScan Indexer Alert" $ALERT_EMAIL
    pm2 restart $INDEXER_NAME
fi

# Check ClickHouse accessibility
if clickhouse-client --query "SELECT 1;" > /dev/null 2>&1; then
    echo "✓ ClickHouse is accessible"
else
    echo "✗ ClickHouse error!"
fi

# Check indexing lag
INDEXING_LAG=$(curl -s -H "Authorization: Bearer $API_KEY" \
  "https://api.supascan.com/v1/indexing/status" | jq -r '.lag_seconds')
if [ $INDEXING_LAG -gt 10 ]; then
    echo "⚠ High indexing lag: ${INDEXING_LAG}s"
fi

# Check API memory usage
API_MEMORY=$(pm2 describe $API_NAME | grep "memory" | awk '{print $4}')
if [ ${API_MEMORY%M} -gt 2000 ]; then
    echo "⚠ High API memory usage: $API_MEMORY"
fi

# Check indexer memory usage
INDEXER_MEMORY=$(pm2 describe $INDEXER_NAME | grep "memory" | awk '{print $4}')
if [ ${INDEXER_MEMORY%M} -gt 8000 ]; then
    echo "⚠ High indexer memory usage: $INDEXER_MEMORY"
fi

# Check recent errors
API_ERROR_COUNT=$(grep -c ERROR logs/supascan-api-$(date +%Y-%m-%d).log)
if [ $API_ERROR_COUNT -gt 20 ]; then
    echo "⚠ High API error count: $API_ERROR_COUNT errors today"
fi

# Check webhook delivery success rate
WEBHOOK_SUCCESS=$(curl -s -H "Authorization: Bearer $API_KEY" \
  "https://api.supascan.com/v1/monitoring/webhooks/stats" | jq -r '.success_rate')
if (( $(echo "$WEBHOOK_SUCCESS < 95" | bc -l) )); then
    echo "⚠ Low webhook success rate: ${WEBHOOK_SUCCESS}%"
fi

Cron Jobs

Add to crontab:

# SupaScan health check every 5 minutes
*/5 * * * * /opt/supascan/health-check.sh >> /opt/supascan/logs/health.log 2>&1

# Daily ClickHouse optimization
0 2 * * * clickhouse-client --query "OPTIMIZE TABLE transactions FINAL"

# Weekly batch processing optimization
0 3 * * 0 /opt/supascan/optimize-batch-processing.sh

# Monthly SupaScan report
0 0 1 * * /opt/supascan/monthly-supascan-report.sh

# Daily custom event filter cleanup
0 1 * * * curl -X POST -H "Authorization: Bearer $SUPASCAN_API_KEY" https://api.supascan.com/v1/monitoring/filters/cleanup

# Hourly webhook delivery retry
0 * * * * curl -X POST -H "Authorization: Bearer $SUPASCAN_API_KEY" https://api.supascan.com/v1/monitoring/webhooks/retry-failed

Monitoring Best Practices

1. Set Up Comprehensive Alerts

  • System Health: High memory usage (>80%), service crashes, indexing lag >10s
  • Performance: API response time >500ms, ClickHouse query time >1s
  • Data Quality: High error rate, webhook delivery failure >5%
  • Custom Events: Filter false positive rate >5%, batch processing delays

2. Regular Reviews

  • Daily: Check logs for errors, review indexing performance, monitor custom event filters
  • Weekly: Review API usage patterns, optimize ClickHouse tables, analyze webhook performance
  • Monthly: Analyze usage patterns, optimize custom filters, review data archival

3. Proactive Maintenance

  • Keep logs rotated and compressed
  • Optimize ClickHouse tables regularly
  • Monitor disk space and data growth
  • Update SupaScan dependencies
  • Review and optimize custom event filters

4. Custom Event Monitoring Best Practices

  • Filter Design: Start with simple filters, gradually add complexity
  • Batch Configuration: Optimize batch sizes based on event frequency
  • Webhook Reliability: Implement retry logic and error handling
  • Performance Monitoring: Track filter accuracy and processing efficiency

5. Documentation

  • Keep incident log for SupaScan issues
  • Document custom event filter configurations
  • Track performance baselines and trends
  • Note recurring issues and solutions
  • Document batch processing optimizations

Troubleshooting Performance Issues

High Memory Usage

  1. Check for memory leaks:

    # Monitor SupaScan memory over time
pm2 describe supascan-api | grep memory
pm2 describe supascan-indexer | grep memory

  2. Optimize ClickHouse memory usage:

    -- Check memory usage by query
SELECT 
    query,
    formatReadableSize(memory_usage) as memory,
    formatReadableSize(peak_memory_usage) as peak_memory
FROM system.query_log 
WHERE memory_usage > 1000000000 
ORDER BY memory_usage DESC;

  3. Restart services periodically:

    # Add to cron for weekly restart
0 4 * * 0 pm2 restart supascan-api supascan-indexer

Slow Response Times

  1. Check Solana RPC latency:

    time curl -X POST $SOLANA_RPC_URL \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"getSlot","params":[],"id":1}'

  2. Analyze ClickHouse queries:

    -- Check slow queries
SELECT 
    query_duration_ms,
    query,
    formatReadableSize(memory_usage) as memory
FROM system.query_log 
WHERE query_duration_ms > 1000 
ORDER BY query_duration_ms DESC 
LIMIT 10;

  3. Optimize custom event filters:

    # Review filter performance
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/monitoring/filters/performance"

Custom Event Filter Issues

  1. High false positive rate:

    # Analyze filter accuracy
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/monitoring/filters/accuracy"

# Optimize filter criteria
curl -X POST \
  -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"filter_id": "your-filter-id", "optimize": true}' \
  https://api.supascan.com/v1/monitoring/filters/optimize

  2. Batch processing delays:

    # Check batch processing queue
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/monitoring/batches/queue-status"

# Adjust batch configuration
curl -X PUT \
  -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"batch_size": 50, "batch_interval_seconds": 60}' \
  https://api.supascan.com/v1/monitoring/batches/config

Webhook Delivery Issues

  1. Failed webhook deliveries:

    # Check webhook delivery status
curl -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  "https://api.supascan.com/v1/monitoring/webhooks/failed"

# Retry failed deliveries
curl -X POST \
  -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  https://api.supascan.com/v1/monitoring/webhooks/retry-failed

  2. Webhook timeout issues:

    # Increase webhook timeout
curl -X PUT \
  -H "Authorization: Bearer $SUPASCAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"timeout_seconds": 30, "retry_attempts": 3}' \
  https://api.supascan.com/v1/monitoring/webhooks/config
Previous
Deployment
Next
Debug Mode