Troubleshooting

Common errors, their root causes, and fixes.

Quick Reference

502 Bad Gateway

Symptom: All PHP pages return 502.

Cause: Nginx can't connect to PHP-FPM backend.

Diagnosis:

# Check if PHP-FPM is running
systemctl status php-fpm-XX

# Check the socket
ls -la /tmp/php-cgi-*.sock

# Check Nginx error log
tail -f /www/wwwlogs/foren.dev.error.log

Common fixes:

# PHP-FPM not running
systemctl start php-fpm-85

# Wrong socket in nginx config
grep fastcgi_pass /www/server/nginx/conf/enable-php.conf
# Should match: fastcgi_pass unix:/tmp/php-cgi-85.sock;

# PHP-FPM has no workers (pm = ondemand)
# Change to pm = dynamic in php-fpm.conf
sed -i 's/pm = ondemand/pm = dynamic/' /www/server/php/XX/etc/php-fpm.conf
systemctl reload php-fpm-XX

# After fixing, verify
curl -sS https://foren.dev/license/api/coforen/health/

MySQL Authentication Failed

Symptom: API endpoints return:

{"ok": false, "db": "Database connection failed: SQLSTATE[HY000] [1045] Access denied"}

Cause: PHP-FPM env variables don't match MySQL credentials, OR the foren_license user doesn't exist.

Diagnosis:

# Test connection with the credentials
mysql -u '<DB_USER>' -p '<DB>' -e 'SELECT 1'

# Check PHP-FPM env
cat /www/server/php/XX/etc/php-fpm.conf | grep FOREN_DB

Common fixes:

# 1. Update PHP-FPM env (if password changed)
# Edit /www/server/php/XX/etc/php-fpm.conf, update FOREN_DB_PASS
systemctl reload php-fpm-XX

# 2. Create the MySQL user (if it doesn't exist)
mysql -u root -p
> CREATE USER '<DB_USER>'@'localhost' IDENTIFIED BY 'STRONG_PASSWORD';
> GRANT ALL PRIVILEGES ON <DB>.* TO '<DB_USER>'@'localhost';
> FLUSH PRIVILEGES;

Note: The MySQL root user may use auth_socket plugin and reject password auth. In that case, use sudo or set up a separate admin user.

500 on Admin Pages

Symptom: Specific admin page returns 500.

Diagnosis:

tail -50 /www/wwwlogs/foren.dev.error.log | grep -A5 'admin/<page>.php'

Common causes and fixes:

1. PHP fatal error (function redeclared)

PHP Fatal error: Cannot redeclare function dt()

Fix: _bootstrap.php defines helpers like dt(), status_badge(). Don't redefine them in individual pages.

2. SQL syntax error (column doesn't exist)

PDOException: SQLSTATE[42S22]: Column not found

Fix: The DB schema may be out of sync. Run migration 011:

mysql -u '<DB_USER>' -p '<DB>' < license/migrations/011_security_and_performance.sql

3. Missing index

PDOException: SQLSTATE[HY000]: General error: 1030 Got error 28

Fix: Disk full, or query optimizer fails. Check indexes:

mysql -u '<DB_USER>' -p '<DB>' -e "SHOW INDEX FROM license_devices"

4. Viewkey missing

RuntimeException: license.v2_view_key is not configured

Fix: Set FOREN_LICENSE_V2_VIEW_KEY in PHP-FPM env, then reload.

Webhooks Not Delivering

Symptom: Events created but no HTTP POST to your endpoint.

Diagnosis:

# Check pending events
mysql -u '<DB_USER>' -p '<DB>' -e "SELECT id, retry_count, last_error, created_at FROM license_events WHERE delivered=0 ORDER BY created_at DESC LIMIT 20"

# Check cron ran
tail /var/log/foren/expiry.log

# Test the cron manually
cd /www/wwwroot/foren.dev && php license/scripts/check_expiring_licenses.php

Common fixes:

# Cron not installed
crontab -e
# Add: 0 9 * * * cd /www/wwwroot/foren.dev && php license/scripts/check_expiring_licenses.php >> /var/log/foren/expiry.log 2>&1

# Environment variables not set in cron
# Wrap with env vars in crontab:
# 0 9 * * * cd /www/wwwroot/foren.dev && FOREN_DB_HOST=127.0.0.1 FOREN_DB_USER=<DB_USER> FOREN_DB_PASS=... php license/scripts/check_expiring_licenses.php >> /var/log/foren/expiry.log 2>&1

# Consumer endpoint down
curl -X POST https://your-app.example.com/webhook -H 'X-License-Webhook-Signature: sha256=test' -d '{}'

Activation Fails

Symptom: POST /license/api/coforen/activate/ returns 400 with error.

"activation_code and machine_fingerprint are required"

Both fields must be present in the request body. Check JSON syntax.

"invalid machine_fingerprint"

The fingerprint must match ^sha256:[a-f0-9]{64}$. Use:

echo -n "$SOME_INPUT" | sha256sum | awk '{print "sha256:"$1}'

"invalid machine_code"

Must be 1-128 chars, [A-Za-z0-9_\-.]+. No spaces, no special chars.

"License is expired or revoked"

Check licenses.is_active and expires_at:

mysql -u '<DB_USER>' -p '<DB>' -e "SELECT id, license_key, is_active, expires_at FROM licenses WHERE license_key='COF-XXXX'"

Signature Verification Fails

Symptom: Client says signature is invalid.

Diagnosis:

# Check the server's public key
curl https://foren.dev/license/api/coforen/public-key/

# Compare with the client's expected key

Common causes:

1. Client has stale public key

Fix: Re-fetch from /public-key/ and re-cache.

2. Server key was rotated

Fix: The old public key is invalidated. All clients must update. Coordinate with clients before rotation.

3. v2 envelope key mismatch

The v2 view key (FOREN_LICENSE_V2_VIEW_KEY) must match between server and clients. If clients use the public key for signature only but a shared key for envelope decryption, both keys must match.

License Module Routes Return 404

Symptom: /license/api/coforen/health/ returns 404.

Diagnosis:

# Check directory exists
ls /www/wwwroot/foren.dev/license/api/coforen/health/

# Check file exists
cat /www/wwwroot/foren.dev/license/api/coforen/health/index.php | head -3

Common fixes:

# Directory missing - re-sync
rsync -av license/api/coforen/health/ user@server:/www/wwwroot/foren.dev/license/api/coforen/health/

# Wrong path in nginx - check fastcgi handler
grep -A3 fastcgi_pass /www/server/nginx/conf/enable-php.conf

Migration Failures

Symptom: mysql < migration.sql fails with errors.

"Duplicate column name"

Migration was already applied. Skip or use migration 011 (idempotent) which handles this case.

"Access denied"

User lacks privileges. Use the admin user:

mysql -u root -p foren_license < migration.sql

Foreign key constraint fails

Data references missing. Run migrations in order.

Performance Issues

Symptom: API responses are slow (>1 second).

Diagnosis:

# Check slow query log
mysqldumpslow /var/log/mysql/slow.log

# Check current connections
mysql -u '<DB_USER>' -p '<DB>' -e "SHOW PROCESSLIST"

Common fixes:

# Run migration 011 to add missing indexes
mysql -u '<DB_USER>' -p '<DB>' < license/migrations/011_security_and_performance.sql

# Increase PHP-FPM workers if pool exhausted
# Edit /www/server/php/XX/etc/php-fpm.conf
# pm.max_children = 30  # (was 5)
systemctl reload php-fpm-XX

Admin Login Fails

Symptom: "用户名或密码错误" (username or password incorrect) — or, if 2FA is enabled, the account reports as locked even with the right password.

Diagnosis:

# Check existence, active flag, lockout state and 2FA state in one shot
mysql -u '<DB_USER>' -p '<DB>' -e "
  SELECT id, username, is_active, failed_logins, locked_until,
         (totp_secret IS NOT NULL) AS has_2fa
  FROM admins"

Three distinct causes, in order of likelihood:

Common fixes:

# Clear the lockout and the failure counter
mysql -u '<DB_USER>' -p '<DB>' -e "
  UPDATE admins SET locked_until=NULL, failed_logins=0, failed_login_at=NULL
  WHERE username='USERNAME'"

# Reactivate a disabled account
mysql -u '<DB_USER>' -p '<DB>' -e "
  UPDATE admins SET is_active=1 WHERE username='USERNAME'"

Reset Password and Hash

To reset an admin password from the command line (replace NEW_PASSWORD and USERNAME):

php -r '
$pdo = new PDO("mysql:host=127.0.0.1;dbname=<DB>", '<DB_USER>', 'PASSWORD');
$hash = password_hash("NEW_PASSWORD", PASSWORD_DEFAULT);
$now  = gmdate("Y-m-d H:i:s");           // UTC, matches now_utc()
$stmt = $pdo->prepare(
  "UPDATE admins SET password_hash=?, updated_at=?, locked_until=NULL, failed_logins=0 WHERE username=?"
);
$stmt->execute([$hash, $now, "USERNAME"]);
echo "Updated: " . $stmt->rowCount() . " row(s)\n";
'

The NEW_PASSWORD must satisfy the administrator policy (≥12 characters with upper case, lower case, a digit and a special character) or the admin will not be able to change it through the UI later.

Locked out of 2FA with no backup codes

If the authenticator app is lost and no backup code remains, an administrator with shell access can clear the TOTP secret. This removes 2FA for that account, so record why in the audit trail:

UPDATE admins SET totp_secret = NULL WHERE username = 'USERNAME';
DELETE FROM admin_2fa_backup_codes WHERE admin_id = (
  SELECT id FROM admins WHERE username = 'USERNAME'
);

Then re-enable 2FA from System → Security at the next login.

2FA Codes Always Rejected

Symptom: The 6-digit code from the authenticator app is never accepted, either at setup confirmation or at login.

Work through these in order:

1. Clock skew. TOTP tolerates ±1 step (±30s). If the server clock drifted

more than that, every code fails.

```bash date -u # compare against a trusted NTP source timedatectl status # check whether the clock is synchronised ```

1. The secret is not base32. Standard authenticator apps require a base32

secret. If the stored or displayed secret contains +, /, = or lowercase letters, the app imported it incorrectly (or the code generated it wrong) and will produce codes that never match. generateSecret() in license/lib/TOTPService.php must emit ^[A-Z2-7]{32}$; this is asserted by tests/test_totp_2fa.php.

1. Timezone / period mismatch. TOTP is period-based, not timezone-based,

so a wrong *system* timezone does not matter — but a device whose clock is minutes off does.

1. Wrong account. Confirm the secret shown during setup belongs to the

account actually being logged into:

```sql SELECT username, (totp_secret IS NOT NULL) AS has_2fa FROM admins; ```

1. The secret is corrupted at rest. totp_secret is AES-256-GCM

ciphertext. If the derived key changed (for example v2_view_key was rotated without re-encrypting), decryption throws and every code fails. Clear 2FA for that account and re-enroll:

```sql UPDATE admins SET totp_secret = NULL WHERE username = 'USERNAME'; ```

Run the regression suite to confirm the implementation itself is sound before suspecting the deployment:

php tests/test_totp_2fa.php

All assertions must pass; the RFC 4226 vectors in particular prove the HMAC and truncation are correct.

Getting More Help

When filing a bug, include:

1. Endpoint URL + HTTP status
2. Request body (omit secrets)
3. Response body
4. Nginx error log snippet (last 20 lines)
5. PHP-FPM error log (if available)
6. Output of license/api/coforen/health/

For urgent issues, contact: see /admin/system.php → Emergency Contacts.

See Also

• DEPLOYMENT.md — Environment variables, cron jobs
• SECURITY.md — Audit results
• DATABASE.md — Schema details for debugging query issues