GCE - What is startpar: service(s) returned failure: google-address-manager google google-startup...

Troubleshooting 'startpar: service(s) returned failure' on Google Compute Engine

Understand and resolve common startup failures like 'google-address-manager', 'google-startup-scripts', and 'google-instance-setup' on GCE instances.

When launching a Google Compute Engine (GCE) instance, you might encounter a startpar: service(s) returned failure message during the boot process. This error typically indicates that one or more critical Google-specific services failed to start correctly. These services are essential for proper instance operation, including network configuration, metadata handling, and script execution. This article will delve into the common causes of these failures and provide systematic troubleshooting steps.

Understanding the 'startpar' Error

The startpar utility is a parallel service starter often used in Debian-based systems. When it reports a failure, it means that one or more services it attempted to start did not exit successfully. In the context of GCE, the services most commonly associated with this error are:

• google-address-manager: Manages internal and external IP addresses, network interfaces, and routing.
• google-startup-scripts: Executes user-provided startup scripts from instance metadata.
• google-instance-setup: Performs initial instance configuration, such as setting the hostname and applying SSH keys.
• google-accounts-manager: Manages user accounts and SSH key distribution.

A failure in any of these services can prevent the instance from booting correctly, leading to network issues, inaccessible SSH, or applications failing to start.

flowchart TD
    A[GCE Instance Boot] --> B{startpar executes services}
    B --> C{google-address-manager}
    B --> D{google-startup-scripts}
    B --> E{google-instance-setup}
    C -- FAIL --> F[startpar: service(s) returned failure]
    D -- FAIL --> F
    E -- FAIL --> F
    C -- SUCCESS --> G[Network Configured]
    D -- SUCCESS --> H[Startup Scripts Run]
    E -- SUCCESS --> I[Instance Setup Complete]
    F --> J[Troubleshooting Required]
    G & H & I --> K[Instance Operational]

Flowchart of GCE Instance Boot and startpar Service Execution

Common Causes and Initial Diagnostics

Several factors can lead to these startup failures. Identifying the root cause often involves inspecting instance logs and configuration. Here are the most common culprits:

1. Corrupted or Missing Google Guest Environment: The Google Guest Environment (GGE) provides the necessary agents and scripts for GCE instances to interact with the Google Cloud platform. If these packages are corrupted, outdated, or missing, core services will fail.
2. Network Configuration Issues: Incorrect network settings, firewall rules, or VPC configurations can prevent google-address-manager from obtaining an IP address or configuring network interfaces.
3. Startup Script Errors: If google-startup-scripts fails, it often means a script provided in the instance metadata has an error, a missing dependency, or incorrect permissions, causing the script to exit with a non-zero status.
4. Disk Space or I/O Issues: Insufficient disk space or underlying disk I/O problems can prevent services from writing logs or accessing necessary files.
5. Kernel or OS-level Problems: Less common, but a corrupted kernel or fundamental OS issues can prevent services from initializing.

To diagnose, the first step is always to check the serial console output and instance logs.

Troubleshooting Steps

Follow these steps to systematically diagnose and resolve startpar failures on your GCE instance.

1. Access Serial Console Output

Navigate to your instance in the Google Cloud Console. Under 'VM instances', select your instance, then go to 'Monitoring' -> 'Serial port 1 (console)'. Look for messages indicating which service failed and any associated error codes or stack traces. This is crucial for understanding the specific failure.

2. Check Instance Logs

If the serial console doesn't provide enough detail, use Cloud Logging. Filter logs by your instance name and look for entries from google-address-manager, google-startup-scripts, google-instance-setup, or systemd units that failed to start. You can also SSH into the instance (if possible) and check /var/log/syslog, /var/log/daemon.log, or journalctl -xe.

3. Verify Google Guest Environment

If you can SSH into the instance, ensure the Google Guest Environment packages are installed and up-to-date. For Debian/Ubuntu:

sudo apt update && sudo apt install --only-upgrade google-compute-engine

For CentOS/RHEL:

sudo yum update google-compute-engine

If you cannot SSH, consider creating a snapshot of the disk and attaching it to a new, working instance to perform these checks.

4. Review Startup Scripts

If google-startup-scripts failed, examine your startup script for syntax errors, missing dependencies, or commands that might exit with a non-zero status. Test the script on a separate, working instance to isolate issues. Remove the startup script from the instance metadata and try rebooting to see if the error persists.

5. Inspect Network Configuration

Ensure your VPC network, subnet, and firewall rules are correctly configured. Verify that the instance has appropriate network tags and that no custom network configurations are interfering with the default GCE network setup. Check if the instance has a valid external IP address (if configured to have one).

6. Recreate Instance with Same Disk

As a last resort, if the issue persists and you suspect OS corruption, create a snapshot of your instance's boot disk. Then, create a new instance using this snapshot as the boot disk. This can sometimes resolve underlying OS issues without losing your data.

gcloud compute instances get-serial-port-output [INSTANCE_NAME] --zone=[ZONE]

Command to retrieve serial console output for a GCE instance.