Symptoms
- A custom log exists in a log group, an agent configuration points at
/var/log/app/*.log, but Log Search shows nothing. - Some instances send logs; others in the same pool don't.
- Logs stopped after an OS hardening change or image update.
The pipeline has four links: plugin running → instance allowed (IAM) → configuration delivered (dynamic group targeting) → files readable and endpoint reachable. Check them in that order.
Step 1: Is the Custom Logs Monitoring plugin enabled and running?
From the console: Instance > Oracle Cloud Agent tab. The plugin Custom Logs Monitoring must be Enabled and Running.
From the CLI:
oci compute instance get --instance-id $INSTANCE \
--query 'data."agent-config"'
oci instance-agent plugin list --instanceagent-id $INSTANCE --compartment-id $COMPARTMENT \
--query 'data[].{name:name, status:status, msg:message}' --output table
Enable it if needed:
oci compute instance update --instance-id $INSTANCE --agent-config \
'{"isMonitoringDisabled":false,"pluginsConfig":[{"name":"Custom Logs Monitoring","desiredState":"ENABLED"}]}' --force
On the instance (Oracle Linux), check the agent itself:
sudo systemctl status oracle-cloud-agent
sudo ls /var/log/oracle-cloud-agent/plugins/unifiedmonitoring/ 2>/dev/null || \
sudo find /var/log/oracle-cloud-agent -maxdepth 3 -name '*.log' | head
The Custom Logs Monitoring plugin runs a Fluentd-based agent under the Oracle Cloud Agent; its own log under /var/log/oracle-cloud-agent/plugins/ tells you about config download and upload errors. Paths differ slightly by agent version.
Tip
On Ubuntu and other non-Oracle images, the Oracle Cloud Agent may not be preinstalled, or may be an older snap. Check the OCI documentation for the supported images and install method for your OS.
Step 2: Dynamic group and policy
The instance authenticates as an instance principal. It needs to be in a dynamic group, and that group needs permission to push logs:
# Dynamic group matching rule (example: all instances in a compartment)
ALL {instance.compartment.id = 'ocid1.compartment.oc1..aaaa...'}
Allow dynamic-group app-instances to use log-content in compartment app-prod
Check the dynamic group rule actually matches the instance (a typo in the compartment OCID is common):
oci iam dynamic-group list --compartment-id $TENANCY \
--query "data[?name=='app-instances'].\"matching-rule\"" --raw-output
Policies are evaluated in the home region; changes can take a few minutes to apply.
Step 3: Does the agent configuration target this instance?
The agent configuration has a host group: one or more dynamic groups (or user groups for some setups). The instance must belong to a dynamic group listed there, or it never downloads the config.
oci logging agent-configuration list --compartment-id $COMPARTMENT \
--query 'data[].{name:"display-name", id:id, enabled:"is-enabled"}' --output table
oci logging agent-configuration get --config-id <config-ocid> \
--query 'data.{groups:"group-association", cfg:"service-configuration"}'
Look at:
is-enabled= true.group-association.group-listincludes the dynamic group from Step 2.service-configuration.sources[].pathsmatches the real file path.destination.log-object-idis the custom log you're searching.
Step 4: File path, permissions and log rotation
ls -l /var/log/app/
sudo -u root head -2 /var/log/app/app.log
Common problems:
| Problem | Effect | Fix |
|---|---|---|
Path in config is /var/log/app/*.log, app writes /var/log/app/app.log.1 and current file is app.current | Pattern doesn't match | Fix the glob |
| Files owned by app user with mode 600 in a 700 directory | Agent can't read them | Relax permissions or add the agent user to the group (check agent run-as user for your version) |
| SELinux enforcing on custom paths | Read denied, logged as AVC | sudo ausearch -m avc -ts recent; add a proper context or policy |
Log rotated with copytruncate too often | Lines skipped | Use rename-based rotation |
| Old lines only | The agent tails new lines; it doesn't upload history | Write a test line after setup |
Write a test line and search for it:
echo "oci-logging-test $(date -u +%FT%TZ) $(hostname)" | sudo tee -a /var/log/app/app.log
search "ocid1.compartment.oc1..aaaa.../app-logs/app-custom"
| where logContent like '%oci-logging-test%'
Step 5: Can the instance reach the Logging service?
Private instances need a Service Gateway (All Services label) or NAT to reach the regional ingestion endpoint. Test HTTPS egress:
curl -sS -o /dev/null -w '%{http_code}\n' https://ingestion.logging.ap-mumbai-1.oci.oraclecloud.com
Any HTTP code (even 404) proves connectivity; a timeout means routing, security rules or a proxy. If you use a proxy, the agent must be configured to use it; check the Oracle Cloud Agent proxy documentation.
Step 6: Parser errors
If the source uses a regex or JSON parser and lines don't match, entries may be dropped or arrive unparsed. Start with the None parser to prove the pipeline works, then add parsing.
Symptom → fix table
| What you see | Fix |
|---|---|
| Plugin "Stopped" or missing | Enable plugin; update Oracle Cloud Agent |
| Agent log shows 401/403 / NotAuthorized | Dynamic group rule, use log-content policy |
| Agent log shows no config | Agent configuration host group doesn't include the instance's dynamic group |
| Config downloaded, nothing uploaded | Path glob, permissions, SELinux |
| Timeouts in agent log | Service Gateway / NAT / proxy |
| Only some pool instances work | Dynamic group rule based on tag or OCID list that new instances don't match |
Prevent it next time
- Use compartment- or tag-based dynamic group rules so new instances in a pool are covered automatically.
- Bake Oracle Cloud Agent and the plugin setting into your image and instance configuration.
- Add a canary line on boot (
logger/echointo the app log) and alert in OCI Monitoring if no log volume arrives for 15 minutes. - Keep agent configuration, dynamic groups and policies together in Terraform.