# n8n CLI observability signals

Status: `VERIFIED`

This package compares four ways to classify `n8n execute --rawOutput` results:

1. process exit code;
2. generic raw-output words such as `error`, `failed`, or `failure`;
3. the exact injected error token;
4. structured execution JSON status.

## Protocol

- Pin `n8nio/n8n:2.30.5` and publish the image digest.
- Run one imported workflow 60 times in a persistent container with no network.
- Odd-numbered runs complete successfully.
- Even-numbered runs throw `EXPECTED_SIGNAL_FAILURE`.
- Record each detector output and a SHA-256 hash of the combined CLI output.
- Keep two normalized samples without publishing full logs or local paths.

## Result

| Detector | Correct | Accuracy | False failures | Missed failures |
| --- | ---: | ---: | ---: | ---: |
| Process exit code | 60 / 60 | 100% | 0 | 0 |
| Generic raw failure words | 30 / 60 | 50% | 30 | 0 |
| Exact injected token | 60 / 60 | 100% | 0 | 0 |
| Structured execution status | 60 / 60 | 100% | 0 | 0 |

The generic detector marked every successful run as failed because normal CLI
output included a task-runner warning containing the word `Failed`. This does
not mean all keyword searches always fail; it shows that generic words over an
unstructured stream are not a safe status contract.

For this pinned CLI test, exit code was a correct binary health signal.
Structured execution JSON provided the status and the error detail needed for
classification. Exact-token search also matched the controlled protocol, but
it depends on knowing every relevant token in advance.

## Files

- `manifest.json`: version, protocol, timestamps and image digest.
- `workflow.json`: importable alternating success/failure workflow.
- `runs.csv`: all 60 detector comparisons and output hashes.
- `samples.json`: normalized examples for one success and one failure.
- `summary.json`: aggregate detector accuracy and limitations.

## Limitations

- Failures are deterministic and synthetic.
- The result covers `n8n execute --rawOutput` on n8n 2.30.5 only.
- The container has no network and does not test third-party APIs.
- The experiment does not cover queue mode, webhooks, retries or external
  observability products.
- Full raw CLI logs are not published because they add machine-specific noise;
  each combined output is represented by its SHA-256 hash.