Get Started¶

This guide deploys API Observer on a Linux server and API Collector beside an application protected by ModSecurity. Collector reads the application's audit log, streams new transactions to Observer, and answers historical log queries from Observer. Repeat the Collector steps for each protected application or node.

The examples use systemd and these placeholder addresses:

Replace the hostnames, paths, service accounts, and ModSecurity log path with values for your environment.

Prerequisites¶

• Linux hosts with systemd. Create dedicated api-observer and api-collector service accounts on their respective hosts.
• A ModSecurity deployment that writes one JSON transaction per line to a persistent audit log. Collector must be able to read that file, including after log rotation.
• Go 1.26.4 or newer and a C compiler on the build host. Both Go modules use DuckDB through CGO. Build binaries for the same operating system and architecture as their target hosts.
• Network routes that allow Collector to reach Observer on TCP 24899 and Observer to reach each Collector on TCP 24900.
• An HTTPS reverse proxy that authenticates dashboard users and forwards to Observer's local HTTP listener.

API Observer currently serves HTTP without built-in authentication, and the gRPC connections use plaintext. Bind the dashboard to loopback behind an authenticated HTTPS reverse proxy. Restrict the gRPC ports to the participating hosts over a trusted private network or encrypted tunnel; do not expose them to the public internet.

Build and install the binaries¶

From a checkout of this repository, build each module separately:

cd api-observer
CGO_ENABLED=1 go build -o api-observer .
cd ../api-collector
CGO_ENABLED=1 go build -o api-collector .

Install the api-observer binary on the Observer host and the api-collector binary on each Collector host, for example under /usr/local/bin/. Also copy api-observer/internal/dashboard/views/assets/ to /opt/api-observer/internal/dashboard/views/assets/ on the Observer host. Observer loads those static assets relative to its working directory; the systemd unit below sets that directory to /opt/api-observer.

Configure API Observer¶

Create /var/lib/api-observer/config.yaml on the Observer host. Give the api-observer service account write access to the file and its parent directory: Observer saves registered nodes and node names back to this configuration. Give the account write access to /var/log/api-observer/ as well.

app_log: /var/log/api-observer/api-observer.log
findings_log: /var/log/api-observer/findings.jsonl
rules: /etc/api-observer/rules.yaml
ingest_address: :24899
dashboard_port: 127.0.0.1:8080
queue_size: 1000
worker_count: 5
nodes: []

Create /etc/api-observer/rules.yaml before starting the service. An empty starting rule set is:

rules: {}

Add detection rules to that file when ready, then restart Observer to load them. The rules path may instead be "" if you want to start without a rule file. Use absolute paths for production configuration and logs.

Create /etc/systemd/system/api-observer.service:

[Unit]
Description=API Observer
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=api-observer
Group=api-observer
WorkingDirectory=/opt/api-observer
Environment=API_OBSERVER_CONFIG=/var/lib/api-observer/config.yaml
Environment=XDG_CONFIG_HOME=/var/lib/api-observer
ExecStart=/usr/local/bin/api-observer start --background=false
Restart=on-failure
StateDirectory=api-observer
LogsDirectory=api-observer
UMask=0077

[Install]
WantedBy=multi-user.target

Start it and inspect its status:

sudo systemctl daemon-reload
sudo systemctl enable --now api-observer
sudo systemctl status api-observer

Point your authenticated HTTPS reverse proxy at 127.0.0.1:8080 on the Observer host. Allow Collector hosts to reach the configured ingest listener on TCP 24899.

Deploy API Collector beside ModSecurity¶

On each protected node, identify the JSON audit log produced by ModSecurity. The example below uses /var/log/modsecurity/audit.json; replace it with the real path. The file must exist before Collector starts, and the api-collector account needs read access to it and its parent directories. Explorer and Analysis query the current file at that path; they do not search rotated archives. Set log rotation and retention with that limitation in mind.

Create /etc/systemd/system/api-collector.service on the Collector host:

[Unit]
Description=API Collector
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=api-collector
Group=api-collector
Environment=XDG_CONFIG_HOME=/var/lib/api-collector
ExecStart=/usr/local/bin/api-collector start --background=false --data-dir=/var/lib/api-collector --file-path=/var/log/modsecurity/audit.json --observer-ingest-addr=observer.internal.example:24899 --query-addr=:24900 --advertise-query-addr=collector.internal.example:24900
Restart=on-failure
StateDirectory=api-collector
UMask=0077

[Install]
WantedBy=multi-user.target

The advertised address must be reachable from Observer. Allow the Observer host to reach this Collector on TCP 24900, and restrict other clients. Keep /var/lib/api-collector/ across restarts and upgrades: Collector stores its stable node ID there.

sudo systemctl daemon-reload
sudo systemctl enable --now api-collector
sudo systemctl status api-collector

Collector starts reading new lines at the end of the audit file. Requests already in the file remain available to historical queries, but they are not streamed as new ingest events when Collector starts. Its first heartbeat is sent within about 15 seconds.

Verify the deployment¶

1. Send an authorized request through the protected application's ModSecurity entry point, after Collector has started.
2. Open https://observer.example.com/nodes. Confirm that the Collector appears and is connected. You can rename it here.
3. Open https://observer.example.com/ for Log Explorer. Search the time range containing the request, then expand a row to inspect the request, response, and findings. The default search range is the last 24 hours.
4. Open https://observer.example.com/analysis. Enter a rule set on the Rules tab, return to Logs, and run a search to evaluate historical records. Rules entered here apply to that analysis run; rules in /etc/api-observer/rules.yaml are loaded by Observer for incoming traffic.

If a node does not appear, check journalctl -u api-collector and journalctl -u api-observer, the audit log permissions, and connectivity to TCP 24899. If a node appears but searches fail, check that Observer can reach its advertised query address on TCP 24900 and that Collector can still read the audit log. A node is marked disconnected if its heartbeats stop for more than 45 seconds.