Operations

Routine checks

Before every exercise

• Confirm the selected cloud scope, home cluster, namespace, resource kind, and name.
• Confirm the entitlement has sufficient scope and executions.
• Confirm the change ticket, owner, window, and rollback responsibility.
• Review capacity, redundancy, current incidents, and recent Monitor events.
• Run Dry run and resolve every warning.

Daily or before use

• Controller, UI, and probe pods are Ready.
• Evidence storage writes succeed.
• Cloud discovery is current and has no identity, IAM, or API warning.
• Entitlement state is understood and operational.

Weekly

• Review failed, blocked, and overdue validations.
• Assign owners for new Risk Register findings.
• Sample evidence links and digests.
• Remove user access that is no longer required.

Backup and restore

For a release named trinetra, Trinetra has two stateful PVCs:

If the Helm release name differs, replace the trinetra- prefix with that release name. Reports and validation evidence are stored separately in the configured object-storage bucket and are not included in either PVC backup.

Cold file backup

Run this procedure in a non-production environment before relying on it. It intentionally stops the two writers so their local databases are copied consistently.

export NS="trinetra-system"
export RELEASE="trinetra"
export BACKUP_DIR="trinetra-backup-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$BACKUP_DIR/license" "$BACKUP_DIR/ui-auth"

kubectl -n "$NS" scale deployment "${RELEASE}-controller" --replicas=0
kubectl -n "$NS" scale deployment "${RELEASE}-ui" --replicas=0
kubectl -n "$NS" wait --for=delete pod \
  -l app.kubernetes.io/instance="$RELEASE",app.kubernetes.io/component=controller \
  --timeout=120s
kubectl -n "$NS" wait --for=delete pod \
  -l app.kubernetes.io/instance="$RELEASE",app.kubernetes.io/component=ui \
  --timeout=120s

Create a temporary helper for the licence PVC, copy all files, and remove the helper:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata: {name: trinetra-license-copy, namespace: ${NS}}
spec:
  restartPolicy: Never
  containers:
    - name: copy
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts: [{name: data, mountPath: /data}]
  volumes:
    - name: data
      persistentVolumeClaim: {claimName: ${RELEASE}-license}
EOF
kubectl -n "$NS" wait --for=condition=Ready pod/trinetra-license-copy --timeout=120s
kubectl -n "$NS" cp trinetra-license-copy:/data/. "$BACKUP_DIR/license"
kubectl -n "$NS" delete pod trinetra-license-copy --wait=true

Repeat for the authentication PVC:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata: {name: trinetra-ui-auth-copy, namespace: ${NS}}
spec:
  restartPolicy: Never
  containers:
    - name: copy
      image: busybox:1.36
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts: [{name: data, mountPath: /data}]
  volumes:
    - name: data
      persistentVolumeClaim: {claimName: ${RELEASE}-ui-auth}
EOF
kubectl -n "$NS" wait --for=condition=Ready pod/trinetra-ui-auth-copy --timeout=120s
kubectl -n "$NS" cp trinetra-ui-auth-copy:/data/. "$BACKUP_DIR/ui-auth"
kubectl -n "$NS" delete pod trinetra-ui-auth-copy --wait=true

tar -czf "${BACKUP_DIR}.tar.gz" "$BACKUP_DIR"
sha256sum "${BACKUP_DIR}.tar.gz" > "${BACKUP_DIR}.tar.gz.sha256"

Store the encrypted backup and checksum in customer-approved storage, then restart the deployments:

kubectl -n "$NS" scale deployment "${RELEASE}-controller" --replicas=1
kubectl -n "$NS" scale deployment "${RELEASE}-ui" --replicas=1
kubectl -n "$NS" rollout status deployment/"${RELEASE}-controller" --timeout=5m
kubectl -n "$NS" rollout status deployment/"${RELEASE}-ui" --timeout=5m

Cold restoration

Restore only into PVCs created for the same Trinetra release and installation. Verify the archive checksum, stop controller and UI as shown above, recreate each corresponding helper pod, and copy the saved contents back:

sha256sum -c trinetra-backup-YYYYMMDDTHHMMSSZ.tar.gz.sha256
tar -xzf trinetra-backup-YYYYMMDDTHHMMSSZ.tar.gz

# After creating trinetra-license-copy with the backup helper manifest,
# stop immediately unless it mounts the intended release PVC:
test "$(kubectl -n "$NS" get pod trinetra-license-copy \
  -o jsonpath='{.spec.volumes[?(@.name=="data")].persistentVolumeClaim.claimName}')" \
  = "${RELEASE}-license"
kubectl -n "$NS" exec trinetra-license-copy -- sh -c 'rm -rf /data/* /data/.[!.]* /data/..?* 2>/dev/null || true'
kubectl -n "$NS" cp ./trinetra-backup-YYYYMMDDTHHMMSSZ/license/. trinetra-license-copy:/data
kubectl -n "$NS" exec trinetra-license-copy -- chown -R 1000:1000 /data
kubectl -n "$NS" delete pod trinetra-license-copy --wait=true

# After creating trinetra-ui-auth-copy with the authentication helper manifest,
# stop immediately unless it mounts the intended release PVC:
test "$(kubectl -n "$NS" get pod trinetra-ui-auth-copy \
  -o jsonpath='{.spec.volumes[?(@.name=="data")].persistentVolumeClaim.claimName}')" \
  = "${RELEASE}-ui-auth"
kubectl -n "$NS" exec trinetra-ui-auth-copy -- sh -c 'rm -rf /data/* /data/.[!.]* /data/..?* 2>/dev/null || true'
kubectl -n "$NS" cp ./trinetra-backup-YYYYMMDDTHHMMSSZ/ui-auth/. trinetra-ui-auth-copy:/data
kubectl -n "$NS" exec trinetra-ui-auth-copy -- chown -R 1000:1000 /data
kubectl -n "$NS" delete pod trinetra-ui-auth-copy --wait=true

Restart controller and UI, then verify sign-in, Licence state, enrolled scope, discovery, one Dry run, History, and access to a sampled evidence object. Keep object versioning and the customer's retention policy enabled for the evidence bucket.

Restoration to a different installation may require licence reissue because entitlements can be bound to the installation ID. Contact LNP Technologies before moving or rebuilding an installation.

Upgrade

1. Review release notes, package version, and image digest.
2. Back up persistent data and export configuration.
3. Review integration, IAM, CRD, and entitlement changes.
4. Upgrade during an approved window.
5. Wait for all deployments to become Ready.
6. Verify sign-in, discovery, Licence status, evidence access, and a Dry run.
7. Record the deployed digest and rollback reference.

Do not reuse a mutable image tag for a different build.

Scaling

The standard deployment uses one replica for each Trinetra component and limits independent concurrent runs. Do not scale stateful components or increase concurrency without validating shared storage, leases, API quotas, and target-estate safety.

Delete or uninstall

Export required reports, preserve evidence, and back up persistent data before deletion. Review volumes, services, load balancers, identities, IAM bindings, secrets, and evidence storage separately; do not assume uninstall removes them. Retain or delete evidence according to customer policy.