Skip to content

Upgrade and expansion

Upgrading a single node (SNSD, SNMD)

  1. Read the release notes and verify the release (Installation).
  2. To be safe, export IAM and configuration: nexoctl admin cluster export local --output backup.zip (see Backup and migration).
  3. Replace the binary (or the package, or the image tag) and restart the service:
sudo apt install ./nexo_1.0.1_amd64.deb     # or: install -m 0755 nexo /usr/local/bin/nexo
sudo systemctl restart nexo
nexo healthcheck --ready && nexo version

The restart stops the server gracefully (SIGTERM: new requests get 503, the ones in flight complete within the shutdown timeout) and interrupts the service for a few seconds.

Rolling upgrade of a cluster

Nodes with the same major version coexist in the same cluster: you upgrade one node at a time without interrupting the service, provided the quorum holds with one node down (see Deployment modes → Choosing the parity). A node with a major version different from the majority does not join the cluster.

For each node, one at a time:

  1. Check that a node can be shut down: /nexo/health/cluster must answer "maintenance": true (and writeQuorum: true):
curl -s http://nexo1.lan:9000/nexo/health/cluster
  1. Install the new binary and restart the service on the node:
sudo apt install ./nexo_1.0.1_amd64.deb && sudo systemctl restart nexo

While stopping, the node answers 503 to /nexo/health/ready: the load balancer excludes it and reads are retried on another node (the configurations in deploy/proxy already do this).

  1. Wait until it is ready and in sync again:
nexo healthcheck --ready                     # on the node; with TLS: --certs-dir /etc/nexo/certs
nexoctl admin info local                     # every node online, the new version on the upgraded node
  1. Move on to the next node. The writes that arrived while the node was down are completed by the healing queue (MRF) when it comes back.

With orchestrators:

  • Kubernetes (Helm): helm upgrade with the new tag updates the StatefulSet one pod at a time; the PodDisruptionBudget (maxUnavailable: 1) protects the quorum during drains.
  • Docker Swarm: docker service update with parallelism: 1 (already set in deploy/swarm/stack.yml).
  • Docker Compose: docker compose up -d --no-deps nexo1, wait for healthy, then nexo2, …

Restart without upgrading

nexoctl admin service restart local          # asks for confirmation; --yes in scripts

Restarts the services of the node that answers the alias's Admin API within the same process (same binary and same static configuration): useful after a problem, not to install a new version. nexoctl admin service stop stops the process (it must be restarted by hand or by the supervisor). They require admin:ServiceRestart and admin:ServiceStop.

Reloading the TLS certificates needs no restart: systemctl reload nexo (SIGHUP) or just replacing the files (see TLS).

Expansion with a new pool

A cluster grows by adding a pool: a group of nodes (possibly with drives of a different size from the first pool). Existing data is not moved.

  1. Prepare the new nodes like the first ones: same Nexo version, same root credentials, same KMS configuration, drives mounted and empty, names resolvable from every node, port 9000 open between all of them.
  2. Work out the new command line: one argument with ellipses per pool, in the same order on every node (the existing pool stays first):
NEXO_ENDPOINTS="http://nexo{1...4}.lan:9000/mnt/disk{1...4} http://nexo{5...8}.lan:9000/mnt/disk{1...4}"

Check it on a new node with nexo server --dry-run (expected pools, sets and parity). 3. Stop all the nodes of the cluster, update NEXO_ENDPOINTS (or the arguments) on every node, old and new, and restart them all. During the bootstrap the nodes check that they have the same topology: a node left with the old command line does not join the cluster. This is a short, planned interruption. 4. The drives of the new pool are formatted with the same deployment ID. Verify with nexoctl admin info local: two pools, every set ok.

After the expansion:

  • the objects already present stay in their pool, even when they are overwritten;
  • new objects go to the pools in proportion to their free space (pools above 95% are excluded);
  • rebalancing data between pools and decommissioning a pool are not available yet.

On Kubernetes the expansion is distributed.pools in the chart (here too with a restart of every pod); with Docker Compose or Swarm you add the services of the new nodes and change command on all of them.

Before upgrading: checklist

  • [ ] backup of IAM and configuration (admin cluster export);
  • [ ] healthy cluster: every node online, no degraded set, empty MRF queue (nexoctl admin info);
  • [ ] free space and inodes on the drives;
  • [ ] synchronized clocks;
  • [ ] same KMS configuration on every node (a node with a KMS different from the majority does not start).