orchestrating applications #6

New Issue

Open

opened 2024-01-18 02:14:31 +00:00 by kiara · 1 comment

kiara commented

2024-01-18 02:14:31 +00:00

Owner

i would be interested in trying hashicorp's nomad for orchestration.

relevant notes on that:

terraform provider: poc
- from nix: nix-nomad
nix service: dev
- raw_exec
- exec - use dropPrivileges = false;
- containers (with NixNG?)
  - docker (needs root) - use dropPrivileges = false; from nomad_1_5
  - error Constraint ${attr.consul.version} semver >= 1.7.0 filtered 1 node
  - ~~using podman with docker alias~~ - will show as unhealthy
  - podman: services.nomad.extraSettingsPlugins = with pkgs; [ nomad-driver-podman ]; - same error as docker
  - singularity (for safety restricts container root to host root) - services.nomad.settings.plugin.nomad-driver-singularity.config.singularity_path = "${pkgs.apptainer}/bin/apptainer"; errors with No argument or block type is named "singularity_path"
- qemu: not sure how to load this driver. invocations need .img file too.
- nix
  - without nomad tho with volumes
    - using SrvOS?
  - thru nomad exec driver as nixpkgs services in flake (if that's possible)?
  - thru container driver (see driver errors above), using image:
    - ~~nixos/nix~~ - alpine over nixos so won't do nixpkgs services
    - wip nixos container
    - made using nix (packages rather than services?) (examples) with some modules
    - ~~made using nix-container~~ - omits {pkgs, lib, ...}: {} so seems harder to use the same config across this and outside containers
    - ~~NixOS containers: can use a config like configuration.nix to make containers~~ - seems not as portable as i'd like, on top of not doing flakes
    - ~~nix2container: make container package using nix from nix~~ - lacks services
    - ~~services-flake~~ - limited to certain services
    - extra-container: can use a config like configuration.nix to make containers - doesn't seem to do flakes
    - flake-containers: can use a config like configuration.nix to make containers from a portable flake
    - made using arion: lets you do docker compose (i.e. can do multiple nodes) thru nix projects from:
      - ~~flake: a flake with packages~~ - lacks services
      - nixos: can use a config like configuration.nix to make containers - doesn't seem to do flakes -- could i just source in a dep from flake? - i think their flake and nixos examples could be combined.
  - nomad-driver-nix (flake): examples include setting packages+command or system flake - rpc error: code = Unknown desc = Couldn't determine flake requisites: json: cannot unmarshal object into Go value of type []*nix2.nixPathInfo on nix path-info with nix 2.18.1 while they used 2022-02-18 @ 9bc03 - 2.6.1+, containing +valid tho not yet -path. i tried running a nixos setup with this older nix, tho i did not manage to actually switch to this older version.
  - nomad-driver-nix2 (non-flake): run Nix batch/service jobs by packages+command (+ flake), isolation as exec. - as above
  - nomad-driver-containerd-nix (flake): sample based on flake+entrypoint. flaky on name containerd-driver vs nomad-driver-containerd, but changes mostly this, and a better example seems their config. - build still fails with Unsupported plugin type
integrating with consul and vault
hetzner
- terraform-based test setup w/ consul
comin with webhooks reenabled?

further concerns:

functional
- network
  - infra
  - orchestration: CNI
  - driver
    - container
    - nix
  - application
- volumes
  - infra: hetznercloud/csi-driver (non-flake, nomad doc)
    - (cf. nixops-hetznercloud also did stuff with this)
  - orchestration: stateful nomad workloads with volumes
  - driver
    - jobs
    - exec
    - container
      - docker
    - nix
      - nomad-driver-nix
  - application
- variables
  - infra
  - orchestration
  - driver
    - container
    - nix
  - application
security
- escalation
- write
- network
- secrets

i would be interested in trying hashicorp's [nomad](https://developer.hashicorp.com/nomad) for orchestration. relevant notes on that: - [x] [terraform provider](https://registry.terraform.io/providers/hashicorp/nomad/latest/docs): [poc](https://code.bij1.org/bij1/terraform-config/compare/main...nomad-tf-poc) - [x] from nix: [nix-nomad](https://tristanpemble.github.io/nix-nomad/) - [nix service](https://mynixos.com/nixpkgs/options/services.nomad): [dev](https://codeberg.org/kiara/cfg/compare/main...hashicorp-wip) - [x] `raw_exec` - [x] `exec` - use `dropPrivileges = false;` - [ ] containers (with [`NixNG`](https://github.com/nix-community/NixNG/)?) - [ ] `docker` (needs root) - use `dropPrivileges = false;` from `nomad_1_5` - error [Constraint `${attr.consul.version} semver >= 1.7.0` filtered 1 node](https://github.com/hashicorp/waypoint/issues/3376) - ~~using podman with docker alias~~ - will show as unhealthy - [ ] `podman`: `services.nomad.extraSettingsPlugins = with pkgs; [ nomad-driver-podman ];` - same error as `docker` - [ ] [`singularity`](https://developer.hashicorp.com/nomad/plugins/drivers/community/singularity) (for safety restricts container root to host root) - `services.nomad.settings.plugin.nomad-driver-singularity.config.singularity_path = "${pkgs.apptainer}/bin/apptainer";` errors with [`No argument or block type is named "singularity_path"`](https://github.com/hpcng/nomad-driver-singularity/issues/47) - [ ] [`qemu`](https://developer.hashicorp.com/nomad/docs/drivers/qemu): not sure how to load this driver. invocations need `.img` file too. - [ ] nix - [ ] without nomad tho with [volume](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/volume)s - [ ] using [SrvOS](https://nix-community.github.io/srvos/)? - [ ] thru nomad `exec` driver as nixpkgs services in flake (if that's possible)? - [ ] thru container driver (see driver errors above), using image: - ~~`nixos/nix`~~ - alpine over nixos so won't do nixpkgs `services` - [ ] [wip nixos container](https://github.com/nix-community/docker-nixpkgs/pull/3/files) - [ ] made using [nix](https://github.com/nix-community/docker-nixpkgs/blob/master/images/curl/default.nix) (packages rather than services?) ([examples](https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/docker/examples.nix#L24-L32)) with [some modules](https://github.com/cloudwatt/nix-container-images?tab=readme-ov-file#supported-nixos-modules) - [ ] ~~made using [`nix-container`](https://nixos.org/manual/nixos/stable/#sec-imperative-containers)~~ - omits `{pkgs, lib, ...}: {}` so seems harder to use the same config across this and outside containers - [ ] ~~[NixOS `containers`](https://nixos.wiki/wiki/NixOS_Containers): can use a config like `configuration.nix` to make containers~~ - seems not as portable as i'd like, on top of not doing flakes - [ ] ~~[`nix2container`](https://github.com/nlewo/nix2container): make container package using nix from nix~~ - lacks services - [ ] ~~[`services-flake`](https://github.com/juspay/services-flake)~~ - limited to [certain services](https://community.flake.parts/services-flake/services) - [ ] [`extra-container`](https://github.com/erikarvstedt/extra-container): can use a config like `configuration.nix` to make containers - doesn't seem to do flakes - [ ] [`flake-containers`](https://github.com/adfaure/flake-containers): can use a config like `configuration.nix` to make containers from a portable flake - [ ] made using [`arion`](https://docs.hercules-ci.com/arion/): lets you do docker compose (i.e. can do multiple nodes) thru nix projects from: - ~~[flake](https://github.com/hercules-ci/arion/tree/main/examples/flake): a flake with packages~~ - lacks services - [nixos](https://github.com/hercules-ci/arion/blob/main/examples/full-nixos/arion-compose.nix): can use a config like `configuration.nix` to make containers - doesn't seem to do flakes -- could i just source in a dep from flake? - i think their flake and nixos examples could be combined. - [ ] [nomad-driver-nix](https://github.com/JanMa/nomad-driver-nspawn/compare/master...input-output-hk:nomad-driver-nix:main) (flake): examples include setting packages+command or system flake - [`rpc error: code = Unknown desc = Couldn't determine flake requisites: json: cannot unmarshal object into Go value of type []*nix2.nixPathInfo`](https://github.com/input-output-hk/nomad-driver-nix/blob/010b09c680887d0cade86e8ac136c3a04609e04a/nix/nspawn.go#L770) on `nix path-info` with nix 2.18.1 while they used 2022-02-18 @ [`9bc03`](https://github.com/NixOS/nix/commit/9bc03adbba5334663901c1136203bc07e4776be9) - 2.6.1+, containing [+valid](https://github.com/NixOS/nix/compare/9bc03adbba5334663901c1136203bc07e4776be9...2.18.1) tho not yet [-path](https://github.com/NixOS/nix/commit/cc46ea163024254d0b74646e1b38b19896d40040). i tried running a nixos setup [with this older nix](https://codeberg.org/kiara/cfg/commit/b275b641f13dc71b3c66c4710d50ba7ef41d5bee), tho i did not manage to actually switch to this older version. - [ ] [nomad-driver-nix2](https://git.deuxfleurs.fr/Deuxfleurs/nomad-driver-nix2) (non-flake): run Nix batch/service jobs by packages+command (+ flake), isolation as `exec`. - [as above](https://git.deuxfleurs.fr/Deuxfleurs/nomad-driver-nix2/issues/1) - [ ] [nomad-driver-containerd-nix](https://github.com/MagicRB/nomad-driver-containerd-nix) (flake): [sample](https://github.com/MagicRB/nomad-driver-containerd-nix/blob/master/example/website.nomad) based on flake+entrypoint. flaky on name `containerd-driver` vs `nomad-driver-containerd`, but changes [mostly this](https://github.com/MagicRB/nomad-driver-containerd-nix/commit/3df2591ba5b279f879fcfe57577746a4da6940ce), and a better example seems [their config](https://git.irunx.org/MagicRB/ra-systems). - build still fails with `Unsupported plugin type` - [ ] [integrating with consul and vault](https://medium.com/@christian.mahardhika/deploy-hashicorp-nomad-consul-vault-locally-e8c2a5d6aa11) - [hetzner](https://community.hetzner.com/tutorials/install-nomad-consul-cluster#step-25---set-up-the-nomad-servers) - [terraform-based test setup w/ consul](https://registry.terraform.io/modules/wenzel-felix/nomad-consul-module/hetzner/latest) - `comin` with [webhooks](https://github.com/nlewo/comin/commit/110717b0a739255f19a2b0ae5f7ec7a98bb148a1) reenabled? further concerns:   - functional - [ ] network - [ ] infra - [ ] orchestration: [CNI](https://developer.hashicorp.com/nomad/docs/networking/cni) - [ ] driver - [ ] container - [ ] nix - [ ] application - [ ] volumes - [ ] infra: [hetznercloud/csi-driver](https://github.com/hetznercloud/csi-driver) (non-flake, [nomad doc](https://github.com/hetznercloud/csi-driver/blob/main/docs/nomad/README.md)) - (cf. [`nixops-hetznercloud`](https://github.com/lukebfox/nixops-hetznercloud) also did stuff with this) - [ ] orchestration: [stateful nomad workloads with volumes](https://developer.hashicorp.com/nomad/tutorials/stateful-workloads/stateful-workloads-host-volumes) - [ ] driver - [ ] [jobs](https://developer.hashicorp.com/nomad/docs/job-specification/volume) - [ ] [exec](https://developer.hashicorp.com/nomad/docs/drivers/exec) - [ ] container - [ ] [docker](https://developer.hashicorp.com/nomad/docs/drivers/docker#volume_driver) - [ ] nix - [ ] [nomad-driver-nix](https://github.com/input-output-hk/nomad-driver-nix/blob/010b09c680887d0cade86e8ac136c3a04609e04a/nix/driver.go) - [ ] application - [ ] variables - [ ] infra - [ ] orchestration - [ ] driver - [ ] container - [ ] nix - [ ] application - security - [ ] escalation - [ ] write - [ ] network - [ ] secrets

kiara commented

2024-02-06 22:50:12 +00:00

Author

Owner

hm maybe skip nomad for now to straight up use nix (+ terraform)?