sk4nz/skz-opack
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

opack-cloud documenation, and image deletion on clean

Browse Source
This commit is contained in:
Samuel Aubertin 2023-09-26 09:51:47 +02:00
parent 17474fcab4
commit dee53a0349
5 changed files with 88 additions and 39 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

42
README.md
View File

@ -87,6 +87,7 @@ Then, `make opack-clean` will remove the VM, but not the vagrant box (in order t
### Overrides Example
In this example, we will demonstrate how to use overrides in **skz-opack** to create an OpenBSD *-current* virtual machine named "run-current" with 4 CPUs and 1GB of RAM.
```make
OPACK_TARGET= run-current
OPACK_SYS_VERSION= snapshots
@ -96,7 +97,7 @@ OPACK_SYS_MEMORY= 1024
include path/to/src/opack.mk
all: opack
vagrant ssh -c "uname -a; sysctl kern.version"
vagrant ssh -c "uname -a; sysctl -n kern.version"
```
In this Makefile:
@ -106,18 +107,47 @@ In this Makefile:
- **OPACK_SYS_CPU** allocates 4 CPUs to the VM.
- **OPACK_SYS_MEMORY** assigns 1GB of RAM to the VM.
Making "opack" a dependency target of "all" will ensure **skz-opack** operates before any command in the "all" target. Try it with `make all` !
Making *opack* a dependency target of *all* will ensure **skz-opack** operates before any command in the *all* target. Try it with `make all` !
Once the "opack" target dependency is finished, the VM is running and the system kernel version is outputted. This example showcases how to customize and run **skz-opack** for your specific needs.
Once the *opack* target dependency is finished, the VM is running and the system kernel version is outputted. This example showcases how to customize and run **skz-opack** for your specific needs.
### GCE example
TODO
Before using this example, you need to prepare the cloud environment:
1. [Create a GCP project](https://cloud.google.com/resource-manager/docs/creating-managing-projects) and declare its name by overriding `OPACK_GCE_PROJECT`.
2. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create) and [obtain its key](https://cloud.google.com/iam/docs/keys-create-delete), which path should be declared with `OPACK_GCE_JSON_KEY`.
3. [Create a GCS Bucket](https://cloud.google.com/storage/docs/creating-buckets) and declare its name with 'OPACK_GCE_BUCKET' (defaults to `OPACK_GCE_PROJECT`).
4. Enable the [Cloud Ressource Manager API](https://console.cloud.google.com/marketplace/product/google/cloudresourcemanager.googleapis.com) in order for the service account to use it with its key.
```make
OPACK_TARGET= testopack-cloud
OPACK_SYS_VERSION= snapshots
OPACK_SYS_DISK_SIZE= 10000
OPACK_GCE_PROJECT= testopack
OPACK_GCE_JSON_KEY= testopack-342b60ffed47.json
OPACK_AUTODISKLABEL_FILE= $(shell realpath ./autodisklabel)
include ../../src/opack.mk
all: opack-cloud
ssh -i $(OPACK_SYS_SSH_PRIVATE_KEY) \
-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
$(OPACK_SYS_USER)@$(shell cat opack-cloud) \
"sysctl -n kern.version; uptime;" 2> /dev/null
```
Here, a custom [disklabel](https://man.openbsd.org/disklabel#AUTOMATIC_DISK_ALLOCATION) path is used (see its [source](./examples/cloud/autodisklabel)).
Alongside the disk size which is increased to 10GB (corresponding to the GCP allocation for `e2-micro` instance), OpenBSD *-current* is used.
Once `opack-cloud` is finished, the VM is up and running in your GCP project. Upon calling `opack-clean`, the VM instance and its boot image will be destroyed.
## Troubleshooting
- To enable debug mode, you can declare 'OPACK_DEBUG=yes' either in your Makefile or at runtime with: 'OPACK_DEBUG=yes make opack'.
- Overrides should happen before the include of "opack.mk".
- Overrides should happen **before** the include of `opack.mk`.
## Architecture
@ -125,7 +155,7 @@ TODO
`OPACK_SYS_USER` can become *root* using the `doas` command.
The `autodisklabel` used for this project is flat, so partitions don't benefit from *nodev*, *nosuid* and *wxallowed* protections.
The default `autodisklabel` used for this project is flat, so partitions don't benefit from *nodev*, *nosuid* and *wxallowed* protections.
### Code structure

21
src/defines.mk
View File

@ -102,9 +102,14 @@ provider "google" {
project = "$${var.project}"
}
module "OPENBSD" {
source = "./module"
module "opack-module" {
source = "./opack-module"
gce_zone = "$${var.zone}"
servers = 1
}
output "ip" {
value = module.opack-module.nat_ip
}
endef
@ -136,5 +141,17 @@ resource "google_compute_instance" "$(OPACK_TARGET)" {
}
}
output "nat_ip" {
value = google_compute_instance.$(OPACK_TARGET)[*].network_interface[0].access_config[0].nat_ip
}
endef
define OPACK_TERRAFORM_IMPORT_CONTENT =
resource "google_compute_image" "$(OPACK_TARGET)" {
# name = "$(OPACK_TARGET)"
}
endef

55
src/opack.mk
View File

@ -2,7 +2,7 @@
.DEFAULT_GOAL := opack
# Enable debugging mode if OPACK_DEBUG is set to 'yes'
OPACK_DEBUG=yes
#OPACK_DEBUG=yes
ifndef OPACK_DEBUG
.SILENT:
@ -67,35 +67,31 @@ $(OPACK_PROVISION_FILE):
echo you need to write $@ && exit 1
id_ed25519:
echo "█ Generating $@"
echo "█ $(shell date +'%H:%M:%S') Generating $@"
ssh-keygen -q -t ed25519 -f $@ -P ""
$(eval OPACK_SYS_SSH_PUBLIC_KEY=$(shell cat id_ed25519.pub))
export OPACK_SYS_SSH_PUBLIC_KEY=$(shell cat id_ed25519.pub 2> /dev/null)
$(OPACK_BOX_FILE): | $(OPACK_PACKER_DIR)/opack.json $(OPACK_PACKER_DIR)/vagrant.key $(OPACK_PROVISION_FILE) $(OPACK_PACKER_HTTP_DIR)/install.conf $(OPACK_PACKER_HTTP_DIR)/autodisklabel
ifndef OPACK_DEBUG
$(call spinner,cd $(OPACK_PACKER_DIR) && (CHECKPOINT_DISABLE=1 PACKER_CACHE_DIR=.$(OPACK_CACHE_DIR) packer build -timestamp-ui opack.json > opack.log && rm -rf $(OPACK_PACKER_DIR)) || echo Error Autoinstalling: check $(OPACK_PACKER_DIR)/opack.log,█ Autoinstalling $(OPACK_BOX_TAG) with Packer)
$(call spinner,cd $(OPACK_PACKER_DIR) && (CHECKPOINT_DISABLE=1 PACKER_CACHE_DIR=.$(OPACK_CACHE_DIR) packer build -timestamp-ui opack.json > opack.log) || echo Error Autoinstalling: check $(OPACK_PACKER_DIR)/opack.log,█ $(shell date +'%H:%M:%S') Autoinstalling $(OPACK_BOX_TAG) with Packer)
else
cd $(OPACK_PACKER_DIR) && CHECKPOINT_DISABLE=1 PACKER_CACHE_DIR=$(OPACK_CACHE_DIR) packer build -timestamp-ui opack.json && rm -rf $(OPACK_PACKER_DIR)
cd $(OPACK_PACKER_DIR) && CHECKPOINT_DISABLE=1 PACKER_CACHE_DIR=$(OPACK_CACHE_DIR) packer build -timestamp-ui opack.json
endif
$(OPACK_META_FILE): $(OPACK_BOX_FILE)
printf '$(subst $(OPACK_BOX_FILE_SHA256),$(shell sha256sum $(OPACK_BOX_FILE) | awk '{print $$1}'),$(subst $(newline),\n,$(OPACK_METADATA_CONTENT)))' > $@
ifndef OPACK_DEBUG
$(call spinner,vagrant box add -f --name $(OPACK_BOX_TAG) $(OPACK_META_FILE) > /dev/null,█ Adding $(OPACK_BOX_FILE) to Vagrant)
$(call spinner,vagrant box add -f --name $(OPACK_BOX_TAG) $(OPACK_META_FILE) > /dev/null,█ $(shell date +'%H:%M:%S') Adding $(OPACK_BOX_FILE) to Vagrant)
else
vagrant box add -f --name $(OPACK_BOX_TAG) $(OPACK_META_FILE)
endif
$(OPACK_PACKER_DIR)/disk.raw.tar.gz: | $(OPACK_PACKER_DIR)/opack-cloud.json id_ed25519 $(OPACK_PROVISION_FILE) $(OPACK_PACKER_HTTP_DIR)/install-cloud.conf $(OPACK_PACKER_HTTP_DIR)/autodisklabel
# -curl -X POST -s -o /dev/null \
# --data-binary @$(OPACK_PACKER_DIR)/bucket.json \
# -H "Authorization: Bearer $(STORAGE_TOKEN)" \
# -H "Content-Type: application/json" \
# "https://storage.googleapis.com/storage/v1/b?project=$(OPACK_GCE_PROJECT)"
# -curl -X DELETE -s -o /dev/null \
# -H "Authorization: Bearer $(IMAGE_TOKEN)" \
# "https://compute.googleapis.com/compute/v1/projects/$(OPACK_GCE_PROJECT)/global/images/$(OPACK_TARGET)"
(cd $(OPACK_PACKER_DIR) && CHECKPOINT_DISABLE=1 PACKER_CACHE_DIR=$(OPACK_CACHE_DIR) packer build -timestamp-ui opack-cloud.json) && rm -rf $(OPACK_PACKER_DIR)
ifndef OPACK_DEBUG
$(call spinner,cd $(OPACK_PACKER_DIR) && (CHECKPOINT_DISABLE=1 PACKER_CACHE_DIR=$(OPACK_CACHE_DIR) packer build -timestamp-ui opack-cloud.json > opack.log),█ $(shell date +'%H:%M:%S') Autoinstalling $(OPACK_TARGET) and uploading GCE image with Packer)
else
(cd $(OPACK_PACKER_DIR) && CHECKPOINT_DISABLE=1 PACKER_CACHE_DIR=$(OPACK_CACHE_DIR) packer build -timestamp-ui opack-cloud.json)
endif
$(OPACK_TERRAFORM_DIR)/main.tf: $(OPACK_TERRAFORM_DIR)
printf '$(subst $(newline),\n,$(OPACK_TERRAFORM_CONTENT))' > $@
@ -103,34 +99,45 @@ $(OPACK_TERRAFORM_DIR)/main.tf: $(OPACK_TERRAFORM_DIR)
$(OPACK_TERRAFORM_MODULE_DIR)/main.tf: $(OPACK_TERRAFORM_MODULE_DIR)
printf '$(subst $(newline),\n,$(OPACK_TERRAFORM_MODULE_CONTENT))' > $@
$(OPACK_TERRAFORM_DIR)/terraform.tfstate: | $(OPACK_PACKER_DIR)/disk.raw.tar.gz $(OPACK_TERRAFORM_DIR)/main.tf $(OPACK_TERRAFORM_MODULE_DIR)/main.tf
cd $(OPACK_TERRAFORM_DIR) && terraform init && terraform apply -auto-approve -state=$@
$(OPACK_TERRAFORM_DIR)/terraform.tfstate: | $(OPACK_PACKER_DIR)/disk.raw.tar.gz $(OPACK_TERRAFORM_DIR)/main.tf $(OPACK_TERRAFORM_MODULE_DIR)/main.tf
ifndef OPACK_DEBUG
$(call spinner,cd $(OPACK_TERRAFORM_DIR) && terraform init >> opack.log && terraform apply -auto-approve >> opack.log && printf '$(subst $(newline),\n,$(OPACK_TERRAFORM_IMPORT_CONTENT))' >> $(OPACK_TERRAFORM_MODULE_DIR)/main.tf && terraform import module.opack-module.google_compute_image.$(OPACK_TARGET) projects/$(OPACK_GCE_PROJECT)/global/images/$(OPACK_TARGET) >> opack.log && printf '$(subst $(newline),\n,$(OPACK_TERRAFORM_MODULE_CONTENT))' > $(OPACK_TERRAFORM_MODULE_DIR)/main.tf,█ $(shell date +'%H:%M:%S') Deploying with Terraform)
else
cd $(OPACK_TERRAFORM_DIR) && terraform init && terraform apply -auto-approve && printf '$(subst $(newline),\n,$(OPACK_TERRAFORM_IMPORT_CONTENT))' >> $(OPACK_TERRAFORM_MODULE_DIR)/main.tf && terraform import module.opack-module.google_compute_image.$(OPACK_TARGET) projects/$(OPACK_GCE_PROJECT)/global/images/$(OPACK_TARGET) && printf '$(subst $(newline),\n,$(OPACK_TERRAFORM_MODULE_CONTENT))' > $(OPACK_TERRAFORM_MODULE_DIR)/main.tf
endif
$(OPACK_VAGRANT_FILE):
printf '$(subst $(newline),\n,$(OPACK_VAGRANT_CONTENT))' > $@
opack: $(OPACK_META_FILE) $(OPACK_VAGRANT_FILE)
ifndef OPACK_DEBUG
$(call spinner,vagrant up > /dev/null,█ Starting $(OPACK_HOSTNAME)-$(OPACK_TARGET) with Vagrant)
$(call spinner,vagrant up > /dev/null,█ $(shell date +'%H:%M:%S') Starting $(OPACK_HOSTNAME)-$(OPACK_TARGET) with Vagrant)
else
vagrant up
endif
echo "█ skz-opack complete, run 'vagrant ssh' to connect to the machine"
echo "█ $(shell date +'%H:%M:%S') skz-opack complete, run 'vagrant ssh' to connect to the machine"
opack-cloud: | $(OPACK_TERRAFORM_DIR)/terraform.tfstate
opack-cloud-ssh: opack-cloud
ssh -i $(OPACK_SYS_SSH_PRIVATE_KEY) $(OPACK_SYS_USER)@$(shell cat opack-cloud 2>/dev/null)
opack-cloud: | $(OPACK_TERRAFORM_DIR)/terraform.tfstate
cd $(OPACK_TERRAFORM_DIR) 2>/dev/null && terraform output -json ip 2>/dev/null | tr -d '"' | tr -d '[' | tr -d ']' > ../$@
while ! nc -z -w 10 $(shell cat opack-cloud) 22 2>/dev/null; do sleep 10; done
echo "█ $(shell date +'%H:%M:%S') skz-opack complete"; echo "▒ You can SSH into $(OPACK_TARGET) using $(OPACK_SYS_SSH_PRIVATE_KEY), or using 'make opack-cloud-ssh':";
opack-clean:
ifndef OPACK_DEBUG
-$(call spinner,vagrant destroy -f >/dev/null; cd opack_deploy_$(OPACK_SHORT_REV) 2>/dev/null && terraform destroy -auto-approve && cd .. && rm -rf opack_deploy_$(OPACK_SHORT_REV); rm -rf $(OPACK_VAGRANT_FILE) opack_installer_$(OPACK_SHORT_REV) *.log ssh-config .vagrant *.json,█ Cleaning up)
-$(call spinner,vagrant destroy -f >/dev/null; cd opack_deploy_$(OPACK_SHORT_REV) 2>/dev/null && terraform destroy -auto-approve >> opack.log && cd .. && rm -rf opack_deploy_$(OPACK_SHORT_REV); rm -rf $(OPACK_VAGRANT_FILE) opack_installer_$(OPACK_SHORT_REV) *.log ssh-config .vagrant opack-cloud,█ $(shell date +'%H:%M:%S') Cleaning up)
else
-vagrant destroy -f
-cd opack_deploy_$(OPACK_SHORT_REV) 2>/dev/null && terraform destroy -auto-approve && cd .. && rm -rf opack_deploy_$(OPACK_SHORT_REV)
-rm -rf $(OPACK_VAGRANT_FILE) opack_installer_$(OPACK_SHORT_REV) *.log ssh-config .vagrant
-cd opack_deploy_$(OPACK_SHORT_REV) 2>/dev/null && terraform destroy -auto-approve >> opack.log
-rm -rf opack_deploy_$(OPACK_SHORT_REV)
-rm -rf $(OPACK_VAGRANT_FILE) opack_installer_$(OPACK_SHORT_REV) *.log ssh-config .vagrant opack-cloud
endif
opack-cleancache:
ifndef OPACK_DEBUG
-$(call spinner,vagrant box remove -f --all $(OPACK_BOX_TAG) 2>/dev/null; rm -rf $(OPACK_CACHE_DIR),█ Cleaning up cache)
-$(call spinner,vagrant box remove -f --all $(OPACK_BOX_TAG) 2>/dev/null; rm -rf $(OPACK_CACHE_DIR),█ $(shell date +'%H:%M:%S') Cleaning up cache)
else
-vagrant box remove -f --all $(OPACK_BOX_TAG) 2>/dev/null
-rm -rf $(OPACK_CACHE_DIR)

7
src/options.mk
View File

@ -133,12 +133,7 @@ OPACK_SYS_SSH_PUBLIC_KEY=$(shell cat $(OPACK_SYS_SSH_PRIVATE_KEY).pub)
OPACK_TERRAFORM_DIR?=opack_deploy_$(OPACK_SHORT_REV)
# Terraform module dir
OPACK_TERRAFORM_MODULE_DIR?=$(OPACK_TERRAFORM_DIR)/module
OPACK_TERRAFORM_DNS_ZONE=sk4-nz-zone
OPACK_TERRAFORM_DNS_NAME=sk4.nz.
OPACK_TERRAFORM_MODULE_DIR?=$(OPACK_TERRAFORM_DIR)/opack-module
ifdef OPACK_DEBUG
$(info ░ BOX $(OPACK_BOX_FILE))

2
src/packerfile-gcp.mk
View File

@ -42,7 +42,7 @@ define OPACK_PACKER_CLOUD_CONTENT =
"echo permit nopass :wheel > /mnt/etc/doas.conf && ",
"echo inet autoconf > /mnt/etc/hostname.vio0 && ",
"echo PubkeyAcceptedAlgorithms +ssh-rsa,ssh-ed25519 >> /mnt/etc/ssh/sshd_config && ",
"echo 'set tty com0' >> /mnt/etc/boot.conf &&",
"echo set tty com0 >> /mnt/etc/boot.conf &&",
"reboot<enter>" ]
} ],
"provisioners": [