Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/validate-examples.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,12 @@ jobs:
# Install yamllint
pip install yamllint

- name: Test ingress gateway targets
run: |
pip install requests
python3 custom-domain/dstack-ingress/scripts/tests/test_gateway_target.py
python3 custom-domain/dstack-ingress/scripts/tests/test_dnsguide.py

- name: Run all checks
run: ./dev.sh check-all

Expand Down
8 changes: 4 additions & 4 deletions custom-domain/dstack-ingress/DNS_PROVIDERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ This guide explains how to configure dstack-ingress to work with different DNS p
### Common Variables (Required for all providers)

- `DOMAIN` - Your custom domain (e.g., `app.example.com`)
- `GATEWAY_DOMAIN` - dstack gateway domain (e.g., `_.dstack-prod5.phala.network`)
- `GATEWAY_DOMAIN` - dstack gateway domain (e.g., `gateway.dstack-prod5.phala.network`)
- `CERTBOT_EMAIL` - Email for Let's Encrypt registration
- `TARGET_ENDPOINT` - Backend application endpoint to proxy to
- `DNS_PROVIDER` - DNS provider to use (`cloudflare`, `linode`, `namecheap`)
Expand Down Expand Up @@ -124,7 +124,7 @@ services:
# Common configuration
- DNS_PROVIDER=linode
- DOMAIN=app.example.com
- GATEWAY_DOMAIN=_.dstack-prod5.phala.network
- GATEWAY_DOMAIN=gateway.dstack-prod5.phala.network
- [email protected]
- TARGET_ENDPOINT=http://backend:8080

Expand All @@ -149,7 +149,7 @@ services:
# Common configuration
- DNS_PROVIDER=namecheap
- DOMAIN=app.example.com
- GATEWAY_DOMAIN=_.dstack-prod5.phala.network
- GATEWAY_DOMAIN=gateway.dstack-prod5.phala.network
- [email protected]
- TARGET_ENDPOINT=http://backend:8080

Expand Down Expand Up @@ -177,7 +177,7 @@ services:
environment:
DNS_PROVIDER: route53
DOMAIN: app.example.com
GATEWAY_DOMAIN: _.${DSTACK_GATEWAY_DOMAIN}
GATEWAY_DOMAIN: gateway.${DSTACK_GATEWAY_DOMAIN}

AWS_REGION: ${AWS_REGION}
AWS_ROLE_ARN: ${AWS_ROLE_ARN}
Expand Down
67 changes: 62 additions & 5 deletions custom-domain/dstack-ingress/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ services:
environment:
- CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN}
- DOMAIN=*.myapp.com
- GATEWAY_DOMAIN=_.dstack-prod5.phala.network
- GATEWAY_DOMAIN=gateway.dstack-prod5.phala.network
- CERTBOT_EMAIL=${CERTBOT_EMAIL}
- SET_CAA=true
- TARGET_ENDPOINT=http://app:80
Expand Down Expand Up @@ -109,7 +109,7 @@ services:
DNS_PROVIDER: cloudflare
CLOUDFLARE_API_TOKEN: ${CLOUDFLARE_API_TOKEN}
CERTBOT_EMAIL: ${CERTBOT_EMAIL}
GATEWAY_DOMAIN: _.dstack-prod5.phala.network
GATEWAY_DOMAIN: gateway.dstack-prod5.phala.network
SET_CAA: true
DOMAINS: |
app.example.com
Expand Down Expand Up @@ -158,7 +158,7 @@ environment:
|----------|-------------|
| `DOMAIN` | Your domain (single-domain mode). Supports wildcards (`*.example.com`) |
| `TARGET_ENDPOINT` | Backend address, e.g. `app:80` or `http://app:80` |
| `GATEWAY_DOMAIN` | dstack gateway domain (e.g. `_.dstack-prod5.phala.network`) |
| `GATEWAY_DOMAIN` | Full gateway CNAME target (e.g. `gateway.dstack-prod5.phala.network`); see migration notes below |
| `ACME_EMAIL` | *(optional)* ACME contact address, in either mode. `CERTBOT_EMAIL` is the historical name and still works. See below — it is optional, and published |
| `DNS_PROVIDER` | DNS provider (`cloudflare`, `linode`, `namecheap`) |

Expand Down Expand Up @@ -330,7 +330,7 @@ services:
- TARGET_ENDPOINT=http://app:80
# Printed as the CNAME target, and used to verify that the hostname
# really resolves to the gateway before issuance starts.
- GATEWAY_DOMAIN=_.dstack-prod5.phala.network
- GATEWAY_DOMAIN=gateway.dstack-prod5.phala.network
# - [email protected] # optional, and published (see below)
# - DNS_SETUP_MODE=wait # default; blocks until the records exist
ports:
Expand All @@ -349,7 +349,7 @@ public DNS until they are visible:
DNS records required for app.example.com
==========================================================================
CNAME app.example.com
-> _.dstack-prod5.phala.network
-> gateway.dstack-prod5.phala.network
TXT _dstack-app-address.app.example.com
-> b1ea785543bbbb19ce9de33744321360992bf63b:443
CAA app.example.com
Expand Down Expand Up @@ -381,6 +381,63 @@ Two consequences:
means updating DNS. `DNS_SETUP_MODE=webhook` exists so this can be automated;
doing it by hand means downtime on every redeploy.

## Gateway CNAME target migration

Use `gateway.<gateway-base-domain>` as the traffic CNAME target. The standalone
underscore label in the legacy `_.<gateway-base-domain>` target is rejected by
hostname-validating resolvers such as Android's DnsResolver (`res_hnok` validates
CNAME targets during host resolution).

### RFC rationale

- [RFC 1034 §3.5](https://www.rfc-editor.org/rfc/rfc1034.html#section-3.5)
gives the preferred hostname syntax: ASCII letters, digits, and hyphens,
starting with a letter and ending with a letter or digit.
[RFC 1123 §2.1](https://www.rfc-editor.org/rfc/rfc1123.html#section-2.1)
also permits a leading digit; underscores remain outside this hostname syntax.
- [RFC 2181 §11](https://www.rfc-editor.org/rfc/rfc2181.html#section-11)
permits arbitrary DNS labels within length limits, including underscores and
CNAME values, and allows applications to impose their own restrictions.
Thus `_.<gateway-base-domain>` is representable in DNS, while the platform's
server target uses hostname-compatible `gateway.<gateway-base-domain>` for
client interoperability. Protocol verification labels such as
`_acme-challenge` and `_dstack-app-address` remain valid for their DNS uses.
- [RFC 2181 §10.1](https://www.rfc-editor.org/rfc/rfc2181.html#section-10.1)
permits only one canonical target per alias. During migration, atomically
replace each traffic alias's CNAME target, keeping exactly one target at that
owner. Retain old and new gateway address records under their separate names
until existing consumers have migrated and their TTLs have elapsed.

### Rollout

`GATEWAY_DOMAIN` is a **complete hostname** and is passed through unchanged to
DNS provider writes, manual DNS instructions, and webhook records. The Compose
examples and E2E default use the new hostname; an existing deployment keeps its
configured target until its environment is updated. The generic
`docker-compose.yaml` reads `GATEWAY_DOMAIN` from your environment. For templates
using `DSTACK_GATEWAY_DOMAIN`, that variable contains only the gateway base domain.

Roll out in this order:

1. The gateway zone operator publishes `gateway.<gateway-base-domain>` pointing
to the same gateway addresses and verifies public resolution. Use A/AAAA
records or a hostname-valid CNAME chain; a CNAME back to the legacy underscore
hostname retains the resolver problem. Check for an existing `gateway` record
and confirm it serves the intended ingress addresses before reusing it.
2. Update `GATEWAY_DOMAIN` in each deployment. DNS-01 mode reconciles the traffic
alias on its next pass, including the alias inside `DELEGATION_ZONE`.
TLS-ALPN-01 mode prints or sends the new target for the operator to publish.
Providers that flatten aliases resolve the new target to address records.
3. Verify the full traffic CNAME chain and application access, including an
Android client, after DNS caches expire. The guide also accepts matching
addresses, so a successful guide check alone does not prove that the old
CNAME chain has been replaced.

Keep the legacy gateway record available for existing clients during migration;
retire it separately after all consumers have moved and their TTLs have elapsed.
Keep `_acme-challenge`, `_dstack-app-address`, and other protocol TXT/delegation
labels unchanged. Updating the examples performs no gateway-zone DNS migration.

## The ACME contact address is optional, and public

`ACME_EMAIL` (or `CERTBOT_EMAIL`) may be left unset in **either** mode. RFC 8555
Expand Down
1 change: 1 addition & 0 deletions custom-domain/dstack-ingress/TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,7 @@ this before suspecting the proxy.

```bash
python3 scripts/tests/test_dnsguide.py # DNS/CAA parsing and record building
python3 scripts/tests/test_gateway_target.py # shipped targets, overrides, protocol labels
bash scripts/tests/test_sanitizers.sh # env var validation
```

Expand Down
2 changes: 1 addition & 1 deletion custom-domain/dstack-ingress/docker-compose.multi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
DNS_PROVIDER: cloudflare
CLOUDFLARE_API_TOKEN: ${CLOUDFLARE_API_TOKEN}
CERTBOT_EMAIL: ${CERTBOT_EMAIL}
GATEWAY_DOMAIN: _.dstack-prod5.phala.network
GATEWAY_DOMAIN: gateway.dstack-prod5.phala.network
SET_CAA: true
DOMAINS: |
app.example.com
Expand Down Expand Up @@ -36,5 +36,5 @@
restart: unless-stopped

volumes:
letsencrypt:

Check failure on line 39 in custom-domain/dstack-ingress/docker-compose.multi.yaml

View workflow job for this annotation

GitHub Actions / Basic Checks (dev.sh)

39:15 [empty-values] empty value in block mapping

Check failure on line 39 in custom-domain/dstack-ingress/docker-compose.multi.yaml

View workflow job for this annotation

GitHub Actions / check-all

39:15 [empty-values] empty value in block mapping
evidences:

Check failure on line 40 in custom-domain/dstack-ingress/docker-compose.multi.yaml

View workflow job for this annotation

GitHub Actions / Basic Checks (dev.sh)

40:13 [empty-values] empty value in block mapping

Check failure on line 40 in custom-domain/dstack-ingress/docker-compose.multi.yaml

View workflow job for this annotation

GitHub Actions / check-all

40:13 [empty-values] empty value in block mapping
4 changes: 2 additions & 2 deletions custom-domain/dstack-ingress/scripts/tests/e2e-test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
# CERTBOT_EMAIL - Email for Let's Encrypt registration
#
# Optional env vars:
# GATEWAY_DOMAIN - dstack gateway domain (default: _.dstack-prod5.phala.network)
# GATEWAY_DOMAIN - dstack gateway domain (default: gateway.dstack-prod5.phala.network)
# IMAGE - dstack-ingress image (default: dstacktee/dstack-ingress:latest)
# INSTANCE_TYPE - CVM instance type (default: tdx.small)
# CERTBOT_STAGING - Use LE staging (default: true)
Expand All @@ -27,7 +27,7 @@ set -uo pipefail
: "${DOMAIN:?DOMAIN is required}"
: "${CLOUDFLARE_API_TOKEN:?CLOUDFLARE_API_TOKEN is required}"
: "${CERTBOT_EMAIL:?CERTBOT_EMAIL is required}"
GATEWAY_DOMAIN="${GATEWAY_DOMAIN:-_.dstack-prod5.phala.network}"
GATEWAY_DOMAIN="${GATEWAY_DOMAIN:-gateway.dstack-prod5.phala.network}"
IMAGE="${IMAGE:-dstacktee/dstack-ingress:latest}"
INSTANCE_TYPE="${INSTANCE_TYPE:-tdx.small}"
CERTBOT_STAGING="${CERTBOT_STAGING:-true}"
Expand Down
84 changes: 84 additions & 0 deletions custom-domain/dstack-ingress/scripts/tests/test_gateway_target.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
#!/usr/bin/env python3
"""Regression tests for gateway targets supplied by the shipped examples.

Run: python3 scripts/tests/test_gateway_target.py
"""

import argparse
import os
from pathlib import Path
import re
import subprocess
import sys
import unittest

SCRIPTS = Path(__file__).resolve().parents[1]
INGRESS = SCRIPTS.parent
REPO = INGRESS.parents[1]
sys.path.insert(0, str(SCRIPTS))

import dnsguide # noqa: E402


class TestGatewayTarget(unittest.TestCase):
def test_multi_domain_compose_target(self):
compose = (INGRESS / "docker-compose.multi.yaml").read_text()
match = re.search(r"GATEWAY_DOMAIN: (\S+)", compose)
assert match is not None, "multi-domain compose must configure a gateway"
target = match.group(1)
self.assertEqual(target, "gateway.dstack-prod5.phala.network")
self.assert_guide_target(target)

def test_k3s_interpolated_target(self):
compose = (REPO / "k3s/docker-compose.yaml").read_text()
match = re.search(r"GATEWAY_DOMAIN=(\S+)", compose)
assert match is not None, "k3s compose must configure a gateway"
target = match.group(1)
target = target.replace("${DSTACK_GATEWAY_DOMAIN}", "cluster.example.net")
self.assertEqual(target, "gateway.cluster.example.net")
self.assert_guide_target(target)

def test_e2e_default_and_explicit_override(self):
script = (SCRIPTS / "tests/e2e-test.sh").read_text()
match = re.search(r"^GATEWAY_DOMAIN=.*$", script, re.MULTILINE)
assert match is not None, "e2e script must configure a gateway"
assignment = match.group(0)
for override, expected in (
(None, "gateway.dstack-prod5.phala.network"),
("edge.operator.example", "edge.operator.example"),
("_.legacy.example", "_.legacy.example"),
):
with self.subTest(override=override):
env = os.environ.copy()
env.pop("GATEWAY_DOMAIN", None)
if override is not None:
env["GATEWAY_DOMAIN"] = override
target = subprocess.check_output(
["bash", "-c", assignment + '\nprintf "%s" "$GATEWAY_DOMAIN"'],
env=env, text=True,
)
self.assertEqual(target, expected)
self.assert_guide_target(target)

def assert_guide_target(self, target):
for challenge in ("dns-01", "tls-alpn-01"):
with self.subTest(challenge=challenge, target=target):
records = dnsguide.build_records(argparse.Namespace(
domain="app.example.com", alias_target=target,
txt_name="_dstack-app-address.app.example.com",
txt_value="deadbeef:443", caa_name="app.example.com",
caa_tag="issue", caa_value="letsencrypt.org",
account_uri="", challenge=challenge,
delegation_zone="deleg.example.net",
include="cname,txt,caa,challenge-cname",
))
self.assertEqual(records[0].value, target)
self.assertEqual(records[1].name, "_dstack-app-address.app.example.com")
self.assertEqual(records[2].name, "_acme-challenge.app.example.com")
self.assertEqual(records[2].value,
"_acme-challenge.app.example.com.deleg.example.net")
self.assertEqual(records[3].value, '0 issue "letsencrypt.org"')


if __name__ == "__main__":
unittest.main(verbosity=2)
2 changes: 1 addition & 1 deletion k3s/docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
- DNS_PROVIDER=cloudflare
- CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN}
- DOMAIN=*.${CLUSTER_DOMAIN}
- GATEWAY_DOMAIN=_.${DSTACK_GATEWAY_DOMAIN}
- GATEWAY_DOMAIN=gateway.${DSTACK_GATEWAY_DOMAIN}
- CERTBOT_EMAIL=${CERTBOT_EMAIL}
- SET_CAA=true
- TARGET_ENDPOINT=k3s:80
Expand All @@ -57,8 +57,8 @@
restart: unless-stopped

volumes:
k3s-data:

Check failure on line 60 in k3s/docker-compose.yaml

View workflow job for this annotation

GitHub Actions / Basic Checks (dev.sh)

60:12 [empty-values] empty value in block mapping

Check failure on line 60 in k3s/docker-compose.yaml

View workflow job for this annotation

GitHub Actions / check-all

60:12 [empty-values] empty value in block mapping
k3s-kubelet:

Check failure on line 61 in k3s/docker-compose.yaml

View workflow job for this annotation

GitHub Actions / Basic Checks (dev.sh)

61:15 [empty-values] empty value in block mapping

Check failure on line 61 in k3s/docker-compose.yaml

View workflow job for this annotation

GitHub Actions / check-all

61:15 [empty-values] empty value in block mapping
k3s-etc:
k3s-log:
cert-data:
Loading