DABYTE DATA DESK · operations

Atomic deploys delete the files that prove you own your site

Answer. An atomic deploy is a staging directory plus one rename, so the document root after the swap contains exactly what the build produced and nothing else. Every file that has to live on the site but is not an output of the generator disappears at that moment: the IndexNow key file, BingSiteAuth.xml, /.well-known/mcp-registry-auth, the Brave Creators verification file. No step fails, no log line is written, and the deploy reports success, because from the deploy's point of view nothing went wrong. The fix has two parts: a persist step that puts those files back onto the staging tree after the build and before the rename, applied by every path that replaces the tree rather than only the one you remember, and a post-deploy probe that reads each ownership path by hand instead of trusting the build.

Published

Who reported it, and when we saw it

15 sources in our monitored set. First seen by us at 2026-08-09 14:59 UTC; our polling resolution is one hour, so this is when we observed it, not when it was published.
SourceLink
https://modelcontextprotocol.io/registry/authentication
https://www.indexnow.org/documentation
https://www.indexnow.org/faq
https://github.com/brave/bat-community-website/blob/master/.well-known
https://support.brave.app/hc/en-us/articles/360021408352-How-do-I-regi
https://www.bing.com/webmasters/help/add-and-verify-site-12184f8b
https://blogs.bing.com/webmaster/september-2019/Import-sites-from-Sear
https://blogs.bing.com/webmaster/august-2019/Bing-Webmaster-Tools-simp
https://jekyllrb.com/docs/configuration/options/
https://capistranorb.com/documentation/getting-started/structure/
https://gohugo.io/commands/hugo/
https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
https://docs.netlify.com/deploy/deploy-overview/
https://www.netlify.com/blog/2021/02/23/terminology-explained-atomic-a
https://docs.netlify.com/extend/develop-and-share/develop-build-plugin

Key facts

What a whole-tree swap actually removes

An atomic deploy is two moves. You copy the build into a staging directory next to the live one, you rename the live directory out of the way, and you rename the staging directory into its place. Visitors never see a half-written tree, and pages that were deleted in the last release do not linger as orphans. The property that makes this safe is the same property that makes it destructive: after the rename, the document root holds exactly the build output, and nothing else survives by accident. Four kinds of file routinely live on a site without ever being produced by the generator. IndexNow expects a plain text file at the site root named after the key itself, containing the key and nothing more; the documentation says 8 to 128 characters and, in the same sentence, both hexadecimal and the wider set a-z, A-Z, 0-9 and dash, so read it as the wider set (indexnow.org, checked on 2026-08-09). Bing Webmaster Tools accepts BingSiteAuth.xml in the root, an msvalidate.01 meta tag, or a CNAME record; it also auto-verifies properties imported from Google Search Console (blogs.bing.com, September 2019) and offers Domain Connect where the DNS provider participates (blogs.bing.com, August 2019). The MCP registry's HTTP domain check reads /.well-known/mcp-registry-auth, a single line of the form v=MCPv1; k=ed25519; p=<base64 public key>, or k=ecdsap384 for an ECDSA P-384 pair; the registry is documented as being in preview, with breaking changes possible before general availability (modelcontextprotocol.io, checked on 2026-08-09). Brave Creators accepts a file at /.well-known/brave-rewards-verification.txt carrying the site domain and a token on labelled Domain: and Token: lines, or the same token as a DNS TXT record of the form brave-ledger-verification=<token> (checked on 2026-08-09). Every one of those was placed once, by hand or by a vendor wizard, and none of them is known to your generator. The generator is not at fault and cannot be: it emits what its templates and data say to emit. The deploy is not at fault either. The gap is architectural, and it opens on the first re-deploy after the file is placed.

Why the failure is quiet

Nothing errors because nothing failed. The build succeeded, the rename succeeded, the site is up, and the pages are the pages you expected. A missing verification file produces a 404 on one path that no human visits and no internal check requests. The consumers of these files read them on their own schedule, not yours, which is what stretches the interval between cause and symptom. The MCP registry reads mcp-registry-auth when you authenticate to publish or update an entry, which for a stable server can be months apart. Bing does not notify you when a verification file disappears, and the property keeps appearing in your Webmaster Tools site list either way, because being listed is not the same as being verified; we have not found documentation of how often Bing re-checks, and this is our own observation on our own properties as of 2026-08-09. IndexNow reads the key file at submission time and answers 403 when the key is not valid, which the documentation defines as key not found, or file found without the key inside it (indexnow.org, checked on 2026-08-09). That is informative, but only if you are submitting, and only if your client reads the response instead of firing and forgetting. By the time any of that surfaces, the deploy that caused it is dozens of deploys back, and the obvious hypotheses are all wrong ones: a vendor change, a CDN rule, an expired token. The correct hypothesis, that a routine publish deleted the file, is the one that leaves no trace to find.

Our own version of it: three writers, one overlay, and a fourth copy

Our handoff records /.well-known/mcp-registry-auth returning 404 on both dabyte.ai and dablock.ai on 2026-08-06, at a point where the deployment plan already listed the file as restored. That was the second time the same class of defect hit us. The first was the IndexNow key, which we closed by making the key an output of the build (backend/sitegen/build_site.py:891-904). The registry proof could not be closed the same way, because a signing proof is not site content and does not belong in the repository, so it went into a persist directory instead, and the persist step was added to the deploy script. The problem was that the deploy script is not the only thing that replaces the tree. Three scripts do: backend/sitegen/deploy_site.sh, which we run by hand after edits; backend/ops/run_monthly.sh, the weekly measurement run; and backend/ops/mh_ingest.sh, an hourly news ingest that rebuilds and republishes whenever feed cells change. Only the first had the overlay. The hourly job did a plain copy-and-rename, so a restored file survived at most an hour, every time. We noticed a day later, and only because the dashboard probes 18 machine surfaces per site on a schedule (backend/ops/dashboard.py:48-52), one of them being the registry path, and one cell turned red. Nothing in the deploy pipeline itself would ever have said a word. The step now lives in one function, apply_persist() in backend/ops/lib_persist.sh, sourced by all three scripts. One copy did survive the deduplication, and it is worth naming rather than glossing: deploy_site.sh sources the library conditionally at line 81, and the remote-ssh branch at line 96 inlines its own cp -a of the persist directory, because the function cannot be sourced on the machine that runs the ssh. So the file whose duplication supplies the moral still contains a hand-maintained copy of the idea, and that copy is now the one to watch. The lesson is duller than the file loss: several hand-maintained copies of one step always diverge, and the copy that diverges is the one that runs most often, because it is the one nobody reviews after the first week.

The persist directory, and its order of operations

The mechanism is small. Keep a directory outside the document root, ours is /opt/media_hub/persist/<site>/, whose internal structure mirrors the site root. Put in it everything that must be served but is not born of the build: ownership proofs for search engines, registry authentication files, key files, wizard-issued verification tokens. Then copy that directory over the staging tree, after the build output has been copied and before the rename that makes it live. Applying it to the staging tree rather than to the live tree matters: the live root is then never, for even one instant, a tree without the proofs in it. The idea is old and borrowed. Capistrano's release layout keeps a shared directory whose linked_files and linked_dirs are symlinked into every release precisely so that state survives deploys. Jekyll cleans its destination on every build by default, and the documentation says plainly that files or folders not created by your site will be removed, with keep_files as the escape hatch. Hugo ships --cleanDestinationDir, described as removing files from the destination that are not found in the static directories. aws s3 sync --delete removes objects in the destination that are absent from the source. Different tools, one hazard, and in each case the vendor has already written down the way out. On hosts with immutable deploys the persist directory has no equivalent, because nothing on the server persists between deploys by design, but the same job moves into the build. Netlify build plugins expose an onPostBuild hook, documented as running after the build command and after Functions bundling, and before the deploy stage, with constants.PUBLISH_DIR pointing at the deploy-ready directory (docs.netlify.com, checked on 2026-08-09). A plugin that writes the verification file into PUBLISH_DIR on every build does exactly what our overlay does, without the file living in the repository. The rule that survives both models is the same: the file must be produced by something that runs every time, or verified somewhere a deploy cannot reach.

Where DNS exists, use DNS

A DNS record lives outside the document root, which means no deploy can reach it. Every vendor discussed here except IndexNow offers a DNS path alongside the file path. Bing Webmaster Tools accepts a CNAME record. The MCP registry supports DNS authentication with a TXT record carrying the same v=MCPv1; k=...; p=<public key> string as the well-known file, using mcp-publisher login dns instead of login http. Brave Creators accepts a TXT record containing brave-ledger-verification=<token>; you can see one in the wild with dig +short TXT basicattentiontoken.org (all checked on 2026-08-09). The rule that follows is mechanical. If you have access to the zone, verify by DNS and never create the file at all. If you do not have access to the zone, use the file, but treat it as an artifact with an owner: either compile it into the build so the generator reproduces it every time, or put it in the persist directory, or write it from a build hook. Placing it by hand into the live document root and moving on is the failure, and it is a failure regardless of which vendor issued it. IndexNow is the exception with no DNS form, so the key file has to exist. Compile it into the build. Note also that the key file is per host: the IndexNow FAQ states that each subdomain is treated as a separate host and that you must create and manage individual key files for each one (indexnow.org/faq, checked on 2026-08-09). Two things beyond that are ours rather than documented. On our hosts a 301 from apex to www meant the key belonged to www and the apex answered 403 (observed 2026-08-06). And being verified is not always the same as being accepted: dablock.ai was added to Bing Webmaster Tools by Google Search Console import, and IndexNow submissions for that host answered 403 with UserForbiddedToAccessSite on 2026-08-05 and 2026-08-06, while dabyte.ai accepted the same key with the key file served the same way. The documented meaning of 403 is an invalid key, which is not what was wrong. Our note records the fix as real verification in Bing Webmaster Tools plus a fresh key for that host; we have not found this response documented anywhere.

The second failure mode of the same swap

Replacing the whole tree is destructive in a second way that has nothing to do with hand-placed files. If the build itself comes up short, because a data source is unreachable, credentials are missing, or a section renders empty, the generator does not crash. It legitimately produces a smaller site, and the swap faithfully publishes it. Indexed URLs become 404s, along with whatever inbound links they had accumulated. Our news layer reads from a database, and without credentials the build silently drops the whole /topics/ section. Two guards cover it, and one of them is weaker than it looks. deploy_site.sh:24-30 refuses to build when it can find neither /opt/media_hub/.env nor SUPABASE_URL in the environment. It does not catch the case that matters most on the server: if the .env file exists but has lost the variable, the first branch wins, the file is sourced, nothing is checked, and the build proceeds. That gap is open in our repository as of 2026-08-09. The second guard is the one that actually holds: all three publishing paths compare page counts before the rename and abort when the fresh build has fewer than 70 percent of the live HTML files, at deploy_site.sh:58, run_monthly.sh:73 and mh_ingest.sh:40. The 0.7 threshold is a judgment call, not a measurement. It encodes the assumption that a build 30 percent smaller than the live site is a source outage rather than an editorial decision. Pick your own number, but pick one, and put it in front of the rename rather than in a report generated afterwards.

Checking after deploy, and what a check still misses

The probe is a loop over the paths that must exist, run after every deploy and failing loudly on the first bad response. Ours runs at origin with an explicit Host header so it also catches vhost misrouting: for p in / /robots.txt /sitemap.xml /llms.txt /index.md /.well-known/mcp.json; do curl -s -o /dev/null -w '%{http_code}' -H 'Host: example.com' http://127.0.0.1$p; done. Run the same list from outside as well, because a CDN cache rule or a bot challenge can turn a healthy origin into a 403 at the edge for exactly these unusual paths. As a current-state check on our own hosts, curl -s -o /dev/null -w '%{http_code} %{content_type}' https://dabyte.ai/.well-known/mcp-registry-auth returned 200 application/json on 2026-08-09, and so did dablock.ai. Our own version of this check is still incomplete, and it is worth stating plainly. deploy_site.sh:146-158 walks 15 paths and exits non-zero on the first non-200, and not one of the 15 is a persist file. The guard that exists does not cover the class of file that gets deleted. The dashboard's list is longer, 18 surfaces at dashboard.py:48-52, and it includes the registry path, which is the only reason anyone noticed. Status codes alone are also not the check. On 2026-08-09 both of our domains return the registry proof with Content-Type: application/json, because the nginx template sets default_type application/json for the entire /.well-known/ location (nginx_vhost.template:106-108). The body is correct and the label is wrong; the registry documentation specifies the path and the body format but not a content type, and we have not tested whether mcp-publisher rejects it, so we are not going to claim it breaks anything. A probe that reads only the number reports that as green. Read the first line of the body and the content type, compare them to what the vendor documented, and write the result down with the date, because a month later there is otherwise no way to say what changed.

Questions this answers

Why not just commit the verification files to the repository so the build emits them?

For some of them that is the right answer, and it is what we did with the IndexNow key: it became a build output at build_site.py:891-904 and stopped disappearing. It does not generalize. A registry authentication file is a domain-ownership proof tied to a private key, and a Brave verification token is account state; neither is site content, and putting them in a repository puts them wherever that repository goes. The persist directory exists for exactly the files you do not want in git but do want on the server.

Does this apply to hosted platforms, or only to self-managed servers?

It applies to hosted platforms too, and the fix changes shape rather than disappearing. Netlify describes deploys as atomic and immutable artifacts and says you create a new deploy instead of pushing individual files, so whether a file is on the site is decided by the deploy, not by anything you left on a server. The persist directory has no equivalent there, but the build hook does the same job: a build plugin's onPostBuild runs after the build and before the deploy stage and can write into constants.PUBLISH_DIR, so the verification file is regenerated on every deploy (docs.netlify.com, checked on 2026-08-09). Verifying by DNS instead removes the question entirely.

How long can a deleted proof go unnoticed?

As long as the interval between checks by whoever reads it. Registry authentication is read when you publish or update an entry. Bing keeps showing the property in your site list whether or not the file is still there, and sends no notification. An IndexNow key is read at submission time and answers 403 if it cannot be found. None of those produce an alert aimed at you. In our case the interval between the deletion and the observation was about a day, and only because an unrelated dashboard happened to probe that path on a schedule.

What is the minimum viable version of this if I do not want to restructure my deploy?

Two things. Move every verification you can into DNS, which removes those files from the blast radius entirely. Then add the remaining paths to whatever already runs after your deploy: a curl loop that exits non-zero on the first response that is not 200 is a few lines and runs in under a second. That will not prevent the deletion, but it converts a silent loss into a failed deploy, which is the whole difference in practice.

Machine access