Builds: isolated builders (development environment)#311
Merged
Conversation
Small changes to use `readthedocs-builder` repository to isolate builds.
humitos
commented
Jul 14, 2026
humitos
marked this pull request as ready for review
July 14, 2026 08:56
humitos
changed the title
humitos
added a commit
to readthedocs/readthedocs.org
that referenced
this pull request
Jul 14, 2026
agjohnson
reviewed
Jul 14, 2026
agjohnson
approved these changes
Jul 15, 2026
Co-authored-by: Anthony <aj@ohess.org>
humitos
added a commit
to readthedocs/readthedocs.org
that referenced
this pull request
Jul 16, 2026
## Description This PR allows us to change the architecture of how we run builds in Read the Docs by using a feature flag (`USE_ISOLATED_BUILDER`) so we can test a handful of project in the new architecture without impacting real users. The architecture document describes in depth how is the implementation for this: [`architecture.md`](https://github.com/readthedocs/readthedocs-builder/blob/celery-on-ec2/docs/architecture.md). Please, read that document and let me know if you have any concern about it. This document is updated with the latest discussion we had: - keep Celery around - build a minimal Celery app to run inside the new minimal build AMI (with pre downloaded Ubuntu images) - that Celery app spins up Docker container and run the `readthedocs-builder/builder` script inside it - Celery worker attend only one task - the AWS instance is killed once the build finishes - keep a warm pool of builders ready to receive builds (builds start immediately) This approach gives us all the features we discussed in our call. ## Notes - ~~There is a [`./dev-run.sh`](https://github.com/readthedocs/readthedocs-builder/blob/celery-on-ec2/scripts/dev-run.sh) script in the repository that I used to develop the Python script that builds the docs inside a container. We are not going to use it in production. You can skip it completely since it just development focused.~~ - Inside the app, we use [`./entrypoint.sh`](https://github.com/readthedocs/readthedocs-builder/blob/celery-on-ec2/scripts/entrypoint.sh) which is the script that runs directly inside the Docker container and handles process signals (SIGTERM, SIGKILL, etc) - We create a new minimal AMI with pre-downloaded Ubuntu images + systemd script to launch the Celery worker - We use a new temporal ASG (`isolated-builders`) to try this new approach - I was able to test it by triggering the Celery task manually from a Django shell. The build passed. The infrastructure works 🎉 - The implementation for HTML upload done in #13011 is not included in `readthedocs-builder` yet. - I added a [`deployment.md`](https://github.com/readthedocs/readthedocs-builder/blob/celery-on-ec2/docs/deployment.md) document that explains the new deployment process for builds. ## Improvements / Features - We can easily have more ASG to match specific needs from customers (bigger AWS instances). The AMI is the same for all the ASGs. - We can have a specific ASG (instance type) for a particular customer without extra work. No need to redeploy these ASG every week. - We don't run any application code next to the build process. - Build instances are ephemeral. They are killed after the build finishes. - We can eventually give users `root` capabilities if we want. - Easy to test different combinations of build instance types. Build queue is a launch template (`user_data`) environment variable. - No need to rebuild AMI on every deploy. - We can deploy new code by merging `main` into `rel` on `readthedocs-builder` repository. The new code will be pulled down immediately on new builds. - No need to wait for the queue/ASG to drain while doing deploys. ## Rollout - Behind a feature flag. We can add a subset of projects to start testing this. - We need this PR to be merged and deployed using the current process. There is no too much changes in this PR, most of the code is isolated under `readthedocs/projects/tasks/isolated_builder.py` file. The rest are new model fields, new settings and path decision depending on the feature flag being enabled or not. - We can deploy this PR and perform live changes on projects with the feature flag by updating `rel` branch on `readthedocs-builds` for quick iterations. ## Decisions / ToDo - The test suite we currently have is not migrated to the new repository. We will need to think and decide what to do there. Since the builder can be ran locally now, we would be able to test more deeply if we want to. - We need to work on the TF code for the infrastructure. - What are the next steps here? --- ## How to review this PR 1. Consider the big picture as a whole (main work at the [`architecture.md`](https://github.com/readthedocs/readthedocs-builder/blob/celery-on-ec2/docs/architecture.md) file) 2. Take a look at the code from this PR finding for something that could break production. Those are important to fix at this point. 3. Review the [`architecture.md`](https://github.com/readthedocs/readthedocs-builder/blob/celery-on-ec2/docs/architecture.md) file and try to load all that information in your mind 😄 4. Look for inconsistencies in that document and potential breaks in data flows and similar. 5. **Don't look at all** at the code from `readthedocs-builder/builder`. I haven't review it deeply either and I'm not worried about it at this point. I will get back to that later, once we have decided to move forward. The code is tested in production and _it works_, tho: it runs a build and connect with the app properly. ## Testing this locally 1. Pull down this PR (`humitos/celery-on-ec2` branch) 2. Clone `readthedocs-builder` next to the other readthedocs repositories you have 3. Pull down branch `humitos/celery-on-ec2` on `readthedocs.org/common` (readthedocs/common#311) 4. Start local development and add `USE_ISOLATED_BUILDER` to a project. 5. Trigger a build for that project. 7. Check the logs for that app as usual. 8. Check the logs for the Docker container with `docker logs -f build-<pk>` ---- * Original discussion: readthedocs/meta#135 * Issue: readthedocs/meta#210 Requires: * readthedocs/common#311 * https://github.com/readthedocs/readthedocs-builder/tree/celery-on-ec2 Initial idea using Fargate/ECS+EC2 (discarded): * #13104 --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Required by: readthedocs/readthedocs.org#13133