Skip to content

Containerised distribution of Guardian main branch - #11

Draft
tonytw1 wants to merge 58 commits into
mainfrom
containerised
Draft

Containerised distribution of Guardian main branch#11
tonytw1 wants to merge 58 commits into
mainfrom
containerised

Conversation

@tonytw1

@tonytw1 tonytw1 commented Jul 29, 2024 •
edited
Loading

Copy link
Copy Markdown
Member

What does this change?

Produces container images for a generalised Grid install based on the Guardian's main branch.

Reasonably opinionated towards "we use AWS services but we don't necessarily run the application on AWS".

  • Disables AWS CloudWatch and Kineses metrics to simplify the AWS dependencies.
  • Switches to ENV variable access tokens for AWS access
  • Alters the service URL scheme to present the Grid microservices under a single hostname (which makes local development and deployment behind a single load balancer simpler)
  • Removes the InnerServiceStatusCheckController as this is less relevant if deployed with a container orchastator.
  • Simplifies Reaper configuration to not need an additional S3 bucket.
  • Removes Guardian specific Composer usage stream consumer and Guardian Content API integration. The Usage API remains intact.
  • Migrates to imgproxy for image previews. Does not build imgops.

How should a reviewer test this change?

How can success be measured?

Who should look at this?

Tested? Documented?

  • locally by committer
  • locally by Guardian reviewer
  • on the Guardian's TEST environment
  • relevant documentation added or amended (if needed)

@tonytw1
tonytw1 force-pushed the containerised branch 2 times, most recently from b0942e9 to 887a0ca Compare August 6, 2024 17:35
@tonytw1
tonytw1 force-pushed the containerised branch 2 times, most recently from 5b3abf6 to 5df2f75 Compare August 30, 2024 19:54
@tonytw1
tonytw1 force-pushed the containerised branch 6 times, most recently from db06496 to 8f64bac Compare September 8, 2024 21:01
@tonytw1
tonytw1 force-pushed the containerised branch 7 times, most recently from e8ad009 to 3ce864b Compare September 25, 2024 11:37
@tonytw1
tonytw1 force-pushed the containerised branch 3 times, most recently from 4810e23 to 779c5c4 Compare October 11, 2024 20:39
tonytw1 added 30 commits September 2, 2026 21:08
@tonytw1
Alter Syndication access check to use URIs not raw host name; allows …
d42c29a 
…all api host name to be internalised. Interface only talks about base URIs.
@tonytw1
Introduce an interface to document the exposed service uris.
accab92 
Split url Services into a trait and a Guardian specific implementation; exposes a few 4th wall breaking direct init's in services.
@tonytw1
Everyone using GuardianUrlSchemeServices directly should take Service…
6efa54e 
…s supplied by common config Service trait;

no need to trouble yourselves with the details of how those URLs are defined.
@tonytw1
Drop GuardianUrlSchemeServices val constructor fields; these allow un…
f0073a7 
…documented access to the private url building concerns.
@tonytw1
Add a Service URL implementation which map services to port numbers o…
0daf5af 
…n the single hostname.

Will work because HTTPS auth is not active.

CORS for single host urls.

Projection end points are on the image-loader service but have seperate config to permit reingession workloads to be on different instances.
@tonytw1
Drop Guardian services URL scheme.
ce45f57 
We have used it to shape the Service interface. It can be dropped now.
@tonytw1
Move all public facing service urls to sub paths under single hostname.
43eba60 
Config single.host.url is exclusively for our single host setup.
@tonytw1
Disable CSRF with is no longer bypassed on a single origin CORS check.
bff26a8 
No longer gets bypassed thanks to preceding CORS check; CORS filter does not appear to tag the request if it passes for same origin.
@tonytw1
[containerised] Download crop asset links click through correctly whe…
dc0a345 
…n media-api is no longer on a different host.
@tonytw1
Delete InnerServiceStatusCheckController
02900f9 
"checks connectivity to all other internal services..." which sounds like something we can let the container orchestrator handle.
InnerServiceStatusCheckController was the only user of Services.allInternalUris

# Conflicts:
#	metadata-editor/app/MetadataEditorComponents.scala
@tonytw1
Simplify reaper paused control to set by config only.
f860b8d 
Removes a bucket dependency.
@tonytw1
[containered] End point to set a digital media usage.
0ab4af7 
Use .metadata as the name of the usage metadata field. Shorter and consistent with Download and Front.
Prefer digital to digital media in the public interface.

More verbose builder method names because of type erasure clashes.

Usage advertises the digital-usage end point.

# Conflicts:
#	usage/app/controllers/UsageApi.scala
@tonytw1
Drop Guardian specific stream ingestion of usages.
2f673a8 
Drop more usages composer references; drops composer url config requirement.

Remove usages UsageGroups from Guardian Content methods.

Remove usages CAPI client and it's config.

Delete usages streaming mode Crier steam listening and CAPI specific reindexForContent end point.

Container consumers are never going to be Guardian internal. External users would integrate their streams by having a microapp or Lambda ping the instance specific usages API. We can drop this Guardian specific code and config.

# Conflicts:
#	usage/app/UsageComponents.scala
@tonytw1
[containerised] Digital usage ids are prefixed with digital for consi…
2d62cbd 
…stency with print.
@tonytw1
[digital-usage] Digital usage webTitle and sectionId are optional.
f40f1f3 
Read back optional webTitle and sectionId correctly from dynamoDB item.

# Conflicts:
#	common-lib/src/main/scala/com/gu/mediaservice/lib/usage/ItemToMediaUsage.scala
#	common-lib/src/main/scala/com/gu/mediaservice/model/usage/DigitalUsageMetadata.scala

# Conflicts:
#	common-lib/src/test/scala/com/gu/mediaservice/lib/usage/ItemToMediaUsageTest.scala
#	usage/test/model/UsageRecordTest.scala
@tonytw1
Unused MediaWrapper class and helpers.
055cb12
@tonytw1
Drop Guardian Content references.
31b28bc
@tonytw1
Unused ResetException.
b7d8052
@tonytw1
Use imgproxy rather than nginx imgops for optimised image previews.
5a058d7 
Faster and lighter. Better colour profile support.

Generate an imgproxy style preview URL. Move service name from imgops to imgproxy.

Disable EXIF autorotation and explicitly correct rotation.
imgproxy does not accept negative rotations.

Be explicit about stripping colour profile to force sRGB.
@tonytw1
Play base project heap size set to 40%; want more heap for given cont…
53b0165 
…ainer size than default ergonomic but mindful of OOM errors.

60% was running close to the OOM kill limit on Java 11 and Java 22 has nudged it into OOM kill.
40% after media-api and collections OOM after Java 25.
@tonytw1
image loader prints java memory settings.
3c9b17f
@tonytw1
Cloudbuild all artifacts as 1 build.
2cb57af 
x86 images only. Multi arch is too slow (14 mins) as buildx repeats the apt-get steps.
Cloudbuild uses node 24 for Kahuna build.

Explicit jdk 11 build in Cloudbuild build.
@tonytw1
Cloudbuild runs Kahuna tests.
8f1485b
@tonytw1
UsageTable methods which can be made private.
3ed614f 
# Conflicts:
#	usage/app/model/UsageTable.scala
@tonytw1
DynamoDB private methods.
0229ff3
@tonytw1
MetadataSqsMessageConsumer.processDeletedImage is private.
b71b075
@tonytw1
Build vips-8.18.4 / JDK25 base image.
6713c94
@tonytw1
Use vips-8.18.4 base image for playImageLoaderProject.
5c236e3
@tonytw1
[containerised] We only need Environment credentials; may speed up fi…
f8e6e30 
…rst hit.
@tonytw1
Nerf EC2MetadataUtils.getInstanceId which has no meaning in a contain…
8c9f198 
…erised world.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tonytw1