Multi tenant - #12

Draft

tonytw1 wants to merge 189 commits into

containerisedfrom

Commits on Sep 2, 2026

Nerf the thrall migration thread; very noisy.
tonytw1
committed
a6a00fe
Delete SyncChecker; not needed yet
tonytw1
committed
0c53747
Delete Scripts; not needed yet and difficult.
tonytw1
committed
07dd5c7
Delete GoodToGoCheck; not instance specific.
tonytw1
committed
3708a07
Introduce an Instance model class and a way to infer it from a request.
tonytw1
committed
c745595
Multi tenant vhosts - Redefine Service URLs to be a function of instance.
tonytw1
committed
c4a4fa3
Marking TODO corsFilter and securityHeadersFilter will need to be reworked to have per request / instance config.
tonytw1
committed
0b26022
Reinstate security headers using an instance specific config.
tonytw1
committed
316ee16
Implement instance specific CORS filter.
tonytw1
committed
ef80422
Reinstate CSRF filter with needed the CORS filter to single bypass on same origin.
tonytw1
committed
a2d4364
Kahuna CSP origin.full and origin.images are believed to be redundant; imgproxy is on the same host as the UI.
tonytw1
committed
7bdc2b5
Resolve TODO. auth and kahuna are on the same hostname as everyother service.
tonytw1
committed
c726d5f
Kahuna auth uri in main.view are correct.
tonytw1
committed
72bdd13
Fix share URL is double //
tonytw1
committed
9ee964d
Initial instance awareness. Prepare upload records a request specific instance field in the upload status table.
tonytw1
committed
404e891
Message instance field moves up from ImageUpdate to all ExternalThrallMessage messages.
tonytw1
committed
c508069
Media API's ElasticSearch class accepts instances inputs into current index alias functions.
tonytw1
committed
c0b1201
Media API's Elastic responses are instance specific.
tonytw1
committed
393f9f4
Step back to service to service calls via the full public URLs so that instance is preserved.
tonytw1
committed
3671738
Introduce a typed Instance and pass it around implicitly.
tonytw1
committed
7c83b15
Patch instance url s"" usages with toString.
tonytw1
committed
78db0cd
URIs take a full Instance rather than Request; use more implicits so minimise the diff with main.
tonytw1
committed
901e90a
Crop paths are prefixed with instance id like originals and thumbs.
tonytw1
committed
e5e7d13
Collections store instance aware using Dynamo composite keys on id + instance.
tonytw1
committed
c06e10d
Instance aware all collections end points.
tonytw1
committed
0b2a2b1
ImageCollectionsStore is instance aware.
tonytw1
committed
1338589
Spike; progating instance throught the RX? channels for simple Usage API end points.
tonytw1
committed
631bcb0
DynamoDB force implicit instance.
tonytw1
committed
f44fe1e
[dynamo] scanForId is instances aware.
tonytw1
committed
35529f1
[dynamo] batchGet query is instance aware.
tonytw1
committed
542926c
[dynamo] get is instance aware.
tonytw1
committed
c6439d5
InstanceAwareDynamoDB asJsObject strips instance key.
tonytw1
committed
4d8b348
deleteItemV2 is instance aware.
tonytw1
committed
1d0521d
Move Leases to id + instance composite key; Scanomo had to be by past; seems to expect all keys to be actual fields of the object.
tonytw1
committed
cc53794
LeaseStoreSpec is instance aware.
tonytw1
committed
897ab15
Upload status table reads are instance aware.
tonytw1
committed
729bb69
Patch up soft deletes table best we can given that Scanmomo doesn't support composite keys on batch and there is no longer any error feedback now BatchWriteItemResult is missing
tonytw1
committed
3c74d19
Just stick instance in soft delete images; not that public facing.
tonytw1
committed
d9f8ea3
[dut] WIP pushing instance down to Usages table for instance aware queries.
tonytw1
committed
3449d98
[dut] Prefix usage grouping with instance slash to make filterable by instance.
tonytw1
committed
be188c8
UsageRecord persists an instance field so that it can be used as the partition key for the by media_id index.
tonytw1
committed
d22556c
UsageRecord persists an instance field so that it can be used as the partition key for the by media_id index.
tonytw1
committed
25a720b
[dynamo] UsageTableTest instance aware tests.
tonytw1
committed
b3e0852
UsageRecord persists an instance field tests.
tonytw1
committed
6b28251
DynamoDB needs instance Partition key like the old InstanceAwareDynamoDB!
tonytw1
committed
ce27185
EditsStoreTest sets up instance Partition key in test table.
tonytw1
committed
8b957d7
EditsStoreTest clean ups.
tonytw1
committed
913da74
Begin integrating the auth provider Kinde.
tonytw1
committed
421ebf4
Logout by clearing play session.
tonytw1
committed
27a85a4
Redirect out of auth.
tonytw1
committed
833f46a
/logout link needs a slash in this setup.
tonytw1
committed
ede5500
Use seperate cookie (not Play session) so it be copied for internal calls.
tonytw1
committed
781b8a6
Flush cookie.
tonytw1
committed
cda0dfb
Logout from Kinde on logout.
tonytw1
committed
be03be1
Auth can run at the root of the domain to auth all instances.
tonytw1
committed
cff6adf
Default exit to /
tonytw1
committed
658badf
auth cookie not been seen on subdomains without an explicit domain set!?
tonytw1
committed
cef7067
Redirect logout back to home page.
tonytw1
committed
38ba9c9
/auth/session needs to be on the same host cause CORS.
tonytw1
committed
8b650e9
Capture loggedin user cookie for use with onBehalfOf like Panda.
tonytw1
committed
bf80093
/auth/session needs to be on the same host cause CORS.
tonytw1
committed
c3fc495
Sign Kinde logged in user cookies.
tonytw1
committed
5ddfcea
Capture all Kinde user details.
tonytw1
committed
b6b4bd1
Resolved TODO; random session backed state for Kinde oauth dance.
tonytw1
committed
3511317
Auth decorates authed user cookie with their instances on oauth callback.
tonytw1
committed
94e0409
Auth provider decorates Principal with allowed instances. Definition of authed becomes 'Are you authenticated and a member of this instance?'
tonytw1
committed
4ef29a4
Use domainRoot is the as the config hook to remove hard coded griddev references.
tonytw1
committed
6daf33a
Inline Await and timeout on Kinde request.
tonytw1
committed
d3a355a
Allow InnerServicePrincipal to access all instances.
tonytw1
committed
84bb156
Setup for non instance principals instances lookup.
tonytw1
committed
dde0de2
Auth calls back to landing service for users instances instead of reading directly from Dynamo.
tonytw1
committed
b6f1737
My instances endpoint for auth moves to config.
tonytw1
committed
b5f09ba
Quiet logging.
tonytw1
committed
e84f0b8
Clean up; useless future.
tonytw1
committed
b5258fc
Introduce CreateInstanceMessage so that thrall can respond to an on demand create index request.
tonytw1
committed
0b2df6b
CreateInstance is added to ExternalThrallMessage and passes through the MessageTranslator.
tonytw1
committed
95bd165
Thrall create instance calls collections to create the root Home collection.
tonytw1
committed
87d9b11
Sign cropper assets URLs.
tonytw1
committed
b97644d
Delete crops is instance aware.
tonytw1
committed
275481b
Thrall polls for list of current instances; setting up for collection of usage data.
tonytw1
committed
a0f770a
Thrall counts total ES images foreach instance.
tonytw1
committed
3e5849b
Thrall knows image count and total file size for instances.
tonytw1
committed
d479991
Thrall announces instance usage onto an SQS queue for instances to pick up.
tonytw1
committed
7639fb6
Thralls call back to instances endpoint is conf.
tonytw1
committed
150c490
Thrall can ensure index on start up to help rebuild Elastic.
tonytw1
committed
8f1dbc9
Push instance all the way down to fileKeyFromId to catch all usages.
tonytw1
committed
45a73ea
Instance specific Reaper
tonytw1
committed
944b618
Clean up; extract method.
tonytw1
committed
42ec983
Renable image-deleted messages.
tonytw1
committed
cf3fc31
Restore image-deleted messages and make instance specific.
tonytw1
committed
649bae6
Marking TODO; no image-deleted notification after hard reap?
tonytw1
committed
8743077
Ingest images from instance specific top level folders in the ingest bucket.
tonytw1
committed
eab1f8f
fileKeyFromId uses implicit Instance for smaller diff.
tonytw1
committed
6dcb3d5
optimisedPngKeyFromId uses implicit Instance for smaller diff.
tonytw1
committed
330ced5
Log principal for blocked request.
tonytw1
committed
67bd991
Enable API key access to ownered instances. Request owner instances by GET parameter rather than auth cookie so that API auth can work as well.
tonytw1
committed
3987d8c
Remove deprecated TODO
tonytw1
committed
4ded919
Clean up.
tonytw1
committed
fe6e240
Logging.
tonytw1
committed
2466c29
Instance usage message adds softDeletedCount field.
tonytw1
committed
f29d21c
Logging.
tonytw1
committed
48c8003
Refactor; extract instance message sender.
tonytw1
committed
80faecb
Instance usage JSON writes.
tonytw1
committed
9b88878
Setting up to send instance message after setup has completed; SQS is probably going to need a type field to multiplex this.
tonytw1
committed
b04124e
Multiplex instance related messages.
tonytw1
committed
3a73f6d
Reaper query build logging.
tonytw1
committed
30fc4c1
Reaper controller logging.
tonytw1
committed
5ce7fac
KeyStore seperates API keys by instance folder.
tonytw1
committed
a243dfe
CSRF filter still problematic.
tonytw1
committed
a1c42c3
Marking TODO.
tonytw1
committed
f00c299
Log /management/healthcheck as debug.
tonytw1
committed
bd39484
Private.
tonytw1
committed
8c34607
Update TODO.
tonytw1
committed
83c5a72
Disable Guardian email parsing of CSV file in bucket usage updates.
tonytw1
committed
77ccb8a
Refactor; QuotaStore in it's own file to make it more visible.
tonytw1
committed
996be40
usage/quotas needs a / prefix.
tonytw1
committed
e1de4bc
Thrall exposes a /config end point so that reaping and hard delete config can be surfaced to users.
tonytw1
committed
055b3ee
Hard delete can delete any image which is in the soft deleted state; regardless of reapable or not.
tonytw1
committed
b9e9d96
Thrall exposes a /config end point so that reaping and hard delete config can be surfaced to users.
tonytw1
committed
965f635
Logging.
tonytw1
committed
9c8da1d
Log delete folder number of files found.
tonytw1
committed
8280dcd
[usage-quotas] Log quota store fetch.
tonytw1
committed
6b51b5b
[usage-quotas] Agency quota + no-op usage.
tonytw1
committed
6f4bdc4
Billing. Setup a new actor to capture usage events.
tonytw1
committed
6bb1325
Move usage events to common lib for reuse.
tonytw1
committed
7d37e18
Usage event for download original.
tonytw1
committed
aa250e0
Usage event SQS sending.
tonytw1
committed
9dc33f7
Usage event includes date.
tonytw1
committed
6427025
Emit api key used event on api auth.
tonytw1
committed
d96c05b
Thrall config end point exposes maybeUploadLimitInBytes for UI.
tonytw1
committed
1c7f110
maybeUploadLimitInBytes is in actual MB (1024 * 1024).
tonytw1
committed
56d8468
Remove source.secureURL from image response.
tonytw1
committed
a5d31af
Set isFeedUpload field on uploaded and ingested images.
tonytw1
committed
7452424
isFeedUpload moves to UploadInfo.
tonytw1
committed
4212bd1
Crops sign source.file to recreate secureUrl.
tonytw1
committed
47e9b64
Correct image if for ingest events.
tonytw1
committed
fa169cb
Clean up; auth instance call simplied.
tonytw1
committed
8ab189c
Spike; machine auth has non user endpoint.
tonytw1
committed
5ef25ca
Api keys have instances attribute; can be checked without an attached Kinde user.
tonytw1
committed
4f00584
Log levels.
tonytw1
committed
faeafd2
api key usage is recorded by key name.
tonytw1
committed
03b5818
Remove persistence.identifier config option which was historically set to 'picdarUrn'.
tonytw1
committed
c4b1160
Debug - log updateStatus
tonytw1
committed
1b4d51f
Debug - prepared to queue request fails.
tonytw1
committed
7009df3
Usage event for user auth.
tonytw1
committed
9e3b1cd
Status.toString does not match the persisted format of the Dynamo tables.
tonytw1
committed
457d1fc
Debug - problem with uploadStatusTableWithCondition
tonytw1
committed
fbbb190
Fix Prepare -> Queued conditional update by burning table and redefining the StatusType enum format as the name string.
tonytw1
committed
c7a4b74
Ingest image presigned uploads need to be into the instance folders.
tonytw1
committed
d15155d
Disable publishChangedSyndicationRightsForPhotoshoot
tonytw1
committed
bb8bd62
Usage event for prepare upload.
tonytw1
committed
92e5e1a
Usage events; would like to capture user or api key inline.
tonytw1
committed
ae21dda
Usage events; would like to capture user or api key inline.
tonytw1
committed
83e56b3
Usage events; api / user recorded for prepare upload.
tonytw1
committed
80224f6
Usage events; api / user recorded for download image.
tonytw1
committed
f7f1163
Move instances to common lib.
tonytw1
committed
ea89fea
Move instances to common lib.
tonytw1
committed
3a6038c
Thrall emits image delete events.
tonytw1
committed
090a810
Preview reapable query by removing the time window when selected as is:reapable in the UI.
tonytw1
committed
b288671
ReapableEligibility checks for isFeedUpload.
tonytw1
committed
3d9367d
Logging.
tonytw1
committed
94a7ef7
ReapableEligibility checks for isFeedUpload.
tonytw1
committed
abd6471
ReapableEligibility checks for isFeedUpload.
tonytw1
committed
e8bb6d8
Log upload original mime type.
tonytw1
committed
013bbbf
Feed ingest file uploader is last folder in path.
tonytw1
committed
b580a66
Feed ingest file uploader is folder after feeds.
tonytw1
committed
6f425cb
Feed ingest file uploader is folder after feeds.
tonytw1
committed
c489d23
Generalise text for Chargable usage rights description.
tonytw1
committed
92653c4
Log total reapable images.
tonytw1
committed
d8500ed
private
tonytw1
committed
662c3d4
Thrall kinesis appName is explicitly set rather than defaulting to the streamName.
tonytw1
committed
4fb564e
Thrall kinesis appName is explicitly set rather than defaulting to the streamName.
tonytw1
committed
ef94bcb
Log stream name for Kinesis put.
tonytw1
committed
847bd3c
Log initial offset.
tonytw1
committed
5f13005
Mulitple KCL streams cannot share the same checkpointing table!
tonytw1
committed
8e0263a
Increase polling interval for unused /notification to 10 minutes.
tonytw1
committed
82eb783
Generalise text for Chargable usage rights description.
tonytw1
committed
4e21e9c
Generalise owned by. Screen grab description uses generalised text.
tonytw1
committed
33725f6
Generalise owned by. Remove org-owned: prefix on image thumbnail mouse overs.
tonytw1
committed
aabaf28
Generalise owned filter names.
tonytw1
committed
9e093f4
Generalise owned by - is suggestion names.
tonytw1
committed
d05b2dc
Generalise the sensitive content explainer.
tonytw1
committed
9f4fa5f
Instances is more testable if injected rather than mixed in.
tonytw1
committed
ecd00c8
Setting up for instance aware migration. Push down the hard-coded dummy instances in the migration sender's constructor.
tonytw1
committed
6ec3bd7
Migration source and status have list of instances.
tonytw1
committed
f38eeef
[multi-tenant] Refactor; group getMyInstances into Instances trait.
tonytw1
committed
a91ac52
[multi-tenant] Do not quietly recover from instances call failures. Needs to be a full visible error.
tonytw1
committed
efe5cca
[multi-tenant] Download export link is instance aware
tonytw1
committed
ee4f936
[multi-tenant] EmbedderMessage is instance aware.
tonytw1
committed
b922262