OutOfMemory error in mirror Lambda function

[dsotirho-ucsc]

Occurred on anvilprod and prod after the merge of the 2026-01-27 promotion (PR #7725 and PR #7726)

https://gitlab.explore.anvilproject.org/ucsc/azul/-/jobs/60484
https://gitlab.azul.data.humancellatlas.org/ucsc/azul/-/jobs/91475

======================================================================
FAIL: test_indexing (integration_test.IndexingIntegrationTest.test_indexing) [mirror_sources_and_files]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builds/ucsc/azul/test/integration_test.py", line 385, in subTest
    yield
  File "/builds/ucsc/azul/test/integration_test.py", line 1796, in _test_mirroring
    self._assert_queues_empty([config.mirror_queue.to_fail.name])
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builds/ucsc/azul/test/integration_test.py", line 1417, in _assert_queues_empty
    self.assertTrue(self.azul_client.is_queue_empty(queue_name))
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: False is not true

[dsotirho-ucsc]

prod

failed message

{
    "queue": "https://sqs.us-east-1.amazonaws.com/542754589326/azul-mirror_fail-prod.fifo",
    "messages": [
        {
            "MessageId": "1be68e81-a947-48fe-8897-bcaf0164bc04",
            "ReceiptHandle": "AQEBIagLKa2iwqUnbf+7YoF67ggKBv6RCmsE3NpDjpzTdi/r8Yhsu1h2819JOdRG64skfbGe+3G4ljn4/NvoX8vIJWOiqSv2ZteAjAszgqgTntntgm5kwqip8DaSXC69rjcSEC21iifSzU0h
tMjy6FmOrFUuRQESUCGXFGKtrhsXKdV3udwdlpoFUk4yysNdYf9r8QqQFj+fEdrxJYjfhBZkxXhgX8fz5NuJlfXq9+WgJ5HEMhiG4cBKpHOgv/KYubue6DUGGkQEyIXqrWioXiuUTO4TdUxIcb7FWmqwQeX0wHs=",
            "MD5OfBody": "2b53afb8e5fbec3ac5de1f95570e65fb",
            "Body": {
                "catalog": "dcp56-it",
                "operation_id": "b0487283-5fe6-498d-9b6d-9ee81e1c06f8",
                "source": {
                    "id": "329f91fa-7ed2-4af0-aaac-fc683420bed0",
                    "spec": "tdr:bigquery:gcp:datarepo-f0f918aa:hca_prod_bd40033154b94fccbff66bb8b079ee1f__20220118_dcp2_20241107_dcp44",
                    "prefix": "a/1",
                    "type": "azul.terra.TDRSourceRef"
                },
                "prefix": "a9",
                "file": {
                    "uuid": "93e2bcb3-3310-4f04-a076-edc6d70edd83",
                    "name": "Adult1_S8_L003_R2_001.fastq.gz",
                    "version": "2021-05-10T23:24:58.882000Z",
                    "drs_uri": "drs://data.terra.bio/v1_329f91fa-7ed2-4af0-aaac-fc683420bed0_623c14af-7536-4601-b3c9-bf16e1690b29",
                    "size": 962376157,
                    "content_type": "binary/octet-stream",
                    "sha256": "a9ff6ebbd732ed1b873df16136ea382c7d9b0c4c5f74355bae6837afc70d7cab",
                    "crc32c": "66e47419",
                    "sha1": "cee9aadb8e4f0d453a657f50a010a478a2b487d0",
                    "s3_etag": "f010845fde24b77c3dccfc730932cc23-15",
                    "type": "azul.plugins.metadata.hca.HCAFile"
                },
                "action": "MirrorFileAction"
            },
            "Attributes": {
                "SenderId": "AROAX4XVRZKHHXD45CIX7:azul-indexer-prod-mirror",
                "ApproximateFirstReceiveTimestamp": "1769835372993",
                "ApproximateReceiveCount": "11",
                "SentTimestamp": "1769835372649",
                "SequenceNumber": "18899822417231855616",
                "MessageDeduplicationId": "1be68e81-a947-48fe-8897-bcaf0164bc04",
                "MessageGroupId": "75404b54-a3f8-5725-a71b-f45791485785",
                "AWSTraceHeader": "Root=1-697d8b67-e5d5d0ff0e7d21b102108a20;Parent=6d5797fa37508867;Sampled=0;Lineage=2:fbad25ae:1",
                "DeadLetterQueueSourceArn": "arn:aws:sqs:us-east-1:542754589326:azul-mirror-prod.fifo"
            },
            "_Attributes": {
                "SentTimestamp": "2026-01-30T20:56:12.649000-08:00",
                "ApproximateFirstReceiveTimestamp": "2026-01-30T20:56:12.993000-08:00"
            }
        }
    ]
}

/aws/lambda/azul-indexer-prod-mirror

CloudWatch Logs Insights
region: us-east-1
log-group-names: /aws/lambda/azul-indexer-prod-mirror
data-sources:
facets:
start-time: 2026-01-31T05:23:52.869Z
end-time: 2026-01-31T05:37:30.000Z
query-string:

fields @timestamp, @message
| filter @message like '27896315-650f-59b9-8327-02e2f150dd98'
| sort @timestamp asc
| limit 10000

---

@timestamp	@message
2026-01-31 05:24:51.491	START RequestId: 27896315-650f-59b9-8327-02e2f150dd98 Version: 7
2026-01-31 05:24:51.492	[INFO] 2026-01-31T05:24:51.492Z 27896315-650f-59b9-8327-02e2f150dd98 azul.indexer.action_controller Worker handling message SQSFifoMessage(body={'catalog': 'dcp56-it', 'operation_id': 'b0487283-5fe6-498d-9b6d-9ee81e1c06f8', 'source': {'id': '329f91fa-7ed2-4af0-aaac-fc683420bed0', 'spec': 'tdr:bigquery:gcp:datarepo-f0f918aa:hca_prod_bd40033154b94fccbff66bb8b079ee1f__20220118_dcp2_20241107_dcp44', 'prefix': 'a/1', 'type': 'azul.terra.TDRSourceRef'}, 'prefix': 'a9', 'file': {'uuid': '93e2bcb3-3310-4f04-a076-edc6d70edd83', 'name': 'Adult1_S8_L003_R2_001.fastq.gz', 'version': '2021-05-10T23:24:58.882000Z', 'drs_uri': 'drs://data.terra.bio/v1_329f91fa-7ed2-4af0-aaac-fc683420bed0_623c14af-7536-4601-b3c9-bf16e1690b29', 'size': 962376157, 'content_type': 'binary/octet-stream', 'sha256': 'a9ff6ebbd732ed1b873df16136ea382c7d9b0c4c5f74355bae6837afc70d7cab', 'crc32c': '66e47419', 'sha1': 'cee9aadb8e4f0d453a657f50a010a478a2b487d0', 's3_etag': 'f010845fde24b77c3dccfc730932cc23-15', 'type': 'azul.plugins.metadata.hca.HCAFile'}, 'action': 'MirrorFileAction'}, attempts=10, id='1be68e81-a947-48fe-8897-bcaf0164bc04', group_id='75404b54-a3f8-5725-a71b-f45791485785', dedup_id='ebb6e599-ab50-5ad0-a99b-054ebec5b594')
2026-01-31 05:24:51.528	[INFO] 2026-01-31T05:24:51.528Z 27896315-650f-59b9-8327-02e2f150dd98 azul.boto3 sqs.GetQueueUrl: Making POST request to https://sqs.us-east-1.amazonaws.com/
2026-01-31 05:24:51.528	[INFO] 2026-01-31T05:24:51.528Z 27896315-650f-59b9-8327-02e2f150dd98 azul.boto3 sqs.GetQueueUrl: … with a request body of length 38 being b'{"QueueName": "azul-mirror-prod.fifo"}'
2026-01-31 05:24:51.575	[INFO] 2026-01-31T05:24:51.575Z 27896315-650f-59b9-8327-02e2f150dd98 azul.boto3 sqs.GetQueueUrl: Got 200 response
2026-01-31 05:24:51.576	[INFO] 2026-01-31T05:24:51.575Z 27896315-650f-59b9-8327-02e2f150dd98 azul.boto3 sqs.GetQueueUrl: … with a response body of length 85 being b'{"QueueUrl":"https://sqs.us-east-1.amazonaws.com/542754589326/azul-mirror-prod.fifo"}'
2026-01-31 05:24:51.673	[INFO] 2026-01-31T05:24:51.673Z 27896315-650f-59b9-8327-02e2f150dd98 azul.deployment Allocated new Boto3 client for 's3' with ID 140536973129248
2026-01-31 05:24:51.676	[INFO] 2026-01-31T05:24:51.676Z 27896315-650f-59b9-8327-02e2f150dd98 azul.boto3 s3.GetObject: Making GET request to https://s3.amazonaws.com/edu-ucsc-gi-platform-hca-prod-mirror-prod.us-east-1/_it/info/a9ff6ebbd732ed1b873df16136ea382c7d9b0c4c5f74355bae6837afc70d7cab.sha256.json
2026-01-31 05:24:51.676	[INFO] 2026-01-31T05:24:51.676Z 27896315-650f-59b9-8327-02e2f150dd98 azul.boto3 s3.GetObject: … without a request body
2026-01-31 05:24:51.728	[INFO] 2026-01-31T05:24:51.728Z 27896315-650f-59b9-8327-02e2f150dd98 azul.boto3 s3.GetObject: Got 404 response
2026-01-31 05:24:51.729	[INFO] 2026-01-31T05:24:51.729Z 27896315-650f-59b9-8327-02e2f150dd98 azul.boto3 s3.GetObject: … with a response body of length 376 being b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>_it/info/a9ff6ebbd732ed1b873df16136ea382c7d9b0c4c5f74355bae6837afc70d7cab.sha256.json</Key><RequestId>V7T7TADHWHGWNEX3</RequestId><HostId>Iqa2LfgtPFcvLrnCpTDaZ/Sc7hzgBAFcIhVAmC7jYceo2sDoSXvepLNxYaXj4zPiF7nKX7l25eSOj+f7YU756FkITQIFRU+p</HostId></Error>'
2026-01-31 05:24:51.731	[INFO] 2026-01-31T05:24:51.731Z 27896315-650f-59b9-8327-02e2f150dd98 azul.boto3 s3.HeadObject: Making HEAD request to https://s3.amazonaws.com/edu-ucsc-gi-platform-hca-prod-mirror-prod.us-east-1/_it/file/a9ff6ebbd732ed1b873df16136ea382c7d9b0c4c5f74355bae6837afc70d7cab.sha256
2026-01-31 05:24:51.731	[INFO] 2026-01-31T05:24:51.731Z 27896315-650f-59b9-8327-02e2f150dd98 azul.boto3 s3.HeadObject: … without a request body
2026-01-31 05:24:51.740	[INFO] 2026-01-31T05:24:51.740Z 27896315-650f-59b9-8327-02e2f150dd98 azul.boto3 s3.HeadObject: Got 404 response
2026-01-31 05:24:51.740	[INFO] 2026-01-31T05:24:51.740Z 27896315-650f-59b9-8327-02e2f150dd98 azul.boto3 s3.HeadObject: … without a response body
2026-01-31 05:24:51.741	[INFO] 2026-01-31T05:24:51.740Z 27896315-650f-59b9-8327-02e2f150dd98 azul.indexer.mirror_service Mirroring file via standard upload: HCAFile(uuid='93e2bcb3-3310-4f04-a076-edc6d70edd83', name='Adult1_S8_L003_R2_001.fastq.gz', version='2021-05-10T23:24:58.882000Z', drs_uri='drs://data.terra.bio/v1_329f91fa-7ed2-4af0-aaac-fc683420bed0_623c14af-7536-4601-b3c9-bf16e1690b29', size=962376157, content_type='binary/octet-stream', sha256='a9ff6ebbd732ed1b873df16136ea382c7d9b0c4c5f74355bae6837afc70d7cab', crc32c='66e47419', sha1='cee9aadb8e4f0d453a657f50a010a478a2b487d0', s3_etag='f010845fde24b77c3dccfc730932cc23-15')
2026-01-31 05:24:51.748	[INFO] 2026-01-31T05:24:51.748Z 27896315-650f-59b9-8327-02e2f150dd98 azul.deployment Allocated new Boto3 client for 'secretsmanager' with ID 140536973131264
2026-01-31 05:24:51.967	[INFO] 2026-01-31T05:24:51.967Z 27896315-650f-59b9-8327-02e2f150dd98 azul.terra Making GET request to 'https://data.terra.bio/ga4gh/drs/v1/objects/v1_329f91fa-7ed2-4af0-aaac-fc683420bed0_623c14af-7536-4601-b3c9-bf16e1690b29'
2026-01-31 05:24:51.967	[INFO] 2026-01-31T05:24:51.967Z 27896315-650f-59b9-8327-02e2f150dd98 azul.terra … without a request body
2026-01-31 05:24:51.968	[INFO] 2026-01-31T05:24:51.968Z 27896315-650f-59b9-8327-02e2f150dd98 azul.terra … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'REDACTED')]
2026-01-31 05:24:52.506	[INFO] 2026-01-31T05:24:52.506Z 27896315-650f-59b9-8327-02e2f150dd98 azul.terra Got 200 response after 0.539s from GET to https://data.terra.bio/ga4gh/drs/v1/objects/v1_329f91fa-7ed2-4af0-aaac-fc683420bed0_623c14af-7536-4601-b3c9-bf16e1690b29
2026-01-31 05:24:52.506	[INFO] 2026-01-31T05:24:52.506Z 27896315-650f-59b9-8327-02e2f150dd98 azul.terra … with response headers HTTPHeaderDict({'Date': 'Sat, 31 Jan 2026 05:24:52 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': 'ekgkkjOe', 'Content-Type': 'application/json', 'Content-Length': '2528', 'Set-Cookie': 'JSESSIONID=4130B0726553DDCF7B105A653084B573; Path=/; Secure; HttpOnly; SameSite=Lax', 'Vary': 'Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2026-01-31 05:24:52.506	[INFO] 2026-01-31T05:24:52.506Z 27896315-650f-59b9-8327-02e2f150dd98 azul.terra … with a response body of length 2528 starting in b'{"id":"v1_329f91fa-7ed2-4af0-aaac-fc683420bed0_623c14af-7536-4601-b3c9-bf16e1690b29","name":"Adult1_S8_L003_R2_001.fastq.gz","self_uri":"drs://data.terra.bio/v1_329f91fa-7ed2-4af0-aaac-fc683420bed0_623c14af-7536-4601-b3c9-bf16e1690b29","size":962376157,"created_time":"2022-01-18T08:04:36.867Z","updated_time":"2022-01-18T08:04:36.867Z","version":"0","mime_type":null,"checksums":[{"checksum":"66e47419","type":"crc32c"},{"checksum":"061c392df8aa5c8c5154e772bd4424b5","type":"md5"}],"access_methods":[{"type":"gs","access_url":{"url":"gs://datarepo-ff9c2603-bucket/81d37c1d-5e68-4442-9ef9-4475c6dc2fba/623c14af-7536-4601-b3c9-bf16e1690b29/Adult1_S8_L003_R2_001.fastq.gz","headers":null},"access_id":"gcp-us-east4*329f91fa-7ed2-4af0-aaac-fc683420bed0","region":"us-east4","authorizations":{"supported_types":["BearerAuth"],"passport_auth_issuers":null,"bearer_auth_issuers":null}},{"type":"https","access_url":{"url":"https://www.googleapis.com/storage/v1/b/datarepo-ff9c2603-bucket/o/81d37c1d-5e68-4442-9ef9-4475c6dc2fba%...'
2026-01-31 05:24:52.507	[INFO] 2026-01-31T05:24:52.507Z 27896315-650f-59b9-8327-02e2f150dd98 azul.terra Making GET request to 'https://data.terra.bio/ga4gh/drs/v1/objects/v1_329f91fa-7ed2-4af0-aaac-fc683420bed0_623c14af-7536-4601-b3c9-bf16e1690b29/access/gcp-us-east4*329f91fa-7ed2-4af0-aaac-fc683420bed0'
2026-01-31 05:24:52.507	[INFO] 2026-01-31T05:24:52.507Z 27896315-650f-59b9-8327-02e2f150dd98 azul.terra … without a request body
2026-01-31 05:24:52.507	[INFO] 2026-01-31T05:24:52.507Z 27896315-650f-59b9-8327-02e2f150dd98 azul.terra … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'REDACTED')]
2026-01-31 05:24:53.100	[INFO] 2026-01-31T05:24:53.100Z 27896315-650f-59b9-8327-02e2f150dd98 azul.terra Got 200 response after 0.593s from GET to https://data.terra.bio/ga4gh/drs/v1/objects/v1_329f91fa-7ed2-4af0-aaac-fc683420bed0_623c14af-7536-4601-b3c9-bf16e1690b29/access/gcp-us-east4*329f91fa-7ed2-4af0-aaac-fc683420bed0
2026-01-31 05:24:53.100	[INFO] 2026-01-31T05:24:53.100Z 27896315-650f-59b9-8327-02e2f150dd98 azul.terra … with response headers HTTPHeaderDict({'Date': 'Sat, 31 Jan 2026 05:24:53 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': '6z0KNnBy', 'Content-Type': 'application/json', 'Content-Length': '1065', 'Set-Cookie': 'JSESSIONID=50657CE364B1635D85273AEFF0097B18; Path=/; Secure; HttpOnly; SameSite=Lax', 'Vary': 'Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2026-01-31 05:24:53.100	[INFO] 2026-01-31T05:24:53.100Z 27896315-650f-59b9-8327-02e2f150dd98 azul.terra … with a response body of length 1065 starting in b'{"url":"https://storage.googleapis.com/datarepo-ff9c2603-bucket/81d37c1d-5e68-4442-9ef9-4475c6dc2fba/623c14af-7536-4601-b3c9-bf16e1690b29/Adult1_S8_L003_R2_001.fastq.gz?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=datarepo-jade-api%40terra-datarepo-production.iam.gserviceaccount.com%2F20260131%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20260131T052453Z&X-Goog-Expires=900&X-Goog-SignedHeaders=host&requestedBy=azul-ucsc-0-public-prod%40platform-hca-prod.iam.gserviceaccount.com&userProject=datarepo-f0f918aa&X-Goog-Signature=5a715...'
2026-01-31 05:24:53.102	[INFO] 2026-01-31T05:24:53.102Z 27896315-650f-59b9-8327-02e2f150dd98 azul.indexer.mirror_service Making GET request to 'https://storage.googleapis.com/datarepo-ff9c2603-bucket/81d37c1d-5e68-4442-9ef9-4475c6dc2fba/623c14af-7536-4601-b3c9-bf16e1690b29/Adult1_S8_L003_R2_001.fastq.gz?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=datarepo-jade-api%40terra-datarepo-production.iam.gserviceaccount.com%2F20260131%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20260131T052453Z&X-Goog-Expires=900&X-Goog-SignedHeaders=host&requestedBy=azul-ucsc-0-public-prod%40platform-hca-prod.iam.gserviceaccount.com&userProject=datarepo-f0f918aa&X-Goog-Signature=5a715...'
2026-01-31 05:24:53.102	[INFO] 2026-01-31T05:24:53.102Z 27896315-650f-59b9-8327-02e2f150dd98 azul.indexer.mirror_service … without a request body
2026-01-31 05:24:53.103	[INFO] 2026-01-31T05:24:53.103Z 27896315-650f-59b9-8327-02e2f150dd98 azul.indexer.mirror_service … without request headers
2026-01-31 05:25:04.218	END RequestId: 27896315-650f-59b9-8327-02e2f150dd98
2026-01-31 05:25:04.218	REPORT RequestId: 27896315-650f-59b9-8327-02e2f150dd98 Duration: 14941.37 ms Billed Duration: 14942 ms Memory Size: 1280 MB Max Memory Used: 1280 MB Status: error Error Type: Runtime.OutOfMemory

---

[dsotirho-ucsc]

anvilprod

failed message

{
    "queue": "https://sqs.us-east-1.amazonaws.com/465330168186/azul-mirror_fail-anvilprod.fifo",
    "messages": [
        {
            "MessageId": "56758192-5c84-40ce-8e17-e69dcd557133",
            "ReceiptHandle": "AQEB9LUsf7jNEGwbDIAPGksjPE5yUQfxK9eQXK+rmKc1VrTyzTGu9PYN2H5aLXBmsMwzI9G/THdujdeX0Fo2YJaDIKqWaq+LHKCzmHZdWLWk06Z4h3N/3wrLeYpX3qVDBUYqm7zo356y0W+BO9yyE6RjUWUvoDhx1aA+1pmFk9+itw/lJkRSiEt7xwdUyqPR5bAiVb/U7/3PEPbGtB8kHLLIsvfaC5/yIoGLWV99FI47XOhWsCyQVhAF3q8t6vG8Ug29KFof7Ot+JlK7Qp5IxcHCW0xFswE0wCHvzriu9twsM4I=",
            "MD5OfBody": "148d8e934c0c50a2cad4bcb87fd28537",
            "Body": {
                "catalog": "anvil12-it",
                "operation_id": "bc2d97a4-e18b-4d08-b67d-5511812f7ed4",
                "source": {
                    "id": "4b79114d-8d84-426e-b48f-482d3606d5bd",
                    "spec": "tdr:bigquery:gcp:datarepo-f5dfb465:ANVIL_ENCORE_293T_20251123_ANV5_202511232035",
                    "prefix": "7c/1",
                    "type": "azul.terra.TDRSourceRef"
                },
                "prefix": "7c9",
                "file": {
                    "uuid": "026b1e4b-5b92-4f62-b5fe-7562b3c8a054",
                    "name": "NT-BG293LV06-37_S3_L002_R2_001.filtered.trimmed.paired.fastq.gz",
                    "version": "2022-06-01T00:00:00.000000Z",
                    "drs_uri": "drs://drs.anv0:v2_d46a3274-6bf4-3a06-a225-ded1ab69c516",
                    "size": 1026721439,
                    "content_type": null,
                    "md5": "7c9444c95682d0eef4f20ce34dfa5003",
                    "type": "azul.plugins.metadata.anvil.AnvilFile"
                },
                "action": "MirrorFileAction"
            },
            "Attributes": {
                "SenderId": "AROAWYV6W4F5NNBDZAIKE:azul-indexer-anvilprod-mirror",
                "ApproximateFirstReceiveTimestamp": "1769834842635",
                "ApproximateReceiveCount": "11",
                "SentTimestamp": "1769834842635",
                "SequenceNumber": "37346566353627359232",
                "MessageDeduplicationId": "56758192-5c84-40ce-8e17-e69dcd557133",
                "MessageGroupId": "94ee701b-4aaf-51a5-94fc-8767938b88f9",
                "AWSTraceHeader": "Root=1-697d8956-84967c8f4466bdc05a7b5845;Parent=5d7e0609451bb45c;Sampled=0;Lineage=2:42007d32:1",
                "DeadLetterQueueSourceArn": "arn:aws:sqs:us-east-1:465330168186:azul-mirror-anvilprod.fifo"
            },
            "_Attributes": {
                "SentTimestamp": "2026-01-30T20:47:22.635000-08:00",
                "ApproximateFirstReceiveTimestamp": "2026-01-30T20:47:22.635000-08:00"
            }
        }
    ]
}

/aws/lambda/azul-indexer-anvilprod-mirror

CloudWatch Logs Insights
region: us-east-1
log-group-names: /aws/lambda/azul-indexer-anvilprod-mirror
data-sources:
facets:
start-time: 2026-01-31T05:14:47.989Z
end-time: 2026-01-31T05:40:30.000Z
query-string:

fields @timestamp, @message
| filter @message like '7528a079-0692-55bc-a264-c18e94a78015'
| sort @timestamp asc
| limit 10000

---

@timestamp	@message
2026-01-31 05:15:55.029	START RequestId: 7528a079-0692-55bc-a264-c18e94a78015 Version: 8
2026-01-31 05:15:55.030	[INFO] 2026-01-31T05:15:55.030Z 7528a079-0692-55bc-a264-c18e94a78015 azul.indexer.action_controller Worker handling message SQSFifoMessage(body={'catalog': 'anvil12-it', 'operation_id': 'bc2d97a4-e18b-4d08-b67d-5511812f7ed4', 'source': {'id': '4b79114d-8d84-426e-b48f-482d3606d5bd', 'spec': 'tdr:bigquery:gcp:datarepo-f5dfb465:ANVIL_ENCORE_293T_20251123_ANV5_202511232035', 'prefix': '7c/1', 'type': 'azul.terra.TDRSourceRef'}, 'prefix': '7c9', 'file': {'uuid': '026b1e4b-5b92-4f62-b5fe-7562b3c8a054', 'name': 'NT-BG293LV06-37_S3_L002_R2_001.filtered.trimmed.paired.fastq.gz', 'version': '2022-06-01T00:00:00.000000Z', 'drs_uri': 'drs://drs.anv0:v2_d46a3274-6bf4-3a06-a225-ded1ab69c516', 'size': 1026721439, 'content_type': None, 'md5': '7c9444c95682d0eef4f20ce34dfa5003', 'type': 'azul.plugins.metadata.anvil.AnvilFile'}, 'action': 'MirrorFileAction'}, attempts=10, id='56758192-5c84-40ce-8e17-e69dcd557133', group_id='94ee701b-4aaf-51a5-94fc-8767938b88f9', dedup_id='52101f54-6df8-5a84-a5a2-2f07cd656f06')
2026-01-31 05:15:55.072	[INFO] 2026-01-31T05:15:55.072Z 7528a079-0692-55bc-a264-c18e94a78015 azul.boto3 sqs.GetQueueUrl: Making POST request to https://sqs.us-east-1.amazonaws.com/
2026-01-31 05:15:55.072	[INFO] 2026-01-31T05:15:55.072Z 7528a079-0692-55bc-a264-c18e94a78015 azul.boto3 sqs.GetQueueUrl: … with a request body of length 43 being b'{"QueueName": "azul-mirror-anvilprod.fifo"}'
2026-01-31 05:15:55.113	[INFO] 2026-01-31T05:15:55.113Z 7528a079-0692-55bc-a264-c18e94a78015 azul.boto3 sqs.GetQueueUrl: Got 200 response
2026-01-31 05:15:55.113	[INFO] 2026-01-31T05:15:55.113Z 7528a079-0692-55bc-a264-c18e94a78015 azul.boto3 sqs.GetQueueUrl: … with a response body of length 90 being b'{"QueueUrl":"https://sqs.us-east-1.amazonaws.com/465330168186/azul-mirror-anvilprod.fifo"}'
2026-01-31 05:15:55.177	[INFO] 2026-01-31T05:15:55.177Z 7528a079-0692-55bc-a264-c18e94a78015 azul.deployment Allocated new Boto3 client for 's3' with ID 139622243713104
2026-01-31 05:15:55.180	[INFO] 2026-01-31T05:15:55.180Z 7528a079-0692-55bc-a264-c18e94a78015 azul.boto3 s3.GetObject: Making GET request to https://s3.amazonaws.com/edu-ucsc-gi-platform-anvil-prod-mirror-anvilprod.us-east-1/_it/info/7c9444c95682d0eef4f20ce34dfa5003.md5.json
2026-01-31 05:15:55.180	[INFO] 2026-01-31T05:15:55.180Z 7528a079-0692-55bc-a264-c18e94a78015 azul.boto3 s3.GetObject: … without a request body
2026-01-31 05:15:55.235	[INFO] 2026-01-31T05:15:55.235Z 7528a079-0692-55bc-a264-c18e94a78015 azul.boto3 s3.GetObject: Got 404 response
2026-01-31 05:15:55.235	[INFO] 2026-01-31T05:15:55.235Z 7528a079-0692-55bc-a264-c18e94a78015 azul.boto3 s3.GetObject: … with a response body of length 321 being b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>_it/info/7c9444c95682d0eef4f20ce34dfa5003.md5.json</Key><RequestId>3EZJ58ZNZ3Z8A3JT</RequestId><HostId>2u1RgLELPeGOJff0Cg9ZNODHZSRIUZuxNgx+3CcAnXKMkd649ZrUyJMcochUKTZXMl8RSjXFwfQ=</HostId></Error>'
2026-01-31 05:15:55.238	[INFO] 2026-01-31T05:15:55.238Z 7528a079-0692-55bc-a264-c18e94a78015 azul.boto3 s3.HeadObject: Making HEAD request to https://s3.amazonaws.com/edu-ucsc-gi-platform-anvil-prod-mirror-anvilprod.us-east-1/_it/file/7c9444c95682d0eef4f20ce34dfa5003.md5
2026-01-31 05:15:55.238	[INFO] 2026-01-31T05:15:55.238Z 7528a079-0692-55bc-a264-c18e94a78015 azul.boto3 s3.HeadObject: … without a request body
2026-01-31 05:15:55.247	[INFO] 2026-01-31T05:15:55.247Z 7528a079-0692-55bc-a264-c18e94a78015 azul.boto3 s3.HeadObject: Got 404 response
2026-01-31 05:15:55.247	[INFO] 2026-01-31T05:15:55.247Z 7528a079-0692-55bc-a264-c18e94a78015 azul.boto3 s3.HeadObject: … without a response body
2026-01-31 05:15:55.248	[INFO] 2026-01-31T05:15:55.248Z 7528a079-0692-55bc-a264-c18e94a78015 azul.indexer.mirror_service Mirroring file via standard upload: AnvilFile(uuid='026b1e4b-5b92-4f62-b5fe-7562b3c8a054', name='NT-BG293LV06-37_S3_L002_R2_001.filtered.trimmed.paired.fastq.gz', version='2022-06-01T00:00:00.000000Z', drs_uri='drs://drs.anv0:v2_d46a3274-6bf4-3a06-a225-ded1ab69c516', size=1026721439, content_type=None, md5='7c9444c95682d0eef4f20ce34dfa5003')
2026-01-31 05:15:55.248	[INFO] 2026-01-31T05:15:55.248Z 7528a079-0692-55bc-a264-c18e94a78015 azul.drs Making GET request to 'https://registry.api.identifiers.org/restApi/namespaces/search/findByPrefix?prefix=drs.anv0'
2026-01-31 05:15:55.249	[INFO] 2026-01-31T05:15:55.248Z 7528a079-0692-55bc-a264-c18e94a78015 azul.drs … without a request body
2026-01-31 05:15:55.249	[INFO] 2026-01-31T05:15:55.249Z 7528a079-0692-55bc-a264-c18e94a78015 azul.drs … without request headers
2026-01-31 05:15:55.359	[INFO] 2026-01-31T05:15:55.359Z 7528a079-0692-55bc-a264-c18e94a78015 azul.drs Got 200 response after 0.111s from GET to https://registry.api.identifiers.org/restApi/namespaces/search/findByPrefix?prefix=drs.anv0
2026-01-31 05:15:55.359	[INFO] 2026-01-31T05:15:55.359Z 7528a079-0692-55bc-a264-c18e94a78015 azul.drs … with response headers HTTPHeaderDict({'Vary': 'Origin, Access-Control-Request-Method, Access-Control-Request-Headers', 'Last-Modified': 'Thu, 01 Jun 2023 09:26:38 GMT', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '0', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'Strict-Transport-Security': 'max-age=31536000 ; includeSubDomains', 'X-Frame-Options': 'DENY', 'Content-Type': 'application/hal+json', 'Date': 'Sat, 31 Jan 2026 05:15:55 GMT', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000', 'Transfer-Encoding': 'chunked'})
2026-01-31 05:15:55.359	[INFO] 2026-01-31T05:15:55.359Z 7528a079-0692-55bc-a264-c18e94a78015 azul.drs … with a response body of length 1836 starting in b'{\n "prefix" : "drs.anv0",\n "mirId" : "MIR:00001014",\n "name" : "AnVIL DRS",\n "pattern" : "^v2_[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$",\n "description" : "This namespace is used for GA4GH Data Repository Service (DRS) access to the USA National Institutes of Health (NIH), National Human Genome Research Institute (NHGRI), Genomic Analysis, Visualization and Informatics Lab-space (AnVIL) program data. AnVIL is a cloud-based genomic data sharing and analysis platform which facilitates integration and computing on and across large datasets generated by NHGRI programs, as well as initiatives funded by the National Institutes of Health (NIH) or by other agencies that support human genomics research. ",\n "created" : "2023-06-01T09:26:38.236+00:00",\n "modified" : "2023-06-01T09:26:38.236+00:00",\n "deprecated" : false,\n "deprecationDate" : null,\n "deprecationOfflineDate" : null,\n "renderDeprecatedLanding" : false,\n "deprecationStatement" : null,\n "infoOnPostmortemA...'
2026-01-31 05:15:55.360	[INFO] 2026-01-31T05:15:55.360Z 7528a079-0692-55bc-a264-c18e94a78015 azul.drs Resolved prefix 'drs.anv0' to namespace ID '3571'
2026-01-31 05:15:55.360	[INFO] 2026-01-31T05:15:55.360Z 7528a079-0692-55bc-a264-c18e94a78015 azul.drs Making GET request to 'https://registry.api.identifiers.org/restApi/resources/search/findAllByNamespaceId?id=3571'
2026-01-31 05:15:55.360	[INFO] 2026-01-31T05:15:55.360Z 7528a079-0692-55bc-a264-c18e94a78015 azul.drs … without a request body
2026-01-31 05:15:55.360	[INFO] 2026-01-31T05:15:55.360Z 7528a079-0692-55bc-a264-c18e94a78015 azul.drs … without request headers
2026-01-31 05:15:55.461	[INFO] 2026-01-31T05:15:55.461Z 7528a079-0692-55bc-a264-c18e94a78015 azul.drs Got 200 response after 0.101s from GET to https://registry.api.identifiers.org/restApi/resources/search/findAllByNamespaceId?id=3571
2026-01-31 05:15:55.461	[INFO] 2026-01-31T05:15:55.461Z 7528a079-0692-55bc-a264-c18e94a78015 azul.drs … with response headers HTTPHeaderDict({'Vary': 'Origin, Access-Control-Request-Method, Access-Control-Request-Headers', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '0', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'Strict-Transport-Security': 'max-age=31536000 ; includeSubDomains', 'X-Frame-Options': 'DENY', 'Content-Type': 'application/hal+json', 'Date': 'Sat, 31 Jan 2026 05:15:55 GMT', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000', 'Transfer-Encoding': 'chunked'})
2026-01-31 05:15:55.461	[INFO] 2026-01-31T05:15:55.461Z 7528a079-0692-55bc-a264-c18e94a78015 azul.drs … with a response body of length 2150 starting in b'{\n "_embedded" : {\n "resources" : [ {\n "mirId" : "MIR:00001011",\n "urlPattern" : "https://data.terra.bio/ga4gh/drs/v1/objects/{$id}",\n "name" : "National Human Genome Research Institute",\n "description" : "NHGRI offers access to reliable and timely information about genomics research and the human genome. Our resources and partnerships help spark scientific curiosity, improve genomic literacy, and foster engagement among learners in different communities.",\n "official" : true,\n "providerCode" : "data.terra.bio",\n "sampleId" : "v2_1b0f158a-86e2-3887-b990-89e3cf2d9082",\n "resourceHomeUrl" : "https://www.genome.gov/",\n "created" : "2023-06-01T09:26:38.248+00:00",\n "modified" : "2023-06-01T09:26:38.248+00:00",\n "deprecated" : false,\n "deprecationDate" : null,\n "deprecationOfflineDate" : null,\n "renderDeprecatedLanding" : false,\n "deprecationStatement" : null,\n "protectedUrls" : false,\n "renderProtectedLanding" : false,\n ...'
2026-01-31 05:15:55.461	[INFO] 2026-01-31T05:15:55.461Z 7528a079-0692-55bc-a264-c18e94a78015 azul.drs Obtained URL pattern 'https://data.terra.bio/ga4gh/drs/v1/objects/{$id}' from resource 'National Human Genome Research Institute'
2026-01-31 05:15:55.468	[INFO] 2026-01-31T05:15:55.468Z 7528a079-0692-55bc-a264-c18e94a78015 azul.deployment Allocated new Boto3 client for 'secretsmanager' with ID 139622243720496
2026-01-31 05:15:55.653	[INFO] 2026-01-31T05:15:55.653Z 7528a079-0692-55bc-a264-c18e94a78015 azul.terra Making GET request to 'https://data.terra.bio/ga4gh/drs/v1/objects/v2_d46a3274-6bf4-3a06-a225-ded1ab69c516'
2026-01-31 05:15:55.653	[INFO] 2026-01-31T05:15:55.653Z 7528a079-0692-55bc-a264-c18e94a78015 azul.terra … without a request body
2026-01-31 05:15:55.654	[INFO] 2026-01-31T05:15:55.654Z 7528a079-0692-55bc-a264-c18e94a78015 azul.terra … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'REDACTED')]
2026-01-31 05:15:56.337	[INFO] 2026-01-31T05:15:56.337Z 7528a079-0692-55bc-a264-c18e94a78015 azul.terra Got 200 response after 0.683s from GET to https://data.terra.bio/ga4gh/drs/v1/objects/v2_d46a3274-6bf4-3a06-a225-ded1ab69c516
2026-01-31 05:15:56.337	[INFO] 2026-01-31T05:15:56.337Z 7528a079-0692-55bc-a264-c18e94a78015 azul.terra … with response headers HTTPHeaderDict({'Date': 'Sat, 31 Jan 2026 05:15:56 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': 'aAkDxZwD', 'Content-Type': 'application/json', 'Content-Length': '2423', 'Set-Cookie': 'JSESSIONID=4B70B7E0D462E22B247002A390F155FF; Path=/; Secure; HttpOnly; SameSite=Lax', 'Vary': 'Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2026-01-31 05:15:56.337	[INFO] 2026-01-31T05:15:56.337Z 7528a079-0692-55bc-a264-c18e94a78015 azul.terra … with a response body of length 2423 starting in b'{"id":"v2_d46a3274-6bf4-3a06-a225-ded1ab69c516","name":"NT-BG293LV06-37_S3_L002_R2_001.filtered.trimmed.paired.fastq.gz","self_uri":"drs://data.terra.bio/v2_d46a3274-6bf4-3a06-a225-ded1ab69c516","size":1026721439,"created_time":"2024-12-03T20:01:07.680Z","updated_time":"2024-12-03T20:01:07.680Z","version":"0","mime_type":null,"checksums":[{"checksum":"a2154296","type":"crc32c"},{"checksum":"7c9444c95682d0eef4f20ce34dfa5003","type":"md5"}],"access_methods":[{"type":"gs","access_url":{"url":"gs://fc-0e48cf31-cd8a-4006-9c7d-17b8c0c5e99e/data_files/fastq/NT-BG293LV06-37_S3_L002_R2_001.filtered.trimmed.paired.fastq.gz","headers":null},"access_id":"gcp-us-central1*4b79114d-8d84-426e-b48f-482d3606d5bd","region":"us-central1","authorizations":{"supported_types":["BearerAuth"],"passport_auth_issuers":null,"bearer_auth_issuers":null}},{"type":"https","access_url":{"url":"https://www.googleapis.com/storage/v1/b/fc-0e48cf31-cd8a-4006-9c7d-17b8c0c5e99e/o/data_files%2Ffastq%2FNT-BG293LV06-37_S3_L002_R2_001.filtered.trim...'
2026-01-31 05:15:56.338	[INFO] 2026-01-31T05:15:56.338Z 7528a079-0692-55bc-a264-c18e94a78015 azul.terra Making GET request to 'https://data.terra.bio/ga4gh/drs/v1/objects/v2_d46a3274-6bf4-3a06-a225-ded1ab69c516/access/gcp-us-central1*4b79114d-8d84-426e-b48f-482d3606d5bd'
2026-01-31 05:15:56.338	[INFO] 2026-01-31T05:15:56.338Z 7528a079-0692-55bc-a264-c18e94a78015 azul.terra … without a request body
2026-01-31 05:15:56.338	[INFO] 2026-01-31T05:15:56.338Z 7528a079-0692-55bc-a264-c18e94a78015 azul.terra … with request headers [('x-goog-api-client', 'cred-type/sa'), ('authorization', 'REDACTED')]
2026-01-31 05:15:57.288	[INFO] 2026-01-31T05:15:57.288Z 7528a079-0692-55bc-a264-c18e94a78015 azul.terra Got 200 response after 0.950s from GET to https://data.terra.bio/ga4gh/drs/v1/objects/v2_d46a3274-6bf4-3a06-a225-ded1ab69c516/access/gcp-us-central1*4b79114d-8d84-426e-b48f-482d3606d5bd
2026-01-31 05:15:57.288	[INFO] 2026-01-31T05:15:57.288Z 7528a079-0692-55bc-a264-c18e94a78015 azul.terra … with response headers HTTPHeaderDict({'Date': 'Sat, 31 Jan 2026 05:15:57 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'Access-Control-Allow-Headers': 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,Accept,Referer,X-App-Id,Origin', 'Access-Control-Allow-Methods': 'GET,POST,DELETE,PUT,PATCH,OPTIONS,HEAD', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains', 'Cache-Control': 'no-cache,no-store,must-revalidate', 'X-Request-ID': 'eZlvzNj6', 'Content-Type': 'application/json', 'Content-Length': '1051', 'Set-Cookie': 'JSESSIONID=CCFBECC943AA4F2BEB5AD3BDB7AE2520; Path=/; Secure; HttpOnly; SameSite=Lax', 'Vary': 'Origin', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000'})
2026-01-31 05:15:57.288	[INFO] 2026-01-31T05:15:57.288Z 7528a079-0692-55bc-a264-c18e94a78015 azul.terra … with a response body of length 1051 starting in b'{"url":"https://storage.googleapis.com/fc-0e48cf31-cd8a-4006-9c7d-17b8c0c5e99e/data_files/fastq/NT-BG293LV06-37_S3_L002_R2_001.filtered.trimmed.paired.fastq.gz?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=tdr-ingest-sa%40datarepo-6f635bc3.iam.gserviceaccount.com%2F20260131%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20260131T051557Z&X-Goog-Expires=900&X-Goog-SignedHeaders=host&requestedBy=azul-ucsc-1-public-anvilprod%40platform-anvil-prod.iam.gserviceaccount.com&userProject=datarepo-f5dfb465&X-Goog-Signature=59549a2a...'
2026-01-31 05:15:57.289	[INFO] 2026-01-31T05:15:57.289Z 7528a079-0692-55bc-a264-c18e94a78015 azul.indexer.mirror_service Making GET request to 'https://storage.googleapis.com/fc-0e48cf31-cd8a-4006-9c7d-17b8c0c5e99e/data_files/fastq/NT-BG293LV06-37_S3_L002_R2_001.filtered.trimmed.paired.fastq.gz?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=tdr-ingest-sa%40datarepo-6f635bc3.iam.gserviceaccount.com%2F20260131%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20260131T051557Z&X-Goog-Expires=900&X-Goog-SignedHeaders=host&requestedBy=azul-ucsc-1-public-anvilprod%40platform-anvil-prod.iam.gserviceaccount.com&userProject=datarepo-f5dfb465&X-Goog-Signature=59549a2a...'
2026-01-31 05:15:57.289	[INFO] 2026-01-31T05:15:57.289Z 7528a079-0692-55bc-a264-c18e94a78015 azul.indexer.mirror_service … without a request body
2026-01-31 05:15:57.290	[INFO] 2026-01-31T05:15:57.290Z 7528a079-0692-55bc-a264-c18e94a78015 azul.indexer.mirror_service … without request headers
2026-01-31 05:16:09.412	END RequestId: 7528a079-0692-55bc-a264-c18e94a78015
2026-01-31 05:16:09.412	REPORT RequestId: 7528a079-0692-55bc-a264-c18e94a78015 Duration: 16515.03 ms Billed Duration: 16516 ms Memory Size: 1280 MB Max Memory Used: 1280 MB Status: error Error Type: Runtime.OutOfMemory

---

[hannes-ucsc]

In prod, I doubled the mirror Lambda memory to 2,560 and fed the failed messages back to the mirror queue. They were handled successfully but required significantly more memory:

CloudWatch Logs Insights
region: us-east-1
log-group-names: /aws/lambda/azul-indexer-prod-mirror
data-sources:
facets:
start-time: -3600s
end-time: 0s
query-string:

fields @timestamp, @message
| filter @message like /REPORT|Worker handling/ 
| sort @timestamp desc
| limit 10000

---

@timestamp	@message
2026-01-31 07:28:38.768	REPORT RequestId: 77f3bdb9-65bb-51cc-a8a3-dc4ed4fca7d1 Duration: 27784.31 ms Billed Duration: 30651 ms Memory Size: 2560 MB Max Memory Used: 1495 MB Init Duration: 2866.55 ms
2026-01-31 07:28:38.688	REPORT RequestId: 689a4f48-d917-5ff8-abe0-2ebf682e0438 Duration: 31595.26 ms Billed Duration: 34328 ms Memory Size: 2560 MB Max Memory Used: 1689 MB Init Duration: 2732.51 ms
2026-01-31 07:28:37.070	REPORT RequestId: faaa1507-69eb-5003-9ade-74f0d8ac061e Duration: 26834.16 ms Billed Duration: 29711 ms Memory Size: 2560 MB Max Memory Used: 1460 MB Init Duration: 2876.19 ms
2026-01-31 07:28:35.071	REPORT RequestId: d48957d6-efe9-54e8-b0f2-36215ed7b2f5 Duration: 28046.95 ms Billed Duration: 30821 ms Memory Size: 2560 MB Max Memory Used: 1496 MB Init Duration: 2773.08 ms
2026-01-31 07:28:10.984	[INFO] 2026-01-31T07:28:10.984Z 77f3bdb9-65bb-51cc-a8a3-dc4ed4fca7d1 azul.indexer.action_controller Worker handling message SQSFifoMessage(body={'catalog': 'dcp56-it', 'operation_id': 'b0487283-5fe6-498d-9b6d-9ee81e1c06f8', 'source': {'id': '329f91fa-7ed2-4af0-aaac-fc683420bed0', 'spec': 'tdr:bigquery:gcp:datarepo-f0f918aa:hca_prod_bd40033154b94fccbff66bb8b079ee1f__20220118_dcp2_20241107_dcp44', 'prefix': 'a/1', 'type': 'azul.terra.TDRSourceRef'}, 'prefix': 'a3', 'file': {'uuid': '34c723b9-9cf3-439f-a6f4-90fedc18706d', 'name': 'Adult2_S34_L001_R1_001.fastq.gz', 'version': '2021-05-10T23:24:59.857000Z', 'drs_uri': 'drs://data.terra.bio/v1_329f91fa-7ed2-4af0-aaac-fc683420bed0_2caacafc-b3d1-4558-b02a-9098e99c90b0', 'size': 961635662, 'content_type': 'binary/octet-stream', 'sha256': 'a38ad66c0afc31d9424043e3250a7abc9dac3348740303a2a159e28e32d69e97', 'crc32c': 'dae91349', 'sha1': 'f11d916a956c28f550767ada8c280877ea3817ff', 's3_etag': '2eecc1472696a458491416da05a0c3aa-15', 'type': 'azul.plugins.metadata.hca.HCAFile'}, 'action': 'MirrorFileAction'}, attempts=2, id='83e8db83-d1f5-400b-8b7d-7262738994ed', group_id='89600769-6f97-51c1-b7a1-cca730a2d84d', dedup_id='5317a414-d407-4ef5-a9e3-5e74404fdc3b')
2026-01-31 07:28:10.236	[INFO] 2026-01-31T07:28:10.236Z faaa1507-69eb-5003-9ade-74f0d8ac061e azul.indexer.action_controller Worker handling message SQSFifoMessage(body={'catalog': 'dcp56-it', 'operation_id': 'b0487283-5fe6-498d-9b6d-9ee81e1c06f8', 'source': {'id': '329f91fa-7ed2-4af0-aaac-fc683420bed0', 'spec': 'tdr:bigquery:gcp:datarepo-f0f918aa:hca_prod_bd40033154b94fccbff66bb8b079ee1f__20220118_dcp2_20241107_dcp44', 'prefix': 'a/1', 'type': 'azul.terra.TDRSourceRef'}, 'prefix': 'a7', 'file': {'uuid': 'bb36dd96-2995-45e8-b128-6e7f28dab8f1', 'name': 'Adult2_S34_L004_R1_001.fastq.gz', 'version': '2021-05-10T23:24:59.970000Z', 'drs_uri': 'drs://data.terra.bio/v1_329f91fa-7ed2-4af0-aaac-fc683420bed0_a2b66ce7-5b7c-40c6-a684-db7db268ec4d', 'size': 943773502, 'content_type': 'binary/octet-stream', 'sha256': 'a74b2dba2fe439d483db19a4b480e44873b668dd0cf926f3e6bada38bd695b3e', 'crc32c': 'ac7eef2d', 'sha1': 'a6b4ea4aee4436dde7625786c813e2d9315409ac', 's3_etag': '5bacf5c5fc57b17722a6bde1d9b050e3-15', 'type': 'azul.plugins.metadata.hca.HCAFile'}, 'action': 'MirrorFileAction'}, attempts=2, id='b6f4ddd9-8d77-4624-9ec9-3493d7472fe8', group_id='7eb0879e-af18-5b70-b105-0c8f3311cd96', dedup_id='2e948b32-db1a-43d1-9711-34bb3f165c89')
2026-01-31 07:28:07.093	[INFO] 2026-01-31T07:28:07.093Z 689a4f48-d917-5ff8-abe0-2ebf682e0438 azul.indexer.action_controller Worker handling message SQSFifoMessage(body={'catalog': 'dcp56-it', 'operation_id': 'b0487283-5fe6-498d-9b6d-9ee81e1c06f8', 'source': {'id': '329f91fa-7ed2-4af0-aaac-fc683420bed0', 'spec': 'tdr:bigquery:gcp:datarepo-f0f918aa:hca_prod_bd40033154b94fccbff66bb8b079ee1f__20220118_dcp2_20241107_dcp44', 'prefix': 'a/1', 'type': 'azul.terra.TDRSourceRef'}, 'prefix': 'a8', 'file': {'uuid': 'a97b90ce-e6ad-4b92-8b97-0c60bcd46031', 'name': 'Adult1_S6_L004_R1_001.fastq.gz', 'version': '2021-05-10T23:24:59.267000Z', 'drs_uri': 'drs://data.terra.bio/v1_329f91fa-7ed2-4af0-aaac-fc683420bed0_ddfc7271-6476-4ba7-b906-652cca6ee75c', 'size': 1063347469, 'content_type': 'binary/octet-stream', 'sha256': 'a847a09b1e72c0bd54fe38e1e20e95ef93f4ab9773cb407336817b8e728a59ce', 'crc32c': '2241f1db', 'sha1': '235a22678aff309b27da73b5d3f7b7fc05540292', 's3_etag': '597c48fa55b4827053118d27ac927223-16', 'type': 'azul.plugins.metadata.hca.HCAFile'}, 'action': 'MirrorFileAction'}, attempts=2, id='7767368e-f8cf-40e6-ad4a-044939551081', group_id='06fbec53-0365-561a-857d-8c4b7f3ee91a', dedup_id='9f3d8967-8b82-4fdd-98c4-fd2770fe1ff7')
2026-01-31 07:28:07.024	[INFO] 2026-01-31T07:28:07.024Z d48957d6-efe9-54e8-b0f2-36215ed7b2f5 azul.indexer.action_controller Worker handling message SQSFifoMessage(body={'catalog': 'dcp56-it', 'operation_id': 'b0487283-5fe6-498d-9b6d-9ee81e1c06f8', 'source': {'id': '329f91fa-7ed2-4af0-aaac-fc683420bed0', 'spec': 'tdr:bigquery:gcp:datarepo-f0f918aa:hca_prod_bd40033154b94fccbff66bb8b079ee1f__20220118_dcp2_20241107_dcp44', 'prefix': 'a/1', 'type': 'azul.terra.TDRSourceRef'}, 'prefix': 'a9', 'file': {'uuid': '93e2bcb3-3310-4f04-a076-edc6d70edd83', 'name': 'Adult1_S8_L003_R2_001.fastq.gz', 'version': '2021-05-10T23:24:58.882000Z', 'drs_uri': 'drs://data.terra.bio/v1_329f91fa-7ed2-4af0-aaac-fc683420bed0_623c14af-7536-4601-b3c9-bf16e1690b29', 'size': 962376157, 'content_type': 'binary/octet-stream', 'sha256': 'a9ff6ebbd732ed1b873df16136ea382c7d9b0c4c5f74355bae6837afc70d7cab', 'crc32c': '66e47419', 'sha1': 'cee9aadb8e4f0d453a657f50a010a478a2b487d0', 's3_etag': 'f010845fde24b77c3dccfc730932cc23-15', 'type': 'azul.plugins.metadata.hca.HCAFile'}, 'action': 'MirrorFileAction'}, attempts=2, id='e0432198-3a26-4eb9-8d0e-abf85a62c469', group_id='75404b54-a3f8-5725-a71b-f45791485785', dedup_id='1be68e81-a947-48fe-8897-bcaf0164bc04')

---

[dsotirho-ucsc]

@hannes-ucsc: "Spike to investigate: attempt to observe the memory consumption increase in lower deployments, attempt to reproduce in personal deployment, and bisect history to narrow which commit might be causing this. My feeling is that it is a memory leak introduced by a commit that upgrades a dependency. If that's correct, bisect further through all the dependencies upgrades in that commit to narrow which dependency is causing this."

[nadove-ucsc]

Unable to reproduce in personal deployment. Ran IT with seed pinned to 12345 on commits d9543eec (the failed pipeline on prod) and 00b8d2439 (the previous promotion), recreating the virtual environment before each deployment. Total memory use by the mirror lambda was 12846 MB for the passing pipeline and 12818 for the failing one. Memory statistics collected using this CloudWatch query:

fields @message | filter @message like 'REPORT' | parse @message "Max Memory Used: * MB" as memory

[nadove-ucsc]

There is no clear trend in the per-invocation memory usage of the mirror lambda in dev across the last 4 months.

Query:

fields @message
| filter @message like 'REPORT' 
| parse @message "Max Memory Used: * MB" as memory
| stats avg(memory) by bin (1d)

Results: https://docs.google.com/spreadsheets/d/19glH4VcqAe6HE4L6s2UufuXySAS_FXh1m288UHjHlL4/edit?gid=0#gid=0