auto_kaggle retries forever when a successful command does not create the submission file

[graysoncooper]

What is wrong

In auto_kaggle, the debug loop counts retries only on the failure branch, so a successful command that does not produce the submission file retries forever and never reaches the debug_max_attempt cutoff.

mle/workflow/kaggle.py initializes debug_attempt = 0 and guards the loop with if debug_attempt > debug_max_attempt. When debugger.analyze_with_log returns status success but the submission file is missing, the code calls coder.debug with a synthetic error report and loops back to the top without incrementing debug_attempt. The only increment lives in the else (non-success) branch, which this path never takes, so the counter never advances toward the stopping condition.

MLE-agent/mle/workflow/kaggle.py

Lines 80 to 108 in a29287e

	debug_attempt = 0
	while True:
	if debug_attempt > debug_max_attempt:
	console.log(f"Debug the code failed with max {debug_max_attempt} attempts. Please check the code manually.")
	break
	with console.status("MLE Debug Agent is executing and debugging the code..."):
	running_cmd = code_report.get('command')
	logs = execute_command(running_cmd)
	debug_report = debugger.analyze_with_log(running_cmd, logs)
	if debug_report.get('status') == 'success':
	# check the submission file
	if not os.path.exists(submission):
	console.log(f"The submission file ({submission}) is not found. Launch the coder to improve...")
	code_report = coder.debug(
	coding_task,
	{
	"status": "error",
	"changes": [
	f"make sure the submission file is generated in {submission}",
	f"make sure the submission file is in the correct format. You can refer to the example submission file: {sub_examples}"
	],
	"suggestion": f"Please update the code related to generating the submission file."
	}
	)
	else:
	break
	else:
	debug_attempt += 1

debug_attempt = 0
while True:
    if debug_attempt > debug_max_attempt:
        console.log(f"Debug the code failed with max {debug_max_attempt} attempts. Please check the code manually.")
        break
    ...
    if debug_report.get('status') == 'success':
        if not os.path.exists(submission):
            console.log(f"The submission file ({submission}) is not found. Launch the coder to improve...")
            code_report = coder.debug(coding_task, { "status": "error", ... })   # loops back, no increment
        else:
            break
    else:
        debug_attempt += 1
        code_report = coder.debug(coding_task, debug_report)

How it manifests

Run mle kaggle --auto (which calls auto_kaggle with debug_max_attempt=5). If the model generates code that runs to completion but writes no submission file, or writes it to the wrong path, every iteration takes the status == 'success' plus missing-submission branch. debug_attempt stays at its initial value, the guard never trips, and the workflow keeps calling execute_command and coder.debug (each an LLM call) until an external timeout or an operator stops it. A model producing code that exits cleanly without the expected output is an ordinary code-generation failure, not an edge case.

Failure scenario

1. auto_kaggle starts with debug_max_attempt=5.
2. Generated code executes successfully, so debug_report['status'] == 'success'.
3. The submission file is absent, so the missing-submission branch runs coder.debug and loops back.
4. debug_attempt is still 0, so the debug_attempt > debug_max_attempt guard is false.
5. Steps 2 to 4 repeat without bound, burning compute and LLM calls, until the process is killed. The intended fallback to manual review after 5 attempts never happens.

Suggested fix

Increment debug_attempt in the missing-submission branch, the same way the else branch does, so both retry paths advance the counter toward debug_max_attempt and the existing manual-review guard applies uniformly.

Automated report: this issue was produced and filed automatically, with no human review before posting. Two independent checks agreed it is a real bug, but if it misreads the code please say so and we will close it.

Found while running Ito (AI code review that runs your application, free for open source) against recently merged PRs. Full analysis.

[MLgeek7325]

您好，您的邮件我已收到！

[dosubot]

@graysoncooper, the bug is confirmed by the code. In the debug loop, debug_attempt is only incremented in the else branch (line 108), but never in the missing-submission branch (lines 92–104). When debug_report['status'] == 'success' and the submission file doesn't exist, the loop spins indefinitely since debug_attempt stays at 0 and the guard at line 82 never trips.

The suggested fix is straightforward — add debug_attempt += 1 in the missing-submission branch:

if debug_report.get('status') == 'success':
    if not os.path.exists(submission):
        console.log(f"The submission file ({submission}) is not found. Launch the coder to improve...")
        debug_attempt += 1  # <-- add this
        code_report = coder.debug(
            coding_task,
            {
                "status": "error",
                "changes": [
                    f"make sure the submission file is generated in {submission}",
                    f"make sure the submission file is in the correct format. You can refer to the example submission file: {sub_examples}"
                ],
                "suggestion": f"Please update the code related to generating the submission file."
            }
        )
    else:
        break
else:
    debug_attempt += 1
    code_report = coder.debug(coding_task, debug_report)

This ensures both retry paths (execution error and missing submission file) advance the counter toward debug_max_attempt, so the existing manual-review fallback applies uniformly.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.