Reloading `i2c_ljca` breaks the `ipu6` driver

[robin-carlier]

This issue might be related to #32, I stumbled upon it while trying to prevent i2c_ljca from breaking hibernation on my machine.

On Debian with kernel 6.11-rc4, using upstreamed ivsc and ipu6 driver, with a Lenovo X1 Carbon gen 10 with ov2740 sensor.

To reproduce:
sudo modprobe -r i2c_ljca; sudo modprobe i2c_ljca; cam -l indicates an error:

[0:15:59.604474783] [9745] ERROR SimplePipeline simple.cpp:1550 No sensor found, while the sensor was correctly detected before reloading the module.

Reloading intel_ipu6_isys or ov2740 without reloading i2c_ljca does not cause this behavior (sensor is detected correctly after reloading the modules).
I tried unloading those two modules prior to reloading i2c_ljca and reloading them after (in all possible orders) without success, so the problem seems to be with i2c_ljca and not with those other two modules.