Revert "Revert "USE_INFERENCE_EXP_MODELS"" #1657
Conversation
hansent
requested review from
PawelPeczek-Roboflow,
grzegorz-roboflow,
probicheaux and
yeldarby
as code owners
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| def preprocess(self, image: Any, **kwargs): | ||
| is_batch = isinstance(image, list) | ||
| images = image if is_batch else [image] | ||
| np_images: List[np.ndarray] = [ | ||
| load_image_rgb( | ||
| v, | ||
| disable_preproc_auto_orient=kwargs.get( | ||
| "disable_preproc_auto_orient", False | ||
| ), | ||
| ) | ||
| for v in images | ||
| ] | ||
| mapped_kwargs = self.map_inference_kwargs(kwargs) | ||
| return self._exp_model.pre_process(np_images, **mapped_kwargs) | ||
|
|
||
| def predict(self, img_in, **kwargs): | ||
| mapped_kwargs = self.map_inference_kwargs(kwargs) | ||
| return self._exp_model.forward(img_in, **mapped_kwargs) | ||
|
|
||
| def postprocess( | ||
| self, | ||
| predictions: Tuple[np.ndarray, ...], | ||
| preprocess_return_metadata: PreprocessingMetadata, | ||
| **kwargs, | ||
| ) -> List[Detections]: | ||
| mapped_kwargs = self.map_inference_kwargs(kwargs) | ||
| detections_list = self._exp_model.post_process( | ||
| predictions, preprocess_return_metadata, **mapped_kwargs |
There was a problem hiding this comment.
The new experimental adapter calls self._exp_model.pre_process, forward, and post_process directly without any synchronization. Other torch-backed models in this repository protect inference with a Lock to avoid concurrent access to shared model state (for example YOLOv8ObjectDetection.predict uses _session_lock). AutoModel instances from inference_exp are PyTorch models as well and are unlikely to be thread-safe. When the server runs with multiple workers or handles concurrent requests, unsynchronized access can trigger CUDA/torch runtime errors or corrupt intermediate buffers. The adapter already imports Lock, so wrapping the model calls in a mutex seems intended and would prevent these race conditions.
Useful? React with 👍 / 👎.
Reverts #1656
Need to run full integration test suite on this branch and resolve transformers issue